Introduction

The computer memory is divided in pages. The standard page size on most systems is 4KB. To keep track on what information is stored in which page, the system uses a page table. Searching in this page table is (in computer terms) a time costly operation. That is why the TLB (Translation Lookaside Buffer) is invented. The TLB is a kind of fast searchable cache. The TLB is faster but limited in size. For systems with a lot of memory, the TLB can not keep track.

If the TLB is not able to satisfy a request, the system falls back to the normal page table. This is called a TLB miss. Such TLB misses are costly (slows down the system) and we want to avoid it. Here is it where Huge Pages come in. By increasing the page size, you reduce the page table and reduce the pressure on the TLB cache.

This section describes how to set up Huge Pages and use it in a KVM/Libvirt enviroment.

Setting up Huge Pages

There are multiple ways to set up huge pages.

1. There the old runtime interface at /proc/sys/vm/nr_hugepages

• This can directly be controlled

$ echo 1024 | sudo tee /proc/sys/vm/nr_hugepages

Or one can let sysctl do that early on boot, do do so put this in /etc/sysctl.conf to set the amount of reserved Huge Pages.

vm.nr_hugepages = 1024

2. there is the new numa and multiple page size aware sysfs interface at

• /sys/devices/system/node/node<NODID>/hugepages/hugepages-<SIZE>kB

This can be directly controlled by echo'ing values to it, just as the old interface. But due to its structure it allows to allocate per node and on different huge page sizes.

3. use the tool hugeadm to control huge pages of different sizes

Check the man page of hugeadm for more details

4. Kernel allocation at boot

Huge pages can later on due to allocation and fragmentation become unavailable. To avoid this being an issue one can let the kernel allocate at boot time. To do so is described in the Kernel documentation.

X. Check huge pages Finally To check the default huge page size and amount of those Huge Pages you can use /proc/meminfo. Only the default size will be listed here, use sysfs or hugeadm for more sizes and how they are allocated across nodes.

cat /proc/meminfo | grep Huge

The output should be something like this:

AnonHugePages:     0 kB
HugePages_Total:   256
HugePages_Free:    256
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

The page size is 2048 KB.

For example, if I would want to reserve 20GB for my guests, I would need 20*1024*1024/2048 = 10240 pages.

Change vm.nr_hugepages to the amount you want and reboot again.

Set up Libvirt to use Huge Pages

Recent releases

In Releases since Xenial (16.04) and later systemd will make default huge page mountpoints available without requiring an administrator to change anything.

Older releases In older releases (Pre Xenial) there was no default huge page mountpoint. There was a KVM helper that made one available, to control it you would edit /etc/default/qemu-kvm. Look for

KVM_HUGEPAGES=0

Change it to:

KVM_HUGEPAGES=1

And finally restart Libvirt to consider your changed configuration:

sudo service libvirt-bin restart

Setting up a guest to use Huge Pages

Stop the guest and configure guest to use Huge Pages

sudo virsh stop {nameOfGuest}
sudo virsh edit {nameOfGuest}

Insert these lines in the config:

<memoryBacking>
<hugepages/>
</memoryBacking>

The above still works these days, but one has to be aware that for more complex systems on newer Releases for different huge page sizes as well as Numa nodes a more powerful but complex definition can be used. This might contain specifying page sizes as well as nodeset references - see the libvirt domain description on that for more.

Afterwards start the guest again:

sudo virsh start {nameOfGuest}

You can check the amount of used Huge Pages to check if your guest is using Huge Pages:

cat /proc/meminfo | grep Huge