Introduction

The computer memory is divided in pages. The standard page size on most systems is 4KB. To keep track on what information is stored in which page, the system uses a page table. Searching in this page table is (in computer terms) a time costly operation. That is why the TLB (Translation Lookaside Buffer) is invented. The TLB is a kind of fast searchable cache. The TLB is faster but limited in size. For systems with a lot of memory, the TLB can not keep track.

If the TLB is not able to satisfy a request, the system falls back to the normal page table. This is called a TLB mis. Such TLB misses are costly (slows down the system) and we want to avoid it. Here is it where Huge Pages come in. By increasing the page size, you reduce the page table and reduce the pressure on the TLB cache.

This section describes how to set up Huge Pages and use it in a KVM/Libvirt enviroment.

Setting up Huge Pages

I have tested this procedure on Ubuntu Trusty Tahr 14.04.1.

Edit /etc/sysctl.conf

sudo nano /etc/sysctl.conf

Put this in /etc/sysctl.conf to set the amount of reserved Huge Pages and to pass the group-ID.

# Allocate 256 HugePageTables (start with a low number but increas it before using it)
vm.nr_hugepages = 256

Reboot the system

sudo reboot

Check the page size and increase the amount of Huge Pages

cat /proc/meminfo | grep Huge

The output should be something like this:

AnonHugePages:     0 kB
HugePages_Total:   256
HugePages_Free:    256
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

The page size is 2048 KB.

For example, if I would want to reserve 20GB for my guests, I would need 20*1024*1024/2048 = 10240 pages.

Change vm.nr_hugepages to the amount you want and reboot again.

Set up Libvirt to use Huge Pages

Edit /etc/default/qemu-kvm

Look for

KVM_HUGEPAGES=0

Change it to

KVM_HUGEPAGES=1

Restart Libvirt

sudo service libvirt-bin restart

Setting up a guest to use Huge Pages

Stop the guest and configure guest to use Huge Pages

sudo virsh stop {nameOfGuest}
sudo virsh edit {nameOfGuest}

Insert these lines in the config:

<memoryBacking>
<hugepages/>
</memoryBacking>

The editor used is vim, to learn how to use it: http://www.openvim.com/tutorial.html

Start the guest again:

sudo virsh start {nameOfGuest}

You can check the amount of used Huge Pages to check if your guest is using Huge Pages:

cat /proc/meminfo | grep Huge

Known issues

There are currently (Ubuntu 15.04 and 15.10) a problems using libvirt-qemu + HugePages + apparmor.

If you enable HugePages in a guest and you start the guest, you get an error not being able to access hugepages.

For more information see:

https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1315402

https://bugs.launchpad.net/ubuntu/vivid/+source/qemu/+bug/1461012


CategoryVirtualization

KVM - Using Hugepages (last edited 2015-11-02 17:52:36 by dominique-ramaekers)