Speed up FreeBSD KVM guests using Virtio

The virtio framework allows guest operating systems running under the Linux KVM hypervisor to take advantage of near-native I/O performance. It works by providing kernel drivers for the guest OS which only work from within a KVM host, exposing I/O functionality through a much thinner layer of code than the traditional full-on emulation of physical hardware. The less code there is between a virtualized guest OS and the host's physical hardware, the faster things will go.

UPDATE: This article applies to FreeBSD versions before 10.0. From 10.0 onward the Virtio drivers are part of the GENERIC kernel so installation on Virtio-enabled VM's works immediately out of the box.

Virtio has been around for a while in Linux but it's still fairly new on FreeBSD, with support only arriving in versions 8.2 and 9.0. That's something of a shame since tools like virt-manager make piss-poor default choices when creating FreeBSD guest VM's. Like configuring PIO-only IDE controllers from the early 90's, limited at 16MB/s throughput.. savages! If you do have a capable version of FreeBSD, loading virtio drivers certainly is worth the effort.

The first step after installing FreeBSD 9 would be to get your system up to date. I use freebsd-update for that, as it works a lot faster than compiling the whole OS from source for a minor patch.

#freebsd-update fetch
#freebsd-update install

These commands bring your system up to date, but you'll need to have the system sources in place in order to build the virtio kernel modules. These modules are still experimental so they aren't in the base system yet. In order to install them you need to install the emulators/virtio-kmod port. First we pull in sources.

#cp /usr/share/examples/cvsup/standard-supfile /root

In the resulting /root/standard-supfile you should look for the part where it says CHANGE_THIS, and modify that into a valid FreeBSD cvsup server. I always change it into cvsup.nl.freebsd.org but your mileage may vary.

#cd /root
#csup ./standard-supfile

The above commands start the download of all the system sources. They'll end up in /usr/src/sys and weigh in at about 200MB so make sure you have enough room to store them.

Now you have everything you need to build the virtio modules. From /usr/ports/emulators/virtio-kmod you run the usual commands to install a port:

#make
#make install
#make clean

This takes a while as it copies tons of source files from /usr/src/sys into the port's work directory, which is why the make clean at the end is important to save you some disk space. These steps give you compatible kernel modules for loading. Now make sure you actually load them at system startup by adding them to /boot/loader.conf like so:

virtio_load="YES"
virtio_pci_load="YES"
virtio_blk_load="YES"
if_vtnet_load="YES"

Apparently you can pick and choose which drivers to use, but I have yet to find a reason not to simply use all of them.

Before any of the virtio magic will work, you'll need to tell FreeBSD to actually use it for its network driver and block devices. By default the virt-manager tool sets up any FreeBSD VM with an emulated Intel gigabit network card, which FreeBSD recognizes as em0. Add the following line to /etc/rc.conf to rename your virtio network card to em0 and use any existing network settings.

ifconfig_vtnet0_name="em0"

You should also change your /etc/fstab file to use virtio block devices instead of the emulated IDE drives. A line like:

/dev/ada0p2 / ufs rw 1

..would become:

/dev/vtbd0p2 / ufs rw 1

You're almost done now. Fire up virt-manager on your host machine and modify the emulated hardware in your FreeBSD VM. Choose 'virtio' for both the emulated network device and for each IDE, SATA or SCSI hard disk you emulate.

Finally, shut down the VM (do not reboot it, really shut it down!), virtually power it back up and things should be hunky-dory with FreeBSD gaining some considerable speed in its network and block I/O layers.