HOWTO set up QEMU with HAX acceleration on NetBSD

Back to homepage

I learnt that Kamil Rytarowski has taken up a challenge of porting HAXM to NetBSD. I found this to be a great learning opportunity to learn about OS development. I decided to help him and check whether his demo of Windows 7 really works. Below are the steps I have taken to get it running.

This post is meant to be a guide on setting up HAXM accelerated QEMU on NetBSD 8.0. The HAXM-capable qemu is currently provided in pkgsrc as wip/qemu-haxm. Apart from that, a kernel patch provided by Kamil is necesary. Below are the steps needed to get all this running on your NetBSD/8.0 system. Command lines prefixed by # are executed as root, $ as regular user

Hardware and software requirements

Installation

1. Install the source sets, patch kernel source and compile a new kernel

Install src xsrc syssrc sets. This can be done during initial system installation or later. In the latter case:

$ wget ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-8.0/source/sets/{src,xsrc,syssrc}.tgz
$ su
# for set in src syssrc xsrc; do tar -zxvf $set.tgz -C / ; done
    

Now patch the kernel source. You will need to patch one file, located in sys/arch/x86/x86/fpu.c

$ cd /usr/src
$ su
# curl http://www.netbsd.org/~kamil/patch-00083-fpu-dna-netbsd8.txt | patch -p0
    

Now, if that patch fails for some reason, just apply it manually, its a very small change. For the record, the patch contents are below

--- sys/arch/x86/x86/fpu.c10 Jul 2018 15:35:26 -00001.12.8.3
+++ sys/arch/x86/x86/fpu.c8 Feb 2019 15:01:40 -0000
@@ -440,9 +440,12 @@
struct pcb *pcb;
int s;

-if (!USERMODE(frame->tf_cs, frame->tf_eflags))
-panic("fpudna from kernel, ip %p, trapframe %p\n",
-    (void *)X86_TF_RIP(frame), frame);
+if (!USERMODE(frame->tf_cs, frame->tf_eflags)){
+  //panic("fpudna from kernel, ip %p, trapframe %p\n",
+  //    (void *)X86_TF_RIP(frame), frame);
+  clts();
+  return;
+}

if (i386_fpu_present == 0) {
send_sigill((void *)X86_TF_RIP(frame));
    

Now that you have patched the kernel, go ahead and build it.

# cd /usr/src
# ./build.sh -m amd64 tools
# ./build.sh -m amd64 kernel=GENERIC
# cp /netbsd /netbsd.old
# cp sys/arch/amd64/compile/obj/GENERIC/netbsd /
# reboot 
    

Should this fail, you have your old kernel backed up, strike the 5 key at the booloader and boot that instead.

Build HAXM

Install the Netwide Assembler (nasm) -- necesary to compile HAXM

# pkg_add nasm
    

Get HAXM sources and build it

$ git clone https://github.com/intel/haxm #OK
$ cd platforms/netbsd #OK
$ make
    

Load the HAXM module

$ cd haxm/platforms/netbsd
$ uname -a    # check uname. kernel MUST be 8.0
$ su
# modload ./haxm.kmod
    

You can run dmesg to verify the module was loaded correctly:

Prepare pkgsrc-wip and build qemu-haxm

Its necesary to use wip/qemu-haxm, so prepare pkgsrc-wip. There is another patch to be applied (for the time being. I asked for git access to pkgsrc-wip and I'll push it as soon as I get my account).

The following commands will install pkgsrc-wip on your system

# cd /usr/pkgsrc
# git clone --depth 1 git://wip.pkgsrc.org/pkgsrc-wip.git wip
    

Apply patch (this might not be necessary)

# cd qemu-haxm
# curl http://polprog.net/papiery/netbsd-haxm/pkgsrc-qemu-haxm.diff | patch
    

Build qemu-haxm with proper options. If you have previously installed qemu, use pkg_delete qemu to remove it

# make PKG_OPTIONS.qemu="sdl gtk3"
# make install
    

Create HAXM devices

Use this script to create HAXM devices and VCPU nodes

It's contents are below

#!/bin/sh
# Makes NetBSD HAXM devices


if [ $(id -u) !=  0 ]; then echo "You need to be root" 1>&2; exit 1; fi 

echo "*** Loading HAXM kmod ***"
modload ./haxm.kmod


echo "*** Making HAX devices ***"
mknod /dev/HAX c 220 0
mkdir /dev/hax_vm
for i in `seq 0 7`; do mkdir /dev/hax_vm0$i; done
for i in `seq 0 7`; do mknod /dev/hax_vm/vm0$i c 222 $i; done

echo "*** Making VCPU Nodes ***"

i=0
while [ $i -lt 128 ]; do
        vm=$(echo "$i/16"|bc)
        vcpu=$(echo "$i%16"|bc)
        echo "i=$i vm=$vm vcpu=$vcpu"
        cmd=$(printf "mknod /dev/hax_vm%02d/vcpu%02d c 221 %d" $vm $vcpu $i)
        $cmd
        let "i+=1"
done
    

Get a test system image

Windows will run only as 32 bit guest for now (due to HAXM not emulating CR8 (TPR) register that Windows uses. CR8 is a new control register added in amd64 used for task prioritization. Currently, a hack ignores CR8 changes in guest). Other amd64 operating systems should be OK. Testing is necessary!!! CR8 emulation should be in HAXM soon, since some people are actively working on it.

Grab a Win 7 x86 image from Microsoft. Select any x86 Win7 and VirtualBox (any platform should do, commands below are for VirtualBox image).

This test image works very well on my quad core, 3.3GHz i3 (!) with 8 GB of RAM test box. Windows is not laggy, mouse pointer is OK, everything works like it should!

Use the commands below to get a qcow2 image

$ unzip [zip file you downloaded]
$ tar -zvf [content].ova
(ignore tar error about corrupted archive, files are intact!)
$ qemu-img convert -f vmdk -O qcow2 [vmdk file] win7live.qcow2
    

OVA file is just a tar archive, so you have to untar it to get the vmdk file. You can remove the vmdk, ovf and zip after you make sure the image works to free up space

Run qemu and enjoy your HAX

# qemu-system-i386 -hda win7live.qcow2 -netdev user,id=nd0 -device rtl8139,netdev=nd0 -accel hax
    

The above command line will launch the windows image with working network. Enjoy

OS gallery

Below there are screenshots of operating systems that boot successfully for now :)

FreeBSD 12.0

Minix 3

Plan 9

Haiku, ToaruOS and KolibriOS - thanks to nia @ !




Sytems that currently fail to run under HAXM:

Win 10 - hangs at the loading screen, run with -m 4G, othwerise errors out on ramdisk creation. 64-bit OS

OpenBSD 6.4 - #GP ? (see src/trap.c) - github issue

Solaris 10 - trap in bootloader - github issue

ReactOS 0.4.6 - BSOD after selecting live mode (device detection dialog runs OK)

Icaros - thanks nia - github issue

Windows XP, Vista - hang host
Windows NT 4.0 - installer broken on normal QEMU
Windows 98 - Issues a VCPU shutdown request (?), emulation stops
QNX 6 - Reported to cause VCPU shutdown (nia)


We were enlightened by a different OS user, that some HAXM guest systems hanging the host kernel is a DoS attack vector :D

Back to homepage