qemu-kvm hangs with DAX

qemu-kvm hangs with DAX

[Yigal Korman]

This is a re-post, I didn't send it to all relevant mailing lists before...

Original below.

Hi everyone,

I have an interesting issue with DAX and KVM - I'm trying to boot a VM
with its memory mapped to a DAX-mounted file (kernel 4.9).

The use case is a bit wacky but I'm trying to recreate something
similar to what clearlinux[1] described (although they don't use this
method anymore).

When mapping the memory to a regular ext4 file, the VM boots fine.
But when mapping to ext4+dax, the VM won't boot or perhaps boots
extremely slowly.
In both cases the FS is on a memory pmem device.

Here's a snippet of how I load things:

mkfs.ext4 /dev/pmem0
mount /dev/pmem0 /mnt
fallocate -l 512M /mnt/mem
qemu-system-x86_64 -nodefconfig -nodefaults \
 -drive if=virtio,file=centos7.qcow2,index=0,media=disk \
 --enable-kvm -serial telnet:localhost:4443,server,nowait \
 -device sga -m 512 -smp 1,sockets=1,cores=1,threads=1 \
 -object memory-backend-file,prealloc=yes,mem-path=/mnt/mem,share=on,size=512M,id=ram
\
 -numa node,nodeid=0,cpus=0,memdev=ram \
 -net nic,model=virtio,vlan=0 \
 -net user,vlan=0,hostname=vm,hostfwd=tcp:127.0.0.1:8001-:22 \
 -name test -monitor telnet:localhost:4444,server,nowait

I use a headless host so I usually connect to the VM with 'telnet
localhost 4443'.

The above works and the VM boots in seconds.
When adding '-o dax' to the mount command, I can catch the grub menu
during boot but it gets stuck.
Sometimes if I wait about 20 minutes, I see some kernel boot messages
appear, but no errors.

I've already tried something Dan Williams suggested - using 'dd'
instead of 'fallocate', but it didn't seem to help.

Also tried profiling the first 30s of qemu boot with 'perf stat' -
doesn't seem any clearer to me but here are the results:

for ext4 w/o DAX:

       4804.688402      task-clock (msec)         #    0.160 CPUs
utilized
            22,389      context-switches          #    0.005 M/sec
               144      cpu-migrations            #    0.030 K/sec
           158,611      page-faults               #    0.033 M/sec
     7,537,184,564      cycles                    #    1.569 GHz
     8,034,998,998      instructions              #    1.07  insn per
cycle
     1,612,266,593      branches                  #  335.561 M/sec
         8,574,733      branch-misses             #    0.53% of all
branches

for ext4 w/ DAX:

      30001.643354      task-clock (msec)         #    1.000 CPUs
utilized
               584      context-switches          #    0.019 K/sec
                12      cpu-migrations            #    0.000 K/sec
           274,575      page-faults               #    0.009 M/sec
     2,131,506,685      cycles                    #    0.071 GHz
     2,252,004,361      instructions              #    1.06  insn per
cycle
       439,086,052      branches                  #   14.635 M/sec
         2,663,760      branch-misses             #    0.61% of all
branches

Seems like w/o DAX, the boot will complete in seconds and the CPU will
remain idle and w/ DAX the CPU is working very hard and there much
more page-faults.

Any thoughts?

Regards,
Yigal

Re: qemu-kvm hangs with DAX

[Paolo Bonzini]

On 24/04/2017 08:16, Yigal Korman wrote:
> This is a re-post, I didn't send it to all relevant mailing lists before...
> 
> Original below.
> 
> Hi everyone,
> 
> I have an interesting issue with DAX and KVM - I'm trying to boot a VM
> with its memory mapped to a DAX-mounted file (kernel 4.9).
> 
> The use case is a bit wacky but I'm trying to recreate something
> similar to what clearlinux[1] described (although they don't use this
> method anymore).
> 
> When mapping the memory to a regular ext4 file, the VM boots fine.
> But when mapping to ext4+dax, the VM won't boot or perhaps boots
> extremely slowly.
> In both cases the FS is on a memory pmem device.
> 
> Here's a snippet of how I load things:
> 
> mkfs.ext4 /dev/pmem0
> mount /dev/pmem0 /mnt
> fallocate -l 512M /mnt/mem
> qemu-system-x86_64 -nodefconfig -nodefaults \
>  -drive if=virtio,file=centos7.qcow2,index=0,media=disk \
>  --enable-kvm -serial telnet:localhost:4443,server,nowait \
>  -device sga -m 512 -smp 1,sockets=1,cores=1,threads=1 \
>  -object memory-backend-file,prealloc=yes,mem-path=/mnt/mem,share=on,size=512M,id=ram
> \
>  -numa node,nodeid=0,cpus=0,memdev=ram \
>  -net nic,model=virtio,vlan=0 \
>  -net user,vlan=0,hostname=vm,hostfwd=tcp:127.0.0.1:8001-:22 \
>  -name test -monitor telnet:localhost:4444,server,nowait
> 
> I use a headless host so I usually connect to the VM with 'telnet
> localhost 4443'.
> 
> The above works and the VM boots in seconds.
> When adding '-o dax' to the mount command, I can catch the grub menu
> during boot but it gets stuck.
> Sometimes if I wait about 20 minutes, I see some kernel boot messages
> appear, but no errors.
> 
> I've already tried something Dan Williams suggested - using 'dd'
> instead of 'fallocate', but it didn't seem to help.
> 
> Also tried profiling the first 30s of qemu boot with 'perf stat' -
> doesn't seem any clearer to me but here are the results:
> 
> for ext4 w/o DAX:
> 
>        4804.688402      task-clock (msec)         #    0.160 CPUs
> utilized
>             22,389      context-switches          #    0.005 M/sec
>                144      cpu-migrations            #    0.030 K/sec
>            158,611      page-faults               #    0.033 M/sec
>      7,537,184,564      cycles                    #    1.569 GHz
>      8,034,998,998      instructions              #    1.07  insn per
> cycle
>      1,612,266,593      branches                  #  335.561 M/sec
>          8,574,733      branch-misses             #    0.53% of all
> branches
> 
> for ext4 w/ DAX:
> 
>       30001.643354      task-clock (msec)         #    1.000 CPUs
> utilized
>                584      context-switches          #    0.019 K/sec
>                 12      cpu-migrations            #    0.000 K/sec
>            274,575      page-faults               #    0.009 M/sec
>      2,131,506,685      cycles                    #    0.071 GHz
>      2,252,004,361      instructions              #    1.06  insn per
> cycle
>        439,086,052      branches                  #   14.635 M/sec
>          2,663,760      branch-misses             #    0.61% of all
> branches
> 
> Seems like w/o DAX, the boot will complete in seconds and the CPU will
> remain idle and w/ DAX the CPU is working very hard and there much
> more page-faults.

Can you try catching a trace with the following command:

trace-cmd record -e kvm -e kvmmmu qemu-kvm *rest of qemu command line*

(requires running as root)? Please send it xz-ipped to me, not the
mailing list, since it can be large.

Paolo