Re: AMD Ryzen KVM/NPT/IOMMU issue

[Graham Neville <grahamneville-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]

[-- Attachment #1.1: Type: text/plain, Size: 4766 bytes --]

Although not related to graphics card performance, there is definitely
another issue with regards to running KVM nested L2 guests when npt=1.

Thought I'd mention this in case it helps with identifying performance
issues with NPT.

I'm unable to start any L2 guests with KVM acceleration (--enable-kvm). As
soon as it attempts to bring up the L2 guest the L1 host crashes, L0 host
remains online. Nothing is printed in either L1 or L0's dmesg.

My L0 is running Arch with 4.11.0-rc6, with qemu 2.8.0. I've tried
different L1 hosts (Ubuntu,Arch) and different kernels right to 4.12-rc5
kernel, along with different qemu versions.

This used to work fine with my Intel i7-4770s setup.

With npt=0, L2 guests can start but performance is dier.

On Wed, Jun 28, 2017 at 7:29 PM, Bridgman, John <John.Bridgman-5C7GfCeVMHo@public.gmane.org>
wrote:

>
>
> From: Alex Williamson <alex.williamson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Sent: June 28, 2017 3:08 PM
> To: Suthikulpanit, Suravee
> Cc: Steven Walter; Nick Sarnie; Paolo Bonzini;
> iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; Matthias Ehrenfeuchter;
> kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Bridgman, John
> Subject: Re: AMD Ryzen KVM/NPT/IOMMU issue
>
> On Thu, 29 Jun 2017 01:53:57 +0700
> Suravee Suthikulpanit <Suravee.Suthikulpanit-5C7GfCeVMHo@public.gmane.org> wrote:
>
> > On 6/29/17 00:26, Steven Walter wrote:
> > >> So, I'm trying to reproduce this issue on the Ryzen system w/ the
> following
> > >> setup:
> > >>
> > >>   * Host kernel v4.11 (with this patch  https://lkml.org/lkml/2017/6/
> 23/295)
> > >>
> > >>   * guest VM RHEL7.3
> > >>
> > >>   * guest graphic driver = radeon
> > >>
> > >>   * qemu-system-x86_64 --version
> > >>     QEMU emulator version 2.9.50 (v2.9.0-1659-g577caa2-dirty)
> > >>
> > >>   * kvm-amd npt=1
> > >>
> > >>   * dGPU is 08:00.0 VGA compatible controller: Advanced Micro
> Devices, Inc.
> > >> [AMD/ATI] Tobago PRO [Radeon R7 360 / R9 360 OEM] (rev 81)
> > >>
> > >>   * qemu-system-x86_64 -smp 4 -enable-kvm -M q35 -m 4096 -cpu host
> -bios
> > >> /usr/share/qemu/bios.bin -device
> > >> ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,
> chassis=1,id=root.1
> > >> -drive file=/sandbox/vm-images/rhel7.3.qcow2,if=virtio,id=disk0 -net
> none
> > >> -vga none -nodefaults -device
> > >> vfio-pci,host=08:00.0,x-vga=on,addr=0.0,multifunction=on,
> bus=root.1,romfile=/sandbox/vm-images/vbios.rom
> > >> -usb -device usb-host,hostbus=3,hostport=1 -device
> > >> usb-host,hostbus=3,hostport=3 -device vfio-pci,host=0000:08:00.1
> -device
> > >> vfio-pci,host=0000:09:00.0
> > >>
> > >> With this setup, I am able to pass-through the dGPU and run the
> following
> > >> test:
> > >>   * Starting up the guest w/ full GNOME GUI on the attached monitor.
> > >>   * glxgears (running @ 60 FPS)
> > >>   * Playing 1080p HD video on Youtube
> > >>
> > >> I am not noticing issues here. What kind of test are you running in
> the
> > >> guest VM?
> > > Try running the open source game "torcs" inside the VM.  I think
> > > you'll find that there's a very noticeable performance different
> > > between npt=1 and npt=0
> >
> > Hm.. actually torcs seems to be running fine w/ npt=1 setup. Although I
> think my
> > driving skill is ~20% worse :(
>
> >A clarification on the issue, it's not that these games/benchmarks
> >don't work or aren't playable, it's that they run slower (as measured by
> >framerate) with npt=1 vs npt=0.  A virtualized guest on AMD hardware is
> >hindered either by lower graphics performance (npt=1) or higher CPU
> >virtualization overhead (npt=0) for high intensity games or graphics
> >workloads.  Intel's equivalent ept feature does not have this issue.  I
> >would encourage looking at the framerate for one mode vs the other
> >before drawing any conclusions on whether it's working well.  Thanks,
> >Alex
>
> One more data point - Nick did some testing with Xen enabling/disabling
> npt and
> found that (a) performance was not affected much whether npt was on or
> off, and
> (b) performance with npt on was pretty close to KVM performance with npt
> off.
>
> This suggests something specific to KVM that doesn't play well with npt,
> although
> I have no idea what that might be. I was going to talk to our folks to see
> if they had
> any suggestions re: ways to narrow down where the performance impact is
> coming
> from, but I ended up going off sick instead.
>
> Haven't gone through the whole thread here yet so apologies if this has
> already been
> mentioned (and Nick sorry for the delay).
>
> _______________________________________________
> iommu mailing list
> iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu
>

[-- Attachment #1.2: Type: text/html, Size: 6552 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]