Re: [PATCH PROTOTYPE 3/6] vfio: Implement support for sparse RAM memory regions

From: Peter Xu <peterx@redhat.com>
To: David Hildenbrand <david@redhat.com>
Cc: Le Tan <tamlokveer@gmail.com>,
	Pankaj Gupta <pankaj.gupta.linux@gmail.com>,
	"Michael S. Tsirkin" <mst@redhat.com>,
	wei.huang2@amd.com, qemu-devel@nongnu.org,
	Luiz Capitulino <lcapitulino@redhat.com>,
	Auger Eric <eric.auger@redhat.com>,
	Alex Williamson <alex.williamson@redhat.com>,
	Wei Yang <richardw.yang@linux.intel.com>,
	Igor Mammedov <imammedo@redhat.com>,
	Paolo Bonzini <pbonzini@redhat.com>,
	"Dr . David Alan Gilbert" <dgilbert@redhat.com>
Subject: Re: [PATCH PROTOTYPE 3/6] vfio: Implement support for sparse RAM memory regions
Date: Wed, 18 Nov 2020 10:23:11 -0500	[thread overview]
Message-ID: <20201118152311.GB29639@xz-x1> (raw)
In-Reply-To: <fcbea24d-c56a-12b4-4a7b-d8faa1e04047@redhat.com>

David,

On Wed, Nov 18, 2020 at 02:04:00PM +0100, David Hildenbrand wrote:
> On 20.10.20 22:44, Peter Xu wrote:
> > On Tue, Oct 20, 2020 at 10:01:12PM +0200, David Hildenbrand wrote:
> >> Thanks ... but I have an AMD system. Will try to find out how to get
> >> that running with AMD :)
> > 
> > May still start with trying intel-iommu first. :) I think it should work for
> > amd hosts too.
> > 
> > Just another FYI - Wei is working on amd-iommu for vfio [1], but it's still
> > during review.
> > 
> > [1] https://lore.kernel.org/qemu-devel/20201002145907.1294353-1-wei.huang2@amd.com/
> > 
> 
> I'm trying to get an iommu setup running (without virtio-mem!),
> but it's a big mess.
> 
> Essential parts of my QEMU cmdline are:
> 
> sudo build/qemu-system-x86_64 \
>     -accel kvm,kernel-irqchip=split \
>     ...
>      device pcie-pci-bridge,addr=1e.0,id=pci.1 \
>     -device vfio-pci,host=0c:00.0,x-vga=on,bus=pci.1,addr=1.0,multifunction=on \
>     -device vfio-pci,host=0c:00.1,bus=pci.1,addr=1.1 \
>     -device intel-iommu,caching-mode=on,intremap=on \

The intel-iommu device needs to be created before the rest of devices.  I
forgot the reason behind, should be related to how the device address spaces
are created.  This rule should apply to all the rest of vIOMMUs, afaiu.

Libvirt guarantees that ordering when VT-d enabled, though when using qemu
cmdline indeed that's hard to identify from the first glance... iirc we tried
to fix this, but I forgot the details, it's just not trivial.

I noticed that this ordering constraint is also missing in the qemu wiki page
of vt-d, so I updated there too, hopefully..

https://wiki.qemu.org/Features/VT-d#Command_Line_Example

> 
> I am running upstream QEMU + Linux -next kernel inside the
> guest on an AMD Ryzen 9 3900X 12-Core Processor.
> I am using SeaBios.
> 
> I tried faking an Intel CPU without luck.
> ("-cpu Skylake-Client,kvm=off,vendor=GenuineIntel")
> 
> As soon as I enable "intel_iommu=on" in my guest kernel, graphics
> stop working (random mess on graphics output) and I get
>   vfio-pci 0000:0c:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0023 address=0xff924000 flags=0x0000]
> in the hypervisor, along with other nice messages.
> 
> I can spot no vfio DMA mappings coming from an iommu, just as if the
> guest wouldn't even try to setup the iommu.
> 
> I tried with
> 1. AMD Radeon RX Vega 56
> 2. Nvidia GT220
> resulting in similar issues.
> 
> I also tried with "-device amd-iommu" with other issues
> (guest won't even boot up). Are my graphics card missing some support or
> is there a fundamental flaw in my setup?

I guess amd-iommu won't work if without Wei Huang's series applied.

> 
> Any clues appreciated.

Please try with above and see whether it works.  Thanks,

-- 
Peter Xu