Re: [RFC 00/18] vfio: Adopt iommufd

From: Eric Auger <eric.auger@redhat.com>
To: Nicolin Chen <nicolinc@nvidia.com>
Cc: Yi Liu <yi.l.liu@intel.com>,
	alex.williamson@redhat.com, cohuck@redhat.com,
	qemu-devel@nongnu.org, david@gibson.dropbear.id.au,
	thuth@redhat.com, farman@linux.ibm.com, mjrosato@linux.ibm.com,
	akrowiak@linux.ibm.com, pasic@linux.ibm.com,
	jjherne@linux.ibm.com, jasowang@redhat.com, kvm@vger.kernel.org,
	jgg@nvidia.com, eric.auger.pro@gmail.com, kevin.tian@intel.com,
	chao.p.peng@intel.com, yi.y.sun@intel.com, peterx@redhat.com
Subject: Re: [RFC 00/18] vfio: Adopt iommufd
Date: Mon, 25 Apr 2022 21:40:02 +0200	[thread overview]
Message-ID: <9ba412e7-85c1-8dd7-9a55-3a0078642bf8@redhat.com> (raw)
In-Reply-To: <Yl4r3Ok61wxCc2zd@Asurada-Nvidia>

Hi Nicolin,

On 4/19/22 5:26 AM, Nicolin Chen wrote:
> On Sun, Apr 17, 2022 at 12:30:40PM +0200, Eric Auger wrote:
>
>>>> - More tests
>>> I did a quick test on my ARM64 platform, using "iommu=smmuv3"
>>> string. The behaviors are different between using default and
>>> using legacy "iommufd=off".
>>>
>>> The legacy pathway exits the VM with:
>>>     vfio 0002:01:00.0:
>>>     failed to setup container for group 1:
>>>     memory listener initialization failed:
>>>     Region smmuv3-iommu-memory-region-16-0:
>>>     device 00.02.0 requires iommu MAP notifier which is not currently supported
>>>
>>> while the iommufd pathway started the VM but reported errors
>>> from host kernel about address translation failures, probably
>>> because of accessing unmapped addresses.
>>>
>>> I found iommufd pathway also calls error_propagate_prepend()
>>> to add to errp for not supporting IOMMU_NOTIFIER_MAP, but it
>>> doesn't get a chance to print errp out. Perhaps there should
>>> be a final error check somewhere to exit?
>> thank you for giving it a try.
>>
>> vsmmuv3 + vfio is not supported as we miss the HW nested stage support
>> and SMMU does not support cache mode. If you want to test viommu on ARM
>> you shall test virtio-iommu+vfio. This should work but this is not yet
>> tested.
> I tried "-device virtio-iommu" and "-device virtio-iommu-pci"
> separately with vfio-pci, but neither seems to work. The host
> SMMU driver reports Translation Faults.
>
> Do you know what commands I should use to run QEMU for that
> combination?
you shall use :

 -device virtio-iommu-pci -device vfio-pci,host=<BDF>

Please make sure the "-device virtio-iommu-pci" is set *before* the
"-device vfio-pci,"

Otherwise the IOMMU MR notifiers are not set properly and this may be
the cause of your physical SMMU translations faults.

Eric
>
>> I pushed a fix for the error notification issue:
>> qemu-for-5.17-rc6-vm-rfcv2-rc0 on my git https://github.com/eauger/qemu.git
> Yes. This fixes the problem. Thanks!
>