From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:42438) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dSg0S-0000XD-R2 for qemu-devel@nongnu.org; Wed, 05 Jul 2017 04:49:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dSg0R-00055v-3j for qemu-devel@nongnu.org; Wed, 05 Jul 2017 04:49:56 -0400 From: Bharat Bhushan Date: Wed, 5 Jul 2017 08:49:43 +0000 Message-ID: References: <1496851287-9428-1-git-send-email-eric.auger@redhat.com> <14c83c14-1e7e-f142-77b0-fa8c873e2d41@redhat.com> <6f22079e-4b4b-6476-77be-51ec83104c0d@redhat.com> <8a7fb667-e6c0-2412-6dca-a8233dd27836@redhat.com> In-Reply-To: <8a7fb667-e6c0-2412-6dca-a8233dd27836@redhat.com> Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Auger Eric , "eric.auger.pro@gmail.com" , "peter.maydell@linaro.org" , "alex.williamson@redhat.com" , "mst@redhat.com" , "qemu-arm@nongnu.org" , "qemu-devel@nongnu.org" , "jean-philippe.brucker@arm.com" Cc: "wei@redhat.com" , "kevin.tian@intel.com" , "marc.zyngier@arm.com" , "tn@semihalf.com" , "will.deacon@arm.com" , "drjones@redhat.com" , "robin.murphy@arm.com" , "christoffer.dall@linaro.org" Hi Eric, > -----Original Message----- > From: Auger Eric [mailto:eric.auger@redhat.com] > Sent: Wednesday, July 05, 2017 2:14 PM > To: Bharat Bhushan ; > eric.auger.pro@gmail.com; peter.maydell@linaro.org; > alex.williamson@redhat.com; mst@redhat.com; qemu-arm@nongnu.org; > qemu-devel@nongnu.org; jean-philippe.brucker@arm.com > Cc: wei@redhat.com; kevin.tian@intel.com; marc.zyngier@arm.com; > tn@semihalf.com; will.deacon@arm.com; drjones@redhat.com; > robin.murphy@arm.com; christoffer.dall@linaro.org > Subject: Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device >=20 > Hi Bharat, >=20 > On 05/07/2017 10:23, Bharat Bhushan wrote: > > Hi Eric, > > > >> -----Original Message----- > >> From: Auger Eric [mailto:eric.auger@redhat.com] > >> Sent: Monday, June 26, 2017 1:25 PM > >> To: Bharat Bhushan ; > >> eric.auger.pro@gmail.com; peter.maydell@linaro.org; > >> alex.williamson@redhat.com; mst@redhat.com; qemu-arm@nongnu.org; > >> qemu-devel@nongnu.org; jean-philippe.brucker@arm.com > >> Cc: wei@redhat.com; kevin.tian@intel.com; marc.zyngier@arm.com; > >> tn@semihalf.com; will.deacon@arm.com; drjones@redhat.com; > >> robin.murphy@arm.com; christoffer.dall@linaro.org > >> Subject: Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device > >> > >> Hi Bharat, > >> > >> On 19/06/2017 09:54, Bharat Bhushan wrote: > >>> Hi Eric, > >>> > >>> I started added replay in virtio-iommu and came across how MSI > >>> interrupts > >> with work with VFIO. > >>> I understand that on intel this works differently but vsmmu will > >>> have same > >> requirement. > >>> kvm-msi-irq-route are added using the msi-address to be translated > >>> by > >> viommu and not the final translated address. > >>> While currently the irqfd framework does not know about emulated > >> iommus (virtio-iommu, vsmmuv3/vintel-iommu). > >>> So in my view we have following options: > >>> - Programming with translated address when setting up > >>> kvm-msi-irq-route > >>> - Route the interrupts via QEMU, which is bad from performance > >>> - vhost-virtio-iommu may solve the problem in long term > >> > >> Sorry for the delay. With regard to the vsmmuv3/vfio integration I > >> think we need to use the guest physical address otherwise the MSI > >> address will not be recognized as an MSI doorbell. > >> > >> Also the fact on ARM we map the MSI doorbell causes an assert in > >> vfio_get_vaddr() as the vITS doorbell is not a RAM region. We will > >> need to handle this specifically. > > > > Also when setup msi-route kvm_irqchip_add_msi_route() we needed to > provide the translated address. > > According to my understanding this is required because kernel does no g= o > through viommu translation when generating interrupt, no? >=20 > yes this is needed when KVM MSI routes are set up, ie. along with GICV3 I= TS. > With GICv2M, qemu direct gsi mapping is used and this is not needed. >=20 > So I do not understand your previous sentence saying "MSI interrupts work= s > without any change". I have almost completed vfio integration with virtio-iommu and now testing = the changes by assigning e1000 device to VM. For this I have changed virtio= -iommu driver to use IOMMU_RESV_MSI rather than sw-msi and this does not ne= ed changed in vfio_get_addr() and kvm_irqchip_add_msi_route() Thanks -Bharat >=20 > Thanks >=20 > Eric >=20 > > > > Thanks > > -Bharat > > > >> > >> Besides I have not looked specifically at the virtio-iommu/vfio > >> integration yet. > >> > >> Thanks > >> > >> Eric > >>> > >>> Is there any other better option I am missing? > >>> > >>> Thanks > >>> -Bharat > >>> > >>>> -----Original Message----- > >>>> From: Auger Eric [mailto:eric.auger@redhat.com] > >>>> Sent: Friday, June 09, 2017 5:24 PM > >>>> To: Bharat Bhushan ; > >>>> eric.auger.pro@gmail.com; peter.maydell@linaro.org; > >>>> alex.williamson@redhat.com; mst@redhat.com; qemu- > arm@nongnu.org; > >>>> qemu-devel@nongnu.org; jean-philippe.brucker@arm.com > >>>> Cc: wei@redhat.com; kevin.tian@intel.com; marc.zyngier@arm.com; > >>>> tn@semihalf.com; will.deacon@arm.com; drjones@redhat.com; > >>>> robin.murphy@arm.com; christoffer.dall@linaro.org > >>>> Subject: Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device > >>>> > >>>> Hi Bharat, > >>>> > >>>> On 09/06/2017 13:30, Bharat Bhushan wrote: > >>>>> Hi Eric, > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: Auger Eric [mailto:eric.auger@redhat.com] > >>>>>> Sent: Friday, June 09, 2017 12:14 PM > >>>>>> To: Bharat Bhushan ; > >>>>>> eric.auger.pro@gmail.com; peter.maydell@linaro.org; > >>>>>> alex.williamson@redhat.com; mst@redhat.com; qemu- > >> arm@nongnu.org; > >>>>>> qemu-devel@nongnu.org; jean-philippe.brucker@arm.com > >>>>>> Cc: will.deacon@arm.com; robin.murphy@arm.com; > >>>> kevin.tian@intel.com; > >>>>>> marc.zyngier@arm.com; christoffer.dall@linaro.org; > >>>>>> drjones@redhat.com; wei@redhat.com; tn@semihalf.com > >>>>>> Subject: Re: [RFC v2 0/8] VIRTIO-IOMMU device > >>>>>> > >>>>>> Hi Bharat, > >>>>>> > >>>>>> On 09/06/2017 08:16, Bharat Bhushan wrote: > >>>>>>> Hi Eric, > >>>>>>> > >>>>>>>> -----Original Message----- > >>>>>>>> From: Eric Auger [mailto:eric.auger@redhat.com] > >>>>>>>> Sent: Wednesday, June 07, 2017 9:31 PM > >>>>>>>> To: eric.auger.pro@gmail.com; eric.auger@redhat.com; > >>>>>>>> peter.maydell@linaro.org; alex.williamson@redhat.com; > >>>>>> mst@redhat.com; > >>>>>>>> qemu-arm@nongnu.org; qemu-devel@nongnu.org; jean- > >>>>>>>> philippe.brucker@arm.com > >>>>>>>> Cc: will.deacon@arm.com; robin.murphy@arm.com; > >>>>>> kevin.tian@intel.com; > >>>>>>>> marc.zyngier@arm.com; christoffer.dall@linaro.org; > >>>>>>>> drjones@redhat.com; wei@redhat.com; tn@semihalf.com; > Bharat > >>>>>> Bhushan > >>>>>>>> > >>>>>>>> Subject: [RFC v2 0/8] VIRTIO-IOMMU device > >>>>>>>> > >>>>>>>> This series implements the virtio-iommu device. This is a proof > >>>>>>>> of concept based on the virtio-iommu specification written by > >>>>>>>> Jean-Philippe > >>>>>> Brucker [1]. > >>>>>>>> This was tested with a guest using the virtio-iommu driver [2] > >>>>>>>> and exposed with a virtio-net-pci using dma ops. > >>>>>>>> > >>>>>>>> The device gets instantiated using the "-device virtio-iommu- > device" > >>>>>>>> option. It currently works with ARM virt machine only as the > >>>>>>>> machine must handle the dt binding between the virtio-mmio > >> "iommu" > >>>>>>>> node and the PCI host bridge node. ACPI booting is not yet > >> supported. > >>>>>>>> > >>>>>>>> This should allow to start some benchmarking activities against > >>>>>>>> pure emulated IOMMU (especially ARM SMMU). > >>>>>>> > >>>>>>> I am testing this on ARM64 and see below continuous error prints: > >>>>>>> > >>>>>>> virtio_iommu_translate sid=3D8 is not known!! > >>>>>>> virtio_iommu_translate sid=3D8 is not known!! > >>>>>>> virtio_iommu_translate sid=3D8 is not known!! > >>>>>>> virtio_iommu_translate sid=3D8 is not known!! > >>>>>>> virtio_iommu_translate sid=3D8 is not known!! > >>>>>>> virtio_iommu_translate sid=3D8 is not known!! > >>>>>>> virtio_iommu_translate sid=3D8 is not known!! > >>>>>>> virtio_iommu_translate sid=3D8 is not known!! > >>>>>>> virtio_iommu_translate sid=3D8 is not known!! > >>>>>>> virtio_iommu_translate sid=3D8 is not known!! > >>>>>>> > >>>>>>> > >>>>>>> Also in guest I do not see device-tree node with virtio-iommu. > >>>>>> do you mean the virtio-mmio with #iommu-cells property? > >>>>>> > >>>>>> This one is created statically by virt machine. I would be > >>>>>> surprised if it were not there. Are you using the virt =3D virt2.1= 0 > machine. > >>>>>> Machines before do not support its instantiation. > >>>>>> > >>>>>> Please can you add a printf in hw/arm/virt.c create_virtio_mmio() > >>>>>> at the moment when this node is created. Also you can add a > >>>>>> printf in > >>>>>> bind_virtio_iommu_device() to make sure the binding with the PCI > >>>>>> host bridge is added on machine init done. > >>>>>> > >>>>>> Also worth to check, CONFIG_VIRTIO_IOMMU=3Dy on guest side. > >>>>> > >>>>> It works on my side. > >>>> Great. > >>>> > >>>> The driver config was disabled and also I was using guest kernel > >>>> which was not have deferred-probing. > >>>> Yes I did not mention in my cover letter the guest I have been > >>>> using is based on Jean-Philippe's branch, featuring deferred IOMMU > probing. > >>>> I I have not tried yet with an upstream guest. > >>>> Now after fixing it works on my side > >>>>> I placed some prints to see dma-map are mapping regions in > >>>>> virtio-iommu, > >>>> it uses emulated iommu. > >>>>> > >>>>> I will continue to add VFIO support now on this and more testing !! > >>>> > >>>> OK. I will do the VFIO integration first on the vsmmuv3 device as I > >>>> already prepared the VFIO replay and hopefully we will sync ;-) > >>>> > >>>> Thanks > >>>> > >>>> Eric > >>>>> > >>>>> Thanks > >>>>> -Bharat > >>>>> > >>>>>> > >>>>>> Thanks > >>>>>> > >>>>>> Eric > >>>>>> > >>>>>>> I am using qemu-tree you mentioned below and iommu-driver > >> patches > >>>>>> published by Jean-P. > >>>>>>> Qemu command line have additional ""-device virtio-iommu- > device". > >>>>>>> What > >>>>>> I am missing ? > >>>>>> > >>>>>> > >>>>>>> > >>>>>>> Thanks > >>>>>>> -Bharat > >>>>>>> > >>>>>>>> > >>>>>>>> Best Regards > >>>>>>>> > >>>>>>>> Eric > >>>>>>>> > >>>>>>>> This series can be found at: > >>>>>>>> https://github.com/eauger/qemu/tree/virtio-iommu-rfcv2 > >>>>>>>> > >>>>>>>> References: > >>>>>>>> [1] [RFC 0/3] virtio-iommu: a paravirtualized IOMMU, [2] [RFC > >>>>>>>> PATCH linux] > >>>>>>>> iommu: Add virtio-iommu driver [3] [RFC PATCH kvmtool 00/15] > >>>>>>>> Add > >>>>>>>> virtio- iommu > >>>>>>>> > >>>>>>>> History: > >>>>>>>> v1 -> v2: > >>>>>>>> - fix redifinition of viommu_as typedef > >>>>>>>> > >>>>>>>> Eric Auger (8): > >>>>>>>> update-linux-headers: import virtio_iommu.h > >>>>>>>> linux-headers: Update for virtio-iommu > >>>>>>>> virtio_iommu: add skeleton > >>>>>>>> virtio-iommu: Decode the command payload > >>>>>>>> virtio_iommu: Add the iommu regions > >>>>>>>> virtio-iommu: Implement the translation and commands > >>>>>>>> hw/arm/virt: Add 2.10 machine type > >>>>>>>> hw/arm/virt: Add virtio-iommu the virt board > >>>>>>>> > >>>>>>>> hw/arm/virt.c | 116 ++++- > >>>>>>>> hw/virtio/Makefile.objs | 1 + > >>>>>>>> hw/virtio/trace-events | 14 + > >>>>>>>> hw/virtio/virtio-iommu.c | 623 > >>>>>> ++++++++++++++++++++++++++ > >>>>>>>> include/hw/arm/virt.h | 5 + > >>>>>>>> include/hw/virtio/virtio-iommu.h | 60 +++ > >>>>>>>> include/standard-headers/linux/virtio_ids.h | 1 + > >>>>>>>> include/standard-headers/linux/virtio_iommu.h | 142 ++++++ > >>>>>>>> linux-headers/linux/virtio_iommu.h | 1 + > >>>>>>>> scripts/update-linux-headers.sh | 3 + > >>>>>>>> 10 files changed, 957 insertions(+), 9 deletions(-) create > >>>>>>>> mode > >>>>>>>> 100644 hw/virtio/virtio-iommu.c create mode 100644 > >>>>>>>> include/hw/virtio/virtio- iommu.h create mode 100644 > >>>>>>>> include/standard- headers/linux/virtio_iommu.h create mode > >>>>>>>> 100644 linux-headers/linux/virtio_iommu.h > >>>>>>>> > >>>>>>>> -- > >>>>>>>> 2.5.5 > >>>>>>> > >>>>> > >>>