From: "Liu, Yi L" <yi.l.liu@intel.com> To: Stefan Hajnoczi <stefanha@gmail.com> Cc: "alex.williamson@redhat.com" <alex.williamson@redhat.com>, "eric.auger@redhat.com" <eric.auger@redhat.com>, "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>, "joro@8bytes.org" <joro@8bytes.org>, "Tian, Kevin" <kevin.tian@intel.com>, "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>, "Raj, Ashok" <ashok.raj@intel.com>, "Tian, Jun J" <jun.j.tian@intel.com>, "Sun, Yi Y" <yi.y.sun@intel.com>, "jean-philippe@linaro.org" <jean-philippe@linaro.org>, "peterx@redhat.com" <peterx@redhat.com>, "Wu, Hao" <hao.wu@intel.com>, "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>, "kvm@vger.kernel.org" <kvm@vger.kernel.org>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org> Subject: RE: [PATCH v2 14/15] vfio: Document dual stage control Date: Wed, 17 Jun 2020 06:27:27 +0000 [thread overview] Message-ID: <DM5PR11MB1435C484283BDCD75F19EDB5C39A0@DM5PR11MB1435.namprd11.prod.outlook.com> (raw) In-Reply-To: <20200615094128.GB1491454@stefanha-x1.localdomain> > From: Stefan Hajnoczi <stefanha@gmail.com> > Sent: Monday, June 15, 2020 5:41 PM > On Thu, Jun 11, 2020 at 05:15:33AM -0700, Liu Yi L wrote: > > > From: Eric Auger <eric.auger@redhat.com> > > > > The VFIO API was enhanced to support nested stage control: a bunch of > > new iotcls and usage guideline. > > > > Let's document the process to follow to set up nested mode. > > > > Cc: Kevin Tian <kevin.tian@intel.com> > > CC: Jacob Pan <jacob.jun.pan@linux.intel.com> > > Cc: Alex Williamson <alex.williamson@redhat.com> > > Cc: Eric Auger <eric.auger@redhat.com> > > Cc: Jean-Philippe Brucker <jean-philippe@linaro.org> > > Cc: Joerg Roedel <joro@8bytes.org> > > Cc: Lu Baolu <baolu.lu@linux.intel.com> > > Signed-off-by: Eric Auger <eric.auger@redhat.com> > > Signed-off-by: Liu Yi L <yi.l.liu@intel.com> > > --- > > v1 -> v2: > > *) new in v2, compared with Eric's original version, pasid table bind > > and fault reporting is removed as this series doesn't cover them. > > Original version from Eric. > > https://lkml.org/lkml/2020/3/20/700 > > > > Documentation/driver-api/vfio.rst | 64 > > +++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 64 insertions(+) > > > > diff --git a/Documentation/driver-api/vfio.rst > > b/Documentation/driver-api/vfio.rst > > index f1a4d3c..06224bd 100644 > > --- a/Documentation/driver-api/vfio.rst > > +++ b/Documentation/driver-api/vfio.rst > > @@ -239,6 +239,70 @@ group and can access them as follows:: > > /* Gratuitous device reset and go... */ > > ioctl(device, VFIO_DEVICE_RESET); > > > > +IOMMU Dual Stage Control > > +------------------------ > > + > > +Some IOMMUs support 2 stages/levels of translation. Stage corresponds > > +to the ARM terminology while level corresponds to Intel's VTD terminology. > > +In the following text we use either without distinction. > > + > > +This is useful when the guest is exposed with a virtual IOMMU and > > +some devices are assigned to the guest through VFIO. Then the guest > > +OS can use stage 1 (GIOVA -> GPA or GVA->GPA), while the hypervisor > > +uses stage 2 for VM isolation (GPA -> HPA). > > + > > +Under dual stage translation, the guest gets ownership of the stage 1 > > +page tables and also owns stage 1 configuration structures. The > > +hypervisor owns the root configuration structure (for security > > +reason), including stage 2 configuration. This works as long > > +configuration structures and page table > > s/as long configuration/as long as configuration/ got it. > > > +format are compatible between the virtual IOMMU and the physical IOMMU. > > s/format/formats/ I see. > > + > > +Assuming the HW supports it, this nested mode is selected by choosing > > +the VFIO_TYPE1_NESTING_IOMMU type through: > > + > > + ioctl(container, VFIO_SET_IOMMU, VFIO_TYPE1_NESTING_IOMMU); > > + > > +This forces the hypervisor to use the stage 2, leaving stage 1 > > +available for guest usage. The guest stage 1 format depends on IOMMU > > +vendor, and it is the same with the nesting configuration method. > > +User space should check the format and configuration method after > > +setting nesting type by > > +using: > > + > > + ioctl(container->fd, VFIO_IOMMU_GET_INFO, &nesting_info); > > + > > +Details can be found in Documentation/userspace-api/iommu.rst. For > > +Intel VT-d, each stage 1 page table is bound to host by: > > + > > + nesting_op->flags = VFIO_IOMMU_NESTING_OP_BIND_PGTBL; > > + memcpy(&nesting_op->data, &bind_data, sizeof(bind_data)); > > + ioctl(container->fd, VFIO_IOMMU_NESTING_OP, nesting_op); > > + > > +As mentioned above, guest OS may use stage 1 for GIOVA->GPA or GVA->GPA. > > +GVA->GPA page tables are available when PASID (Process Address Space > > +GVA->ID) > > +is exposed to guest. e.g. guest with PASID-capable devices assigned. > > +For such page table binding, the bind_data should include PASID info, > > +which is allocated by guest itself or by host. This depends on > > +hardware vendor e.g. Intel VT-d requires to allocate PASID from host. > > +This requirement is available by VFIO_IOMMU_GET_INFO. User space > > +could allocate PASID from host by: > > + > > + req.flags = VFIO_IOMMU_ALLOC_PASID; > > + ioctl(container, VFIO_IOMMU_PASID_REQUEST, &req); > > It is not clear how the userspace application determines whether PASIDs must be > allocated from the host via VFIO_IOMMU_PASID_REQUEST or if the guest itself can > allocate PASIDs. The text mentions VFIO_IOMMU_GET_INFO but what exactly > should the userspace application check? For VT-d, spec 3.0 introduced Virtual Cmd interface for PASID allocation, guest request PASID from host if it detects the interface. Application should check the IOMMU_NESTING_FEAT_SYSWIDE_PASID setting in the below info reported by VFIO_IOMMU_GET_INFO. And virtual VT-d should not report SVA related capabilities to guest if SYSWIDE_PASID is not supported by kernel. +struct iommu_nesting_info { + __u32 size; + __u32 format; + __u32 features; +#define IOMMU_NESTING_FEAT_SYSWIDE_PASID (1 << 0) +#define IOMMU_NESTING_FEAT_BIND_PGTBL (1 << 1) +#define IOMMU_NESTING_FEAT_CACHE_INVLD (1 << 2) + __u32 flags; + __u8 data[]; +}; https://lore.kernel.org/linux-iommu/1591877734-66527-3-git-send-email-yi.l.liu@intel.com/ Regards, Yi Liu
WARNING: multiple messages have this Message-ID (diff)
From: "Liu, Yi L" <yi.l.liu@intel.com> To: Stefan Hajnoczi <stefanha@gmail.com> Cc: "jean-philippe@linaro.org" <jean-philippe@linaro.org>, "Tian, Kevin" <kevin.tian@intel.com>, "Raj, Ashok" <ashok.raj@intel.com>, "kvm@vger.kernel.org" <kvm@vger.kernel.org>, "Sun, Yi Y" <yi.y.sun@intel.com>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "alex.williamson@redhat.com" <alex.williamson@redhat.com>, "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>, "Wu, Hao" <hao.wu@intel.com>, "Tian, Jun J" <jun.j.tian@intel.com> Subject: RE: [PATCH v2 14/15] vfio: Document dual stage control Date: Wed, 17 Jun 2020 06:27:27 +0000 [thread overview] Message-ID: <DM5PR11MB1435C484283BDCD75F19EDB5C39A0@DM5PR11MB1435.namprd11.prod.outlook.com> (raw) In-Reply-To: <20200615094128.GB1491454@stefanha-x1.localdomain> > From: Stefan Hajnoczi <stefanha@gmail.com> > Sent: Monday, June 15, 2020 5:41 PM > On Thu, Jun 11, 2020 at 05:15:33AM -0700, Liu Yi L wrote: > > > From: Eric Auger <eric.auger@redhat.com> > > > > The VFIO API was enhanced to support nested stage control: a bunch of > > new iotcls and usage guideline. > > > > Let's document the process to follow to set up nested mode. > > > > Cc: Kevin Tian <kevin.tian@intel.com> > > CC: Jacob Pan <jacob.jun.pan@linux.intel.com> > > Cc: Alex Williamson <alex.williamson@redhat.com> > > Cc: Eric Auger <eric.auger@redhat.com> > > Cc: Jean-Philippe Brucker <jean-philippe@linaro.org> > > Cc: Joerg Roedel <joro@8bytes.org> > > Cc: Lu Baolu <baolu.lu@linux.intel.com> > > Signed-off-by: Eric Auger <eric.auger@redhat.com> > > Signed-off-by: Liu Yi L <yi.l.liu@intel.com> > > --- > > v1 -> v2: > > *) new in v2, compared with Eric's original version, pasid table bind > > and fault reporting is removed as this series doesn't cover them. > > Original version from Eric. > > https://lkml.org/lkml/2020/3/20/700 > > > > Documentation/driver-api/vfio.rst | 64 > > +++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 64 insertions(+) > > > > diff --git a/Documentation/driver-api/vfio.rst > > b/Documentation/driver-api/vfio.rst > > index f1a4d3c..06224bd 100644 > > --- a/Documentation/driver-api/vfio.rst > > +++ b/Documentation/driver-api/vfio.rst > > @@ -239,6 +239,70 @@ group and can access them as follows:: > > /* Gratuitous device reset and go... */ > > ioctl(device, VFIO_DEVICE_RESET); > > > > +IOMMU Dual Stage Control > > +------------------------ > > + > > +Some IOMMUs support 2 stages/levels of translation. Stage corresponds > > +to the ARM terminology while level corresponds to Intel's VTD terminology. > > +In the following text we use either without distinction. > > + > > +This is useful when the guest is exposed with a virtual IOMMU and > > +some devices are assigned to the guest through VFIO. Then the guest > > +OS can use stage 1 (GIOVA -> GPA or GVA->GPA), while the hypervisor > > +uses stage 2 for VM isolation (GPA -> HPA). > > + > > +Under dual stage translation, the guest gets ownership of the stage 1 > > +page tables and also owns stage 1 configuration structures. The > > +hypervisor owns the root configuration structure (for security > > +reason), including stage 2 configuration. This works as long > > +configuration structures and page table > > s/as long configuration/as long as configuration/ got it. > > > +format are compatible between the virtual IOMMU and the physical IOMMU. > > s/format/formats/ I see. > > + > > +Assuming the HW supports it, this nested mode is selected by choosing > > +the VFIO_TYPE1_NESTING_IOMMU type through: > > + > > + ioctl(container, VFIO_SET_IOMMU, VFIO_TYPE1_NESTING_IOMMU); > > + > > +This forces the hypervisor to use the stage 2, leaving stage 1 > > +available for guest usage. The guest stage 1 format depends on IOMMU > > +vendor, and it is the same with the nesting configuration method. > > +User space should check the format and configuration method after > > +setting nesting type by > > +using: > > + > > + ioctl(container->fd, VFIO_IOMMU_GET_INFO, &nesting_info); > > + > > +Details can be found in Documentation/userspace-api/iommu.rst. For > > +Intel VT-d, each stage 1 page table is bound to host by: > > + > > + nesting_op->flags = VFIO_IOMMU_NESTING_OP_BIND_PGTBL; > > + memcpy(&nesting_op->data, &bind_data, sizeof(bind_data)); > > + ioctl(container->fd, VFIO_IOMMU_NESTING_OP, nesting_op); > > + > > +As mentioned above, guest OS may use stage 1 for GIOVA->GPA or GVA->GPA. > > +GVA->GPA page tables are available when PASID (Process Address Space > > +GVA->ID) > > +is exposed to guest. e.g. guest with PASID-capable devices assigned. > > +For such page table binding, the bind_data should include PASID info, > > +which is allocated by guest itself or by host. This depends on > > +hardware vendor e.g. Intel VT-d requires to allocate PASID from host. > > +This requirement is available by VFIO_IOMMU_GET_INFO. User space > > +could allocate PASID from host by: > > + > > + req.flags = VFIO_IOMMU_ALLOC_PASID; > > + ioctl(container, VFIO_IOMMU_PASID_REQUEST, &req); > > It is not clear how the userspace application determines whether PASIDs must be > allocated from the host via VFIO_IOMMU_PASID_REQUEST or if the guest itself can > allocate PASIDs. The text mentions VFIO_IOMMU_GET_INFO but what exactly > should the userspace application check? For VT-d, spec 3.0 introduced Virtual Cmd interface for PASID allocation, guest request PASID from host if it detects the interface. Application should check the IOMMU_NESTING_FEAT_SYSWIDE_PASID setting in the below info reported by VFIO_IOMMU_GET_INFO. And virtual VT-d should not report SVA related capabilities to guest if SYSWIDE_PASID is not supported by kernel. +struct iommu_nesting_info { + __u32 size; + __u32 format; + __u32 features; +#define IOMMU_NESTING_FEAT_SYSWIDE_PASID (1 << 0) +#define IOMMU_NESTING_FEAT_BIND_PGTBL (1 << 1) +#define IOMMU_NESTING_FEAT_CACHE_INVLD (1 << 2) + __u32 flags; + __u8 data[]; +}; https://lore.kernel.org/linux-iommu/1591877734-66527-3-git-send-email-yi.l.liu@intel.com/ Regards, Yi Liu _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
next prev parent reply other threads:[~2020-06-17 6:27 UTC|newest] Thread overview: 74+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-06-11 12:15 [PATCH v2 00/15] vfio: expose virtual Shared Virtual Addressing to VMs Liu Yi L 2020-06-11 12:15 ` Liu Yi L 2020-06-11 12:15 ` [PATCH v2 01/15] vfio/type1: Refactor vfio_iommu_type1_ioctl() Liu Yi L 2020-06-11 12:15 ` Liu Yi L 2020-06-11 12:15 ` [PATCH v2 02/15] iommu: Report domain nesting info Liu Yi L 2020-06-11 12:15 ` Liu Yi L 2020-06-11 19:30 ` Alex Williamson 2020-06-11 19:30 ` Alex Williamson 2020-06-12 9:05 ` Liu, Yi L 2020-06-12 9:05 ` Liu, Yi L 2020-06-15 1:22 ` Tian, Kevin 2020-06-15 1:22 ` Tian, Kevin 2020-06-15 6:04 ` Liu, Yi L 2020-06-15 6:04 ` Liu, Yi L 2020-06-16 1:56 ` Tian, Kevin 2020-06-16 1:56 ` Tian, Kevin 2020-06-16 2:24 ` Liu, Yi L 2020-06-16 2:24 ` Liu, Yi L 2020-06-17 14:39 ` Jean-Philippe Brucker 2020-06-17 14:39 ` Jean-Philippe Brucker 2020-06-18 11:46 ` Liu, Yi L 2020-06-18 11:46 ` Liu, Yi L 2020-06-11 12:15 ` [PATCH v2 03/15] vfio/type1: Report iommu nesting info to userspace Liu Yi L 2020-06-11 12:15 ` Liu Yi L 2020-06-11 12:15 ` [PATCH v2 04/15] vfio: Add PASID allocation/free support Liu Yi L 2020-06-11 12:15 ` Liu Yi L 2020-06-11 12:15 ` [PATCH v2 05/15] iommu/vt-d: Support setting ioasid set to domain Liu Yi L 2020-06-11 12:15 ` Liu Yi L 2020-06-11 12:15 ` [PATCH v2 06/15] vfio/type1: Add VFIO_IOMMU_PASID_REQUEST (alloc/free) Liu Yi L 2020-06-11 12:15 ` Liu Yi L 2020-06-11 12:15 ` [PATCH v2 07/15] iommu/uapi: Add iommu_gpasid_unbind_data Liu Yi L 2020-06-11 12:15 ` Liu Yi L 2020-06-11 12:15 ` [PATCH v2 08/15] iommu: Pass domain and unbind_data to sva_unbind_gpasid() Liu Yi L 2020-06-11 12:15 ` Liu Yi L 2020-06-11 12:15 ` [PATCH v2 09/15] iommu/vt-d: Check ownership for PASIDs from user-space Liu Yi L 2020-06-11 12:15 ` Liu Yi L 2020-06-11 12:15 ` [PATCH v2 10/15] vfio/type1: Support binding guest page tables to PASID Liu Yi L 2020-06-11 12:15 ` Liu Yi L 2020-06-11 12:15 ` [PATCH v2 11/15] vfio/type1: Allow invalidating first-level/stage IOMMU cache Liu Yi L 2020-06-11 12:15 ` Liu Yi L 2020-06-11 12:15 ` [PATCH v2 12/15] vfio/type1: Add vSVA support for IOMMU-backed mdevs Liu Yi L 2020-06-11 12:15 ` Liu Yi L 2020-06-11 12:15 ` [PATCH v2 13/15] vfio/pci: Expose PCIe PASID capability to guest Liu Yi L 2020-06-11 12:15 ` Liu Yi L 2020-06-11 12:15 ` [PATCH v2 14/15] vfio: Document dual stage control Liu Yi L 2020-06-11 12:15 ` Liu Yi L 2020-06-15 9:41 ` Stefan Hajnoczi 2020-06-15 9:41 ` Stefan Hajnoczi 2020-06-17 6:27 ` Liu, Yi L [this message] 2020-06-17 6:27 ` Liu, Yi L 2020-06-22 12:51 ` Stefan Hajnoczi 2020-06-22 12:51 ` Stefan Hajnoczi 2020-06-23 6:43 ` Liu, Yi L 2020-06-23 6:43 ` Liu, Yi L 2020-06-11 12:15 ` [PATCH v2 15/15] iommu/vt-d: Support reporting nesting capability info Liu Yi L 2020-06-11 12:15 ` Liu Yi L 2020-06-15 10:02 ` [PATCH v2 00/15] vfio: expose virtual Shared Virtual Addressing to VMs Stefan Hajnoczi 2020-06-15 10:02 ` Stefan Hajnoczi 2020-06-15 12:39 ` Liu, Yi L 2020-06-15 12:39 ` Liu, Yi L 2020-06-16 15:34 ` Stefan Hajnoczi 2020-06-16 15:34 ` Stefan Hajnoczi 2020-06-16 2:26 ` Tian, Kevin 2020-06-16 2:26 ` Tian, Kevin 2020-06-16 15:49 ` Stefan Hajnoczi 2020-06-16 15:49 ` Stefan Hajnoczi 2020-06-16 16:09 ` Peter Xu 2020-06-16 16:09 ` Peter Xu 2020-06-22 12:49 ` Stefan Hajnoczi 2020-06-22 12:49 ` Stefan Hajnoczi 2020-06-16 17:00 ` Raj, Ashok 2020-06-16 17:00 ` Raj, Ashok 2020-06-22 12:49 ` Stefan Hajnoczi 2020-06-22 12:49 ` Stefan Hajnoczi
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=DM5PR11MB1435C484283BDCD75F19EDB5C39A0@DM5PR11MB1435.namprd11.prod.outlook.com \ --to=yi.l.liu@intel.com \ --cc=alex.williamson@redhat.com \ --cc=ashok.raj@intel.com \ --cc=baolu.lu@linux.intel.com \ --cc=eric.auger@redhat.com \ --cc=hao.wu@intel.com \ --cc=iommu@lists.linux-foundation.org \ --cc=jacob.jun.pan@linux.intel.com \ --cc=jean-philippe@linaro.org \ --cc=joro@8bytes.org \ --cc=jun.j.tian@intel.com \ --cc=kevin.tian@intel.com \ --cc=kvm@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=peterx@redhat.com \ --cc=stefanha@gmail.com \ --cc=yi.y.sun@intel.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.