IOMMU Archive on lore.kernel.org
 help / color / Atom feed
From: Liu Yi L <yi.l.liu@intel.com>
To: alex.williamson@redhat.com, eric.auger@redhat.com,
	baolu.lu@linux.intel.com, joro@8bytes.org
Cc: jean-philippe@linaro.org, kevin.tian@intel.com,
	ashok.raj@intel.com, kvm@vger.kernel.org, stefanha@gmail.com,
	jun.j.tian@intel.com, iommu@lists.linux-foundation.org,
	linux-kernel@vger.kernel.org, yi.y.sun@intel.com,
	hao.wu@intel.com
Subject: [PATCH v6 14/15] vfio: Document dual stage control
Date: Mon, 27 Jul 2020 23:27:43 -0700
Message-ID: <1595917664-33276-15-git-send-email-yi.l.liu@intel.com> (raw)
In-Reply-To: <1595917664-33276-1-git-send-email-yi.l.liu@intel.com>

From: Eric Auger <eric.auger@redhat.com>

The VFIO API was enhanced to support nested stage control: a bunch of
new ioctls and usage guideline.

Let's document the process to follow to set up nested mode.

Cc: Kevin Tian <kevin.tian@intel.com>
CC: Jacob Pan <jacob.jun.pan@linux.intel.com>
Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: Eric Auger <eric.auger@redhat.com>
Cc: Jean-Philippe Brucker <jean-philippe@linaro.org>
Cc: Joerg Roedel <joro@8bytes.org>
Cc: Lu Baolu <baolu.lu@linux.intel.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Liu Yi L <yi.l.liu@intel.com>
---
v5 -> v6:
*) tweak per Eric's comments.

v3 -> v4:
*) add review-by from Stefan Hajnoczi

v2 -> v3:
*) address comments from Stefan Hajnoczi

v1 -> v2:
*) new in v2, compared with Eric's original version, pasid table bind
   and fault reporting is removed as this series doesn't cover them.
   Original version from Eric.
   https://lkml.org/lkml/2020/3/20/700
---
 Documentation/driver-api/vfio.rst | 75 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 75 insertions(+)

diff --git a/Documentation/driver-api/vfio.rst b/Documentation/driver-api/vfio.rst
index f1a4d3c..c0d43f0 100644
--- a/Documentation/driver-api/vfio.rst
+++ b/Documentation/driver-api/vfio.rst
@@ -239,6 +239,81 @@ group and can access them as follows::
 	/* Gratuitous device reset and go... */
 	ioctl(device, VFIO_DEVICE_RESET);
 
+IOMMU Dual Stage Control
+------------------------
+
+Some IOMMUs support 2 stages/levels of translation. Stage corresponds
+to the ARM terminology while level corresponds to Intel's terminology.
+In the following text we use either without distinction.
+
+This is useful when the guest is exposed with a virtual IOMMU and some
+devices are assigned to the guest through VFIO. Then the guest OS can
+use stage-1 (GIOVA -> GPA or GVA->GPA), while the hypervisor uses stage
+2 for VM isolation (GPA -> HPA).
+
+Under dual stage translation, the guest gets ownership of the stage-1 page
+tables and also owns stage-1 configuration structures. The hypervisor owns
+the root configuration structure (for security reason), including stage-2
+configuration. This works as long as configuration structures and page table
+formats are compatible between the virtual IOMMU and the physical IOMMU.
+
+Assuming the HW supports it, this nested mode is selected by choosing the
+VFIO_TYPE1_NESTING_IOMMU type through:
+
+    ioctl(container, VFIO_SET_IOMMU, VFIO_TYPE1_NESTING_IOMMU);
+
+This forces the hypervisor to use the stage-2, leaving stage-1 available
+for guest usage. The stage-1 format and binding method are vendor specific
+and reported in nesting cap (VFIO_IOMMU_TYPE1_INFO_CAP_NESTING) through
+VFIO_IOMMU_GET_INFO:
+
+    ioctl(container->fd, VFIO_IOMMU_GET_INFO, &nesting_info);
+
+The nesting cap info is available only after NESTING_IOMMU is selected.
+If underlying IOMMU doesn't support nesting, VFIO_SET_IOMMU fails and
+userspace should try other IOMMU types. Details of the nesting cap info
+can be found in Documentation/userspace-api/iommu.rst.
+
+The stage-1 page table can be bound to the IOMMU in two methods: directly
+or indirectly. Direct binding requires userspace to notify VFIO of every
+guest stage-1 page table binding, while indirect binding allows userspace
+to bind once with an intermediate structure (e.g. PASID table) which
+indirectly links to guest stage-1 page tables. The actual binding method
+depends on IOMMU vendor. Currently only the direct binding capability (
+IOMMU_NESTING_FEAT_BIND_PGTBL) is supported:
+
+    nesting_op->flags = VFIO_IOMMU_NESTING_OP_BIND_PGTBL;
+    memcpy(&nesting_op->data, &bind_data, sizeof(bind_data));
+    ioctl(container->fd, VFIO_IOMMU_NESTING_OP, nesting_op);
+
+When multiple stage-1 page tables are supported on a device, each page
+table is associated with a PASID (Process Address Space ID) to differentiate
+with each other. In such case, userspace should include PASID in the
+bind_data when issuing direct binding request.
+
+PASID could be managed per-device or system-wide which, again, depends on
+IOMMU vendor and is reported in nesting cap info. When system-wide policy
+is reported (IOMMU_NESTING_FEAT_SYSWIDE_PASID), e.g. as by Intel platforms,
+userspace *must* allocate PASID from VFIO before attempting binding of
+stage-1 page table:
+
+    req.flags = VFIO_IOMMU_ALLOC_PASID;
+    ioctl(container, VFIO_IOMMU_PASID_REQUEST, &req);
+
+Once the stage-1 page table is bound to the IOMMU, the guest is allowed to
+fully manage its mapping at its disposal. The IOMMU walks nested stage-1
+and stage-2 page tables when serving DMA requests from assigned device, and
+may cache the stage-1 mapping in the IOTLB. When required (IOMMU_NESTING_
+FEAT_CACHE_INVLD), userspace *must* forward guest stage-1 invalidation to
+the host, so the IOTLB is invalidated:
+
+    nesting_op->flags = VFIO_IOMMU_NESTING_OP_CACHE_INVLD;
+    memcpy(&nesting_op->data, &cache_inv_data, sizeof(cache_inv_data));
+    ioctl(container->fd, VFIO_IOMMU_NESTING_OP, nesting_op);
+
+Forwarded invalidations can happen at various granularity levels (page
+level, context level, etc.)
+
 VFIO User API
 -------------------------------------------------------------------------------
 
-- 
2.7.4

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

  parent reply index

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-07-28  6:27 [PATCH v6 00/15] vfio: expose virtual Shared Virtual Addressing to VMs Liu Yi L
2020-07-28  6:27 ` [PATCH v6 01/15] vfio/type1: Refactor vfio_iommu_type1_ioctl() Liu Yi L
2020-07-28 15:53   ` Alex Williamson
2020-07-29  2:20     ` Liu, Yi L
2020-07-28  6:27 ` [PATCH v6 02/15] iommu: Report domain nesting info Liu Yi L
2020-08-13 12:52   ` Auger Eric
2020-08-14  7:15     ` Liu, Yi L
2020-08-16 12:40       ` Auger Eric
2020-08-18  4:21         ` Jacob Pan
2020-08-18  6:59           ` Auger Eric
2020-07-28  6:27 ` [PATCH v6 03/15] iommu/smmu: Report empty " Liu Yi L
2020-07-28  6:27 ` [PATCH v6 04/15] vfio/type1: Report iommu nesting info to userspace Liu Yi L
2020-08-13 13:19   ` Auger Eric
2020-08-14  7:36     ` Liu, Yi L
2020-08-20 19:52   ` Alex Williamson
2020-08-21  0:52     ` Liu, Yi L
2020-07-28  6:27 ` [PATCH v6 05/15] vfio: Add PASID allocation/free support Liu Yi L
2020-08-13 15:07   ` Auger Eric
2020-08-14  7:40     ` Liu, Yi L
2020-07-28  6:27 ` [PATCH v6 06/15] iommu/vt-d: Support setting ioasid set to domain Liu Yi L
2020-08-13 15:06   ` Auger Eric
2020-08-14  8:04     ` Liu, Yi L
2020-08-16 12:42       ` Auger Eric
2020-07-28  6:27 ` [PATCH v6 07/15] vfio/type1: Add VFIO_IOMMU_PASID_REQUEST (alloc/free) Liu Yi L
2020-08-15 16:30   ` Auger Eric
2020-08-17  5:23     ` Liu, Yi L
2020-08-20 20:51   ` Alex Williamson
2020-08-21  0:37     ` Liu, Yi L
2020-08-21  1:49       ` Alex Williamson
2020-08-21  2:18         ` Liu, Yi L
2020-07-28  6:27 ` [PATCH v6 08/15] iommu: Pass domain to sva_unbind_gpasid() Liu Yi L
2020-08-20 21:06   ` Alex Williamson
2020-08-21  0:18     ` Liu, Yi L
2020-08-21 13:09     ` Auger Eric
2020-07-28  6:27 ` [PATCH v6 09/15] iommu/vt-d: Check ownership for PASIDs from user-space Liu Yi L
2020-08-15 16:30   ` Auger Eric
2020-08-17  5:38     ` Liu, Yi L
2020-07-28  6:27 ` [PATCH v6 10/15] vfio/type1: Support binding guest page tables to PASID Liu Yi L
2020-08-16 11:29   ` Auger Eric
2020-08-17  6:30     ` Liu, Yi L
2020-07-28  6:27 ` [PATCH v6 11/15] vfio/type1: Allow invalidating first-level/stage IOMMU cache Liu Yi L
2020-08-16 11:35   ` Auger Eric
2020-08-17  6:30     ` Liu, Yi L
2020-07-28  6:27 ` [PATCH v6 12/15] vfio/type1: Add vSVA support for IOMMU-backed mdevs Liu Yi L
2020-08-20 21:48   ` Alex Williamson
2020-08-21  0:53     ` Liu, Yi L
2020-07-28  6:27 ` [PATCH v6 13/15] vfio/pci: Expose PCIe PASID capability to guest Liu Yi L
2020-07-28  6:27 ` Liu Yi L [this message]
2020-08-16 11:51   ` [PATCH v6 14/15] vfio: Document dual stage control Auger Eric
2020-08-17  7:00     ` Liu, Yi L
2020-08-17  7:40       ` Eric Auger
2020-08-17  7:43         ` Liu, Yi L
2020-07-28  6:27 ` [PATCH v6 15/15] iommu/vt-d: Support reporting nesting capability info Liu Yi L
2020-08-16 12:01   ` Auger Eric
2020-08-17  7:05     ` Liu, Yi L
2020-08-17  7:42       ` Auger Eric
2020-08-17  7:45         ` Liu, Yi L

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1595917664-33276-15-git-send-email-yi.l.liu@intel.com \
    --to=yi.l.liu@intel.com \
    --cc=alex.williamson@redhat.com \
    --cc=ashok.raj@intel.com \
    --cc=baolu.lu@linux.intel.com \
    --cc=eric.auger@redhat.com \
    --cc=hao.wu@intel.com \
    --cc=iommu@lists.linux-foundation.org \
    --cc=jean-philippe@linaro.org \
    --cc=joro@8bytes.org \
    --cc=jun.j.tian@intel.com \
    --cc=kevin.tian@intel.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=stefanha@gmail.com \
    --cc=yi.y.sun@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

IOMMU Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-iommu/0 linux-iommu/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-iommu linux-iommu/ https://lore.kernel.org/linux-iommu \
		iommu@lists.linux-foundation.org
	public-inbox-index linux-iommu

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.linux-foundation.lists.iommu


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git