From: Yi Liu <yi.l.liu@intel.com>
To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com
Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com,
eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org,
mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com,
yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com,
jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com,
lulu@redhat.com, suravee.suthikulpanit@amd.com,
intel-gvt-dev@lists.freedesktop.org,
intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org,
xudong.hao@intel.com, yan.y.zhao@intel.com,
terrence.xu@intel.com
Subject: [PATCH v6 24/24] docs: vfio: Add vfio device cdev description
Date: Wed, 8 Mar 2023 05:29:03 -0800 [thread overview]
Message-ID: <20230308132903.465159-25-yi.l.liu@intel.com> (raw)
In-Reply-To: <20230308132903.465159-1-yi.l.liu@intel.com>
This gives notes for userspace applications on device cdev usage.
Signed-off-by: Yi Liu <yi.l.liu@intel.com>
---
Documentation/driver-api/vfio.rst | 125 ++++++++++++++++++++++++++++++
1 file changed, 125 insertions(+)
diff --git a/Documentation/driver-api/vfio.rst b/Documentation/driver-api/vfio.rst
index 44527420f20d..227940e5224f 100644
--- a/Documentation/driver-api/vfio.rst
+++ b/Documentation/driver-api/vfio.rst
@@ -239,6 +239,123 @@ group and can access them as follows::
/* Gratuitous device reset and go... */
ioctl(device, VFIO_DEVICE_RESET);
+IOMMUFD and vfio_iommu_type1
+----------------------------
+
+IOMMUFD is the new user API to manage I/O page tables from userspace.
+It intends to be the portal of delivering advanced userspace DMA
+features (nested translation [5], PASID [6], etc.) and backward
+compatible with the vfio_iommu_type1 driver. Eventually vfio_iommu_type1
+will be deprecated.
+
+With the backward compatibility, no change is required for legacy VFIO
+drivers or applications to connect a VFIO device to IOMMUFD.
+
+ When CONFIG_IOMMUFD_VFIO_CONTAINER=n, VFIO container still provides
+ /dev/vfio/vfio which connects to vfio_iommu_type1. To disable VFIO
+ container and vfio_iommu_type1, the administrator could symbol link
+ /dev/vfio/vfio to /dev/iommu to enable VFIO container emulation
+ in IOMMUFD.
+
+ When CONFIG_IOMMUFD_VFIO_CONTAINER=y, IOMMUFD directly provides
+ /dev/vfio/vfio while the VFIO container and vfio_iommu_type1 are
+ explicitly disabled.
+
+VFIO Device cdev
+----------------
+
+Traditionally user acquires a device fd via VFIO_GROUP_GET_DEVICE_FD
+in a VFIO group.
+
+With CONFIG_VFIO_DEVICE_CDEV=y the user can now acquire a device fd
+by directly opening a character device /dev/vfio/devices/vfioX where
+"X" is the number allocated uniquely by VFIO for registered devices.
+
+The cdev only works with IOMMUFD. Both VFIO drivers and applications
+must adapt to the new cdev security model which requires using
+VFIO_DEVICE_BIND_IOMMUFD to claim DMA ownership before starting to
+actually use the device. Once bind succeeds then a VFIO device can
+be fully accessed by the user.
+
+VFIO device cdev doesn't rely on VFIO group/container/iommu drivers.
+Hence those modules can be fully compiled out in an environment
+where no legacy VFIO application exists.
+
+So far SPAPR does not support IOMMUFD yet. So it cannot support device
+cdev either.
+
+Device cdev Example
+-------------------
+
+Assume user wants to access PCI device 0000:6a:01.0::
+
+ $ ls /sys/bus/pci/devices/0000:6a:01.0/vfio-dev/
+ vfio0
+
+This device is therefore represented as vfio0. The user can verify
+its existence::
+
+ $ ls -l /dev/vfio/devices/vfio0
+ crw------- 1 root root 511, 0 Feb 16 01:22 /dev/vfio/devices/vfio0
+ $ cat /sys/bus/pci/devices/0000:6a:01.0/vfio-dev/vfio0/dev
+ 511:0
+ $ ls -l /dev/char/511\:0
+ lrwxrwxrwx 1 root root 21 Feb 16 01:22 /dev/char/511:0 -> ../vfio/devices/vfio0
+
+Then provide the user with access to the device if unprivileged
+operation is desired::
+
+ $ chown user:user /dev/vfio/devices/vfio0
+
+Finally the user could get cdev fd by::
+
+ cdev_fd = open("/dev/vfio/devices/vfio0", O_RDWR);
+
+An opened cdev_fd doesn't give the user any permission of accessing
+the device except binding the cdev_fd to an iommufd. After that point
+then the device is fully accessible including attaching it to an
+IOMMUFD IOAS/HWPT to enable userspace DMA::
+
+ struct vfio_device_bind_iommufd bind = {
+ .argsz = sizeof(bind),
+ .flags = 0,
+ };
+ struct iommu_ioas_alloc alloc_data = {
+ .size = sizeof(alloc_data),
+ .flags = 0,
+ };
+ struct vfio_device_attach_iommufd_pt attach_data = {
+ .argsz = sizeof(attach_data),
+ .flags = 0,
+ };
+ struct iommu_ioas_map map = {
+ .size = sizeof(map),
+ .flags = IOMMU_IOAS_MAP_READABLE |
+ IOMMU_IOAS_MAP_WRITEABLE |
+ IOMMU_IOAS_MAP_FIXED_IOVA,
+ .__reserved = 0,
+ };
+
+ iommufd = open("/dev/iommu", O_RDWR);
+
+ bind.iommufd = iommufd; // negative iommufd means vfio-noiommu mode
+ ioctl(cdev_fd, VFIO_DEVICE_BIND_IOMMUFD, &bind);
+
+ ioctl(iommufd, IOMMU_IOAS_ALLOC, &alloc_data);
+ attach_data.pt_id = alloc_data.out_ioas_id;
+ ioctl(cdev_fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach_data);
+
+ /* Allocate some space and setup a DMA mapping */
+ map.user_va = (int64_t)mmap(0, 1024 * 1024, PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);
+ map.iova = 0; /* 1MB starting at 0x0 from device view */
+ map.length = 1024 * 1024;
+ map.ioas_id = alloc_data.out_ioas_id;;
+
+ ioctl(iommufd, IOMMU_IOAS_MAP, &map);
+
+ /* Other device operations as stated in "VFIO Usage Example" */
+
VFIO User API
-------------------------------------------------------------------------------
@@ -566,3 +683,11 @@ This implementation has some specifics:
\-0d.1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
+
+.. [5] Nested translation is an IOMMU feature which supports two stage
+ address translations. This improves the address translation efficiency
+ in IOMMU virtualization.
+
+.. [6] PASID stands for Process Address Space ID, introduced by PCI
+ Express. It is a prerequisite for Shared Virtual Addressing (SVA)
+ and Scalable I/O Virtualization (Scalable IOV).
--
2.34.1
prev parent reply other threads:[~2023-03-08 13:34 UTC|newest]
Thread overview: 101+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-03-08 13:28 [PATCH v6 00/24] cover-letter: Add vfio_device cdev for iommufd support Yi Liu
2023-03-08 13:28 ` [PATCH v6 01/24] vfio: Allocate per device file structure Yi Liu
2023-03-08 13:28 ` [PATCH v6 02/24] vfio: Refine vfio file kAPIs for KVM Yi Liu
2023-03-08 13:28 ` [PATCH v6 03/24] vfio: Accept vfio device file in the KVM facing kAPI Yi Liu
2023-03-08 13:28 ` [PATCH v6 04/24] kvm/vfio: Rename kvm_vfio_group to prepare for accepting vfio device fd Yi Liu
2023-03-08 13:28 ` [PATCH v6 05/24] kvm/vfio: Accept vfio device file from userspace Yi Liu
2023-03-22 14:10 ` Xu Yilun
2023-03-28 3:48 ` Liu, Yi L
2023-03-08 13:28 ` [PATCH v6 06/24] vfio: Pass struct vfio_device_file * to vfio_device_open/close() Yi Liu
2023-03-08 13:28 ` [PATCH v6 07/24] vfio: Block device access via device fd until device is opened Yi Liu
2023-03-10 4:50 ` Tian, Kevin
2023-03-08 13:28 ` [PATCH v6 08/24] vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset() Yi Liu
2023-03-08 13:28 ` [PATCH v6 09/24] vfio/pci: Only need to check opened devices in the dev_set for hot reset Yi Liu
2023-03-10 5:00 ` Tian, Kevin
2023-03-08 13:28 ` [PATCH v6 10/24] vfio/pci: Rename the helpers and data in hot reset path to accept device fd Yi Liu
2023-03-10 5:01 ` Tian, Kevin
2023-03-08 13:28 ` [PATCH v6 11/24] vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl Yi Liu
2023-03-10 5:08 ` Tian, Kevin
2023-03-10 5:38 ` Liu, Yi L
2023-03-08 13:28 ` [PATCH v6 12/24] vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET Yi Liu
2023-03-10 5:31 ` Tian, Kevin
2023-03-10 6:04 ` Liu, Yi L
2023-03-10 9:08 ` Tian, Kevin
2023-03-10 17:42 ` Jason Gunthorpe
2023-03-15 22:53 ` Alex Williamson
2023-03-15 23:31 ` Tian, Kevin
2023-03-16 3:54 ` [offlist] " Liu, Yi L
2023-03-16 6:09 ` Tian, Kevin
2023-03-16 6:28 ` Liu, Yi L
2023-03-16 6:49 ` Nicolin Chen
2023-03-16 13:22 ` Liu, Yi L
2023-03-16 21:27 ` Nicolin Chen
2023-03-16 18:45 ` Alex Williamson
2023-03-16 23:29 ` Tian, Kevin
2023-03-17 0:22 ` Alex Williamson
2023-03-17 0:57 ` Tian, Kevin
2023-03-17 15:15 ` Alex Williamson
2023-03-20 17:14 ` Jason Gunthorpe
2023-03-20 22:52 ` Alex Williamson
2023-03-20 23:39 ` Jason Gunthorpe
2023-03-21 20:31 ` Alex Williamson
2023-03-21 20:50 ` Jason Gunthorpe
2023-03-21 21:01 ` Alex Williamson
2023-03-21 22:20 ` Jason Gunthorpe
2023-03-21 22:47 ` Alex Williamson
2023-03-22 4:42 ` Liu, Yi L
2023-03-22 12:23 ` Alex Williamson
2023-03-22 12:27 ` Jason Gunthorpe
2023-03-22 12:36 ` Alex Williamson
2023-03-22 12:47 ` Jason Gunthorpe
2023-03-24 9:09 ` Tian, Kevin
2023-03-24 13:14 ` Jason Gunthorpe
2023-03-22 8:17 ` Liu, Yi L
2023-03-22 12:17 ` Jason Gunthorpe
2023-03-22 13:33 ` Liu, Yi L
2023-03-22 13:43 ` Jason Gunthorpe
2023-03-23 3:15 ` Liu, Yi L
2023-03-23 12:02 ` Jason Gunthorpe
2023-03-24 9:25 ` Liu, Yi L
2023-03-27 11:57 ` Liu, Yi L
2023-03-08 13:28 ` [PATCH v6 13/24] vfio/iommufd: Split the compat_ioas attach out from vfio_iommufd_bind() Yi Liu
2023-03-10 8:08 ` Tian, Kevin
2023-03-10 8:22 ` Liu, Yi L
2023-03-10 9:10 ` Tian, Kevin
2023-03-11 10:24 ` Liu, Yi L
2023-03-13 2:06 ` Tian, Kevin
2023-03-08 13:28 ` [PATCH v6 14/24] vfio: Add cdev_device_open_cnt to vfio_group Yi Liu
2023-03-08 13:28 ` [PATCH v6 15/24] vfio: Make vfio_device_open() single open for device cdev path Yi Liu
2023-03-08 13:28 ` [PATCH v6 16/24] vfio: Make vfio_device_first_open() to cover the noiommu mode in " Yi Liu
2023-03-10 8:30 ` Tian, Kevin
2023-03-08 13:28 ` [PATCH v6 17/24] vfio-iommufd: Make vfio_iommufd_bind() selectively return devid Yi Liu
2023-03-10 8:31 ` Tian, Kevin
2023-03-08 13:28 ` [PATCH v6 18/24] vfio-iommufd: Add detach_ioas support for physical VFIO devices Yi Liu
2023-03-08 13:28 ` [PATCH v6 19/24] vfio-iommufd: Add detach_ioas support for emulated " Yi Liu
2023-03-10 23:42 ` Nicolin Chen
2023-03-15 6:15 ` Liu, Yi L
2023-03-15 6:25 ` Nicolin Chen
2023-03-08 13:28 ` [PATCH v6 20/24] vfio: Add cdev for vfio_device Yi Liu
2023-03-10 8:48 ` Tian, Kevin
2023-03-10 9:59 ` Liu, Yi L
2023-03-08 13:29 ` [PATCH v6 21/24] vfio: Add VFIO_DEVICE_BIND_IOMMUFD Yi Liu
2023-03-10 9:01 ` Tian, Kevin
2023-03-10 9:58 ` Liu, Yi L
2023-03-10 10:06 ` Tian, Kevin
2023-03-15 4:40 ` Liu, Yi L
2023-03-15 6:57 ` Tian, Kevin
2023-03-20 14:09 ` Jason Gunthorpe
2023-03-20 14:31 ` Yi Liu
2023-03-20 17:16 ` Jason Gunthorpe
2023-03-21 1:30 ` Tian, Kevin
2023-03-21 12:00 ` Jason Gunthorpe
2023-03-21 14:37 ` Liu, Yi L
2023-03-21 14:41 ` Jason Gunthorpe
2023-03-21 14:51 ` Liu, Yi L
2023-03-21 14:58 ` Jason Gunthorpe
2023-03-21 15:10 ` Liu, Yi L
2023-03-21 16:54 ` Jason Gunthorpe
2023-03-08 13:29 ` [PATCH v6 22/24] vfio: Add VFIO_DEVICE_AT[DE]TACH_IOMMUFD_PT Yi Liu
2023-03-08 13:29 ` [PATCH v6 23/24] vfio: Compile group optionally Yi Liu
2023-03-10 9:03 ` Tian, Kevin
2023-03-08 13:29 ` Yi Liu [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230308132903.465159-25-yi.l.liu@intel.com \
--to=yi.l.liu@intel.com \
--cc=alex.williamson@redhat.com \
--cc=chao.p.peng@linux.intel.com \
--cc=cohuck@redhat.com \
--cc=eric.auger@redhat.com \
--cc=intel-gfx@lists.freedesktop.org \
--cc=intel-gvt-dev@lists.freedesktop.org \
--cc=jasowang@redhat.com \
--cc=jgg@nvidia.com \
--cc=joro@8bytes.org \
--cc=kevin.tian@intel.com \
--cc=kvm@vger.kernel.org \
--cc=linux-s390@vger.kernel.org \
--cc=lulu@redhat.com \
--cc=mjrosato@linux.ibm.com \
--cc=nicolinc@nvidia.com \
--cc=peterx@redhat.com \
--cc=robin.murphy@arm.com \
--cc=shameerali.kolothum.thodi@huawei.com \
--cc=suravee.suthikulpanit@amd.com \
--cc=terrence.xu@intel.com \
--cc=xudong.hao@intel.com \
--cc=yan.y.zhao@intel.com \
--cc=yi.y.sun@linux.intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).