linux-arm-kernel.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
* [RFC PATCH v3 0/8] Add IOPF support for VFIO passthrough
@ 2021-04-09  3:44 Shenming Lu
  2021-04-09  3:44 ` [RFC PATCH v3 1/8] iommu: Evolve the device fault reporting framework Shenming Lu
                   ` (9 more replies)
  0 siblings, 10 replies; 40+ messages in thread
From: Shenming Lu @ 2021-04-09  3:44 UTC (permalink / raw)
  To: Alex Williamson, Cornelia Huck, Will Deacon, Robin Murphy,
	Joerg Roedel, Jean-Philippe Brucker, Eric Auger, kvm,
	linux-kernel, linux-arm-kernel, iommu, linux-api
  Cc: Kevin Tian, Lu Baolu, yi.l.liu, Christoph Hellwig,
	Jonathan Cameron, Barry Song, wanghaibin.wang, yuzenghui,
	lushenming

Hi,

Requesting for your comments and suggestions. :-)

The static pinning and mapping problem in VFIO and possible solutions
have been discussed a lot [1, 2]. One of the solutions is to add I/O
Page Fault support for VFIO devices. Different from those relatively
complicated software approaches such as presenting a vIOMMU that provides
the DMA buffer information (might include para-virtualized optimizations),
IOPF mainly depends on the hardware faulting capability, such as the PCIe
PRI extension or Arm SMMU stall model. What's more, the IOPF support in
the IOMMU driver has already been implemented in SVA [3]. So we add IOPF
support for VFIO passthrough based on the IOPF part of SVA in this series.

We have measured its performance with UADK [4] (passthrough an accelerator
to a VM(1U16G)) on Hisilicon Kunpeng920 board (and compared with host SVA):

Run hisi_sec_test...
 - with varying sending times and message lengths
 - with/without IOPF enabled (speed slowdown)

when msg_len = 1MB (and PREMAP_LEN (in Patch 4) = 1):
            slowdown (num of faults)
 times      VFIO IOPF      host SVA
 1          63.4% (518)    82.8% (512)
 100        22.9% (1058)   47.9% (1024)
 1000       2.6% (1071)    8.5% (1024)

when msg_len = 10MB (and PREMAP_LEN = 512):
            slowdown (num of faults)
 times      VFIO IOPF
 1          32.6% (13)
 100        3.5% (26)
 1000       1.6% (26)

History:

v2 -> v3
 - Nit fixes.
 - No reason to disable reporting the unrecoverable faults. (baolu)
 - Maintain a global IOPF enabled group list.
 - Split the pre-mapping optimization to be a separate patch.
 - Add selective faulting support (use vfio_pin_pages to indicate the
   non-faultable scope and add a new struct vfio_range to record it,
   untested). (Kevin)

v1 -> v2
 - Numerous improvements following the suggestions. Thanks a lot to all
   of you.

Note that PRI is not supported at the moment since there is no hardware.

Links:
[1] Lesokhin I, et al. Page Fault Support for Network Controllers. In ASPLOS,
    2016.
[2] Tian K, et al. coIOMMU: A Virtual IOMMU with Cooperative DMA Buffer Tracking
    for Efficient Memory Management in Direct I/O. In USENIX ATC, 2020.
[3] https://patchwork.kernel.org/project/linux-arm-kernel/cover/20210401154718.307519-1-jean-philippe@linaro.org/
[4] https://github.com/Linaro/uadk

Thanks,
Shenming


Shenming Lu (8):
  iommu: Evolve the device fault reporting framework
  vfio/type1: Add a page fault handler
  vfio/type1: Add an MMU notifier to avoid pinning
  vfio/type1: Pre-map more pages than requested in the IOPF handling
  vfio/type1: VFIO_IOMMU_ENABLE_IOPF
  vfio/type1: No need to statically pin and map if IOPF enabled
  vfio/type1: Add selective DMA faulting support
  vfio: Add nested IOPF support

 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c   |    3 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |   18 +-
 drivers/iommu/iommu.c                         |   56 +-
 drivers/vfio/vfio.c                           |   85 +-
 drivers/vfio/vfio_iommu_type1.c               | 1000 ++++++++++++++++-
 include/linux/iommu.h                         |   19 +-
 include/linux/vfio.h                          |   13 +
 include/uapi/linux/iommu.h                    |    4 +
 include/uapi/linux/vfio.h                     |    6 +
 9 files changed, 1181 insertions(+), 23 deletions(-)

-- 
2.19.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 40+ messages in thread

end of thread, other threads:[~2021-06-01  4:38 UTC | newest]

Thread overview: 40+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-09  3:44 [RFC PATCH v3 0/8] Add IOPF support for VFIO passthrough Shenming Lu
2021-04-09  3:44 ` [RFC PATCH v3 1/8] iommu: Evolve the device fault reporting framework Shenming Lu
2021-05-18 18:58   ` Alex Williamson
2021-05-21  6:37     ` Shenming Lu
2021-04-09  3:44 ` [RFC PATCH v3 2/8] vfio/type1: Add a page fault handler Shenming Lu
2021-05-18 18:58   ` Alex Williamson
2021-05-21  6:38     ` Shenming Lu
2021-05-24 22:11       ` Alex Williamson
2021-05-27 11:16         ` Shenming Lu
2021-04-09  3:44 ` [RFC PATCH v3 3/8] vfio/type1: Add an MMU notifier to avoid pinning Shenming Lu
2021-05-18 18:58   ` Alex Williamson
2021-05-21  6:37     ` Shenming Lu
2021-04-09  3:44 ` [RFC PATCH v3 4/8] vfio/type1: Pre-map more pages than requested in the IOPF handling Shenming Lu
2021-05-18 18:58   ` Alex Williamson
2021-05-21  6:37     ` Shenming Lu
2021-04-09  3:44 ` [RFC PATCH v3 5/8] vfio/type1: VFIO_IOMMU_ENABLE_IOPF Shenming Lu
2021-05-18 18:58   ` Alex Williamson
2021-05-21  6:38     ` Shenming Lu
2021-05-24 22:11       ` Alex Williamson
2021-05-27 11:15         ` Shenming Lu
2021-04-09  3:44 ` [RFC PATCH v3 6/8] vfio/type1: No need to statically pin and map if IOPF enabled Shenming Lu
2021-05-18 18:58   ` Alex Williamson
2021-05-21  6:39     ` Shenming Lu
2021-04-09  3:44 ` [RFC PATCH v3 7/8] vfio/type1: Add selective DMA faulting support Shenming Lu
2021-05-18 18:58   ` Alex Williamson
2021-05-21  6:39     ` Shenming Lu
2021-04-09  3:44 ` [RFC PATCH v3 8/8] vfio: Add nested IOPF support Shenming Lu
2021-05-18 18:58   ` Alex Williamson
2021-05-21  7:59     ` Shenming Lu
2021-05-24 13:11       ` Shenming Lu
2021-05-24 22:11         ` Alex Williamson
2021-05-27 11:03           ` Shenming Lu
2021-05-27 11:18             ` Lu Baolu
2021-06-01  4:36               ` Shenming Lu
2021-04-26  1:41 ` [RFC PATCH v3 0/8] Add IOPF support for VFIO passthrough Shenming Lu
2021-05-11 11:30   ` Shenming Lu
2021-05-18 18:57 ` Alex Williamson
2021-05-21  6:37   ` Shenming Lu
2021-05-24 22:11     ` Alex Williamson
2021-05-27 11:25       ` Shenming Lu

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).