[PATCH v3] vfio iommu: Add dma available capability

[PATCH v3] vfio iommu: Add dma available capability

[Matthew Rosato]

Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container") added
a limit to the number of concurrent DMA requests for a vfio container.
However, lazy unmapping in s390 can in fact cause quite a large number of
outstanding DMA requests to build up prior to being purged, potentially 
the entire guest DMA space.  This results in unexpected 'VFIO_MAP_DMA
failed: No space left on device' conditions seen in QEMU.

This patch proposes to provide the remaining number of allowable DMA 
requests via the VFIO_IOMMU_GET_INFO ioctl as a new capability.  A 
subsequent patchset to QEMU would collect this information and use it in 
s390 PCI support to tap the guest on the shoulder before overrunning the 
vfio limit.

Changes from v2:
- Typos / fixed stale comment block

Matthew Rosato (1):
  vfio iommu: Add dma available capability

 drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
 include/uapi/linux/vfio.h       | 15 +++++++++++++++
 2 files changed, 32 insertions(+)

-- 
1.8.3.1

[PATCH v3] vfio iommu: Add dma available capability

[Matthew Rosato]

Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container")
added the ability to limit the number of memory backed DMA mappings.
However on s390x, when lazy mapping is in use, we use a very large
number of concurrent mappings.  Let's provide the current allowable
number of DMA mappings to userspace via the IOMMU info chain so that
userspace can take appropriate mitigation.

Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
 include/uapi/linux/vfio.h       | 15 +++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 5fbf0c1..15e21db 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -2609,6 +2609,20 @@ static int vfio_iommu_migration_build_caps(struct vfio_iommu *iommu,
 	return vfio_info_add_capability(caps, &cap_mig.header, sizeof(cap_mig));
 }
 
+static int vfio_iommu_dma_avail_build_caps(struct vfio_iommu *iommu,
+					   struct vfio_info_cap *caps)
+{
+	struct vfio_iommu_type1_info_dma_avail cap_dma_avail;
+
+	cap_dma_avail.header.id = VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL;
+	cap_dma_avail.header.version = 1;
+
+	cap_dma_avail.avail = iommu->dma_avail;
+
+	return vfio_info_add_capability(caps, &cap_dma_avail.header,
+					sizeof(cap_dma_avail));
+}
+
 static int vfio_iommu_type1_get_info(struct vfio_iommu *iommu,
 				     unsigned long arg)
 {
@@ -2642,6 +2656,9 @@ static int vfio_iommu_type1_get_info(struct vfio_iommu *iommu,
 	ret = vfio_iommu_migration_build_caps(iommu, &caps);
 
 	if (!ret)
+		ret = vfio_iommu_dma_avail_build_caps(iommu, &caps);
+
+	if (!ret)
 		ret = vfio_iommu_iova_build_caps(iommu, &caps);
 
 	mutex_unlock(&iommu->lock);
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index 9204705..3891e03 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -1039,6 +1039,21 @@ struct vfio_iommu_type1_info_cap_migration {
 	__u64	max_dirty_bitmap_size;		/* in bytes */
 };
 
+/*
+ * The DMA available capability allows to report the current number of
+ * simultaneously outstanding DMA mappings that are allowed.
+ *
+ * The structure below defines version 1 of this capability.
+ *
+ * avail: specifies the current number of outstanding DMA mappings allowed.
+ */
+#define VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL 3
+
+struct vfio_iommu_type1_info_dma_avail {
+	struct	vfio_info_cap_header header;
+	__u32	avail;
+};
+
 #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
 
 /**
-- 
1.8.3.1

Re: [PATCH v3] vfio iommu: Add dma available capability

[Matthew Rosato]

On 9/15/20 3:05 PM, Matthew Rosato wrote:
> Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container") added
> a limit to the number of concurrent DMA requests for a vfio container.
> However, lazy unmapping in s390 can in fact cause quite a large number of
> outstanding DMA requests to build up prior to being purged, potentially
> the entire guest DMA space.  This results in unexpected 'VFIO_MAP_DMA
> failed: No space left on device' conditions seen in QEMU.
> 
> This patch proposes to provide the remaining number of allowable DMA
> requests via the VFIO_IOMMU_GET_INFO ioctl as a new capability.  A
> subsequent patchset to QEMU would collect this information and use it in
> s390 PCI support to tap the guest on the shoulder before overrunning the
> vfio limit.

Link to latest QEMU patchset:
https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg05505.html

> 
> Changes from v2:
> - Typos / fixed stale comment block
> 
> Matthew Rosato (1):
>    vfio iommu: Add dma available capability
> 
>   drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
>   include/uapi/linux/vfio.h       | 15 +++++++++++++++
>   2 files changed, 32 insertions(+)
> 

Re: [PATCH v3] vfio iommu: Add dma available capability

[Cornelia Huck]

On Tue, 15 Sep 2020 15:05:18 -0400
Matthew Rosato <mjrosato@linux.ibm.com> wrote:

> Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container")
> added the ability to limit the number of memory backed DMA mappings.
> However on s390x, when lazy mapping is in use, we use a very large
> number of concurrent mappings.  Let's provide the current allowable
> number of DMA mappings to userspace via the IOMMU info chain so that
> userspace can take appropriate mitigation.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> ---
>  drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
>  include/uapi/linux/vfio.h       | 15 +++++++++++++++
>  2 files changed, 32 insertions(+)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>