[PATCH 00/10] Connect VFIO to IOMMUFD

[PATCH 00/10] Connect VFIO to IOMMUFD

[Jason Gunthorpe]

This series provides an alternative container layer for VFIO implemented
using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
not be compiled in.

At this point iommufd can be injected by passing in a iommfd FD to
VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
to obtain the compat IOAS and then connect up all the VFIO drivers as
appropriate.

This is temporary stopping point, a following series will provide a way to
directly open a VFIO device FD and directly connect it to IOMMUFD using
native ioctls that can expose the IOMMUFD features like hwpt, future
vPASID and dynamic attachment.

This series, in compat mode, has passed all the qemu tests we have
available, including the test suites for the Intel GVT mdev. Aside from
the temporary limitation with P2P memory this is belived to be fully
compatible with VFIO.

This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd

It requires the iommufd series:

https://lore.kernel.org/r/0-v3-402a7d6459de+24b-iommufd_jgg@nvidia.com

Jason Gunthorpe (10):
  vfio: Move vfio_device driver open/close code to a function
  vfio: Move vfio_device_assign_container() into
    vfio_device_first_open()
  vfio: Rename vfio_device_assign/unassign_container()
  vfio: Move storage of allow_unsafe_interrupts to vfio_main.c
  vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for
    vfio_file_enforced_coherent()
  vfio-iommufd: Allow iommufd to be used in place of a container fd
  vfio-iommufd: Support iommufd for physical VFIO devices
  vfio-iommufd: Support iommufd for emulated VFIO devices
  vfio: Make vfio_container optionally compiled
  iommufd: Allow iommufd to supply /dev/vfio/vfio

 drivers/gpu/drm/i915/gvt/kvmgt.c              |   3 +
 drivers/iommu/iommufd/Kconfig                 |  12 +
 drivers/iommu/iommufd/main.c                  |  35 +-
 drivers/s390/cio/vfio_ccw_ops.c               |   3 +
 drivers/s390/crypto/vfio_ap_ops.c             |   3 +
 drivers/vfio/Kconfig                          |  38 ++-
 drivers/vfio/Makefile                         |   5 +-
 drivers/vfio/container.c                      | 136 ++------
 drivers/vfio/fsl-mc/vfio_fsl_mc.c             |   3 +
 drivers/vfio/iommufd.c                        | 161 +++++++++
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    |   6 +
 drivers/vfio/pci/mlx5/main.c                  |   3 +
 drivers/vfio/pci/vfio_pci.c                   |   3 +
 drivers/vfio/platform/vfio_amba.c             |   3 +
 drivers/vfio/platform/vfio_platform.c         |   3 +
 drivers/vfio/vfio.h                           | 100 +++++-
 drivers/vfio/vfio_iommu_type1.c               |   5 +-
 drivers/vfio/vfio_main.c                      | 318 ++++++++++++++----
 include/linux/vfio.h                          |  39 +++
 19 files changed, 681 insertions(+), 198 deletions(-)
 create mode 100644 drivers/vfio/iommufd.c


base-commit: 3bec937e94942a6aee8854be1c1f5cc2b92d15e2
-- 
2.38.0

[PATCH 00/10] Connect VFIO to IOMMUFD

[Jason Gunthorpe]

This series provides an alternative container layer for VFIO implemented
using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
not be compiled in.

At this point iommufd can be injected by passing in a iommfd FD to
VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
to obtain the compat IOAS and then connect up all the VFIO drivers as
appropriate.

This is temporary stopping point, a following series will provide a way to
directly open a VFIO device FD and directly connect it to IOMMUFD using
native ioctls that can expose the IOMMUFD features like hwpt, future
vPASID and dynamic attachment.

This series, in compat mode, has passed all the qemu tests we have
available, including the test suites for the Intel GVT mdev. Aside from
the temporary limitation with P2P memory this is belived to be fully
compatible with VFIO.

This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd

It requires the iommufd series:

https://lore.kernel.org/r/0-v3-402a7d6459de+24b-iommufd_jgg@nvidia.com

Jason Gunthorpe (10):
  vfio: Move vfio_device driver open/close code to a function
  vfio: Move vfio_device_assign_container() into
    vfio_device_first_open()
  vfio: Rename vfio_device_assign/unassign_container()
  vfio: Move storage of allow_unsafe_interrupts to vfio_main.c
  vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for
    vfio_file_enforced_coherent()
  vfio-iommufd: Allow iommufd to be used in place of a container fd
  vfio-iommufd: Support iommufd for physical VFIO devices
  vfio-iommufd: Support iommufd for emulated VFIO devices
  vfio: Make vfio_container optionally compiled
  iommufd: Allow iommufd to supply /dev/vfio/vfio

 drivers/gpu/drm/i915/gvt/kvmgt.c              |   3 +
 drivers/iommu/iommufd/Kconfig                 |  12 +
 drivers/iommu/iommufd/main.c                  |  35 +-
 drivers/s390/cio/vfio_ccw_ops.c               |   3 +
 drivers/s390/crypto/vfio_ap_ops.c             |   3 +
 drivers/vfio/Kconfig                          |  38 ++-
 drivers/vfio/Makefile                         |   5 +-
 drivers/vfio/container.c                      | 136 ++------
 drivers/vfio/fsl-mc/vfio_fsl_mc.c             |   3 +
 drivers/vfio/iommufd.c                        | 161 +++++++++
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    |   6 +
 drivers/vfio/pci/mlx5/main.c                  |   3 +
 drivers/vfio/pci/vfio_pci.c                   |   3 +
 drivers/vfio/platform/vfio_amba.c             |   3 +
 drivers/vfio/platform/vfio_platform.c         |   3 +
 drivers/vfio/vfio.h                           | 100 +++++-
 drivers/vfio/vfio_iommu_type1.c               |   5 +-
 drivers/vfio/vfio_main.c                      | 318 ++++++++++++++----
 include/linux/vfio.h                          |  39 +++
 19 files changed, 681 insertions(+), 198 deletions(-)
 create mode 100644 drivers/vfio/iommufd.c


base-commit: 3bec937e94942a6aee8854be1c1f5cc2b92d15e2
-- 
2.38.0

[Intel-gfx] [PATCH 00/10] Connect VFIO to IOMMUFD

[Jason Gunthorpe]

This series provides an alternative container layer for VFIO implemented
using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
not be compiled in.

At this point iommufd can be injected by passing in a iommfd FD to
VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
to obtain the compat IOAS and then connect up all the VFIO drivers as
appropriate.

This is temporary stopping point, a following series will provide a way to
directly open a VFIO device FD and directly connect it to IOMMUFD using
native ioctls that can expose the IOMMUFD features like hwpt, future
vPASID and dynamic attachment.

This series, in compat mode, has passed all the qemu tests we have
available, including the test suites for the Intel GVT mdev. Aside from
the temporary limitation with P2P memory this is belived to be fully
compatible with VFIO.

This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd

It requires the iommufd series:

https://lore.kernel.org/r/0-v3-402a7d6459de+24b-iommufd_jgg@nvidia.com

Jason Gunthorpe (10):
  vfio: Move vfio_device driver open/close code to a function
  vfio: Move vfio_device_assign_container() into
    vfio_device_first_open()
  vfio: Rename vfio_device_assign/unassign_container()
  vfio: Move storage of allow_unsafe_interrupts to vfio_main.c
  vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for
    vfio_file_enforced_coherent()
  vfio-iommufd: Allow iommufd to be used in place of a container fd
  vfio-iommufd: Support iommufd for physical VFIO devices
  vfio-iommufd: Support iommufd for emulated VFIO devices
  vfio: Make vfio_container optionally compiled
  iommufd: Allow iommufd to supply /dev/vfio/vfio

 drivers/gpu/drm/i915/gvt/kvmgt.c              |   3 +
 drivers/iommu/iommufd/Kconfig                 |  12 +
 drivers/iommu/iommufd/main.c                  |  35 +-
 drivers/s390/cio/vfio_ccw_ops.c               |   3 +
 drivers/s390/crypto/vfio_ap_ops.c             |   3 +
 drivers/vfio/Kconfig                          |  38 ++-
 drivers/vfio/Makefile                         |   5 +-
 drivers/vfio/container.c                      | 136 ++------
 drivers/vfio/fsl-mc/vfio_fsl_mc.c             |   3 +
 drivers/vfio/iommufd.c                        | 161 +++++++++
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    |   6 +
 drivers/vfio/pci/mlx5/main.c                  |   3 +
 drivers/vfio/pci/vfio_pci.c                   |   3 +
 drivers/vfio/platform/vfio_amba.c             |   3 +
 drivers/vfio/platform/vfio_platform.c         |   3 +
 drivers/vfio/vfio.h                           | 100 +++++-
 drivers/vfio/vfio_iommu_type1.c               |   5 +-
 drivers/vfio/vfio_main.c                      | 318 ++++++++++++++----
 include/linux/vfio.h                          |  39 +++
 19 files changed, 681 insertions(+), 198 deletions(-)
 create mode 100644 drivers/vfio/iommufd.c


base-commit: 3bec937e94942a6aee8854be1c1f5cc2b92d15e2
-- 
2.38.0

[PATCH 01/10] vfio: Move vfio_device driver open/close code to a function

[Jason Gunthorpe]

This error unwind is getting complicated. Move all the code into two
pair'd function. The functions should be called when the open_count == 1
after incrementing/before decrementing.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/vfio_main.c | 95 ++++++++++++++++++++++------------------
 1 file changed, 53 insertions(+), 42 deletions(-)

diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 2d168793d4e1ce..d043383fc3ba2b 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -734,6 +734,51 @@ bool vfio_assert_device_open(struct vfio_device *device)
 	return !WARN_ON_ONCE(!READ_ONCE(device->open_count));
 }
 
+static int vfio_device_first_open(struct vfio_device *device)
+{
+	int ret;
+
+	lockdep_assert_held(&device->dev_set->lock);
+
+	if (!try_module_get(device->dev->driver->owner))
+		return -ENODEV;
+
+	/*
+	 * Here we pass the KVM pointer with the group under the read lock.  If
+	 * the device driver will use it, it must obtain a reference and release
+	 * it during close_device.
+	 */
+	mutex_lock(&device->group->group_lock);
+	device->kvm = device->group->kvm;
+	if (device->ops->open_device) {
+		ret = device->ops->open_device(device);
+		if (ret)
+			goto err_module_put;
+	}
+	vfio_device_container_register(device);
+	mutex_unlock(&device->group->group_lock);
+	return 0;
+
+err_module_put:
+	device->kvm = NULL;
+	mutex_unlock(&device->group->group_lock);
+	module_put(device->dev->driver->owner);
+	return ret;
+}
+
+static void vfio_device_last_close(struct vfio_device *device)
+{
+	lockdep_assert_held(&device->dev_set->lock);
+
+	mutex_lock(&device->group->group_lock);
+	vfio_device_container_unregister(device);
+	if (device->ops->close_device)
+		device->ops->close_device(device);
+	device->kvm = NULL;
+	mutex_unlock(&device->group->group_lock);
+	module_put(device->dev->driver->owner);
+}
+
 static struct file *vfio_device_open(struct vfio_device *device)
 {
 	struct file *filep;
@@ -745,29 +790,12 @@ static struct file *vfio_device_open(struct vfio_device *device)
 	if (ret)
 		return ERR_PTR(ret);
 
-	if (!try_module_get(device->dev->driver->owner)) {
-		ret = -ENODEV;
-		goto err_unassign_container;
-	}
-
 	mutex_lock(&device->dev_set->lock);
 	device->open_count++;
 	if (device->open_count == 1) {
-		/*
-		 * Here we pass the KVM pointer with the group under the read
-		 * lock.  If the device driver will use it, it must obtain a
-		 * reference and release it during close_device.
-		 */
-		mutex_lock(&device->group->group_lock);
-		device->kvm = device->group->kvm;
-
-		if (device->ops->open_device) {
-			ret = device->ops->open_device(device);
-			if (ret)
-				goto err_undo_count;
-		}
-		vfio_device_container_register(device);
-		mutex_unlock(&device->group->group_lock);
+		ret = vfio_device_first_open(device);
+		if (ret)
+			goto err_unassign_container;
 	}
 	mutex_unlock(&device->dev_set->lock);
 
@@ -800,20 +828,11 @@ static struct file *vfio_device_open(struct vfio_device *device)
 
 err_close_device:
 	mutex_lock(&device->dev_set->lock);
-	mutex_lock(&device->group->group_lock);
-	if (device->open_count == 1 && device->ops->close_device) {
-		device->ops->close_device(device);
-
-		vfio_device_container_unregister(device);
-	}
-err_undo_count:
-	mutex_unlock(&device->group->group_lock);
+	if (device->open_count == 1)
+		vfio_device_last_close(device);
+err_unassign_container:
 	device->open_count--;
-	if (device->open_count == 0 && device->kvm)
-		device->kvm = NULL;
 	mutex_unlock(&device->dev_set->lock);
-	module_put(device->dev->driver->owner);
-err_unassign_container:
 	vfio_device_unassign_container(device);
 	return ERR_PTR(ret);
 }
@@ -1016,19 +1035,11 @@ static int vfio_device_fops_release(struct inode *inode, struct file *filep)
 
 	mutex_lock(&device->dev_set->lock);
 	vfio_assert_device_open(device);
-	mutex_lock(&device->group->group_lock);
-	if (device->open_count == 1 && device->ops->close_device)
-		device->ops->close_device(device);
-
-	vfio_device_container_unregister(device);
-	mutex_unlock(&device->group->group_lock);
+	if (device->open_count == 1)
+		vfio_device_last_close(device);
 	device->open_count--;
-	if (device->open_count == 0)
-		device->kvm = NULL;
 	mutex_unlock(&device->dev_set->lock);
 
-	module_put(device->dev->driver->owner);
-
 	vfio_device_unassign_container(device);
 
 	vfio_device_put_registration(device);
-- 
2.38.0

[PATCH 01/10] vfio: Move vfio_device driver open/close code to a function

[Jason Gunthorpe]

This error unwind is getting complicated. Move all the code into two
pair'd function. The functions should be called when the open_count == 1
after incrementing/before decrementing.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/vfio_main.c | 95 ++++++++++++++++++++++------------------
 1 file changed, 53 insertions(+), 42 deletions(-)

diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 2d168793d4e1ce..d043383fc3ba2b 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -734,6 +734,51 @@ bool vfio_assert_device_open(struct vfio_device *device)
 	return !WARN_ON_ONCE(!READ_ONCE(device->open_count));
 }
 
+static int vfio_device_first_open(struct vfio_device *device)
+{
+	int ret;
+
+	lockdep_assert_held(&device->dev_set->lock);
+
+	if (!try_module_get(device->dev->driver->owner))
+		return -ENODEV;
+
+	/*
+	 * Here we pass the KVM pointer with the group under the read lock.  If
+	 * the device driver will use it, it must obtain a reference and release
+	 * it during close_device.
+	 */
+	mutex_lock(&device->group->group_lock);
+	device->kvm = device->group->kvm;
+	if (device->ops->open_device) {
+		ret = device->ops->open_device(device);
+		if (ret)
+			goto err_module_put;
+	}
+	vfio_device_container_register(device);
+	mutex_unlock(&device->group->group_lock);
+	return 0;
+
+err_module_put:
+	device->kvm = NULL;
+	mutex_unlock(&device->group->group_lock);
+	module_put(device->dev->driver->owner);
+	return ret;
+}
+
+static void vfio_device_last_close(struct vfio_device *device)
+{
+	lockdep_assert_held(&device->dev_set->lock);
+
+	mutex_lock(&device->group->group_lock);
+	vfio_device_container_unregister(device);
+	if (device->ops->close_device)
+		device->ops->close_device(device);
+	device->kvm = NULL;
+	mutex_unlock(&device->group->group_lock);
+	module_put(device->dev->driver->owner);
+}
+
 static struct file *vfio_device_open(struct vfio_device *device)
 {
 	struct file *filep;
@@ -745,29 +790,12 @@ static struct file *vfio_device_open(struct vfio_device *device)
 	if (ret)
 		return ERR_PTR(ret);
 
-	if (!try_module_get(device->dev->driver->owner)) {
-		ret = -ENODEV;
-		goto err_unassign_container;
-	}
-
 	mutex_lock(&device->dev_set->lock);
 	device->open_count++;
 	if (device->open_count == 1) {
-		/*
-		 * Here we pass the KVM pointer with the group under the read
-		 * lock.  If the device driver will use it, it must obtain a
-		 * reference and release it during close_device.
-		 */
-		mutex_lock(&device->group->group_lock);
-		device->kvm = device->group->kvm;
-
-		if (device->ops->open_device) {
-			ret = device->ops->open_device(device);
-			if (ret)
-				goto err_undo_count;
-		}
-		vfio_device_container_register(device);
-		mutex_unlock(&device->group->group_lock);
+		ret = vfio_device_first_open(device);
+		if (ret)
+			goto err_unassign_container;
 	}
 	mutex_unlock(&device->dev_set->lock);
 
@@ -800,20 +828,11 @@ static struct file *vfio_device_open(struct vfio_device *device)
 
 err_close_device:
 	mutex_lock(&device->dev_set->lock);
-	mutex_lock(&device->group->group_lock);
-	if (device->open_count == 1 && device->ops->close_device) {
-		device->ops->close_device(device);
-
-		vfio_device_container_unregister(device);
-	}
-err_undo_count:
-	mutex_unlock(&device->group->group_lock);
+	if (device->open_count == 1)
+		vfio_device_last_close(device);
+err_unassign_container:
 	device->open_count--;
-	if (device->open_count == 0 && device->kvm)
-		device->kvm = NULL;
 	mutex_unlock(&device->dev_set->lock);
-	module_put(device->dev->driver->owner);
-err_unassign_container:
 	vfio_device_unassign_container(device);
 	return ERR_PTR(ret);
 }
@@ -1016,19 +1035,11 @@ static int vfio_device_fops_release(struct inode *inode, struct file *filep)
 
 	mutex_lock(&device->dev_set->lock);
 	vfio_assert_device_open(device);
-	mutex_lock(&device->group->group_lock);
-	if (device->open_count == 1 && device->ops->close_device)
-		device->ops->close_device(device);
-
-	vfio_device_container_unregister(device);
-	mutex_unlock(&device->group->group_lock);
+	if (device->open_count == 1)
+		vfio_device_last_close(device);
 	device->open_count--;
-	if (device->open_count == 0)
-		device->kvm = NULL;
 	mutex_unlock(&device->dev_set->lock);
 
-	module_put(device->dev->driver->owner);
-
 	vfio_device_unassign_container(device);
 
 	vfio_device_put_registration(device);
-- 
2.38.0

[Intel-gfx] [PATCH 01/10] vfio: Move vfio_device driver open/close code to a function

[Jason Gunthorpe]

This error unwind is getting complicated. Move all the code into two
pair'd function. The functions should be called when the open_count == 1
after incrementing/before decrementing.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/vfio_main.c | 95 ++++++++++++++++++++++------------------
 1 file changed, 53 insertions(+), 42 deletions(-)

diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 2d168793d4e1ce..d043383fc3ba2b 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -734,6 +734,51 @@ bool vfio_assert_device_open(struct vfio_device *device)
 	return !WARN_ON_ONCE(!READ_ONCE(device->open_count));
 }
 
+static int vfio_device_first_open(struct vfio_device *device)
+{
+	int ret;
+
+	lockdep_assert_held(&device->dev_set->lock);
+
+	if (!try_module_get(device->dev->driver->owner))
+		return -ENODEV;
+
+	/*
+	 * Here we pass the KVM pointer with the group under the read lock.  If
+	 * the device driver will use it, it must obtain a reference and release
+	 * it during close_device.
+	 */
+	mutex_lock(&device->group->group_lock);
+	device->kvm = device->group->kvm;
+	if (device->ops->open_device) {
+		ret = device->ops->open_device(device);
+		if (ret)
+			goto err_module_put;
+	}
+	vfio_device_container_register(device);
+	mutex_unlock(&device->group->group_lock);
+	return 0;
+
+err_module_put:
+	device->kvm = NULL;
+	mutex_unlock(&device->group->group_lock);
+	module_put(device->dev->driver->owner);
+	return ret;
+}
+
+static void vfio_device_last_close(struct vfio_device *device)
+{
+	lockdep_assert_held(&device->dev_set->lock);
+
+	mutex_lock(&device->group->group_lock);
+	vfio_device_container_unregister(device);
+	if (device->ops->close_device)
+		device->ops->close_device(device);
+	device->kvm = NULL;
+	mutex_unlock(&device->group->group_lock);
+	module_put(device->dev->driver->owner);
+}
+
 static struct file *vfio_device_open(struct vfio_device *device)
 {
 	struct file *filep;
@@ -745,29 +790,12 @@ static struct file *vfio_device_open(struct vfio_device *device)
 	if (ret)
 		return ERR_PTR(ret);
 
-	if (!try_module_get(device->dev->driver->owner)) {
-		ret = -ENODEV;
-		goto err_unassign_container;
-	}
-
 	mutex_lock(&device->dev_set->lock);
 	device->open_count++;
 	if (device->open_count == 1) {
-		/*
-		 * Here we pass the KVM pointer with the group under the read
-		 * lock.  If the device driver will use it, it must obtain a
-		 * reference and release it during close_device.
-		 */
-		mutex_lock(&device->group->group_lock);
-		device->kvm = device->group->kvm;
-
-		if (device->ops->open_device) {
-			ret = device->ops->open_device(device);
-			if (ret)
-				goto err_undo_count;
-		}
-		vfio_device_container_register(device);
-		mutex_unlock(&device->group->group_lock);
+		ret = vfio_device_first_open(device);
+		if (ret)
+			goto err_unassign_container;
 	}
 	mutex_unlock(&device->dev_set->lock);
 
@@ -800,20 +828,11 @@ static struct file *vfio_device_open(struct vfio_device *device)
 
 err_close_device:
 	mutex_lock(&device->dev_set->lock);
-	mutex_lock(&device->group->group_lock);
-	if (device->open_count == 1 && device->ops->close_device) {
-		device->ops->close_device(device);
-
-		vfio_device_container_unregister(device);
-	}
-err_undo_count:
-	mutex_unlock(&device->group->group_lock);
+	if (device->open_count == 1)
+		vfio_device_last_close(device);
+err_unassign_container:
 	device->open_count--;
-	if (device->open_count == 0 && device->kvm)
-		device->kvm = NULL;
 	mutex_unlock(&device->dev_set->lock);
-	module_put(device->dev->driver->owner);
-err_unassign_container:
 	vfio_device_unassign_container(device);
 	return ERR_PTR(ret);
 }
@@ -1016,19 +1035,11 @@ static int vfio_device_fops_release(struct inode *inode, struct file *filep)
 
 	mutex_lock(&device->dev_set->lock);
 	vfio_assert_device_open(device);
-	mutex_lock(&device->group->group_lock);
-	if (device->open_count == 1 && device->ops->close_device)
-		device->ops->close_device(device);
-
-	vfio_device_container_unregister(device);
-	mutex_unlock(&device->group->group_lock);
+	if (device->open_count == 1)
+		vfio_device_last_close(device);
 	device->open_count--;
-	if (device->open_count == 0)
-		device->kvm = NULL;
 	mutex_unlock(&device->dev_set->lock);
 
-	module_put(device->dev->driver->owner);
-
 	vfio_device_unassign_container(device);
 
 	vfio_device_put_registration(device);
-- 
2.38.0

[PATCH 02/10] vfio: Move vfio_device_assign_container() into vfio_device_first_open()

[Jason Gunthorpe]

The only thing this function does is assert the group has an assigned
container and incrs refcounts.

The overall model we have is that once a conatiner_users refcount is
incremented it cannot be de-assigned from the group -
vfio_group_ioctl_unset_container() will fail and the group FD cannot be
closed.

Thus we do not need to check this on evey device FD open, just the
first. Reorganize the code so that only the first open and last close
manages the container.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/container.c |  4 ++--
 drivers/vfio/vfio_main.c | 18 ++++++++----------
 2 files changed, 10 insertions(+), 12 deletions(-)

diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
index d74164abbf401d..dd79a66ec62cad 100644
--- a/drivers/vfio/container.c
+++ b/drivers/vfio/container.c
@@ -531,11 +531,11 @@ int vfio_device_assign_container(struct vfio_device *device)
 
 void vfio_device_unassign_container(struct vfio_device *device)
 {
-	mutex_lock(&device->group->group_lock);
+	lockdep_assert_held_write(&device->group->group_lock);
+
 	WARN_ON(device->group->container_users <= 1);
 	device->group->container_users--;
 	fput(device->group->opened_file);
-	mutex_unlock(&device->group->group_lock);
 }
 
 /*
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index d043383fc3ba2b..204443ba3b3cd9 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -749,16 +749,22 @@ static int vfio_device_first_open(struct vfio_device *device)
 	 * it during close_device.
 	 */
 	mutex_lock(&device->group->group_lock);
+	ret = vfio_device_assign_container(device);
+	if (ret)
+		goto err_module_put;
+
 	device->kvm = device->group->kvm;
 	if (device->ops->open_device) {
 		ret = device->ops->open_device(device);
 		if (ret)
-			goto err_module_put;
+			goto err_container;
 	}
 	vfio_device_container_register(device);
 	mutex_unlock(&device->group->group_lock);
 	return 0;
 
+err_container:
+	vfio_device_unassign_container(device);
 err_module_put:
 	device->kvm = NULL;
 	mutex_unlock(&device->group->group_lock);
@@ -775,6 +781,7 @@ static void vfio_device_last_close(struct vfio_device *device)
 	if (device->ops->close_device)
 		device->ops->close_device(device);
 	device->kvm = NULL;
+	vfio_device_unassign_container(device);
 	mutex_unlock(&device->group->group_lock);
 	module_put(device->dev->driver->owner);
 }
@@ -784,12 +791,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
 	struct file *filep;
 	int ret;
 
-	mutex_lock(&device->group->group_lock);
-	ret = vfio_device_assign_container(device);
-	mutex_unlock(&device->group->group_lock);
-	if (ret)
-		return ERR_PTR(ret);
-
 	mutex_lock(&device->dev_set->lock);
 	device->open_count++;
 	if (device->open_count == 1) {
@@ -833,7 +834,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
 err_unassign_container:
 	device->open_count--;
 	mutex_unlock(&device->dev_set->lock);
-	vfio_device_unassign_container(device);
 	return ERR_PTR(ret);
 }
 
@@ -1040,8 +1040,6 @@ static int vfio_device_fops_release(struct inode *inode, struct file *filep)
 	device->open_count--;
 	mutex_unlock(&device->dev_set->lock);
 
-	vfio_device_unassign_container(device);
-
 	vfio_device_put_registration(device);
 
 	return 0;
-- 
2.38.0

[Intel-gfx] [PATCH 02/10] vfio: Move vfio_device_assign_container() into vfio_device_first_open()

[Jason Gunthorpe]

The only thing this function does is assert the group has an assigned
container and incrs refcounts.

The overall model we have is that once a conatiner_users refcount is
incremented it cannot be de-assigned from the group -
vfio_group_ioctl_unset_container() will fail and the group FD cannot be
closed.

Thus we do not need to check this on evey device FD open, just the
first. Reorganize the code so that only the first open and last close
manages the container.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/container.c |  4 ++--
 drivers/vfio/vfio_main.c | 18 ++++++++----------
 2 files changed, 10 insertions(+), 12 deletions(-)

diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
index d74164abbf401d..dd79a66ec62cad 100644
--- a/drivers/vfio/container.c
+++ b/drivers/vfio/container.c
@@ -531,11 +531,11 @@ int vfio_device_assign_container(struct vfio_device *device)
 
 void vfio_device_unassign_container(struct vfio_device *device)
 {
-	mutex_lock(&device->group->group_lock);
+	lockdep_assert_held_write(&device->group->group_lock);
+
 	WARN_ON(device->group->container_users <= 1);
 	device->group->container_users--;
 	fput(device->group->opened_file);
-	mutex_unlock(&device->group->group_lock);
 }
 
 /*
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index d043383fc3ba2b..204443ba3b3cd9 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -749,16 +749,22 @@ static int vfio_device_first_open(struct vfio_device *device)
 	 * it during close_device.
 	 */
 	mutex_lock(&device->group->group_lock);
+	ret = vfio_device_assign_container(device);
+	if (ret)
+		goto err_module_put;
+
 	device->kvm = device->group->kvm;
 	if (device->ops->open_device) {
 		ret = device->ops->open_device(device);
 		if (ret)
-			goto err_module_put;
+			goto err_container;
 	}
 	vfio_device_container_register(device);
 	mutex_unlock(&device->group->group_lock);
 	return 0;
 
+err_container:
+	vfio_device_unassign_container(device);
 err_module_put:
 	device->kvm = NULL;
 	mutex_unlock(&device->group->group_lock);
@@ -775,6 +781,7 @@ static void vfio_device_last_close(struct vfio_device *device)
 	if (device->ops->close_device)
 		device->ops->close_device(device);
 	device->kvm = NULL;
+	vfio_device_unassign_container(device);
 	mutex_unlock(&device->group->group_lock);
 	module_put(device->dev->driver->owner);
 }
@@ -784,12 +791,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
 	struct file *filep;
 	int ret;
 
-	mutex_lock(&device->group->group_lock);
-	ret = vfio_device_assign_container(device);
-	mutex_unlock(&device->group->group_lock);
-	if (ret)
-		return ERR_PTR(ret);
-
 	mutex_lock(&device->dev_set->lock);
 	device->open_count++;
 	if (device->open_count == 1) {
@@ -833,7 +834,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
 err_unassign_container:
 	device->open_count--;
 	mutex_unlock(&device->dev_set->lock);
-	vfio_device_unassign_container(device);
 	return ERR_PTR(ret);
 }
 
@@ -1040,8 +1040,6 @@ static int vfio_device_fops_release(struct inode *inode, struct file *filep)
 	device->open_count--;
 	mutex_unlock(&device->dev_set->lock);
 
-	vfio_device_unassign_container(device);
-
 	vfio_device_put_registration(device);
 
 	return 0;
-- 
2.38.0

[PATCH 02/10] vfio: Move vfio_device_assign_container() into vfio_device_first_open()

[Jason Gunthorpe]

The only thing this function does is assert the group has an assigned
container and incrs refcounts.

The overall model we have is that once a conatiner_users refcount is
incremented it cannot be de-assigned from the group -
vfio_group_ioctl_unset_container() will fail and the group FD cannot be
closed.

Thus we do not need to check this on evey device FD open, just the
first. Reorganize the code so that only the first open and last close
manages the container.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/container.c |  4 ++--
 drivers/vfio/vfio_main.c | 18 ++++++++----------
 2 files changed, 10 insertions(+), 12 deletions(-)

diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
index d74164abbf401d..dd79a66ec62cad 100644
--- a/drivers/vfio/container.c
+++ b/drivers/vfio/container.c
@@ -531,11 +531,11 @@ int vfio_device_assign_container(struct vfio_device *device)
 
 void vfio_device_unassign_container(struct vfio_device *device)
 {
-	mutex_lock(&device->group->group_lock);
+	lockdep_assert_held_write(&device->group->group_lock);
+
 	WARN_ON(device->group->container_users <= 1);
 	device->group->container_users--;
 	fput(device->group->opened_file);
-	mutex_unlock(&device->group->group_lock);
 }
 
 /*
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index d043383fc3ba2b..204443ba3b3cd9 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -749,16 +749,22 @@ static int vfio_device_first_open(struct vfio_device *device)
 	 * it during close_device.
 	 */
 	mutex_lock(&device->group->group_lock);
+	ret = vfio_device_assign_container(device);
+	if (ret)
+		goto err_module_put;
+
 	device->kvm = device->group->kvm;
 	if (device->ops->open_device) {
 		ret = device->ops->open_device(device);
 		if (ret)
-			goto err_module_put;
+			goto err_container;
 	}
 	vfio_device_container_register(device);
 	mutex_unlock(&device->group->group_lock);
 	return 0;
 
+err_container:
+	vfio_device_unassign_container(device);
 err_module_put:
 	device->kvm = NULL;
 	mutex_unlock(&device->group->group_lock);
@@ -775,6 +781,7 @@ static void vfio_device_last_close(struct vfio_device *device)
 	if (device->ops->close_device)
 		device->ops->close_device(device);
 	device->kvm = NULL;
+	vfio_device_unassign_container(device);
 	mutex_unlock(&device->group->group_lock);
 	module_put(device->dev->driver->owner);
 }
@@ -784,12 +791,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
 	struct file *filep;
 	int ret;
 
-	mutex_lock(&device->group->group_lock);
-	ret = vfio_device_assign_container(device);
-	mutex_unlock(&device->group->group_lock);
-	if (ret)
-		return ERR_PTR(ret);
-
 	mutex_lock(&device->dev_set->lock);
 	device->open_count++;
 	if (device->open_count == 1) {
@@ -833,7 +834,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
 err_unassign_container:
 	device->open_count--;
 	mutex_unlock(&device->dev_set->lock);
-	vfio_device_unassign_container(device);
 	return ERR_PTR(ret);
 }
 
@@ -1040,8 +1040,6 @@ static int vfio_device_fops_release(struct inode *inode, struct file *filep)
 	device->open_count--;
 	mutex_unlock(&device->dev_set->lock);
 
-	vfio_device_unassign_container(device);
-
 	vfio_device_put_registration(device);
 
 	return 0;
-- 
2.38.0

[PATCH 03/10] vfio: Rename vfio_device_assign/unassign_container()

[Jason Gunthorpe]

These functions don't really assign anything anymore, they just increment
some refcounts and do a sanity check. Call them
vfio_group_[un]use_container()

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/container.c | 14 ++++++--------
 drivers/vfio/vfio.h      |  4 ++--
 drivers/vfio/vfio_main.c |  6 +++---
 3 files changed, 11 insertions(+), 13 deletions(-)

diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
index dd79a66ec62cad..499777930b08fa 100644
--- a/drivers/vfio/container.c
+++ b/drivers/vfio/container.c
@@ -511,10 +511,8 @@ void vfio_group_detach_container(struct vfio_group *group)
 	vfio_container_put(container);
 }
 
-int vfio_device_assign_container(struct vfio_device *device)
+int vfio_group_use_container(struct vfio_group *group)
 {
-	struct vfio_group *group = device->group;
-
 	lockdep_assert_held(&group->group_lock);
 
 	if (!group->container || !group->container->iommu_driver ||
@@ -529,13 +527,13 @@ int vfio_device_assign_container(struct vfio_device *device)
 	return 0;
 }
 
-void vfio_device_unassign_container(struct vfio_device *device)
+void vfio_group_unuse_container(struct vfio_group *group)
 {
-	lockdep_assert_held_write(&device->group->group_lock);
+	lockdep_assert_held(&group->group_lock);
 
-	WARN_ON(device->group->container_users <= 1);
-	device->group->container_users--;
-	fput(device->group->opened_file);
+	WARN_ON(group->container_users <= 1);
+	group->container_users--;
+	fput(group->opened_file);
 }
 
 /*
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index bcad54bbab08c4..f95f4925b83bbd 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -112,8 +112,8 @@ void vfio_unregister_iommu_driver(const struct vfio_iommu_driver_ops *ops);
 bool vfio_assert_device_open(struct vfio_device *device);
 
 struct vfio_container *vfio_container_from_file(struct file *filep);
-int vfio_device_assign_container(struct vfio_device *device);
-void vfio_device_unassign_container(struct vfio_device *device);
+int vfio_group_use_container(struct vfio_group *group);
+void vfio_group_unuse_container(struct vfio_group *group);
 int vfio_container_attach_group(struct vfio_container *container,
 				struct vfio_group *group);
 void vfio_group_detach_container(struct vfio_group *group);
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 204443ba3b3cd9..8d809ecd982b39 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -749,7 +749,7 @@ static int vfio_device_first_open(struct vfio_device *device)
 	 * it during close_device.
 	 */
 	mutex_lock(&device->group->group_lock);
-	ret = vfio_device_assign_container(device);
+	ret = vfio_group_use_container(device->group);
 	if (ret)
 		goto err_module_put;
 
@@ -764,7 +764,7 @@ static int vfio_device_first_open(struct vfio_device *device)
 	return 0;
 
 err_container:
-	vfio_device_unassign_container(device);
+	vfio_group_unuse_container(device->group);
 err_module_put:
 	device->kvm = NULL;
 	mutex_unlock(&device->group->group_lock);
@@ -781,7 +781,7 @@ static void vfio_device_last_close(struct vfio_device *device)
 	if (device->ops->close_device)
 		device->ops->close_device(device);
 	device->kvm = NULL;
-	vfio_device_unassign_container(device);
+	vfio_group_unuse_container(device->group);
 	mutex_unlock(&device->group->group_lock);
 	module_put(device->dev->driver->owner);
 }
-- 
2.38.0

[PATCH 03/10] vfio: Rename vfio_device_assign/unassign_container()

[Jason Gunthorpe]

These functions don't really assign anything anymore, they just increment
some refcounts and do a sanity check. Call them
vfio_group_[un]use_container()

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/container.c | 14 ++++++--------
 drivers/vfio/vfio.h      |  4 ++--
 drivers/vfio/vfio_main.c |  6 +++---
 3 files changed, 11 insertions(+), 13 deletions(-)

diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
index dd79a66ec62cad..499777930b08fa 100644
--- a/drivers/vfio/container.c
+++ b/drivers/vfio/container.c
@@ -511,10 +511,8 @@ void vfio_group_detach_container(struct vfio_group *group)
 	vfio_container_put(container);
 }
 
-int vfio_device_assign_container(struct vfio_device *device)
+int vfio_group_use_container(struct vfio_group *group)
 {
-	struct vfio_group *group = device->group;
-
 	lockdep_assert_held(&group->group_lock);
 
 	if (!group->container || !group->container->iommu_driver ||
@@ -529,13 +527,13 @@ int vfio_device_assign_container(struct vfio_device *device)
 	return 0;
 }
 
-void vfio_device_unassign_container(struct vfio_device *device)
+void vfio_group_unuse_container(struct vfio_group *group)
 {
-	lockdep_assert_held_write(&device->group->group_lock);
+	lockdep_assert_held(&group->group_lock);
 
-	WARN_ON(device->group->container_users <= 1);
-	device->group->container_users--;
-	fput(device->group->opened_file);
+	WARN_ON(group->container_users <= 1);
+	group->container_users--;
+	fput(group->opened_file);
 }
 
 /*
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index bcad54bbab08c4..f95f4925b83bbd 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -112,8 +112,8 @@ void vfio_unregister_iommu_driver(const struct vfio_iommu_driver_ops *ops);
 bool vfio_assert_device_open(struct vfio_device *device);
 
 struct vfio_container *vfio_container_from_file(struct file *filep);
-int vfio_device_assign_container(struct vfio_device *device);
-void vfio_device_unassign_container(struct vfio_device *device);
+int vfio_group_use_container(struct vfio_group *group);
+void vfio_group_unuse_container(struct vfio_group *group);
 int vfio_container_attach_group(struct vfio_container *container,
 				struct vfio_group *group);
 void vfio_group_detach_container(struct vfio_group *group);
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 204443ba3b3cd9..8d809ecd982b39 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -749,7 +749,7 @@ static int vfio_device_first_open(struct vfio_device *device)
 	 * it during close_device.
 	 */
 	mutex_lock(&device->group->group_lock);
-	ret = vfio_device_assign_container(device);
+	ret = vfio_group_use_container(device->group);
 	if (ret)
 		goto err_module_put;
 
@@ -764,7 +764,7 @@ static int vfio_device_first_open(struct vfio_device *device)
 	return 0;
 
 err_container:
-	vfio_device_unassign_container(device);
+	vfio_group_unuse_container(device->group);
 err_module_put:
 	device->kvm = NULL;
 	mutex_unlock(&device->group->group_lock);
@@ -781,7 +781,7 @@ static void vfio_device_last_close(struct vfio_device *device)
 	if (device->ops->close_device)
 		device->ops->close_device(device);
 	device->kvm = NULL;
-	vfio_device_unassign_container(device);
+	vfio_group_unuse_container(device->group);
 	mutex_unlock(&device->group->group_lock);
 	module_put(device->dev->driver->owner);
 }
-- 
2.38.0

[Intel-gfx] [PATCH 03/10] vfio: Rename vfio_device_assign/unassign_container()

[Jason Gunthorpe]

These functions don't really assign anything anymore, they just increment
some refcounts and do a sanity check. Call them
vfio_group_[un]use_container()

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/container.c | 14 ++++++--------
 drivers/vfio/vfio.h      |  4 ++--
 drivers/vfio/vfio_main.c |  6 +++---
 3 files changed, 11 insertions(+), 13 deletions(-)

diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
index dd79a66ec62cad..499777930b08fa 100644
--- a/drivers/vfio/container.c
+++ b/drivers/vfio/container.c
@@ -511,10 +511,8 @@ void vfio_group_detach_container(struct vfio_group *group)
 	vfio_container_put(container);
 }
 
-int vfio_device_assign_container(struct vfio_device *device)
+int vfio_group_use_container(struct vfio_group *group)
 {
-	struct vfio_group *group = device->group;
-
 	lockdep_assert_held(&group->group_lock);
 
 	if (!group->container || !group->container->iommu_driver ||
@@ -529,13 +527,13 @@ int vfio_device_assign_container(struct vfio_device *device)
 	return 0;
 }
 
-void vfio_device_unassign_container(struct vfio_device *device)
+void vfio_group_unuse_container(struct vfio_group *group)
 {
-	lockdep_assert_held_write(&device->group->group_lock);
+	lockdep_assert_held(&group->group_lock);
 
-	WARN_ON(device->group->container_users <= 1);
-	device->group->container_users--;
-	fput(device->group->opened_file);
+	WARN_ON(group->container_users <= 1);
+	group->container_users--;
+	fput(group->opened_file);
 }
 
 /*
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index bcad54bbab08c4..f95f4925b83bbd 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -112,8 +112,8 @@ void vfio_unregister_iommu_driver(const struct vfio_iommu_driver_ops *ops);
 bool vfio_assert_device_open(struct vfio_device *device);
 
 struct vfio_container *vfio_container_from_file(struct file *filep);
-int vfio_device_assign_container(struct vfio_device *device);
-void vfio_device_unassign_container(struct vfio_device *device);
+int vfio_group_use_container(struct vfio_group *group);
+void vfio_group_unuse_container(struct vfio_group *group);
 int vfio_container_attach_group(struct vfio_container *container,
 				struct vfio_group *group);
 void vfio_group_detach_container(struct vfio_group *group);
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 204443ba3b3cd9..8d809ecd982b39 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -749,7 +749,7 @@ static int vfio_device_first_open(struct vfio_device *device)
 	 * it during close_device.
 	 */
 	mutex_lock(&device->group->group_lock);
-	ret = vfio_device_assign_container(device);
+	ret = vfio_group_use_container(device->group);
 	if (ret)
 		goto err_module_put;
 
@@ -764,7 +764,7 @@ static int vfio_device_first_open(struct vfio_device *device)
 	return 0;
 
 err_container:
-	vfio_device_unassign_container(device);
+	vfio_group_unuse_container(device->group);
 err_module_put:
 	device->kvm = NULL;
 	mutex_unlock(&device->group->group_lock);
@@ -781,7 +781,7 @@ static void vfio_device_last_close(struct vfio_device *device)
 	if (device->ops->close_device)
 		device->ops->close_device(device);
 	device->kvm = NULL;
-	vfio_device_unassign_container(device);
+	vfio_group_unuse_container(device->group);
 	mutex_unlock(&device->group->group_lock);
 	module_put(device->dev->driver->owner);
 }
-- 
2.38.0

[PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

This legacy module knob has become uAPI, when set on the vfio_iommu_type1
it disables some security protections in the iommu drivers. Move the
storage for this knob to vfio_main.c so that iommufd can access it too.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/vfio.h             | 2 ++
 drivers/vfio/vfio_iommu_type1.c | 5 ++---
 drivers/vfio/vfio_main.c        | 3 +++
 3 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index f95f4925b83bbd..54e5a8e0834ccb 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -130,4 +130,6 @@ extern bool vfio_noiommu __read_mostly;
 enum { vfio_noiommu = false };
 #endif
 
+extern bool vfio_allow_unsafe_interrupts;
+
 #endif
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 23c24fe98c00d4..186e33a006d314 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -44,9 +44,8 @@
 #define DRIVER_AUTHOR   "Alex Williamson <alex.williamson@redhat.com>"
 #define DRIVER_DESC     "Type1 IOMMU driver for VFIO"
 
-static bool allow_unsafe_interrupts;
 module_param_named(allow_unsafe_interrupts,
-		   allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
+		   vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
 MODULE_PARM_DESC(allow_unsafe_interrupts,
 		 "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
 
@@ -2282,7 +2281,7 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
 		    iommu_group_for_each_dev(iommu_group, (void *)IOMMU_CAP_INTR_REMAP,
 					     vfio_iommu_device_capable);
 
-	if (!allow_unsafe_interrupts && !msi_remap) {
+	if (!vfio_allow_unsafe_interrupts && !msi_remap) {
 		pr_warn("%s: No interrupt remapping support.  Use the module param \"allow_unsafe_interrupts\" to enable VFIO IOMMU support on this platform\n",
 		       __func__);
 		ret = -EPERM;
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 8d809ecd982b39..1e414b2c48a511 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -51,6 +51,9 @@ static struct vfio {
 	struct ida			device_ida;
 } vfio;
 
+bool vfio_allow_unsafe_interrupts;
+EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
+
 static DEFINE_XARRAY(vfio_device_set_xa);
 static const struct file_operations vfio_group_fops;
 
-- 
2.38.0

[PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

This legacy module knob has become uAPI, when set on the vfio_iommu_type1
it disables some security protections in the iommu drivers. Move the
storage for this knob to vfio_main.c so that iommufd can access it too.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/vfio.h             | 2 ++
 drivers/vfio/vfio_iommu_type1.c | 5 ++---
 drivers/vfio/vfio_main.c        | 3 +++
 3 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index f95f4925b83bbd..54e5a8e0834ccb 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -130,4 +130,6 @@ extern bool vfio_noiommu __read_mostly;
 enum { vfio_noiommu = false };
 #endif
 
+extern bool vfio_allow_unsafe_interrupts;
+
 #endif
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 23c24fe98c00d4..186e33a006d314 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -44,9 +44,8 @@
 #define DRIVER_AUTHOR   "Alex Williamson <alex.williamson@redhat.com>"
 #define DRIVER_DESC     "Type1 IOMMU driver for VFIO"
 
-static bool allow_unsafe_interrupts;
 module_param_named(allow_unsafe_interrupts,
-		   allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
+		   vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
 MODULE_PARM_DESC(allow_unsafe_interrupts,
 		 "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
 
@@ -2282,7 +2281,7 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
 		    iommu_group_for_each_dev(iommu_group, (void *)IOMMU_CAP_INTR_REMAP,
 					     vfio_iommu_device_capable);
 
-	if (!allow_unsafe_interrupts && !msi_remap) {
+	if (!vfio_allow_unsafe_interrupts && !msi_remap) {
 		pr_warn("%s: No interrupt remapping support.  Use the module param \"allow_unsafe_interrupts\" to enable VFIO IOMMU support on this platform\n",
 		       __func__);
 		ret = -EPERM;
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 8d809ecd982b39..1e414b2c48a511 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -51,6 +51,9 @@ static struct vfio {
 	struct ida			device_ida;
 } vfio;
 
+bool vfio_allow_unsafe_interrupts;
+EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
+
 static DEFINE_XARRAY(vfio_device_set_xa);
 static const struct file_operations vfio_group_fops;
 
-- 
2.38.0

[Intel-gfx] [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

This legacy module knob has become uAPI, when set on the vfio_iommu_type1
it disables some security protections in the iommu drivers. Move the
storage for this knob to vfio_main.c so that iommufd can access it too.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/vfio.h             | 2 ++
 drivers/vfio/vfio_iommu_type1.c | 5 ++---
 drivers/vfio/vfio_main.c        | 3 +++
 3 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index f95f4925b83bbd..54e5a8e0834ccb 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -130,4 +130,6 @@ extern bool vfio_noiommu __read_mostly;
 enum { vfio_noiommu = false };
 #endif
 
+extern bool vfio_allow_unsafe_interrupts;
+
 #endif
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 23c24fe98c00d4..186e33a006d314 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -44,9 +44,8 @@
 #define DRIVER_AUTHOR   "Alex Williamson <alex.williamson@redhat.com>"
 #define DRIVER_DESC     "Type1 IOMMU driver for VFIO"
 
-static bool allow_unsafe_interrupts;
 module_param_named(allow_unsafe_interrupts,
-		   allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
+		   vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
 MODULE_PARM_DESC(allow_unsafe_interrupts,
 		 "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
 
@@ -2282,7 +2281,7 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
 		    iommu_group_for_each_dev(iommu_group, (void *)IOMMU_CAP_INTR_REMAP,
 					     vfio_iommu_device_capable);
 
-	if (!allow_unsafe_interrupts && !msi_remap) {
+	if (!vfio_allow_unsafe_interrupts && !msi_remap) {
 		pr_warn("%s: No interrupt remapping support.  Use the module param \"allow_unsafe_interrupts\" to enable VFIO IOMMU support on this platform\n",
 		       __func__);
 		ret = -EPERM;
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 8d809ecd982b39..1e414b2c48a511 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -51,6 +51,9 @@ static struct vfio {
 	struct ida			device_ida;
 } vfio;
 
+bool vfio_allow_unsafe_interrupts;
+EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
+
 static DEFINE_XARRAY(vfio_device_set_xa);
 static const struct file_operations vfio_group_fops;
 
-- 
2.38.0

[PATCH 05/10] vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()

[Jason Gunthorpe]

iommufd doesn't establish the iommu_domains until after the device FD is
opened, even if the container has been set. This design is part of moving
away from the group centric iommu APIs.

This is fine, except that the normal sequence of establishing the kvm
wbindv won't work:

   group = open("/dev/vfio/XX")
   ioctl(group, VFIO_GROUP_SET_CONTAINER)
   ioctl(kvm, KVM_DEV_VFIO_GROUP_ADD)
   ioctl(group, VFIO_GROUP_GET_DEVICE_FD)

As the domains don't start existing until GET_DEVICE_FD. Further,
GET_DEVICE_FD requires that KVM_DEV_VFIO_GROUP_ADD already be done as that
is what sets the group->kvm and thus device->kvm for the driver to use
during open.

Now that we have device centric cap ops and the new
IOMMU_CAP_ENFORCE_CACHE_COHERENCY we know what the iommu_domain will be
capable of without having to create it. Use this to compute
vfio_file_enforced_coherent() and resolve the ordering problems.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/container.c |  5 +++--
 drivers/vfio/vfio.h      |  2 --
 drivers/vfio/vfio_main.c | 27 ++++++++++++++-------------
 3 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
index 499777930b08fa..d97747dfb05d02 100644
--- a/drivers/vfio/container.c
+++ b/drivers/vfio/container.c
@@ -188,8 +188,9 @@ void vfio_device_container_unregister(struct vfio_device *device)
 			device->group->container->iommu_data, device);
 }
 
-long vfio_container_ioctl_check_extension(struct vfio_container *container,
-					  unsigned long arg)
+static long
+vfio_container_ioctl_check_extension(struct vfio_container *container,
+				     unsigned long arg)
 {
 	struct vfio_iommu_driver *driver;
 	long ret = 0;
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index 54e5a8e0834ccb..247590334e14b0 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -119,8 +119,6 @@ int vfio_container_attach_group(struct vfio_container *container,
 void vfio_group_detach_container(struct vfio_group *group);
 void vfio_device_container_register(struct vfio_device *device);
 void vfio_device_container_unregister(struct vfio_device *device);
-long vfio_container_ioctl_check_extension(struct vfio_container *container,
-					  unsigned long arg);
 int __init vfio_container_init(void);
 void vfio_container_cleanup(void);
 
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 1e414b2c48a511..a8d1fbfcc3ddad 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -1625,24 +1625,25 @@ EXPORT_SYMBOL_GPL(vfio_file_is_group);
 bool vfio_file_enforced_coherent(struct file *file)
 {
 	struct vfio_group *group = file->private_data;
-	bool ret;
+	struct vfio_device *device;
+	bool ret = true;
 
 	if (!vfio_file_is_group(file))
 		return true;
 
-	mutex_lock(&group->group_lock);
-	if (group->container) {
-		ret = vfio_container_ioctl_check_extension(group->container,
-							   VFIO_DMA_CC_IOMMU);
-	} else {
-		/*
-		 * Since the coherency state is determined only once a container
-		 * is attached the user must do so before they can prove they
-		 * have permission.
-		 */
-		ret = true;
+	/*
+	 * If the device does not have IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
+	 * any domain later attached to it will also not support it.
+	 */
+	mutex_lock(&group->device_lock);
+	list_for_each_entry(device, &group->device_list, group_next) {
+		if (!device_iommu_capable(device->dev,
+					  IOMMU_CAP_ENFORCE_CACHE_COHERENCY)) {
+			ret = false;
+			break;
+		}
 	}
-	mutex_unlock(&group->group_lock);
+	mutex_unlock(&group->device_lock);
 	return ret;
 }
 EXPORT_SYMBOL_GPL(vfio_file_enforced_coherent);
-- 
2.38.0

[PATCH 05/10] vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()

[Jason Gunthorpe]

iommufd doesn't establish the iommu_domains until after the device FD is
opened, even if the container has been set. This design is part of moving
away from the group centric iommu APIs.

This is fine, except that the normal sequence of establishing the kvm
wbindv won't work:

   group = open("/dev/vfio/XX")
   ioctl(group, VFIO_GROUP_SET_CONTAINER)
   ioctl(kvm, KVM_DEV_VFIO_GROUP_ADD)
   ioctl(group, VFIO_GROUP_GET_DEVICE_FD)

As the domains don't start existing until GET_DEVICE_FD. Further,
GET_DEVICE_FD requires that KVM_DEV_VFIO_GROUP_ADD already be done as that
is what sets the group->kvm and thus device->kvm for the driver to use
during open.

Now that we have device centric cap ops and the new
IOMMU_CAP_ENFORCE_CACHE_COHERENCY we know what the iommu_domain will be
capable of without having to create it. Use this to compute
vfio_file_enforced_coherent() and resolve the ordering problems.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/container.c |  5 +++--
 drivers/vfio/vfio.h      |  2 --
 drivers/vfio/vfio_main.c | 27 ++++++++++++++-------------
 3 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
index 499777930b08fa..d97747dfb05d02 100644
--- a/drivers/vfio/container.c
+++ b/drivers/vfio/container.c
@@ -188,8 +188,9 @@ void vfio_device_container_unregister(struct vfio_device *device)
 			device->group->container->iommu_data, device);
 }
 
-long vfio_container_ioctl_check_extension(struct vfio_container *container,
-					  unsigned long arg)
+static long
+vfio_container_ioctl_check_extension(struct vfio_container *container,
+				     unsigned long arg)
 {
 	struct vfio_iommu_driver *driver;
 	long ret = 0;
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index 54e5a8e0834ccb..247590334e14b0 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -119,8 +119,6 @@ int vfio_container_attach_group(struct vfio_container *container,
 void vfio_group_detach_container(struct vfio_group *group);
 void vfio_device_container_register(struct vfio_device *device);
 void vfio_device_container_unregister(struct vfio_device *device);
-long vfio_container_ioctl_check_extension(struct vfio_container *container,
-					  unsigned long arg);
 int __init vfio_container_init(void);
 void vfio_container_cleanup(void);
 
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 1e414b2c48a511..a8d1fbfcc3ddad 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -1625,24 +1625,25 @@ EXPORT_SYMBOL_GPL(vfio_file_is_group);
 bool vfio_file_enforced_coherent(struct file *file)
 {
 	struct vfio_group *group = file->private_data;
-	bool ret;
+	struct vfio_device *device;
+	bool ret = true;
 
 	if (!vfio_file_is_group(file))
 		return true;
 
-	mutex_lock(&group->group_lock);
-	if (group->container) {
-		ret = vfio_container_ioctl_check_extension(group->container,
-							   VFIO_DMA_CC_IOMMU);
-	} else {
-		/*
-		 * Since the coherency state is determined only once a container
-		 * is attached the user must do so before they can prove they
-		 * have permission.
-		 */
-		ret = true;
+	/*
+	 * If the device does not have IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
+	 * any domain later attached to it will also not support it.
+	 */
+	mutex_lock(&group->device_lock);
+	list_for_each_entry(device, &group->device_list, group_next) {
+		if (!device_iommu_capable(device->dev,
+					  IOMMU_CAP_ENFORCE_CACHE_COHERENCY)) {
+			ret = false;
+			break;
+		}
 	}
-	mutex_unlock(&group->group_lock);
+	mutex_unlock(&group->device_lock);
 	return ret;
 }
 EXPORT_SYMBOL_GPL(vfio_file_enforced_coherent);
-- 
2.38.0

[Intel-gfx] [PATCH 05/10] vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()

[Jason Gunthorpe]

iommufd doesn't establish the iommu_domains until after the device FD is
opened, even if the container has been set. This design is part of moving
away from the group centric iommu APIs.

This is fine, except that the normal sequence of establishing the kvm
wbindv won't work:

   group = open("/dev/vfio/XX")
   ioctl(group, VFIO_GROUP_SET_CONTAINER)
   ioctl(kvm, KVM_DEV_VFIO_GROUP_ADD)
   ioctl(group, VFIO_GROUP_GET_DEVICE_FD)

As the domains don't start existing until GET_DEVICE_FD. Further,
GET_DEVICE_FD requires that KVM_DEV_VFIO_GROUP_ADD already be done as that
is what sets the group->kvm and thus device->kvm for the driver to use
during open.

Now that we have device centric cap ops and the new
IOMMU_CAP_ENFORCE_CACHE_COHERENCY we know what the iommu_domain will be
capable of without having to create it. Use this to compute
vfio_file_enforced_coherent() and resolve the ordering problems.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/container.c |  5 +++--
 drivers/vfio/vfio.h      |  2 --
 drivers/vfio/vfio_main.c | 27 ++++++++++++++-------------
 3 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
index 499777930b08fa..d97747dfb05d02 100644
--- a/drivers/vfio/container.c
+++ b/drivers/vfio/container.c
@@ -188,8 +188,9 @@ void vfio_device_container_unregister(struct vfio_device *device)
 			device->group->container->iommu_data, device);
 }
 
-long vfio_container_ioctl_check_extension(struct vfio_container *container,
-					  unsigned long arg)
+static long
+vfio_container_ioctl_check_extension(struct vfio_container *container,
+				     unsigned long arg)
 {
 	struct vfio_iommu_driver *driver;
 	long ret = 0;
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index 54e5a8e0834ccb..247590334e14b0 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -119,8 +119,6 @@ int vfio_container_attach_group(struct vfio_container *container,
 void vfio_group_detach_container(struct vfio_group *group);
 void vfio_device_container_register(struct vfio_device *device);
 void vfio_device_container_unregister(struct vfio_device *device);
-long vfio_container_ioctl_check_extension(struct vfio_container *container,
-					  unsigned long arg);
 int __init vfio_container_init(void);
 void vfio_container_cleanup(void);
 
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 1e414b2c48a511..a8d1fbfcc3ddad 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -1625,24 +1625,25 @@ EXPORT_SYMBOL_GPL(vfio_file_is_group);
 bool vfio_file_enforced_coherent(struct file *file)
 {
 	struct vfio_group *group = file->private_data;
-	bool ret;
+	struct vfio_device *device;
+	bool ret = true;
 
 	if (!vfio_file_is_group(file))
 		return true;
 
-	mutex_lock(&group->group_lock);
-	if (group->container) {
-		ret = vfio_container_ioctl_check_extension(group->container,
-							   VFIO_DMA_CC_IOMMU);
-	} else {
-		/*
-		 * Since the coherency state is determined only once a container
-		 * is attached the user must do so before they can prove they
-		 * have permission.
-		 */
-		ret = true;
+	/*
+	 * If the device does not have IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
+	 * any domain later attached to it will also not support it.
+	 */
+	mutex_lock(&group->device_lock);
+	list_for_each_entry(device, &group->device_list, group_next) {
+		if (!device_iommu_capable(device->dev,
+					  IOMMU_CAP_ENFORCE_CACHE_COHERENCY)) {
+			ret = false;
+			break;
+		}
 	}
-	mutex_unlock(&group->group_lock);
+	mutex_unlock(&group->device_lock);
 	return ret;
 }
 EXPORT_SYMBOL_GPL(vfio_file_enforced_coherent);
-- 
2.38.0

[PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Jason Gunthorpe]

This makes VFIO_GROUP_SET_CONTAINER accept both a vfio container FD and an
iommufd.

In iommufd mode an IOAS will exist after the SET_CONTAINER, but it will
not be attached to any groups.

From a VFIO perspective this means that the VFIO_GROUP_GET_STATUS and
VFIO_GROUP_FLAGS_VIABLE works subtly differently. With the container FD
the iommu_group_claim_dma_owner() is done during SET_CONTAINER but for
IOMMFD this is done during VFIO_GROUP_GET_DEVICE_FD. Meaning that
VFIO_GROUP_FLAGS_VIABLE could be set but GET_DEVICE_FD will fail due to
viability.

As GET_DEVICE_FD can fail for many reasons already this is not expected to
be a meaningful difference.

Reorganize the tests for if the group has an assigned container or iommu
into a vfio_group_has_iommu() function and consolidate all the duplicated
WARN_ON's etc related to this.

Call container functions only if a container is actually present on the
group.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/Kconfig     |  1 +
 drivers/vfio/container.c |  7 ++--
 drivers/vfio/vfio.h      |  2 ++
 drivers/vfio/vfio_main.c | 76 ++++++++++++++++++++++++++++++++--------
 4 files changed, 69 insertions(+), 17 deletions(-)

diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
index 86c381ceb9a1e9..1118d322eec97d 100644
--- a/drivers/vfio/Kconfig
+++ b/drivers/vfio/Kconfig
@@ -2,6 +2,7 @@
 menuconfig VFIO
 	tristate "VFIO Non-Privileged userspace driver framework"
 	select IOMMU_API
+	depends on IOMMUFD || !IOMMUFD
 	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM || ARM64)
 	select INTERVAL_TREE
 	help
diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
index d97747dfb05d02..8772dad6808539 100644
--- a/drivers/vfio/container.c
+++ b/drivers/vfio/container.c
@@ -516,8 +516,11 @@ int vfio_group_use_container(struct vfio_group *group)
 {
 	lockdep_assert_held(&group->group_lock);
 
-	if (!group->container || !group->container->iommu_driver ||
-	    WARN_ON(!group->container_users))
+	/*
+	 * The container fd has been assigned with VFIO_GROUP_SET_CONTAINER but
+	 * VFIO_SET_IOMMU hasn't been done yet.
+	 */
+	if (!group->container->iommu_driver)
 		return -EINVAL;
 
 	if (group->type == VFIO_NO_IOMMU && !capable(CAP_SYS_RAWIO))
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index 247590334e14b0..985e13d52989ca 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -10,6 +10,7 @@
 #include <linux/cdev.h>
 #include <linux/module.h>
 
+struct iommufd_ctx;
 struct iommu_group;
 struct vfio_device;
 struct vfio_container;
@@ -60,6 +61,7 @@ struct vfio_group {
 	struct kvm			*kvm;
 	struct file			*opened_file;
 	struct blocking_notifier_head	notifier;
+	struct iommufd_ctx		*iommufd;
 };
 
 /* events for the backend driver notify callback */
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index a8d1fbfcc3ddad..cf0ea744de931e 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -35,6 +35,7 @@
 #include <linux/pm_runtime.h>
 #include <linux/interval_tree.h>
 #include <linux/iova_bitmap.h>
+#include <linux/iommufd.h>
 #include "vfio.h"
 
 #define DRIVER_VERSION	"0.3"
@@ -665,6 +666,16 @@ EXPORT_SYMBOL_GPL(vfio_unregister_group_dev);
 /*
  * VFIO Group fd, /dev/vfio/$GROUP
  */
+static bool vfio_group_has_iommu(struct vfio_group *group)
+{
+	lockdep_assert_held(&group->group_lock);
+	if (!group->container)
+		WARN_ON(group->container_users);
+	else
+		WARN_ON(!group->container_users);
+	return group->container || group->iommufd;
+}
+
 /*
  * VFIO_GROUP_UNSET_CONTAINER should fail if there are other users or
  * if there was no container to unset.  Since the ioctl is called on
@@ -676,15 +687,21 @@ static int vfio_group_ioctl_unset_container(struct vfio_group *group)
 	int ret = 0;
 
 	mutex_lock(&group->group_lock);
-	if (!group->container) {
+	if (!vfio_group_has_iommu(group)) {
 		ret = -EINVAL;
 		goto out_unlock;
 	}
-	if (group->container_users != 1) {
-		ret = -EBUSY;
-		goto out_unlock;
+	if (group->container) {
+		if (group->container_users != 1) {
+			ret = -EBUSY;
+			goto out_unlock;
+		}
+		vfio_group_detach_container(group);
+	}
+	if (group->iommufd) {
+		iommufd_ctx_put(group->iommufd);
+		group->iommufd = NULL;
 	}
-	vfio_group_detach_container(group);
 
 out_unlock:
 	mutex_unlock(&group->group_lock);
@@ -695,6 +712,7 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
 					  int __user *arg)
 {
 	struct vfio_container *container;
+	struct iommufd_ctx *iommufd;
 	struct fd f;
 	int ret;
 	int fd;
@@ -707,7 +725,7 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
 		return -EBADF;
 
 	mutex_lock(&group->group_lock);
-	if (group->container || WARN_ON(group->container_users)) {
+	if (vfio_group_has_iommu(group)) {
 		ret = -EINVAL;
 		goto out_unlock;
 	}
@@ -717,12 +735,23 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
 	}
 
 	container = vfio_container_from_file(f.file);
-	ret = -EINVAL;
 	if (container) {
 		ret = vfio_container_attach_group(container, group);
 		goto out_unlock;
 	}
 
+	iommufd = iommufd_ctx_from_file(f.file);
+	if (!IS_ERR(iommufd)) {
+		u32 ioas_id;
+
+		group->iommufd = iommufd;
+		ret = iommufd_vfio_compat_ioas_id(iommufd, &ioas_id);
+		goto out_unlock;
+	}
+
+	/* The FD passed is not recognized. */
+	ret = -EBADF;
+
 out_unlock:
 	mutex_unlock(&group->group_lock);
 	fdput(f);
@@ -752,9 +781,16 @@ static int vfio_device_first_open(struct vfio_device *device)
 	 * it during close_device.
 	 */
 	mutex_lock(&device->group->group_lock);
-	ret = vfio_group_use_container(device->group);
-	if (ret)
+	if (!vfio_group_has_iommu(device->group)) {
+		ret = -EINVAL;
 		goto err_module_put;
+	}
+
+	if (device->group->container) {
+		ret = vfio_group_use_container(device->group);
+		if (ret)
+			goto err_module_put;
+	}
 
 	device->kvm = device->group->kvm;
 	if (device->ops->open_device) {
@@ -762,14 +798,16 @@ static int vfio_device_first_open(struct vfio_device *device)
 		if (ret)
 			goto err_container;
 	}
-	vfio_device_container_register(device);
+	if (device->group->container)
+		vfio_device_container_register(device);
 	mutex_unlock(&device->group->group_lock);
 	return 0;
 
 err_container:
-	vfio_group_unuse_container(device->group);
-err_module_put:
+	if (device->group->container)
+		vfio_group_unuse_container(device->group);
 	device->kvm = NULL;
+err_module_put:
 	mutex_unlock(&device->group->group_lock);
 	module_put(device->dev->driver->owner);
 	return ret;
@@ -780,11 +818,13 @@ static void vfio_device_last_close(struct vfio_device *device)
 	lockdep_assert_held(&device->dev_set->lock);
 
 	mutex_lock(&device->group->group_lock);
-	vfio_device_container_unregister(device);
+	if (device->group->container)
+		vfio_device_container_unregister(device);
 	if (device->ops->close_device)
 		device->ops->close_device(device);
 	device->kvm = NULL;
-	vfio_group_unuse_container(device->group);
+	if (device->group->container)
+		vfio_group_unuse_container(device->group);
 	mutex_unlock(&device->group->group_lock);
 	module_put(device->dev->driver->owner);
 }
@@ -900,7 +940,7 @@ static int vfio_group_ioctl_get_status(struct vfio_group *group,
 		return -ENODEV;
 	}
 
-	if (group->container)
+	if (group->container || group->iommufd)
 		status.flags |= VFIO_GROUP_FLAGS_CONTAINER_SET |
 				VFIO_GROUP_FLAGS_VIABLE;
 	else if (!iommu_group_dma_owner_claimed(group->iommu_group))
@@ -983,6 +1023,10 @@ static int vfio_group_fops_release(struct inode *inode, struct file *filep)
 	WARN_ON(group->notifier.head);
 	if (group->container)
 		vfio_group_detach_container(group);
+	if (group->iommufd) {
+		iommufd_ctx_put(group->iommufd);
+		group->iommufd = NULL;
+	}
 	group->opened_file = NULL;
 	mutex_unlock(&group->group_lock);
 	return 0;
@@ -1879,6 +1923,8 @@ static void __exit vfio_cleanup(void)
 module_init(vfio_init);
 module_exit(vfio_cleanup);
 
+MODULE_IMPORT_NS(IOMMUFD);
+MODULE_IMPORT_NS(IOMMUFD_VFIO);
 MODULE_VERSION(DRIVER_VERSION);
 MODULE_LICENSE("GPL v2");
 MODULE_AUTHOR(DRIVER_AUTHOR);
-- 
2.38.0

[PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Jason Gunthorpe]

This makes VFIO_GROUP_SET_CONTAINER accept both a vfio container FD and an
iommufd.

In iommufd mode an IOAS will exist after the SET_CONTAINER, but it will
not be attached to any groups.

From a VFIO perspective this means that the VFIO_GROUP_GET_STATUS and
VFIO_GROUP_FLAGS_VIABLE works subtly differently. With the container FD
the iommu_group_claim_dma_owner() is done during SET_CONTAINER but for
IOMMFD this is done during VFIO_GROUP_GET_DEVICE_FD. Meaning that
VFIO_GROUP_FLAGS_VIABLE could be set but GET_DEVICE_FD will fail due to
viability.

As GET_DEVICE_FD can fail for many reasons already this is not expected to
be a meaningful difference.

Reorganize the tests for if the group has an assigned container or iommu
into a vfio_group_has_iommu() function and consolidate all the duplicated
WARN_ON's etc related to this.

Call container functions only if a container is actually present on the
group.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/Kconfig     |  1 +
 drivers/vfio/container.c |  7 ++--
 drivers/vfio/vfio.h      |  2 ++
 drivers/vfio/vfio_main.c | 76 ++++++++++++++++++++++++++++++++--------
 4 files changed, 69 insertions(+), 17 deletions(-)

diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
index 86c381ceb9a1e9..1118d322eec97d 100644
--- a/drivers/vfio/Kconfig
+++ b/drivers/vfio/Kconfig
@@ -2,6 +2,7 @@
 menuconfig VFIO
 	tristate "VFIO Non-Privileged userspace driver framework"
 	select IOMMU_API
+	depends on IOMMUFD || !IOMMUFD
 	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM || ARM64)
 	select INTERVAL_TREE
 	help
diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
index d97747dfb05d02..8772dad6808539 100644
--- a/drivers/vfio/container.c
+++ b/drivers/vfio/container.c
@@ -516,8 +516,11 @@ int vfio_group_use_container(struct vfio_group *group)
 {
 	lockdep_assert_held(&group->group_lock);
 
-	if (!group->container || !group->container->iommu_driver ||
-	    WARN_ON(!group->container_users))
+	/*
+	 * The container fd has been assigned with VFIO_GROUP_SET_CONTAINER but
+	 * VFIO_SET_IOMMU hasn't been done yet.
+	 */
+	if (!group->container->iommu_driver)
 		return -EINVAL;
 
 	if (group->type == VFIO_NO_IOMMU && !capable(CAP_SYS_RAWIO))
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index 247590334e14b0..985e13d52989ca 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -10,6 +10,7 @@
 #include <linux/cdev.h>
 #include <linux/module.h>
 
+struct iommufd_ctx;
 struct iommu_group;
 struct vfio_device;
 struct vfio_container;
@@ -60,6 +61,7 @@ struct vfio_group {
 	struct kvm			*kvm;
 	struct file			*opened_file;
 	struct blocking_notifier_head	notifier;
+	struct iommufd_ctx		*iommufd;
 };
 
 /* events for the backend driver notify callback */
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index a8d1fbfcc3ddad..cf0ea744de931e 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -35,6 +35,7 @@
 #include <linux/pm_runtime.h>
 #include <linux/interval_tree.h>
 #include <linux/iova_bitmap.h>
+#include <linux/iommufd.h>
 #include "vfio.h"
 
 #define DRIVER_VERSION	"0.3"
@@ -665,6 +666,16 @@ EXPORT_SYMBOL_GPL(vfio_unregister_group_dev);
 /*
  * VFIO Group fd, /dev/vfio/$GROUP
  */
+static bool vfio_group_has_iommu(struct vfio_group *group)
+{
+	lockdep_assert_held(&group->group_lock);
+	if (!group->container)
+		WARN_ON(group->container_users);
+	else
+		WARN_ON(!group->container_users);
+	return group->container || group->iommufd;
+}
+
 /*
  * VFIO_GROUP_UNSET_CONTAINER should fail if there are other users or
  * if there was no container to unset.  Since the ioctl is called on
@@ -676,15 +687,21 @@ static int vfio_group_ioctl_unset_container(struct vfio_group *group)
 	int ret = 0;
 
 	mutex_lock(&group->group_lock);
-	if (!group->container) {
+	if (!vfio_group_has_iommu(group)) {
 		ret = -EINVAL;
 		goto out_unlock;
 	}
-	if (group->container_users != 1) {
-		ret = -EBUSY;
-		goto out_unlock;
+	if (group->container) {
+		if (group->container_users != 1) {
+			ret = -EBUSY;
+			goto out_unlock;
+		}
+		vfio_group_detach_container(group);
+	}
+	if (group->iommufd) {
+		iommufd_ctx_put(group->iommufd);
+		group->iommufd = NULL;
 	}
-	vfio_group_detach_container(group);
 
 out_unlock:
 	mutex_unlock(&group->group_lock);
@@ -695,6 +712,7 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
 					  int __user *arg)
 {
 	struct vfio_container *container;
+	struct iommufd_ctx *iommufd;
 	struct fd f;
 	int ret;
 	int fd;
@@ -707,7 +725,7 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
 		return -EBADF;
 
 	mutex_lock(&group->group_lock);
-	if (group->container || WARN_ON(group->container_users)) {
+	if (vfio_group_has_iommu(group)) {
 		ret = -EINVAL;
 		goto out_unlock;
 	}
@@ -717,12 +735,23 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
 	}
 
 	container = vfio_container_from_file(f.file);
-	ret = -EINVAL;
 	if (container) {
 		ret = vfio_container_attach_group(container, group);
 		goto out_unlock;
 	}
 
+	iommufd = iommufd_ctx_from_file(f.file);
+	if (!IS_ERR(iommufd)) {
+		u32 ioas_id;
+
+		group->iommufd = iommufd;
+		ret = iommufd_vfio_compat_ioas_id(iommufd, &ioas_id);
+		goto out_unlock;
+	}
+
+	/* The FD passed is not recognized. */
+	ret = -EBADF;
+
 out_unlock:
 	mutex_unlock(&group->group_lock);
 	fdput(f);
@@ -752,9 +781,16 @@ static int vfio_device_first_open(struct vfio_device *device)
 	 * it during close_device.
 	 */
 	mutex_lock(&device->group->group_lock);
-	ret = vfio_group_use_container(device->group);
-	if (ret)
+	if (!vfio_group_has_iommu(device->group)) {
+		ret = -EINVAL;
 		goto err_module_put;
+	}
+
+	if (device->group->container) {
+		ret = vfio_group_use_container(device->group);
+		if (ret)
+			goto err_module_put;
+	}
 
 	device->kvm = device->group->kvm;
 	if (device->ops->open_device) {
@@ -762,14 +798,16 @@ static int vfio_device_first_open(struct vfio_device *device)
 		if (ret)
 			goto err_container;
 	}
-	vfio_device_container_register(device);
+	if (device->group->container)
+		vfio_device_container_register(device);
 	mutex_unlock(&device->group->group_lock);
 	return 0;
 
 err_container:
-	vfio_group_unuse_container(device->group);
-err_module_put:
+	if (device->group->container)
+		vfio_group_unuse_container(device->group);
 	device->kvm = NULL;
+err_module_put:
 	mutex_unlock(&device->group->group_lock);
 	module_put(device->dev->driver->owner);
 	return ret;
@@ -780,11 +818,13 @@ static void vfio_device_last_close(struct vfio_device *device)
 	lockdep_assert_held(&device->dev_set->lock);
 
 	mutex_lock(&device->group->group_lock);
-	vfio_device_container_unregister(device);
+	if (device->group->container)
+		vfio_device_container_unregister(device);
 	if (device->ops->close_device)
 		device->ops->close_device(device);
 	device->kvm = NULL;
-	vfio_group_unuse_container(device->group);
+	if (device->group->container)
+		vfio_group_unuse_container(device->group);
 	mutex_unlock(&device->group->group_lock);
 	module_put(device->dev->driver->owner);
 }
@@ -900,7 +940,7 @@ static int vfio_group_ioctl_get_status(struct vfio_group *group,
 		return -ENODEV;
 	}
 
-	if (group->container)
+	if (group->container || group->iommufd)
 		status.flags |= VFIO_GROUP_FLAGS_CONTAINER_SET |
 				VFIO_GROUP_FLAGS_VIABLE;
 	else if (!iommu_group_dma_owner_claimed(group->iommu_group))
@@ -983,6 +1023,10 @@ static int vfio_group_fops_release(struct inode *inode, struct file *filep)
 	WARN_ON(group->notifier.head);
 	if (group->container)
 		vfio_group_detach_container(group);
+	if (group->iommufd) {
+		iommufd_ctx_put(group->iommufd);
+		group->iommufd = NULL;
+	}
 	group->opened_file = NULL;
 	mutex_unlock(&group->group_lock);
 	return 0;
@@ -1879,6 +1923,8 @@ static void __exit vfio_cleanup(void)
 module_init(vfio_init);
 module_exit(vfio_cleanup);
 
+MODULE_IMPORT_NS(IOMMUFD);
+MODULE_IMPORT_NS(IOMMUFD_VFIO);
 MODULE_VERSION(DRIVER_VERSION);
 MODULE_LICENSE("GPL v2");
 MODULE_AUTHOR(DRIVER_AUTHOR);
-- 
2.38.0

[Intel-gfx] [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Jason Gunthorpe]

This makes VFIO_GROUP_SET_CONTAINER accept both a vfio container FD and an
iommufd.

In iommufd mode an IOAS will exist after the SET_CONTAINER, but it will
not be attached to any groups.

From a VFIO perspective this means that the VFIO_GROUP_GET_STATUS and
VFIO_GROUP_FLAGS_VIABLE works subtly differently. With the container FD
the iommu_group_claim_dma_owner() is done during SET_CONTAINER but for
IOMMFD this is done during VFIO_GROUP_GET_DEVICE_FD. Meaning that
VFIO_GROUP_FLAGS_VIABLE could be set but GET_DEVICE_FD will fail due to
viability.

As GET_DEVICE_FD can fail for many reasons already this is not expected to
be a meaningful difference.

Reorganize the tests for if the group has an assigned container or iommu
into a vfio_group_has_iommu() function and consolidate all the duplicated
WARN_ON's etc related to this.

Call container functions only if a container is actually present on the
group.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/Kconfig     |  1 +
 drivers/vfio/container.c |  7 ++--
 drivers/vfio/vfio.h      |  2 ++
 drivers/vfio/vfio_main.c | 76 ++++++++++++++++++++++++++++++++--------
 4 files changed, 69 insertions(+), 17 deletions(-)

diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
index 86c381ceb9a1e9..1118d322eec97d 100644
--- a/drivers/vfio/Kconfig
+++ b/drivers/vfio/Kconfig
@@ -2,6 +2,7 @@
 menuconfig VFIO
 	tristate "VFIO Non-Privileged userspace driver framework"
 	select IOMMU_API
+	depends on IOMMUFD || !IOMMUFD
 	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM || ARM64)
 	select INTERVAL_TREE
 	help
diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
index d97747dfb05d02..8772dad6808539 100644
--- a/drivers/vfio/container.c
+++ b/drivers/vfio/container.c
@@ -516,8 +516,11 @@ int vfio_group_use_container(struct vfio_group *group)
 {
 	lockdep_assert_held(&group->group_lock);
 
-	if (!group->container || !group->container->iommu_driver ||
-	    WARN_ON(!group->container_users))
+	/*
+	 * The container fd has been assigned with VFIO_GROUP_SET_CONTAINER but
+	 * VFIO_SET_IOMMU hasn't been done yet.
+	 */
+	if (!group->container->iommu_driver)
 		return -EINVAL;
 
 	if (group->type == VFIO_NO_IOMMU && !capable(CAP_SYS_RAWIO))
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index 247590334e14b0..985e13d52989ca 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -10,6 +10,7 @@
 #include <linux/cdev.h>
 #include <linux/module.h>
 
+struct iommufd_ctx;
 struct iommu_group;
 struct vfio_device;
 struct vfio_container;
@@ -60,6 +61,7 @@ struct vfio_group {
 	struct kvm			*kvm;
 	struct file			*opened_file;
 	struct blocking_notifier_head	notifier;
+	struct iommufd_ctx		*iommufd;
 };
 
 /* events for the backend driver notify callback */
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index a8d1fbfcc3ddad..cf0ea744de931e 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -35,6 +35,7 @@
 #include <linux/pm_runtime.h>
 #include <linux/interval_tree.h>
 #include <linux/iova_bitmap.h>
+#include <linux/iommufd.h>
 #include "vfio.h"
 
 #define DRIVER_VERSION	"0.3"
@@ -665,6 +666,16 @@ EXPORT_SYMBOL_GPL(vfio_unregister_group_dev);
 /*
  * VFIO Group fd, /dev/vfio/$GROUP
  */
+static bool vfio_group_has_iommu(struct vfio_group *group)
+{
+	lockdep_assert_held(&group->group_lock);
+	if (!group->container)
+		WARN_ON(group->container_users);
+	else
+		WARN_ON(!group->container_users);
+	return group->container || group->iommufd;
+}
+
 /*
  * VFIO_GROUP_UNSET_CONTAINER should fail if there are other users or
  * if there was no container to unset.  Since the ioctl is called on
@@ -676,15 +687,21 @@ static int vfio_group_ioctl_unset_container(struct vfio_group *group)
 	int ret = 0;
 
 	mutex_lock(&group->group_lock);
-	if (!group->container) {
+	if (!vfio_group_has_iommu(group)) {
 		ret = -EINVAL;
 		goto out_unlock;
 	}
-	if (group->container_users != 1) {
-		ret = -EBUSY;
-		goto out_unlock;
+	if (group->container) {
+		if (group->container_users != 1) {
+			ret = -EBUSY;
+			goto out_unlock;
+		}
+		vfio_group_detach_container(group);
+	}
+	if (group->iommufd) {
+		iommufd_ctx_put(group->iommufd);
+		group->iommufd = NULL;
 	}
-	vfio_group_detach_container(group);
 
 out_unlock:
 	mutex_unlock(&group->group_lock);
@@ -695,6 +712,7 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
 					  int __user *arg)
 {
 	struct vfio_container *container;
+	struct iommufd_ctx *iommufd;
 	struct fd f;
 	int ret;
 	int fd;
@@ -707,7 +725,7 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
 		return -EBADF;
 
 	mutex_lock(&group->group_lock);
-	if (group->container || WARN_ON(group->container_users)) {
+	if (vfio_group_has_iommu(group)) {
 		ret = -EINVAL;
 		goto out_unlock;
 	}
@@ -717,12 +735,23 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
 	}
 
 	container = vfio_container_from_file(f.file);
-	ret = -EINVAL;
 	if (container) {
 		ret = vfio_container_attach_group(container, group);
 		goto out_unlock;
 	}
 
+	iommufd = iommufd_ctx_from_file(f.file);
+	if (!IS_ERR(iommufd)) {
+		u32 ioas_id;
+
+		group->iommufd = iommufd;
+		ret = iommufd_vfio_compat_ioas_id(iommufd, &ioas_id);
+		goto out_unlock;
+	}
+
+	/* The FD passed is not recognized. */
+	ret = -EBADF;
+
 out_unlock:
 	mutex_unlock(&group->group_lock);
 	fdput(f);
@@ -752,9 +781,16 @@ static int vfio_device_first_open(struct vfio_device *device)
 	 * it during close_device.
 	 */
 	mutex_lock(&device->group->group_lock);
-	ret = vfio_group_use_container(device->group);
-	if (ret)
+	if (!vfio_group_has_iommu(device->group)) {
+		ret = -EINVAL;
 		goto err_module_put;
+	}
+
+	if (device->group->container) {
+		ret = vfio_group_use_container(device->group);
+		if (ret)
+			goto err_module_put;
+	}
 
 	device->kvm = device->group->kvm;
 	if (device->ops->open_device) {
@@ -762,14 +798,16 @@ static int vfio_device_first_open(struct vfio_device *device)
 		if (ret)
 			goto err_container;
 	}
-	vfio_device_container_register(device);
+	if (device->group->container)
+		vfio_device_container_register(device);
 	mutex_unlock(&device->group->group_lock);
 	return 0;
 
 err_container:
-	vfio_group_unuse_container(device->group);
-err_module_put:
+	if (device->group->container)
+		vfio_group_unuse_container(device->group);
 	device->kvm = NULL;
+err_module_put:
 	mutex_unlock(&device->group->group_lock);
 	module_put(device->dev->driver->owner);
 	return ret;
@@ -780,11 +818,13 @@ static void vfio_device_last_close(struct vfio_device *device)
 	lockdep_assert_held(&device->dev_set->lock);
 
 	mutex_lock(&device->group->group_lock);
-	vfio_device_container_unregister(device);
+	if (device->group->container)
+		vfio_device_container_unregister(device);
 	if (device->ops->close_device)
 		device->ops->close_device(device);
 	device->kvm = NULL;
-	vfio_group_unuse_container(device->group);
+	if (device->group->container)
+		vfio_group_unuse_container(device->group);
 	mutex_unlock(&device->group->group_lock);
 	module_put(device->dev->driver->owner);
 }
@@ -900,7 +940,7 @@ static int vfio_group_ioctl_get_status(struct vfio_group *group,
 		return -ENODEV;
 	}
 
-	if (group->container)
+	if (group->container || group->iommufd)
 		status.flags |= VFIO_GROUP_FLAGS_CONTAINER_SET |
 				VFIO_GROUP_FLAGS_VIABLE;
 	else if (!iommu_group_dma_owner_claimed(group->iommu_group))
@@ -983,6 +1023,10 @@ static int vfio_group_fops_release(struct inode *inode, struct file *filep)
 	WARN_ON(group->notifier.head);
 	if (group->container)
 		vfio_group_detach_container(group);
+	if (group->iommufd) {
+		iommufd_ctx_put(group->iommufd);
+		group->iommufd = NULL;
+	}
 	group->opened_file = NULL;
 	mutex_unlock(&group->group_lock);
 	return 0;
@@ -1879,6 +1923,8 @@ static void __exit vfio_cleanup(void)
 module_init(vfio_init);
 module_exit(vfio_cleanup);
 
+MODULE_IMPORT_NS(IOMMUFD);
+MODULE_IMPORT_NS(IOMMUFD_VFIO);
 MODULE_VERSION(DRIVER_VERSION);
 MODULE_LICENSE("GPL v2");
 MODULE_AUTHOR(DRIVER_AUTHOR);
-- 
2.38.0

[PATCH 07/10] vfio-iommufd: Support iommufd for physical VFIO devices

[Jason Gunthorpe]

This creates the iommufd_device for the physical VFIO drivers. These are
all the drivers that are calling vfio_register_group_dev() and expect the
type1 code to setup a real iommu_domain against their parent struct
device.

The design gives the driver a choice in how it gets connected to iommufd
by providing bind_iommufd/unbind_iommufd/attach_ioas callbacks to
implement as required. The core code provides three default callbacks for
physical mode using a real iommu_domain. This is suitable for drivers
using vfio_register_group_dev()

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/Makefile                         |   1 +
 drivers/vfio/fsl-mc/vfio_fsl_mc.c             |   3 +
 drivers/vfio/iommufd.c                        | 104 ++++++++++++++++++
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    |   6 +
 drivers/vfio/pci/mlx5/main.c                  |   3 +
 drivers/vfio/pci/vfio_pci.c                   |   3 +
 drivers/vfio/platform/vfio_amba.c             |   3 +
 drivers/vfio/platform/vfio_platform.c         |   3 +
 drivers/vfio/vfio.h                           |  15 +++
 drivers/vfio/vfio_main.c                      |  13 ++-
 include/linux/vfio.h                          |  25 +++++
 11 files changed, 177 insertions(+), 2 deletions(-)
 create mode 100644 drivers/vfio/iommufd.c

diff --git a/drivers/vfio/Makefile b/drivers/vfio/Makefile
index b693a1169286f8..3863922529ef20 100644
--- a/drivers/vfio/Makefile
+++ b/drivers/vfio/Makefile
@@ -6,6 +6,7 @@ obj-$(CONFIG_VFIO) += vfio.o
 vfio-y += vfio_main.o \
 	  iova_bitmap.o \
 	  container.o
+vfio-$(CONFIG_IOMMUFD) += iommufd.o
 
 obj-$(CONFIG_VFIO_VIRQFD) += vfio_virqfd.o
 obj-$(CONFIG_VFIO_IOMMU_TYPE1) += vfio_iommu_type1.o
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index b16874e913e4f5..5cd4bb47644039 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -592,6 +592,9 @@ static const struct vfio_device_ops vfio_fsl_mc_ops = {
 	.read		= vfio_fsl_mc_read,
 	.write		= vfio_fsl_mc_write,
 	.mmap		= vfio_fsl_mc_mmap,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
 };
 
 static struct fsl_mc_driver vfio_fsl_mc_driver = {
diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
new file mode 100644
index 00000000000000..8280bb32ee677c
--- /dev/null
+++ b/drivers/vfio/iommufd.c
@@ -0,0 +1,104 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2021-2022, NVIDIA CORPORATION & AFFILIATES
+ */
+#include <linux/vfio.h>
+#include <linux/iommufd.h>
+
+#include "vfio.h"
+
+MODULE_IMPORT_NS(IOMMUFD);
+MODULE_IMPORT_NS(IOMMUFD_VFIO);
+
+int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx)
+{
+	u32 ioas_id;
+	u32 device_id;
+	int ret;
+
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	/*
+	 * If the driver doesn't provide this op then it means the device does
+	 * not do DMA at all. So nothing to do.
+	 */
+	if (!vdev->ops->bind_iommufd)
+		return 0;
+
+	ret = vdev->ops->bind_iommufd(vdev, ictx, &device_id);
+	if (ret)
+		return ret;
+
+	if (vdev->ops->attach_ioas) {
+		ret = iommufd_vfio_compat_ioas_id(ictx, &ioas_id);
+		if (ret)
+			goto err_unbind;
+		ret = vdev->ops->attach_ioas(vdev, &ioas_id);
+		if (ret)
+			goto err_unbind;
+		vdev->iommufd_attached = true;
+	}
+
+	/*
+	 * The legacy path has no way to return the device id or the selected
+	 * pt_id
+	 */
+	return 0;
+
+err_unbind:
+	if (vdev->ops->unbind_iommufd)
+		vdev->ops->unbind_iommufd(vdev);
+	return ret;
+}
+
+void vfio_iommufd_unbind(struct vfio_device *vdev)
+{
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	if (!vdev->iommufd_device)
+		return;
+
+	if (vdev->ops->unbind_iommufd)
+		vdev->ops->unbind_iommufd(vdev);
+}
+
+/*
+ * The physical standard ops mean that the iommufd_device is bound to the
+ * physical device vdev->dev that was provided to vfio_init_group_dev(). Drivers
+ * using this ops set should call vfio_register_group_dev()
+ */
+int vfio_iommufd_physical_bind(struct vfio_device *vdev,
+			       struct iommufd_ctx *ictx, u32 *out_device_id)
+{
+	struct iommufd_device *idev;
+
+	idev = iommufd_device_bind(ictx, vdev->dev, out_device_id);
+	if (IS_ERR(idev))
+		return PTR_ERR(idev);
+	vdev->iommufd_device = idev;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_physical_bind);
+
+void vfio_iommufd_physical_unbind(struct vfio_device *vdev)
+{
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	if (vdev->iommufd_attached) {
+		iommufd_device_detach(vdev->iommufd_device);
+		vdev->iommufd_attached = false;
+	}
+	iommufd_device_unbind(vdev->iommufd_device);
+	vdev->iommufd_device = NULL;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_physical_unbind);
+
+int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id)
+{
+	unsigned int flags = 0;
+
+	if (vfio_allow_unsafe_interrupts)
+		flags |= IOMMUFD_ATTACH_FLAGS_ALLOW_UNSAFE_INTERRUPT;
+	return iommufd_device_attach(vdev->iommufd_device, pt_id, flags);
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_physical_attach_ioas);
diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
index 39eeca18a0f7c8..40019b11c5a969 100644
--- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
@@ -1246,6 +1246,9 @@ static const struct vfio_device_ops hisi_acc_vfio_pci_migrn_ops = {
 	.mmap = hisi_acc_vfio_pci_mmap,
 	.request = vfio_pci_core_request,
 	.match = vfio_pci_core_match,
+	.bind_iommufd = vfio_iommufd_physical_bind,
+	.unbind_iommufd = vfio_iommufd_physical_unbind,
+	.attach_ioas = vfio_iommufd_physical_attach_ioas,
 };
 
 static const struct vfio_device_ops hisi_acc_vfio_pci_ops = {
@@ -1261,6 +1264,9 @@ static const struct vfio_device_ops hisi_acc_vfio_pci_ops = {
 	.mmap = vfio_pci_core_mmap,
 	.request = vfio_pci_core_request,
 	.match = vfio_pci_core_match,
+	.bind_iommufd = vfio_iommufd_physical_bind,
+	.unbind_iommufd = vfio_iommufd_physical_unbind,
+	.attach_ioas = vfio_iommufd_physical_attach_ioas,
 };
 
 static int hisi_acc_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c
index fd6ccb8454a24a..32d1f38d351e7e 100644
--- a/drivers/vfio/pci/mlx5/main.c
+++ b/drivers/vfio/pci/mlx5/main.c
@@ -623,6 +623,9 @@ static const struct vfio_device_ops mlx5vf_pci_ops = {
 	.mmap = vfio_pci_core_mmap,
 	.request = vfio_pci_core_request,
 	.match = vfio_pci_core_match,
+	.bind_iommufd = vfio_iommufd_physical_bind,
+	.unbind_iommufd = vfio_iommufd_physical_unbind,
+	.attach_ioas = vfio_iommufd_physical_attach_ioas,
 };
 
 static int mlx5vf_pci_probe(struct pci_dev *pdev,
diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
index 1d4919edfbde48..29091ee2e9849b 100644
--- a/drivers/vfio/pci/vfio_pci.c
+++ b/drivers/vfio/pci/vfio_pci.c
@@ -138,6 +138,9 @@ static const struct vfio_device_ops vfio_pci_ops = {
 	.mmap		= vfio_pci_core_mmap,
 	.request	= vfio_pci_core_request,
 	.match		= vfio_pci_core_match,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
 };
 
 static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
diff --git a/drivers/vfio/platform/vfio_amba.c b/drivers/vfio/platform/vfio_amba.c
index eaea63e5294c58..5a046098d0bdf4 100644
--- a/drivers/vfio/platform/vfio_amba.c
+++ b/drivers/vfio/platform/vfio_amba.c
@@ -117,6 +117,9 @@ static const struct vfio_device_ops vfio_amba_ops = {
 	.read		= vfio_platform_read,
 	.write		= vfio_platform_write,
 	.mmap		= vfio_platform_mmap,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
 };
 
 static const struct amba_id pl330_ids[] = {
diff --git a/drivers/vfio/platform/vfio_platform.c b/drivers/vfio/platform/vfio_platform.c
index 82cedcebfd9022..b87c3b70878341 100644
--- a/drivers/vfio/platform/vfio_platform.c
+++ b/drivers/vfio/platform/vfio_platform.c
@@ -106,6 +106,9 @@ static const struct vfio_device_ops vfio_platform_ops = {
 	.read		= vfio_platform_read,
 	.write		= vfio_platform_write,
 	.mmap		= vfio_platform_mmap,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
 };
 
 static struct platform_driver vfio_platform_driver = {
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index 985e13d52989ca..809f2e8523968e 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -124,6 +124,21 @@ void vfio_device_container_unregister(struct vfio_device *device);
 int __init vfio_container_init(void);
 void vfio_container_cleanup(void);
 
+#if IS_ENABLED(CONFIG_IOMMUFD)
+int vfio_iommufd_bind(struct vfio_device *device, struct iommufd_ctx *ictx);
+void vfio_iommufd_unbind(struct vfio_device *device);
+#else
+static inline int vfio_iommufd_bind(struct vfio_device *device,
+				    struct iommufd_ctx *ictx)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void vfio_iommufd_unbind(struct vfio_device *device)
+{
+}
+#endif
+
 #ifdef CONFIG_VFIO_NOIOMMU
 extern bool vfio_noiommu __read_mostly;
 #else
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index cf0ea744de931e..bfbda04af1ffda 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -528,6 +528,11 @@ static int __vfio_register_dev(struct vfio_device *device,
 	if (IS_ERR(group))
 		return PTR_ERR(group);
 
+	if (WARN_ON(device->ops->bind_iommufd &&
+		    (!device->ops->unbind_iommufd ||
+		     !device->ops->attach_ioas)))
+		return -EINVAL;
+
 	/*
 	 * If the driver doesn't specify a set then the device is added to a
 	 * singleton set just for itself.
@@ -790,6 +795,10 @@ static int vfio_device_first_open(struct vfio_device *device)
 		ret = vfio_group_use_container(device->group);
 		if (ret)
 			goto err_module_put;
+	} else if (device->group->iommufd) {
+		ret = vfio_iommufd_bind(device, device->group->iommufd);
+		if (ret)
+			goto err_module_put;
 	}
 
 	device->kvm = device->group->kvm;
@@ -806,6 +815,7 @@ static int vfio_device_first_open(struct vfio_device *device)
 err_container:
 	if (device->group->container)
 		vfio_group_unuse_container(device->group);
+	vfio_iommufd_unbind(device);
 	device->kvm = NULL;
 err_module_put:
 	mutex_unlock(&device->group->group_lock);
@@ -825,6 +835,7 @@ static void vfio_device_last_close(struct vfio_device *device)
 	device->kvm = NULL;
 	if (device->group->container)
 		vfio_group_unuse_container(device->group);
+	vfio_iommufd_unbind(device);
 	mutex_unlock(&device->group->group_lock);
 	module_put(device->dev->driver->owner);
 }
@@ -1923,8 +1934,6 @@ static void __exit vfio_cleanup(void)
 module_init(vfio_init);
 module_exit(vfio_cleanup);
 
-MODULE_IMPORT_NS(IOMMUFD);
-MODULE_IMPORT_NS(IOMMUFD_VFIO);
 MODULE_VERSION(DRIVER_VERSION);
 MODULE_LICENSE("GPL v2");
 MODULE_AUTHOR(DRIVER_AUTHOR);
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index e7cebeb875dd1a..a7fc4d747dc226 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -17,6 +17,8 @@
 #include <linux/iova_bitmap.h>
 
 struct kvm;
+struct iommufd_ctx;
+struct iommufd_device;
 
 /*
  * VFIO devices can be placed in a set, this allows all devices to share this
@@ -54,6 +56,10 @@ struct vfio_device {
 	struct completion comp;
 	struct list_head group_next;
 	struct list_head iommu_entry;
+#if IS_ENABLED(CONFIG_IOMMUFD)
+	struct iommufd_device *iommufd_device;
+	bool iommufd_attached;
+#endif
 };
 
 /**
@@ -80,6 +86,10 @@ struct vfio_device_ops {
 	char	*name;
 	int	(*init)(struct vfio_device *vdev);
 	void	(*release)(struct vfio_device *vdev);
+	int	(*bind_iommufd)(struct vfio_device *vdev,
+				struct iommufd_ctx *ictx, u32 *out_device_id);
+	void	(*unbind_iommufd)(struct vfio_device *vdev);
+	int	(*attach_ioas)(struct vfio_device *vdev, u32 *pt_id);
 	int	(*open_device)(struct vfio_device *vdev);
 	void	(*close_device)(struct vfio_device *vdev);
 	ssize_t	(*read)(struct vfio_device *vdev, char __user *buf,
@@ -96,6 +106,21 @@ struct vfio_device_ops {
 				  void __user *arg, size_t argsz);
 };
 
+#if IS_ENABLED(CONFIG_IOMMUFD)
+int vfio_iommufd_physical_bind(struct vfio_device *vdev,
+			       struct iommufd_ctx *ictx, u32 *out_device_id);
+void vfio_iommufd_physical_unbind(struct vfio_device *vdev);
+int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
+#else
+#define vfio_iommufd_physical_bind                                      \
+	((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx,   \
+		  u32 *out_device_id)) NULL)
+#define vfio_iommufd_physical_unbind \
+	((void (*)(struct vfio_device *vdev)) NULL)
+#define vfio_iommufd_physical_attach_ioas \
+	((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL)
+#endif
+
 /**
  * @migration_set_state: Optional callback to change the migration state for
  *         devices that support migration. It's mandatory for
-- 
2.38.0

[PATCH 07/10] vfio-iommufd: Support iommufd for physical VFIO devices

[Jason Gunthorpe]

This creates the iommufd_device for the physical VFIO drivers. These are
all the drivers that are calling vfio_register_group_dev() and expect the
type1 code to setup a real iommu_domain against their parent struct
device.

The design gives the driver a choice in how it gets connected to iommufd
by providing bind_iommufd/unbind_iommufd/attach_ioas callbacks to
implement as required. The core code provides three default callbacks for
physical mode using a real iommu_domain. This is suitable for drivers
using vfio_register_group_dev()

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/Makefile                         |   1 +
 drivers/vfio/fsl-mc/vfio_fsl_mc.c             |   3 +
 drivers/vfio/iommufd.c                        | 104 ++++++++++++++++++
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    |   6 +
 drivers/vfio/pci/mlx5/main.c                  |   3 +
 drivers/vfio/pci/vfio_pci.c                   |   3 +
 drivers/vfio/platform/vfio_amba.c             |   3 +
 drivers/vfio/platform/vfio_platform.c         |   3 +
 drivers/vfio/vfio.h                           |  15 +++
 drivers/vfio/vfio_main.c                      |  13 ++-
 include/linux/vfio.h                          |  25 +++++
 11 files changed, 177 insertions(+), 2 deletions(-)
 create mode 100644 drivers/vfio/iommufd.c

diff --git a/drivers/vfio/Makefile b/drivers/vfio/Makefile
index b693a1169286f8..3863922529ef20 100644
--- a/drivers/vfio/Makefile
+++ b/drivers/vfio/Makefile
@@ -6,6 +6,7 @@ obj-$(CONFIG_VFIO) += vfio.o
 vfio-y += vfio_main.o \
 	  iova_bitmap.o \
 	  container.o
+vfio-$(CONFIG_IOMMUFD) += iommufd.o
 
 obj-$(CONFIG_VFIO_VIRQFD) += vfio_virqfd.o
 obj-$(CONFIG_VFIO_IOMMU_TYPE1) += vfio_iommu_type1.o
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index b16874e913e4f5..5cd4bb47644039 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -592,6 +592,9 @@ static const struct vfio_device_ops vfio_fsl_mc_ops = {
 	.read		= vfio_fsl_mc_read,
 	.write		= vfio_fsl_mc_write,
 	.mmap		= vfio_fsl_mc_mmap,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
 };
 
 static struct fsl_mc_driver vfio_fsl_mc_driver = {
diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
new file mode 100644
index 00000000000000..8280bb32ee677c
--- /dev/null
+++ b/drivers/vfio/iommufd.c
@@ -0,0 +1,104 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2021-2022, NVIDIA CORPORATION & AFFILIATES
+ */
+#include <linux/vfio.h>
+#include <linux/iommufd.h>
+
+#include "vfio.h"
+
+MODULE_IMPORT_NS(IOMMUFD);
+MODULE_IMPORT_NS(IOMMUFD_VFIO);
+
+int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx)
+{
+	u32 ioas_id;
+	u32 device_id;
+	int ret;
+
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	/*
+	 * If the driver doesn't provide this op then it means the device does
+	 * not do DMA at all. So nothing to do.
+	 */
+	if (!vdev->ops->bind_iommufd)
+		return 0;
+
+	ret = vdev->ops->bind_iommufd(vdev, ictx, &device_id);
+	if (ret)
+		return ret;
+
+	if (vdev->ops->attach_ioas) {
+		ret = iommufd_vfio_compat_ioas_id(ictx, &ioas_id);
+		if (ret)
+			goto err_unbind;
+		ret = vdev->ops->attach_ioas(vdev, &ioas_id);
+		if (ret)
+			goto err_unbind;
+		vdev->iommufd_attached = true;
+	}
+
+	/*
+	 * The legacy path has no way to return the device id or the selected
+	 * pt_id
+	 */
+	return 0;
+
+err_unbind:
+	if (vdev->ops->unbind_iommufd)
+		vdev->ops->unbind_iommufd(vdev);
+	return ret;
+}
+
+void vfio_iommufd_unbind(struct vfio_device *vdev)
+{
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	if (!vdev->iommufd_device)
+		return;
+
+	if (vdev->ops->unbind_iommufd)
+		vdev->ops->unbind_iommufd(vdev);
+}
+
+/*
+ * The physical standard ops mean that the iommufd_device is bound to the
+ * physical device vdev->dev that was provided to vfio_init_group_dev(). Drivers
+ * using this ops set should call vfio_register_group_dev()
+ */
+int vfio_iommufd_physical_bind(struct vfio_device *vdev,
+			       struct iommufd_ctx *ictx, u32 *out_device_id)
+{
+	struct iommufd_device *idev;
+
+	idev = iommufd_device_bind(ictx, vdev->dev, out_device_id);
+	if (IS_ERR(idev))
+		return PTR_ERR(idev);
+	vdev->iommufd_device = idev;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_physical_bind);
+
+void vfio_iommufd_physical_unbind(struct vfio_device *vdev)
+{
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	if (vdev->iommufd_attached) {
+		iommufd_device_detach(vdev->iommufd_device);
+		vdev->iommufd_attached = false;
+	}
+	iommufd_device_unbind(vdev->iommufd_device);
+	vdev->iommufd_device = NULL;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_physical_unbind);
+
+int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id)
+{
+	unsigned int flags = 0;
+
+	if (vfio_allow_unsafe_interrupts)
+		flags |= IOMMUFD_ATTACH_FLAGS_ALLOW_UNSAFE_INTERRUPT;
+	return iommufd_device_attach(vdev->iommufd_device, pt_id, flags);
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_physical_attach_ioas);
diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
index 39eeca18a0f7c8..40019b11c5a969 100644
--- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
@@ -1246,6 +1246,9 @@ static const struct vfio_device_ops hisi_acc_vfio_pci_migrn_ops = {
 	.mmap = hisi_acc_vfio_pci_mmap,
 	.request = vfio_pci_core_request,
 	.match = vfio_pci_core_match,
+	.bind_iommufd = vfio_iommufd_physical_bind,
+	.unbind_iommufd = vfio_iommufd_physical_unbind,
+	.attach_ioas = vfio_iommufd_physical_attach_ioas,
 };
 
 static const struct vfio_device_ops hisi_acc_vfio_pci_ops = {
@@ -1261,6 +1264,9 @@ static const struct vfio_device_ops hisi_acc_vfio_pci_ops = {
 	.mmap = vfio_pci_core_mmap,
 	.request = vfio_pci_core_request,
 	.match = vfio_pci_core_match,
+	.bind_iommufd = vfio_iommufd_physical_bind,
+	.unbind_iommufd = vfio_iommufd_physical_unbind,
+	.attach_ioas = vfio_iommufd_physical_attach_ioas,
 };
 
 static int hisi_acc_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c
index fd6ccb8454a24a..32d1f38d351e7e 100644
--- a/drivers/vfio/pci/mlx5/main.c
+++ b/drivers/vfio/pci/mlx5/main.c
@@ -623,6 +623,9 @@ static const struct vfio_device_ops mlx5vf_pci_ops = {
 	.mmap = vfio_pci_core_mmap,
 	.request = vfio_pci_core_request,
 	.match = vfio_pci_core_match,
+	.bind_iommufd = vfio_iommufd_physical_bind,
+	.unbind_iommufd = vfio_iommufd_physical_unbind,
+	.attach_ioas = vfio_iommufd_physical_attach_ioas,
 };
 
 static int mlx5vf_pci_probe(struct pci_dev *pdev,
diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
index 1d4919edfbde48..29091ee2e9849b 100644
--- a/drivers/vfio/pci/vfio_pci.c
+++ b/drivers/vfio/pci/vfio_pci.c
@@ -138,6 +138,9 @@ static const struct vfio_device_ops vfio_pci_ops = {
 	.mmap		= vfio_pci_core_mmap,
 	.request	= vfio_pci_core_request,
 	.match		= vfio_pci_core_match,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
 };
 
 static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
diff --git a/drivers/vfio/platform/vfio_amba.c b/drivers/vfio/platform/vfio_amba.c
index eaea63e5294c58..5a046098d0bdf4 100644
--- a/drivers/vfio/platform/vfio_amba.c
+++ b/drivers/vfio/platform/vfio_amba.c
@@ -117,6 +117,9 @@ static const struct vfio_device_ops vfio_amba_ops = {
 	.read		= vfio_platform_read,
 	.write		= vfio_platform_write,
 	.mmap		= vfio_platform_mmap,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
 };
 
 static const struct amba_id pl330_ids[] = {
diff --git a/drivers/vfio/platform/vfio_platform.c b/drivers/vfio/platform/vfio_platform.c
index 82cedcebfd9022..b87c3b70878341 100644
--- a/drivers/vfio/platform/vfio_platform.c
+++ b/drivers/vfio/platform/vfio_platform.c
@@ -106,6 +106,9 @@ static const struct vfio_device_ops vfio_platform_ops = {
 	.read		= vfio_platform_read,
 	.write		= vfio_platform_write,
 	.mmap		= vfio_platform_mmap,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
 };
 
 static struct platform_driver vfio_platform_driver = {
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index 985e13d52989ca..809f2e8523968e 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -124,6 +124,21 @@ void vfio_device_container_unregister(struct vfio_device *device);
 int __init vfio_container_init(void);
 void vfio_container_cleanup(void);
 
+#if IS_ENABLED(CONFIG_IOMMUFD)
+int vfio_iommufd_bind(struct vfio_device *device, struct iommufd_ctx *ictx);
+void vfio_iommufd_unbind(struct vfio_device *device);
+#else
+static inline int vfio_iommufd_bind(struct vfio_device *device,
+				    struct iommufd_ctx *ictx)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void vfio_iommufd_unbind(struct vfio_device *device)
+{
+}
+#endif
+
 #ifdef CONFIG_VFIO_NOIOMMU
 extern bool vfio_noiommu __read_mostly;
 #else
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index cf0ea744de931e..bfbda04af1ffda 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -528,6 +528,11 @@ static int __vfio_register_dev(struct vfio_device *device,
 	if (IS_ERR(group))
 		return PTR_ERR(group);
 
+	if (WARN_ON(device->ops->bind_iommufd &&
+		    (!device->ops->unbind_iommufd ||
+		     !device->ops->attach_ioas)))
+		return -EINVAL;
+
 	/*
 	 * If the driver doesn't specify a set then the device is added to a
 	 * singleton set just for itself.
@@ -790,6 +795,10 @@ static int vfio_device_first_open(struct vfio_device *device)
 		ret = vfio_group_use_container(device->group);
 		if (ret)
 			goto err_module_put;
+	} else if (device->group->iommufd) {
+		ret = vfio_iommufd_bind(device, device->group->iommufd);
+		if (ret)
+			goto err_module_put;
 	}
 
 	device->kvm = device->group->kvm;
@@ -806,6 +815,7 @@ static int vfio_device_first_open(struct vfio_device *device)
 err_container:
 	if (device->group->container)
 		vfio_group_unuse_container(device->group);
+	vfio_iommufd_unbind(device);
 	device->kvm = NULL;
 err_module_put:
 	mutex_unlock(&device->group->group_lock);
@@ -825,6 +835,7 @@ static void vfio_device_last_close(struct vfio_device *device)
 	device->kvm = NULL;
 	if (device->group->container)
 		vfio_group_unuse_container(device->group);
+	vfio_iommufd_unbind(device);
 	mutex_unlock(&device->group->group_lock);
 	module_put(device->dev->driver->owner);
 }
@@ -1923,8 +1934,6 @@ static void __exit vfio_cleanup(void)
 module_init(vfio_init);
 module_exit(vfio_cleanup);
 
-MODULE_IMPORT_NS(IOMMUFD);
-MODULE_IMPORT_NS(IOMMUFD_VFIO);
 MODULE_VERSION(DRIVER_VERSION);
 MODULE_LICENSE("GPL v2");
 MODULE_AUTHOR(DRIVER_AUTHOR);
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index e7cebeb875dd1a..a7fc4d747dc226 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -17,6 +17,8 @@
 #include <linux/iova_bitmap.h>
 
 struct kvm;
+struct iommufd_ctx;
+struct iommufd_device;
 
 /*
  * VFIO devices can be placed in a set, this allows all devices to share this
@@ -54,6 +56,10 @@ struct vfio_device {
 	struct completion comp;
 	struct list_head group_next;
 	struct list_head iommu_entry;
+#if IS_ENABLED(CONFIG_IOMMUFD)
+	struct iommufd_device *iommufd_device;
+	bool iommufd_attached;
+#endif
 };
 
 /**
@@ -80,6 +86,10 @@ struct vfio_device_ops {
 	char	*name;
 	int	(*init)(struct vfio_device *vdev);
 	void	(*release)(struct vfio_device *vdev);
+	int	(*bind_iommufd)(struct vfio_device *vdev,
+				struct iommufd_ctx *ictx, u32 *out_device_id);
+	void	(*unbind_iommufd)(struct vfio_device *vdev);
+	int	(*attach_ioas)(struct vfio_device *vdev, u32 *pt_id);
 	int	(*open_device)(struct vfio_device *vdev);
 	void	(*close_device)(struct vfio_device *vdev);
 	ssize_t	(*read)(struct vfio_device *vdev, char __user *buf,
@@ -96,6 +106,21 @@ struct vfio_device_ops {
 				  void __user *arg, size_t argsz);
 };
 
+#if IS_ENABLED(CONFIG_IOMMUFD)
+int vfio_iommufd_physical_bind(struct vfio_device *vdev,
+			       struct iommufd_ctx *ictx, u32 *out_device_id);
+void vfio_iommufd_physical_unbind(struct vfio_device *vdev);
+int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
+#else
+#define vfio_iommufd_physical_bind                                      \
+	((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx,   \
+		  u32 *out_device_id)) NULL)
+#define vfio_iommufd_physical_unbind \
+	((void (*)(struct vfio_device *vdev)) NULL)
+#define vfio_iommufd_physical_attach_ioas \
+	((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL)
+#endif
+
 /**
  * @migration_set_state: Optional callback to change the migration state for
  *         devices that support migration. It's mandatory for
-- 
2.38.0

[Intel-gfx] [PATCH 07/10] vfio-iommufd: Support iommufd for physical VFIO devices

[Jason Gunthorpe]

This creates the iommufd_device for the physical VFIO drivers. These are
all the drivers that are calling vfio_register_group_dev() and expect the
type1 code to setup a real iommu_domain against their parent struct
device.

The design gives the driver a choice in how it gets connected to iommufd
by providing bind_iommufd/unbind_iommufd/attach_ioas callbacks to
implement as required. The core code provides three default callbacks for
physical mode using a real iommu_domain. This is suitable for drivers
using vfio_register_group_dev()

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/Makefile                         |   1 +
 drivers/vfio/fsl-mc/vfio_fsl_mc.c             |   3 +
 drivers/vfio/iommufd.c                        | 104 ++++++++++++++++++
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    |   6 +
 drivers/vfio/pci/mlx5/main.c                  |   3 +
 drivers/vfio/pci/vfio_pci.c                   |   3 +
 drivers/vfio/platform/vfio_amba.c             |   3 +
 drivers/vfio/platform/vfio_platform.c         |   3 +
 drivers/vfio/vfio.h                           |  15 +++
 drivers/vfio/vfio_main.c                      |  13 ++-
 include/linux/vfio.h                          |  25 +++++
 11 files changed, 177 insertions(+), 2 deletions(-)
 create mode 100644 drivers/vfio/iommufd.c

diff --git a/drivers/vfio/Makefile b/drivers/vfio/Makefile
index b693a1169286f8..3863922529ef20 100644
--- a/drivers/vfio/Makefile
+++ b/drivers/vfio/Makefile
@@ -6,6 +6,7 @@ obj-$(CONFIG_VFIO) += vfio.o
 vfio-y += vfio_main.o \
 	  iova_bitmap.o \
 	  container.o
+vfio-$(CONFIG_IOMMUFD) += iommufd.o
 
 obj-$(CONFIG_VFIO_VIRQFD) += vfio_virqfd.o
 obj-$(CONFIG_VFIO_IOMMU_TYPE1) += vfio_iommu_type1.o
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index b16874e913e4f5..5cd4bb47644039 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -592,6 +592,9 @@ static const struct vfio_device_ops vfio_fsl_mc_ops = {
 	.read		= vfio_fsl_mc_read,
 	.write		= vfio_fsl_mc_write,
 	.mmap		= vfio_fsl_mc_mmap,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
 };
 
 static struct fsl_mc_driver vfio_fsl_mc_driver = {
diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
new file mode 100644
index 00000000000000..8280bb32ee677c
--- /dev/null
+++ b/drivers/vfio/iommufd.c
@@ -0,0 +1,104 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2021-2022, NVIDIA CORPORATION & AFFILIATES
+ */
+#include <linux/vfio.h>
+#include <linux/iommufd.h>
+
+#include "vfio.h"
+
+MODULE_IMPORT_NS(IOMMUFD);
+MODULE_IMPORT_NS(IOMMUFD_VFIO);
+
+int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx)
+{
+	u32 ioas_id;
+	u32 device_id;
+	int ret;
+
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	/*
+	 * If the driver doesn't provide this op then it means the device does
+	 * not do DMA at all. So nothing to do.
+	 */
+	if (!vdev->ops->bind_iommufd)
+		return 0;
+
+	ret = vdev->ops->bind_iommufd(vdev, ictx, &device_id);
+	if (ret)
+		return ret;
+
+	if (vdev->ops->attach_ioas) {
+		ret = iommufd_vfio_compat_ioas_id(ictx, &ioas_id);
+		if (ret)
+			goto err_unbind;
+		ret = vdev->ops->attach_ioas(vdev, &ioas_id);
+		if (ret)
+			goto err_unbind;
+		vdev->iommufd_attached = true;
+	}
+
+	/*
+	 * The legacy path has no way to return the device id or the selected
+	 * pt_id
+	 */
+	return 0;
+
+err_unbind:
+	if (vdev->ops->unbind_iommufd)
+		vdev->ops->unbind_iommufd(vdev);
+	return ret;
+}
+
+void vfio_iommufd_unbind(struct vfio_device *vdev)
+{
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	if (!vdev->iommufd_device)
+		return;
+
+	if (vdev->ops->unbind_iommufd)
+		vdev->ops->unbind_iommufd(vdev);
+}
+
+/*
+ * The physical standard ops mean that the iommufd_device is bound to the
+ * physical device vdev->dev that was provided to vfio_init_group_dev(). Drivers
+ * using this ops set should call vfio_register_group_dev()
+ */
+int vfio_iommufd_physical_bind(struct vfio_device *vdev,
+			       struct iommufd_ctx *ictx, u32 *out_device_id)
+{
+	struct iommufd_device *idev;
+
+	idev = iommufd_device_bind(ictx, vdev->dev, out_device_id);
+	if (IS_ERR(idev))
+		return PTR_ERR(idev);
+	vdev->iommufd_device = idev;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_physical_bind);
+
+void vfio_iommufd_physical_unbind(struct vfio_device *vdev)
+{
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	if (vdev->iommufd_attached) {
+		iommufd_device_detach(vdev->iommufd_device);
+		vdev->iommufd_attached = false;
+	}
+	iommufd_device_unbind(vdev->iommufd_device);
+	vdev->iommufd_device = NULL;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_physical_unbind);
+
+int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id)
+{
+	unsigned int flags = 0;
+
+	if (vfio_allow_unsafe_interrupts)
+		flags |= IOMMUFD_ATTACH_FLAGS_ALLOW_UNSAFE_INTERRUPT;
+	return iommufd_device_attach(vdev->iommufd_device, pt_id, flags);
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_physical_attach_ioas);
diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
index 39eeca18a0f7c8..40019b11c5a969 100644
--- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
@@ -1246,6 +1246,9 @@ static const struct vfio_device_ops hisi_acc_vfio_pci_migrn_ops = {
 	.mmap = hisi_acc_vfio_pci_mmap,
 	.request = vfio_pci_core_request,
 	.match = vfio_pci_core_match,
+	.bind_iommufd = vfio_iommufd_physical_bind,
+	.unbind_iommufd = vfio_iommufd_physical_unbind,
+	.attach_ioas = vfio_iommufd_physical_attach_ioas,
 };
 
 static const struct vfio_device_ops hisi_acc_vfio_pci_ops = {
@@ -1261,6 +1264,9 @@ static const struct vfio_device_ops hisi_acc_vfio_pci_ops = {
 	.mmap = vfio_pci_core_mmap,
 	.request = vfio_pci_core_request,
 	.match = vfio_pci_core_match,
+	.bind_iommufd = vfio_iommufd_physical_bind,
+	.unbind_iommufd = vfio_iommufd_physical_unbind,
+	.attach_ioas = vfio_iommufd_physical_attach_ioas,
 };
 
 static int hisi_acc_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c
index fd6ccb8454a24a..32d1f38d351e7e 100644
--- a/drivers/vfio/pci/mlx5/main.c
+++ b/drivers/vfio/pci/mlx5/main.c
@@ -623,6 +623,9 @@ static const struct vfio_device_ops mlx5vf_pci_ops = {
 	.mmap = vfio_pci_core_mmap,
 	.request = vfio_pci_core_request,
 	.match = vfio_pci_core_match,
+	.bind_iommufd = vfio_iommufd_physical_bind,
+	.unbind_iommufd = vfio_iommufd_physical_unbind,
+	.attach_ioas = vfio_iommufd_physical_attach_ioas,
 };
 
 static int mlx5vf_pci_probe(struct pci_dev *pdev,
diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
index 1d4919edfbde48..29091ee2e9849b 100644
--- a/drivers/vfio/pci/vfio_pci.c
+++ b/drivers/vfio/pci/vfio_pci.c
@@ -138,6 +138,9 @@ static const struct vfio_device_ops vfio_pci_ops = {
 	.mmap		= vfio_pci_core_mmap,
 	.request	= vfio_pci_core_request,
 	.match		= vfio_pci_core_match,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
 };
 
 static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
diff --git a/drivers/vfio/platform/vfio_amba.c b/drivers/vfio/platform/vfio_amba.c
index eaea63e5294c58..5a046098d0bdf4 100644
--- a/drivers/vfio/platform/vfio_amba.c
+++ b/drivers/vfio/platform/vfio_amba.c
@@ -117,6 +117,9 @@ static const struct vfio_device_ops vfio_amba_ops = {
 	.read		= vfio_platform_read,
 	.write		= vfio_platform_write,
 	.mmap		= vfio_platform_mmap,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
 };
 
 static const struct amba_id pl330_ids[] = {
diff --git a/drivers/vfio/platform/vfio_platform.c b/drivers/vfio/platform/vfio_platform.c
index 82cedcebfd9022..b87c3b70878341 100644
--- a/drivers/vfio/platform/vfio_platform.c
+++ b/drivers/vfio/platform/vfio_platform.c
@@ -106,6 +106,9 @@ static const struct vfio_device_ops vfio_platform_ops = {
 	.read		= vfio_platform_read,
 	.write		= vfio_platform_write,
 	.mmap		= vfio_platform_mmap,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
 };
 
 static struct platform_driver vfio_platform_driver = {
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index 985e13d52989ca..809f2e8523968e 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -124,6 +124,21 @@ void vfio_device_container_unregister(struct vfio_device *device);
 int __init vfio_container_init(void);
 void vfio_container_cleanup(void);
 
+#if IS_ENABLED(CONFIG_IOMMUFD)
+int vfio_iommufd_bind(struct vfio_device *device, struct iommufd_ctx *ictx);
+void vfio_iommufd_unbind(struct vfio_device *device);
+#else
+static inline int vfio_iommufd_bind(struct vfio_device *device,
+				    struct iommufd_ctx *ictx)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void vfio_iommufd_unbind(struct vfio_device *device)
+{
+}
+#endif
+
 #ifdef CONFIG_VFIO_NOIOMMU
 extern bool vfio_noiommu __read_mostly;
 #else
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index cf0ea744de931e..bfbda04af1ffda 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -528,6 +528,11 @@ static int __vfio_register_dev(struct vfio_device *device,
 	if (IS_ERR(group))
 		return PTR_ERR(group);
 
+	if (WARN_ON(device->ops->bind_iommufd &&
+		    (!device->ops->unbind_iommufd ||
+		     !device->ops->attach_ioas)))
+		return -EINVAL;
+
 	/*
 	 * If the driver doesn't specify a set then the device is added to a
 	 * singleton set just for itself.
@@ -790,6 +795,10 @@ static int vfio_device_first_open(struct vfio_device *device)
 		ret = vfio_group_use_container(device->group);
 		if (ret)
 			goto err_module_put;
+	} else if (device->group->iommufd) {
+		ret = vfio_iommufd_bind(device, device->group->iommufd);
+		if (ret)
+			goto err_module_put;
 	}
 
 	device->kvm = device->group->kvm;
@@ -806,6 +815,7 @@ static int vfio_device_first_open(struct vfio_device *device)
 err_container:
 	if (device->group->container)
 		vfio_group_unuse_container(device->group);
+	vfio_iommufd_unbind(device);
 	device->kvm = NULL;
 err_module_put:
 	mutex_unlock(&device->group->group_lock);
@@ -825,6 +835,7 @@ static void vfio_device_last_close(struct vfio_device *device)
 	device->kvm = NULL;
 	if (device->group->container)
 		vfio_group_unuse_container(device->group);
+	vfio_iommufd_unbind(device);
 	mutex_unlock(&device->group->group_lock);
 	module_put(device->dev->driver->owner);
 }
@@ -1923,8 +1934,6 @@ static void __exit vfio_cleanup(void)
 module_init(vfio_init);
 module_exit(vfio_cleanup);
 
-MODULE_IMPORT_NS(IOMMUFD);
-MODULE_IMPORT_NS(IOMMUFD_VFIO);
 MODULE_VERSION(DRIVER_VERSION);
 MODULE_LICENSE("GPL v2");
 MODULE_AUTHOR(DRIVER_AUTHOR);
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index e7cebeb875dd1a..a7fc4d747dc226 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -17,6 +17,8 @@
 #include <linux/iova_bitmap.h>
 
 struct kvm;
+struct iommufd_ctx;
+struct iommufd_device;
 
 /*
  * VFIO devices can be placed in a set, this allows all devices to share this
@@ -54,6 +56,10 @@ struct vfio_device {
 	struct completion comp;
 	struct list_head group_next;
 	struct list_head iommu_entry;
+#if IS_ENABLED(CONFIG_IOMMUFD)
+	struct iommufd_device *iommufd_device;
+	bool iommufd_attached;
+#endif
 };
 
 /**
@@ -80,6 +86,10 @@ struct vfio_device_ops {
 	char	*name;
 	int	(*init)(struct vfio_device *vdev);
 	void	(*release)(struct vfio_device *vdev);
+	int	(*bind_iommufd)(struct vfio_device *vdev,
+				struct iommufd_ctx *ictx, u32 *out_device_id);
+	void	(*unbind_iommufd)(struct vfio_device *vdev);
+	int	(*attach_ioas)(struct vfio_device *vdev, u32 *pt_id);
 	int	(*open_device)(struct vfio_device *vdev);
 	void	(*close_device)(struct vfio_device *vdev);
 	ssize_t	(*read)(struct vfio_device *vdev, char __user *buf,
@@ -96,6 +106,21 @@ struct vfio_device_ops {
 				  void __user *arg, size_t argsz);
 };
 
+#if IS_ENABLED(CONFIG_IOMMUFD)
+int vfio_iommufd_physical_bind(struct vfio_device *vdev,
+			       struct iommufd_ctx *ictx, u32 *out_device_id);
+void vfio_iommufd_physical_unbind(struct vfio_device *vdev);
+int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
+#else
+#define vfio_iommufd_physical_bind                                      \
+	((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx,   \
+		  u32 *out_device_id)) NULL)
+#define vfio_iommufd_physical_unbind \
+	((void (*)(struct vfio_device *vdev)) NULL)
+#define vfio_iommufd_physical_attach_ioas \
+	((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL)
+#endif
+
 /**
  * @migration_set_state: Optional callback to change the migration state for
  *         devices that support migration. It's mandatory for
-- 
2.38.0

[PATCH 08/10] vfio-iommufd: Support iommufd for emulated VFIO devices

[Jason Gunthorpe]

Emulated VFIO devices are calling vfio_register_emulated_iommu_dev() and
consist of all the mdev drivers.

Like the physical drivers, support for iommufd is provided by the driver
supplying the correct correct standard ops. Provide ops from the core that
duplicate what vfio_register_emulated_iommu_dev() does.

Emulated drivers are where it is more likely to see variation in the
iommfd support ops. For instance IDXD will probably need to setup both a
iommfd_device context linked to a PASID and an iommufd_access context to
support all their mdev operations.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/gpu/drm/i915/gvt/kvmgt.c  |   3 +
 drivers/s390/cio/vfio_ccw_ops.c   |   3 +
 drivers/s390/crypto/vfio_ap_ops.c |   3 +
 drivers/vfio/container.c          | 108 ++++++-----------------------
 drivers/vfio/iommufd.c            |  57 ++++++++++++++++
 drivers/vfio/vfio.h               |  10 ++-
 drivers/vfio/vfio_main.c          | 110 +++++++++++++++++++++++++++++-
 include/linux/vfio.h              |  14 ++++
 8 files changed, 217 insertions(+), 91 deletions(-)

diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 7a45e5360caf2d..579b230a0f58d9 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -1474,6 +1474,9 @@ static const struct vfio_device_ops intel_vgpu_dev_ops = {
 	.mmap		= intel_vgpu_mmap,
 	.ioctl		= intel_vgpu_ioctl,
 	.dma_unmap	= intel_vgpu_dma_unmap,
+	.bind_iommufd	= vfio_iommufd_emulated_bind,
+	.unbind_iommufd = vfio_iommufd_emulated_unbind,
+	.attach_ioas	= vfio_iommufd_emulated_attach_ioas,
 };
 
 static int intel_vgpu_probe(struct mdev_device *mdev)
diff --git a/drivers/s390/cio/vfio_ccw_ops.c b/drivers/s390/cio/vfio_ccw_ops.c
index 6ae4d012d80084..560453d99c24fc 100644
--- a/drivers/s390/cio/vfio_ccw_ops.c
+++ b/drivers/s390/cio/vfio_ccw_ops.c
@@ -588,6 +588,9 @@ static const struct vfio_device_ops vfio_ccw_dev_ops = {
 	.ioctl = vfio_ccw_mdev_ioctl,
 	.request = vfio_ccw_mdev_request,
 	.dma_unmap = vfio_ccw_dma_unmap,
+	.bind_iommufd = vfio_iommufd_emulated_bind,
+	.unbind_iommufd = vfio_iommufd_emulated_unbind,
+	.attach_ioas = vfio_iommufd_emulated_attach_ioas,
 };
 
 struct mdev_driver vfio_ccw_mdev_driver = {
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 0b4cc8c597ae67..bb7776d207924f 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -1789,6 +1789,9 @@ static const struct vfio_device_ops vfio_ap_matrix_dev_ops = {
 	.close_device = vfio_ap_mdev_close_device,
 	.ioctl = vfio_ap_mdev_ioctl,
 	.dma_unmap = vfio_ap_mdev_dma_unmap,
+	.bind_iommufd = vfio_iommufd_emulated_bind,
+	.unbind_iommufd = vfio_iommufd_emulated_unbind,
+	.attach_ioas = vfio_iommufd_emulated_attach_ioas,
 };
 
 static struct mdev_driver vfio_ap_matrix_driver = {
diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
index 8772dad6808539..0388f2e33447eb 100644
--- a/drivers/vfio/container.c
+++ b/drivers/vfio/container.c
@@ -540,113 +540,45 @@ void vfio_group_unuse_container(struct vfio_group *group)
 	fput(group->opened_file);
 }
 
-/*
- * Pin contiguous user pages and return their associated host pages for local
- * domain only.
- * @device [in]  : device
- * @iova [in]    : starting IOVA of user pages to be pinned.
- * @npage [in]   : count of pages to be pinned.  This count should not
- *		   be greater than VFIO_PIN_PAGES_MAX_ENTRIES.
- * @prot [in]    : protection flags
- * @pages[out]   : array of host pages
- * Return error or number of pages pinned.
- *
- * A driver may only call this function if the vfio_device was created
- * by vfio_register_emulated_iommu_dev().
- */
-int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova,
-		   int npage, int prot, struct page **pages)
+int vfio_container_pin_pages(struct vfio_container *container,
+			     struct iommu_group *iommu_group, dma_addr_t iova,
+			     int npage, int prot, struct page **pages)
 {
-	struct vfio_container *container;
-	struct vfio_group *group = device->group;
-	struct vfio_iommu_driver *driver;
-	int ret;
-
-	if (!pages || !npage || !vfio_assert_device_open(device))
-		return -EINVAL;
+	/* group->container cannot change while a vfio device is open */
+	struct vfio_iommu_driver *driver = container->iommu_driver;
 
 	if (npage > VFIO_PIN_PAGES_MAX_ENTRIES)
 		return -E2BIG;
 
 	/* group->container cannot change while a vfio device is open */
-	container = group->container;
 	driver = container->iommu_driver;
-	if (likely(driver && driver->ops->pin_pages))
-		ret = driver->ops->pin_pages(container->iommu_data,
-					     group->iommu_group, iova,
-					     npage, prot, pages);
-	else
-		ret = -ENOTTY;
-
-	return ret;
+	if (unlikely(!driver || !driver->ops->pin_pages))
+		return -ENOTTY;
+	return driver->ops->pin_pages(container->iommu_data, iommu_group, iova,
+				      npage, prot, pages);
 }
-EXPORT_SYMBOL(vfio_pin_pages);
 
-/*
- * Unpin contiguous host pages for local domain only.
- * @device [in]  : device
- * @iova [in]    : starting address of user pages to be unpinned.
- * @npage [in]   : count of pages to be unpinned.  This count should not
- *                 be greater than VFIO_PIN_PAGES_MAX_ENTRIES.
- */
-void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova, int npage)
+void vfio_container_unpin_pages(struct vfio_container *container,
+				dma_addr_t iova, int npage)
 {
-	struct vfio_container *container;
-	struct vfio_iommu_driver *driver;
-
 	if (WARN_ON(npage <= 0 || npage > VFIO_PIN_PAGES_MAX_ENTRIES))
 		return;
 
-	if (WARN_ON(!vfio_assert_device_open(device)))
-		return;
-
-	/* group->container cannot change while a vfio device is open */
-	container = device->group->container;
-	driver = container->iommu_driver;
-
-	driver->ops->unpin_pages(container->iommu_data, iova, npage);
+	container->iommu_driver->ops->unpin_pages(container->iommu_data, iova,
+						  npage);
 }
-EXPORT_SYMBOL(vfio_unpin_pages);
 
-/*
- * This interface allows the CPUs to perform some sort of virtual DMA on
- * behalf of the device.
- *
- * CPUs read/write from/into a range of IOVAs pointing to user space memory
- * into/from a kernel buffer.
- *
- * As the read/write of user space memory is conducted via the CPUs and is
- * not a real device DMA, it is not necessary to pin the user space memory.
- *
- * @device [in]		: VFIO device
- * @iova [in]		: base IOVA of a user space buffer
- * @data [in]		: pointer to kernel buffer
- * @len [in]		: kernel buffer length
- * @write		: indicate read or write
- * Return error code on failure or 0 on success.
- */
-int vfio_dma_rw(struct vfio_device *device, dma_addr_t iova, void *data,
-		size_t len, bool write)
+int vfio_container_dma_rw(struct vfio_container *container, dma_addr_t iova,
+			  void *data, size_t len, bool write)
 {
-	struct vfio_container *container;
-	struct vfio_iommu_driver *driver;
-	int ret = 0;
-
-	if (!data || len <= 0 || !vfio_assert_device_open(device))
-		return -EINVAL;
-
 	/* group->container cannot change while a vfio device is open */
-	container = device->group->container;
-	driver = container->iommu_driver;
+	struct vfio_iommu_driver *driver = container->iommu_driver;
 
-	if (likely(driver && driver->ops->dma_rw))
-		ret = driver->ops->dma_rw(container->iommu_data,
-					  iova, data, len, write);
-	else
-		ret = -ENOTTY;
-	return ret;
+	if (unlikely(!driver || !driver->ops->dma_rw))
+		return -ENOTTY;
+	return driver->ops->dma_rw(container->iommu_data, iova, data, len,
+				   write);
 }
-EXPORT_SYMBOL(vfio_dma_rw);
 
 int __init vfio_container_init(void)
 {
diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
index 8280bb32ee677c..40eb6931ab2321 100644
--- a/drivers/vfio/iommufd.c
+++ b/drivers/vfio/iommufd.c
@@ -102,3 +102,60 @@ int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id)
 	return iommufd_device_attach(vdev->iommufd_device, pt_id, flags);
 }
 EXPORT_SYMBOL_GPL(vfio_iommufd_physical_attach_ioas);
+
+/*
+ * The emulated standard ops mean that vfio_device is going to use the
+ * "mdev path" and will call vfio_pin_pages()/vfio_dma_rw(). Drivers using this
+ * ops set should call vfio_register_emulated_iommu_dev().
+ */
+
+static void vfio_emulated_unmap(void *data, unsigned long iova,
+				unsigned long length)
+{
+	struct vfio_device *vdev = data;
+
+	vdev->ops->dma_unmap(vdev, iova, length);
+}
+
+static const struct iommufd_access_ops vfio_user_ops = {
+	.unmap = vfio_emulated_unmap,
+};
+
+int vfio_iommufd_emulated_bind(struct vfio_device *vdev,
+			       struct iommufd_ctx *ictx, u32 *out_device_id)
+{
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	vdev->iommufd_ictx = ictx;
+	iommufd_ctx_get(ictx);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_emulated_bind);
+
+void vfio_iommufd_emulated_unbind(struct vfio_device *vdev)
+{
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	if (vdev->iommufd_access) {
+		iommufd_access_destroy(vdev->iommufd_access);
+		vdev->iommufd_access = NULL;
+	}
+	iommufd_ctx_put(vdev->iommufd_ictx);
+	vdev->iommufd_ictx = NULL;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_emulated_unbind);
+
+int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id)
+{
+	struct iommufd_access *user;
+
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	user = iommufd_access_create(vdev->iommufd_ictx, *pt_id, &vfio_user_ops,
+				     vdev);
+	if (IS_ERR(user))
+		return PTR_ERR(user);
+	vdev->iommufd_access = user;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_emulated_attach_ioas);
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index 809f2e8523968e..d57a08afb5cf5c 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -111,8 +111,6 @@ struct vfio_iommu_driver {
 int vfio_register_iommu_driver(const struct vfio_iommu_driver_ops *ops);
 void vfio_unregister_iommu_driver(const struct vfio_iommu_driver_ops *ops);
 
-bool vfio_assert_device_open(struct vfio_device *device);
-
 struct vfio_container *vfio_container_from_file(struct file *filep);
 int vfio_group_use_container(struct vfio_group *group);
 void vfio_group_unuse_container(struct vfio_group *group);
@@ -121,6 +119,14 @@ int vfio_container_attach_group(struct vfio_container *container,
 void vfio_group_detach_container(struct vfio_group *group);
 void vfio_device_container_register(struct vfio_device *device);
 void vfio_device_container_unregister(struct vfio_device *device);
+int vfio_container_pin_pages(struct vfio_container *container,
+			     struct iommu_group *iommu_group, dma_addr_t iova,
+			     int npage, int prot, struct page **pages);
+void vfio_container_unpin_pages(struct vfio_container *container,
+				dma_addr_t iova, int npage);
+int vfio_container_dma_rw(struct vfio_container *container, dma_addr_t iova,
+			  void *data, size_t len, bool write);
+
 int __init vfio_container_init(void);
 void vfio_container_cleanup(void);
 
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index bfbda04af1ffda..9b837efbddb6db 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -766,7 +766,7 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
 static const struct file_operations vfio_device_fops;
 
 /* true if the vfio_device has open_device() called but not close_device() */
-bool vfio_assert_device_open(struct vfio_device *device)
+static bool vfio_assert_device_open(struct vfio_device *device)
 {
 	return !WARN_ON_ONCE(!READ_ONCE(device->open_count));
 }
@@ -1861,6 +1861,114 @@ int vfio_set_irqs_validate_and_prepare(struct vfio_irq_set *hdr, int num_irqs,
 }
 EXPORT_SYMBOL(vfio_set_irqs_validate_and_prepare);
 
+/*
+ * Pin contiguous user pages and return their associated host pages for local
+ * domain only.
+ * @device [in]  : device
+ * @iova [in]    : starting IOVA of user pages to be pinned.
+ * @npage [in]   : count of pages to be pinned.  This count should not
+ *		   be greater than VFIO_PIN_PAGES_MAX_ENTRIES.
+ * @prot [in]    : protection flags
+ * @pages[out]   : array of host pages
+ * Return error or number of pages pinned.
+ *
+ * A driver may only call this function if the vfio_device was created
+ * by vfio_register_emulated_iommu_dev() due to vfio_container_pin_pages().
+ */
+int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova,
+		   int npage, int prot, struct page **pages)
+{
+	if (!pages || !npage || !vfio_assert_device_open(device))
+		return -EINVAL;
+	if (device->group->container)
+		return vfio_container_pin_pages(device->group->container,
+						device->group->iommu_group,
+						iova, npage, prot, pages);
+	if (device->iommufd_access) {
+		int ret;
+
+		if (iova > ULONG_MAX)
+			return -EINVAL;
+		ret = iommufd_access_pin_pages(
+			device->iommufd_access, iova, npage * PAGE_SIZE, pages,
+			(prot & IOMMU_WRITE) ? IOMMUFD_ACCESS_RW_WRITE : 0);
+		if (ret)
+			return ret;
+		return npage;
+	}
+	return -EINVAL;
+}
+EXPORT_SYMBOL(vfio_pin_pages);
+
+/*
+ * Unpin contiguous host pages for local domain only.
+ * @device [in]  : device
+ * @iova [in]    : starting address of user pages to be unpinned.
+ * @npage [in]   : count of pages to be unpinned.  This count should not
+ *                 be greater than VFIO_PIN_PAGES_MAX_ENTRIES.
+ */
+void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova, int npage)
+{
+	if (WARN_ON(!vfio_assert_device_open(device)))
+		return;
+
+	if (device->group->container) {
+		vfio_container_unpin_pages(device->group->container, iova,
+					   npage);
+	} else if (device->iommufd_access) {
+		if (WARN_ON(iova > ULONG_MAX))
+			return;
+		iommufd_access_unpin_pages(device->iommufd_access, iova,
+					   npage * PAGE_SIZE);
+	}
+}
+EXPORT_SYMBOL(vfio_unpin_pages);
+
+/*
+ * This interface allows the CPUs to perform some sort of virtual DMA on
+ * behalf of the device.
+ *
+ * CPUs read/write from/into a range of IOVAs pointing to user space memory
+ * into/from a kernel buffer.
+ *
+ * As the read/write of user space memory is conducted via the CPUs and is
+ * not a real device DMA, it is not necessary to pin the user space memory.
+ *
+ * @device [in]		: VFIO device
+ * @iova [in]		: base IOVA of a user space buffer
+ * @data [in]		: pointer to kernel buffer
+ * @len [in]		: kernel buffer length
+ * @write		: indicate read or write
+ * Return error code on failure or 0 on success.
+ */
+int vfio_dma_rw(struct vfio_device *device, dma_addr_t iova, void *data,
+		size_t len, bool write)
+{
+	if (!data || len <= 0 || !vfio_assert_device_open(device))
+		return -EINVAL;
+
+	if (device->group->container)
+		return vfio_container_dma_rw(device->group->container, iova,
+					     data, len, write);
+
+	if (device->iommufd_access) {
+		unsigned int flags = 0;
+
+		if (iova > ULONG_MAX)
+			return -EINVAL;
+
+		/* VFIO historically tries to auto-detect a kthread */
+		if (!current->mm)
+			flags |= IOMMUFD_ACCESS_RW_KTHREAD;
+		if (write)
+			flags |= IOMMUFD_ACCESS_RW_WRITE;
+		return iommufd_access_rw(device->iommufd_access, iova, data,
+					 len, flags);
+	}
+	return -EINVAL;
+}
+EXPORT_SYMBOL(vfio_dma_rw);
+
 /*
  * Module/class support
  */
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index a7fc4d747dc226..d5f84f98c0fa8f 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -19,6 +19,7 @@
 struct kvm;
 struct iommufd_ctx;
 struct iommufd_device;
+struct iommufd_access;
 
 /*
  * VFIO devices can be placed in a set, this allows all devices to share this
@@ -56,8 +57,10 @@ struct vfio_device {
 	struct completion comp;
 	struct list_head group_next;
 	struct list_head iommu_entry;
+	struct iommufd_access *iommufd_access;
 #if IS_ENABLED(CONFIG_IOMMUFD)
 	struct iommufd_device *iommufd_device;
+	struct iommufd_ctx *iommufd_ictx;
 	bool iommufd_attached;
 #endif
 };
@@ -111,6 +114,10 @@ int vfio_iommufd_physical_bind(struct vfio_device *vdev,
 			       struct iommufd_ctx *ictx, u32 *out_device_id);
 void vfio_iommufd_physical_unbind(struct vfio_device *vdev);
 int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
+int vfio_iommufd_emulated_bind(struct vfio_device *vdev,
+			       struct iommufd_ctx *ictx, u32 *out_device_id);
+void vfio_iommufd_emulated_unbind(struct vfio_device *vdev);
+int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
 #else
 #define vfio_iommufd_physical_bind                                      \
 	((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx,   \
@@ -119,6 +126,13 @@ int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
 	((void (*)(struct vfio_device *vdev)) NULL)
 #define vfio_iommufd_physical_attach_ioas \
 	((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL)
+#define vfio_iommufd_emulated_bind                                      \
+	((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx,   \
+		  u32 *out_device_id)) NULL)
+#define vfio_iommufd_emulated_unbind \
+	((void (*)(struct vfio_device *vdev)) NULL)
+#define vfio_iommufd_emulated_attach_ioas \
+	((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL)
 #endif
 
 /**
-- 
2.38.0

[PATCH 08/10] vfio-iommufd: Support iommufd for emulated VFIO devices

[Jason Gunthorpe]

Emulated VFIO devices are calling vfio_register_emulated_iommu_dev() and
consist of all the mdev drivers.

Like the physical drivers, support for iommufd is provided by the driver
supplying the correct correct standard ops. Provide ops from the core that
duplicate what vfio_register_emulated_iommu_dev() does.

Emulated drivers are where it is more likely to see variation in the
iommfd support ops. For instance IDXD will probably need to setup both a
iommfd_device context linked to a PASID and an iommufd_access context to
support all their mdev operations.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/gpu/drm/i915/gvt/kvmgt.c  |   3 +
 drivers/s390/cio/vfio_ccw_ops.c   |   3 +
 drivers/s390/crypto/vfio_ap_ops.c |   3 +
 drivers/vfio/container.c          | 108 ++++++-----------------------
 drivers/vfio/iommufd.c            |  57 ++++++++++++++++
 drivers/vfio/vfio.h               |  10 ++-
 drivers/vfio/vfio_main.c          | 110 +++++++++++++++++++++++++++++-
 include/linux/vfio.h              |  14 ++++
 8 files changed, 217 insertions(+), 91 deletions(-)

diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 7a45e5360caf2d..579b230a0f58d9 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -1474,6 +1474,9 @@ static const struct vfio_device_ops intel_vgpu_dev_ops = {
 	.mmap		= intel_vgpu_mmap,
 	.ioctl		= intel_vgpu_ioctl,
 	.dma_unmap	= intel_vgpu_dma_unmap,
+	.bind_iommufd	= vfio_iommufd_emulated_bind,
+	.unbind_iommufd = vfio_iommufd_emulated_unbind,
+	.attach_ioas	= vfio_iommufd_emulated_attach_ioas,
 };
 
 static int intel_vgpu_probe(struct mdev_device *mdev)
diff --git a/drivers/s390/cio/vfio_ccw_ops.c b/drivers/s390/cio/vfio_ccw_ops.c
index 6ae4d012d80084..560453d99c24fc 100644
--- a/drivers/s390/cio/vfio_ccw_ops.c
+++ b/drivers/s390/cio/vfio_ccw_ops.c
@@ -588,6 +588,9 @@ static const struct vfio_device_ops vfio_ccw_dev_ops = {
 	.ioctl = vfio_ccw_mdev_ioctl,
 	.request = vfio_ccw_mdev_request,
 	.dma_unmap = vfio_ccw_dma_unmap,
+	.bind_iommufd = vfio_iommufd_emulated_bind,
+	.unbind_iommufd = vfio_iommufd_emulated_unbind,
+	.attach_ioas = vfio_iommufd_emulated_attach_ioas,
 };
 
 struct mdev_driver vfio_ccw_mdev_driver = {
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 0b4cc8c597ae67..bb7776d207924f 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -1789,6 +1789,9 @@ static const struct vfio_device_ops vfio_ap_matrix_dev_ops = {
 	.close_device = vfio_ap_mdev_close_device,
 	.ioctl = vfio_ap_mdev_ioctl,
 	.dma_unmap = vfio_ap_mdev_dma_unmap,
+	.bind_iommufd = vfio_iommufd_emulated_bind,
+	.unbind_iommufd = vfio_iommufd_emulated_unbind,
+	.attach_ioas = vfio_iommufd_emulated_attach_ioas,
 };
 
 static struct mdev_driver vfio_ap_matrix_driver = {
diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
index 8772dad6808539..0388f2e33447eb 100644
--- a/drivers/vfio/container.c
+++ b/drivers/vfio/container.c
@@ -540,113 +540,45 @@ void vfio_group_unuse_container(struct vfio_group *group)
 	fput(group->opened_file);
 }
 
-/*
- * Pin contiguous user pages and return their associated host pages for local
- * domain only.
- * @device [in]  : device
- * @iova [in]    : starting IOVA of user pages to be pinned.
- * @npage [in]   : count of pages to be pinned.  This count should not
- *		   be greater than VFIO_PIN_PAGES_MAX_ENTRIES.
- * @prot [in]    : protection flags
- * @pages[out]   : array of host pages
- * Return error or number of pages pinned.
- *
- * A driver may only call this function if the vfio_device was created
- * by vfio_register_emulated_iommu_dev().
- */
-int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova,
-		   int npage, int prot, struct page **pages)
+int vfio_container_pin_pages(struct vfio_container *container,
+			     struct iommu_group *iommu_group, dma_addr_t iova,
+			     int npage, int prot, struct page **pages)
 {
-	struct vfio_container *container;
-	struct vfio_group *group = device->group;
-	struct vfio_iommu_driver *driver;
-	int ret;
-
-	if (!pages || !npage || !vfio_assert_device_open(device))
-		return -EINVAL;
+	/* group->container cannot change while a vfio device is open */
+	struct vfio_iommu_driver *driver = container->iommu_driver;
 
 	if (npage > VFIO_PIN_PAGES_MAX_ENTRIES)
 		return -E2BIG;
 
 	/* group->container cannot change while a vfio device is open */
-	container = group->container;
 	driver = container->iommu_driver;
-	if (likely(driver && driver->ops->pin_pages))
-		ret = driver->ops->pin_pages(container->iommu_data,
-					     group->iommu_group, iova,
-					     npage, prot, pages);
-	else
-		ret = -ENOTTY;
-
-	return ret;
+	if (unlikely(!driver || !driver->ops->pin_pages))
+		return -ENOTTY;
+	return driver->ops->pin_pages(container->iommu_data, iommu_group, iova,
+				      npage, prot, pages);
 }
-EXPORT_SYMBOL(vfio_pin_pages);
 
-/*
- * Unpin contiguous host pages for local domain only.
- * @device [in]  : device
- * @iova [in]    : starting address of user pages to be unpinned.
- * @npage [in]   : count of pages to be unpinned.  This count should not
- *                 be greater than VFIO_PIN_PAGES_MAX_ENTRIES.
- */
-void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova, int npage)
+void vfio_container_unpin_pages(struct vfio_container *container,
+				dma_addr_t iova, int npage)
 {
-	struct vfio_container *container;
-	struct vfio_iommu_driver *driver;
-
 	if (WARN_ON(npage <= 0 || npage > VFIO_PIN_PAGES_MAX_ENTRIES))
 		return;
 
-	if (WARN_ON(!vfio_assert_device_open(device)))
-		return;
-
-	/* group->container cannot change while a vfio device is open */
-	container = device->group->container;
-	driver = container->iommu_driver;
-
-	driver->ops->unpin_pages(container->iommu_data, iova, npage);
+	container->iommu_driver->ops->unpin_pages(container->iommu_data, iova,
+						  npage);
 }
-EXPORT_SYMBOL(vfio_unpin_pages);
 
-/*
- * This interface allows the CPUs to perform some sort of virtual DMA on
- * behalf of the device.
- *
- * CPUs read/write from/into a range of IOVAs pointing to user space memory
- * into/from a kernel buffer.
- *
- * As the read/write of user space memory is conducted via the CPUs and is
- * not a real device DMA, it is not necessary to pin the user space memory.
- *
- * @device [in]		: VFIO device
- * @iova [in]		: base IOVA of a user space buffer
- * @data [in]		: pointer to kernel buffer
- * @len [in]		: kernel buffer length
- * @write		: indicate read or write
- * Return error code on failure or 0 on success.
- */
-int vfio_dma_rw(struct vfio_device *device, dma_addr_t iova, void *data,
-		size_t len, bool write)
+int vfio_container_dma_rw(struct vfio_container *container, dma_addr_t iova,
+			  void *data, size_t len, bool write)
 {
-	struct vfio_container *container;
-	struct vfio_iommu_driver *driver;
-	int ret = 0;
-
-	if (!data || len <= 0 || !vfio_assert_device_open(device))
-		return -EINVAL;
-
 	/* group->container cannot change while a vfio device is open */
-	container = device->group->container;
-	driver = container->iommu_driver;
+	struct vfio_iommu_driver *driver = container->iommu_driver;
 
-	if (likely(driver && driver->ops->dma_rw))
-		ret = driver->ops->dma_rw(container->iommu_data,
-					  iova, data, len, write);
-	else
-		ret = -ENOTTY;
-	return ret;
+	if (unlikely(!driver || !driver->ops->dma_rw))
+		return -ENOTTY;
+	return driver->ops->dma_rw(container->iommu_data, iova, data, len,
+				   write);
 }
-EXPORT_SYMBOL(vfio_dma_rw);
 
 int __init vfio_container_init(void)
 {
diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
index 8280bb32ee677c..40eb6931ab2321 100644
--- a/drivers/vfio/iommufd.c
+++ b/drivers/vfio/iommufd.c
@@ -102,3 +102,60 @@ int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id)
 	return iommufd_device_attach(vdev->iommufd_device, pt_id, flags);
 }
 EXPORT_SYMBOL_GPL(vfio_iommufd_physical_attach_ioas);
+
+/*
+ * The emulated standard ops mean that vfio_device is going to use the
+ * "mdev path" and will call vfio_pin_pages()/vfio_dma_rw(). Drivers using this
+ * ops set should call vfio_register_emulated_iommu_dev().
+ */
+
+static void vfio_emulated_unmap(void *data, unsigned long iova,
+				unsigned long length)
+{
+	struct vfio_device *vdev = data;
+
+	vdev->ops->dma_unmap(vdev, iova, length);
+}
+
+static const struct iommufd_access_ops vfio_user_ops = {
+	.unmap = vfio_emulated_unmap,
+};
+
+int vfio_iommufd_emulated_bind(struct vfio_device *vdev,
+			       struct iommufd_ctx *ictx, u32 *out_device_id)
+{
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	vdev->iommufd_ictx = ictx;
+	iommufd_ctx_get(ictx);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_emulated_bind);
+
+void vfio_iommufd_emulated_unbind(struct vfio_device *vdev)
+{
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	if (vdev->iommufd_access) {
+		iommufd_access_destroy(vdev->iommufd_access);
+		vdev->iommufd_access = NULL;
+	}
+	iommufd_ctx_put(vdev->iommufd_ictx);
+	vdev->iommufd_ictx = NULL;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_emulated_unbind);
+
+int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id)
+{
+	struct iommufd_access *user;
+
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	user = iommufd_access_create(vdev->iommufd_ictx, *pt_id, &vfio_user_ops,
+				     vdev);
+	if (IS_ERR(user))
+		return PTR_ERR(user);
+	vdev->iommufd_access = user;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_emulated_attach_ioas);
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index 809f2e8523968e..d57a08afb5cf5c 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -111,8 +111,6 @@ struct vfio_iommu_driver {
 int vfio_register_iommu_driver(const struct vfio_iommu_driver_ops *ops);
 void vfio_unregister_iommu_driver(const struct vfio_iommu_driver_ops *ops);
 
-bool vfio_assert_device_open(struct vfio_device *device);
-
 struct vfio_container *vfio_container_from_file(struct file *filep);
 int vfio_group_use_container(struct vfio_group *group);
 void vfio_group_unuse_container(struct vfio_group *group);
@@ -121,6 +119,14 @@ int vfio_container_attach_group(struct vfio_container *container,
 void vfio_group_detach_container(struct vfio_group *group);
 void vfio_device_container_register(struct vfio_device *device);
 void vfio_device_container_unregister(struct vfio_device *device);
+int vfio_container_pin_pages(struct vfio_container *container,
+			     struct iommu_group *iommu_group, dma_addr_t iova,
+			     int npage, int prot, struct page **pages);
+void vfio_container_unpin_pages(struct vfio_container *container,
+				dma_addr_t iova, int npage);
+int vfio_container_dma_rw(struct vfio_container *container, dma_addr_t iova,
+			  void *data, size_t len, bool write);
+
 int __init vfio_container_init(void);
 void vfio_container_cleanup(void);
 
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index bfbda04af1ffda..9b837efbddb6db 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -766,7 +766,7 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
 static const struct file_operations vfio_device_fops;
 
 /* true if the vfio_device has open_device() called but not close_device() */
-bool vfio_assert_device_open(struct vfio_device *device)
+static bool vfio_assert_device_open(struct vfio_device *device)
 {
 	return !WARN_ON_ONCE(!READ_ONCE(device->open_count));
 }
@@ -1861,6 +1861,114 @@ int vfio_set_irqs_validate_and_prepare(struct vfio_irq_set *hdr, int num_irqs,
 }
 EXPORT_SYMBOL(vfio_set_irqs_validate_and_prepare);
 
+/*
+ * Pin contiguous user pages and return their associated host pages for local
+ * domain only.
+ * @device [in]  : device
+ * @iova [in]    : starting IOVA of user pages to be pinned.
+ * @npage [in]   : count of pages to be pinned.  This count should not
+ *		   be greater than VFIO_PIN_PAGES_MAX_ENTRIES.
+ * @prot [in]    : protection flags
+ * @pages[out]   : array of host pages
+ * Return error or number of pages pinned.
+ *
+ * A driver may only call this function if the vfio_device was created
+ * by vfio_register_emulated_iommu_dev() due to vfio_container_pin_pages().
+ */
+int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova,
+		   int npage, int prot, struct page **pages)
+{
+	if (!pages || !npage || !vfio_assert_device_open(device))
+		return -EINVAL;
+	if (device->group->container)
+		return vfio_container_pin_pages(device->group->container,
+						device->group->iommu_group,
+						iova, npage, prot, pages);
+	if (device->iommufd_access) {
+		int ret;
+
+		if (iova > ULONG_MAX)
+			return -EINVAL;
+		ret = iommufd_access_pin_pages(
+			device->iommufd_access, iova, npage * PAGE_SIZE, pages,
+			(prot & IOMMU_WRITE) ? IOMMUFD_ACCESS_RW_WRITE : 0);
+		if (ret)
+			return ret;
+		return npage;
+	}
+	return -EINVAL;
+}
+EXPORT_SYMBOL(vfio_pin_pages);
+
+/*
+ * Unpin contiguous host pages for local domain only.
+ * @device [in]  : device
+ * @iova [in]    : starting address of user pages to be unpinned.
+ * @npage [in]   : count of pages to be unpinned.  This count should not
+ *                 be greater than VFIO_PIN_PAGES_MAX_ENTRIES.
+ */
+void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova, int npage)
+{
+	if (WARN_ON(!vfio_assert_device_open(device)))
+		return;
+
+	if (device->group->container) {
+		vfio_container_unpin_pages(device->group->container, iova,
+					   npage);
+	} else if (device->iommufd_access) {
+		if (WARN_ON(iova > ULONG_MAX))
+			return;
+		iommufd_access_unpin_pages(device->iommufd_access, iova,
+					   npage * PAGE_SIZE);
+	}
+}
+EXPORT_SYMBOL(vfio_unpin_pages);
+
+/*
+ * This interface allows the CPUs to perform some sort of virtual DMA on
+ * behalf of the device.
+ *
+ * CPUs read/write from/into a range of IOVAs pointing to user space memory
+ * into/from a kernel buffer.
+ *
+ * As the read/write of user space memory is conducted via the CPUs and is
+ * not a real device DMA, it is not necessary to pin the user space memory.
+ *
+ * @device [in]		: VFIO device
+ * @iova [in]		: base IOVA of a user space buffer
+ * @data [in]		: pointer to kernel buffer
+ * @len [in]		: kernel buffer length
+ * @write		: indicate read or write
+ * Return error code on failure or 0 on success.
+ */
+int vfio_dma_rw(struct vfio_device *device, dma_addr_t iova, void *data,
+		size_t len, bool write)
+{
+	if (!data || len <= 0 || !vfio_assert_device_open(device))
+		return -EINVAL;
+
+	if (device->group->container)
+		return vfio_container_dma_rw(device->group->container, iova,
+					     data, len, write);
+
+	if (device->iommufd_access) {
+		unsigned int flags = 0;
+
+		if (iova > ULONG_MAX)
+			return -EINVAL;
+
+		/* VFIO historically tries to auto-detect a kthread */
+		if (!current->mm)
+			flags |= IOMMUFD_ACCESS_RW_KTHREAD;
+		if (write)
+			flags |= IOMMUFD_ACCESS_RW_WRITE;
+		return iommufd_access_rw(device->iommufd_access, iova, data,
+					 len, flags);
+	}
+	return -EINVAL;
+}
+EXPORT_SYMBOL(vfio_dma_rw);
+
 /*
  * Module/class support
  */
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index a7fc4d747dc226..d5f84f98c0fa8f 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -19,6 +19,7 @@
 struct kvm;
 struct iommufd_ctx;
 struct iommufd_device;
+struct iommufd_access;
 
 /*
  * VFIO devices can be placed in a set, this allows all devices to share this
@@ -56,8 +57,10 @@ struct vfio_device {
 	struct completion comp;
 	struct list_head group_next;
 	struct list_head iommu_entry;
+	struct iommufd_access *iommufd_access;
 #if IS_ENABLED(CONFIG_IOMMUFD)
 	struct iommufd_device *iommufd_device;
+	struct iommufd_ctx *iommufd_ictx;
 	bool iommufd_attached;
 #endif
 };
@@ -111,6 +114,10 @@ int vfio_iommufd_physical_bind(struct vfio_device *vdev,
 			       struct iommufd_ctx *ictx, u32 *out_device_id);
 void vfio_iommufd_physical_unbind(struct vfio_device *vdev);
 int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
+int vfio_iommufd_emulated_bind(struct vfio_device *vdev,
+			       struct iommufd_ctx *ictx, u32 *out_device_id);
+void vfio_iommufd_emulated_unbind(struct vfio_device *vdev);
+int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
 #else
 #define vfio_iommufd_physical_bind                                      \
 	((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx,   \
@@ -119,6 +126,13 @@ int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
 	((void (*)(struct vfio_device *vdev)) NULL)
 #define vfio_iommufd_physical_attach_ioas \
 	((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL)
+#define vfio_iommufd_emulated_bind                                      \
+	((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx,   \
+		  u32 *out_device_id)) NULL)
+#define vfio_iommufd_emulated_unbind \
+	((void (*)(struct vfio_device *vdev)) NULL)
+#define vfio_iommufd_emulated_attach_ioas \
+	((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL)
 #endif
 
 /**
-- 
2.38.0

[Intel-gfx] [PATCH 08/10] vfio-iommufd: Support iommufd for emulated VFIO devices

[Jason Gunthorpe]

Emulated VFIO devices are calling vfio_register_emulated_iommu_dev() and
consist of all the mdev drivers.

Like the physical drivers, support for iommufd is provided by the driver
supplying the correct correct standard ops. Provide ops from the core that
duplicate what vfio_register_emulated_iommu_dev() does.

Emulated drivers are where it is more likely to see variation in the
iommfd support ops. For instance IDXD will probably need to setup both a
iommfd_device context linked to a PASID and an iommufd_access context to
support all their mdev operations.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/gpu/drm/i915/gvt/kvmgt.c  |   3 +
 drivers/s390/cio/vfio_ccw_ops.c   |   3 +
 drivers/s390/crypto/vfio_ap_ops.c |   3 +
 drivers/vfio/container.c          | 108 ++++++-----------------------
 drivers/vfio/iommufd.c            |  57 ++++++++++++++++
 drivers/vfio/vfio.h               |  10 ++-
 drivers/vfio/vfio_main.c          | 110 +++++++++++++++++++++++++++++-
 include/linux/vfio.h              |  14 ++++
 8 files changed, 217 insertions(+), 91 deletions(-)

diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 7a45e5360caf2d..579b230a0f58d9 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -1474,6 +1474,9 @@ static const struct vfio_device_ops intel_vgpu_dev_ops = {
 	.mmap		= intel_vgpu_mmap,
 	.ioctl		= intel_vgpu_ioctl,
 	.dma_unmap	= intel_vgpu_dma_unmap,
+	.bind_iommufd	= vfio_iommufd_emulated_bind,
+	.unbind_iommufd = vfio_iommufd_emulated_unbind,
+	.attach_ioas	= vfio_iommufd_emulated_attach_ioas,
 };
 
 static int intel_vgpu_probe(struct mdev_device *mdev)
diff --git a/drivers/s390/cio/vfio_ccw_ops.c b/drivers/s390/cio/vfio_ccw_ops.c
index 6ae4d012d80084..560453d99c24fc 100644
--- a/drivers/s390/cio/vfio_ccw_ops.c
+++ b/drivers/s390/cio/vfio_ccw_ops.c
@@ -588,6 +588,9 @@ static const struct vfio_device_ops vfio_ccw_dev_ops = {
 	.ioctl = vfio_ccw_mdev_ioctl,
 	.request = vfio_ccw_mdev_request,
 	.dma_unmap = vfio_ccw_dma_unmap,
+	.bind_iommufd = vfio_iommufd_emulated_bind,
+	.unbind_iommufd = vfio_iommufd_emulated_unbind,
+	.attach_ioas = vfio_iommufd_emulated_attach_ioas,
 };
 
 struct mdev_driver vfio_ccw_mdev_driver = {
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 0b4cc8c597ae67..bb7776d207924f 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -1789,6 +1789,9 @@ static const struct vfio_device_ops vfio_ap_matrix_dev_ops = {
 	.close_device = vfio_ap_mdev_close_device,
 	.ioctl = vfio_ap_mdev_ioctl,
 	.dma_unmap = vfio_ap_mdev_dma_unmap,
+	.bind_iommufd = vfio_iommufd_emulated_bind,
+	.unbind_iommufd = vfio_iommufd_emulated_unbind,
+	.attach_ioas = vfio_iommufd_emulated_attach_ioas,
 };
 
 static struct mdev_driver vfio_ap_matrix_driver = {
diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
index 8772dad6808539..0388f2e33447eb 100644
--- a/drivers/vfio/container.c
+++ b/drivers/vfio/container.c
@@ -540,113 +540,45 @@ void vfio_group_unuse_container(struct vfio_group *group)
 	fput(group->opened_file);
 }
 
-/*
- * Pin contiguous user pages and return their associated host pages for local
- * domain only.
- * @device [in]  : device
- * @iova [in]    : starting IOVA of user pages to be pinned.
- * @npage [in]   : count of pages to be pinned.  This count should not
- *		   be greater than VFIO_PIN_PAGES_MAX_ENTRIES.
- * @prot [in]    : protection flags
- * @pages[out]   : array of host pages
- * Return error or number of pages pinned.
- *
- * A driver may only call this function if the vfio_device was created
- * by vfio_register_emulated_iommu_dev().
- */
-int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova,
-		   int npage, int prot, struct page **pages)
+int vfio_container_pin_pages(struct vfio_container *container,
+			     struct iommu_group *iommu_group, dma_addr_t iova,
+			     int npage, int prot, struct page **pages)
 {
-	struct vfio_container *container;
-	struct vfio_group *group = device->group;
-	struct vfio_iommu_driver *driver;
-	int ret;
-
-	if (!pages || !npage || !vfio_assert_device_open(device))
-		return -EINVAL;
+	/* group->container cannot change while a vfio device is open */
+	struct vfio_iommu_driver *driver = container->iommu_driver;
 
 	if (npage > VFIO_PIN_PAGES_MAX_ENTRIES)
 		return -E2BIG;
 
 	/* group->container cannot change while a vfio device is open */
-	container = group->container;
 	driver = container->iommu_driver;
-	if (likely(driver && driver->ops->pin_pages))
-		ret = driver->ops->pin_pages(container->iommu_data,
-					     group->iommu_group, iova,
-					     npage, prot, pages);
-	else
-		ret = -ENOTTY;
-
-	return ret;
+	if (unlikely(!driver || !driver->ops->pin_pages))
+		return -ENOTTY;
+	return driver->ops->pin_pages(container->iommu_data, iommu_group, iova,
+				      npage, prot, pages);
 }
-EXPORT_SYMBOL(vfio_pin_pages);
 
-/*
- * Unpin contiguous host pages for local domain only.
- * @device [in]  : device
- * @iova [in]    : starting address of user pages to be unpinned.
- * @npage [in]   : count of pages to be unpinned.  This count should not
- *                 be greater than VFIO_PIN_PAGES_MAX_ENTRIES.
- */
-void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova, int npage)
+void vfio_container_unpin_pages(struct vfio_container *container,
+				dma_addr_t iova, int npage)
 {
-	struct vfio_container *container;
-	struct vfio_iommu_driver *driver;
-
 	if (WARN_ON(npage <= 0 || npage > VFIO_PIN_PAGES_MAX_ENTRIES))
 		return;
 
-	if (WARN_ON(!vfio_assert_device_open(device)))
-		return;
-
-	/* group->container cannot change while a vfio device is open */
-	container = device->group->container;
-	driver = container->iommu_driver;
-
-	driver->ops->unpin_pages(container->iommu_data, iova, npage);
+	container->iommu_driver->ops->unpin_pages(container->iommu_data, iova,
+						  npage);
 }
-EXPORT_SYMBOL(vfio_unpin_pages);
 
-/*
- * This interface allows the CPUs to perform some sort of virtual DMA on
- * behalf of the device.
- *
- * CPUs read/write from/into a range of IOVAs pointing to user space memory
- * into/from a kernel buffer.
- *
- * As the read/write of user space memory is conducted via the CPUs and is
- * not a real device DMA, it is not necessary to pin the user space memory.
- *
- * @device [in]		: VFIO device
- * @iova [in]		: base IOVA of a user space buffer
- * @data [in]		: pointer to kernel buffer
- * @len [in]		: kernel buffer length
- * @write		: indicate read or write
- * Return error code on failure or 0 on success.
- */
-int vfio_dma_rw(struct vfio_device *device, dma_addr_t iova, void *data,
-		size_t len, bool write)
+int vfio_container_dma_rw(struct vfio_container *container, dma_addr_t iova,
+			  void *data, size_t len, bool write)
 {
-	struct vfio_container *container;
-	struct vfio_iommu_driver *driver;
-	int ret = 0;
-
-	if (!data || len <= 0 || !vfio_assert_device_open(device))
-		return -EINVAL;
-
 	/* group->container cannot change while a vfio device is open */
-	container = device->group->container;
-	driver = container->iommu_driver;
+	struct vfio_iommu_driver *driver = container->iommu_driver;
 
-	if (likely(driver && driver->ops->dma_rw))
-		ret = driver->ops->dma_rw(container->iommu_data,
-					  iova, data, len, write);
-	else
-		ret = -ENOTTY;
-	return ret;
+	if (unlikely(!driver || !driver->ops->dma_rw))
+		return -ENOTTY;
+	return driver->ops->dma_rw(container->iommu_data, iova, data, len,
+				   write);
 }
-EXPORT_SYMBOL(vfio_dma_rw);
 
 int __init vfio_container_init(void)
 {
diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
index 8280bb32ee677c..40eb6931ab2321 100644
--- a/drivers/vfio/iommufd.c
+++ b/drivers/vfio/iommufd.c
@@ -102,3 +102,60 @@ int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id)
 	return iommufd_device_attach(vdev->iommufd_device, pt_id, flags);
 }
 EXPORT_SYMBOL_GPL(vfio_iommufd_physical_attach_ioas);
+
+/*
+ * The emulated standard ops mean that vfio_device is going to use the
+ * "mdev path" and will call vfio_pin_pages()/vfio_dma_rw(). Drivers using this
+ * ops set should call vfio_register_emulated_iommu_dev().
+ */
+
+static void vfio_emulated_unmap(void *data, unsigned long iova,
+				unsigned long length)
+{
+	struct vfio_device *vdev = data;
+
+	vdev->ops->dma_unmap(vdev, iova, length);
+}
+
+static const struct iommufd_access_ops vfio_user_ops = {
+	.unmap = vfio_emulated_unmap,
+};
+
+int vfio_iommufd_emulated_bind(struct vfio_device *vdev,
+			       struct iommufd_ctx *ictx, u32 *out_device_id)
+{
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	vdev->iommufd_ictx = ictx;
+	iommufd_ctx_get(ictx);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_emulated_bind);
+
+void vfio_iommufd_emulated_unbind(struct vfio_device *vdev)
+{
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	if (vdev->iommufd_access) {
+		iommufd_access_destroy(vdev->iommufd_access);
+		vdev->iommufd_access = NULL;
+	}
+	iommufd_ctx_put(vdev->iommufd_ictx);
+	vdev->iommufd_ictx = NULL;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_emulated_unbind);
+
+int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id)
+{
+	struct iommufd_access *user;
+
+	lockdep_assert_held(&vdev->dev_set->lock);
+
+	user = iommufd_access_create(vdev->iommufd_ictx, *pt_id, &vfio_user_ops,
+				     vdev);
+	if (IS_ERR(user))
+		return PTR_ERR(user);
+	vdev->iommufd_access = user;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_iommufd_emulated_attach_ioas);
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index 809f2e8523968e..d57a08afb5cf5c 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -111,8 +111,6 @@ struct vfio_iommu_driver {
 int vfio_register_iommu_driver(const struct vfio_iommu_driver_ops *ops);
 void vfio_unregister_iommu_driver(const struct vfio_iommu_driver_ops *ops);
 
-bool vfio_assert_device_open(struct vfio_device *device);
-
 struct vfio_container *vfio_container_from_file(struct file *filep);
 int vfio_group_use_container(struct vfio_group *group);
 void vfio_group_unuse_container(struct vfio_group *group);
@@ -121,6 +119,14 @@ int vfio_container_attach_group(struct vfio_container *container,
 void vfio_group_detach_container(struct vfio_group *group);
 void vfio_device_container_register(struct vfio_device *device);
 void vfio_device_container_unregister(struct vfio_device *device);
+int vfio_container_pin_pages(struct vfio_container *container,
+			     struct iommu_group *iommu_group, dma_addr_t iova,
+			     int npage, int prot, struct page **pages);
+void vfio_container_unpin_pages(struct vfio_container *container,
+				dma_addr_t iova, int npage);
+int vfio_container_dma_rw(struct vfio_container *container, dma_addr_t iova,
+			  void *data, size_t len, bool write);
+
 int __init vfio_container_init(void);
 void vfio_container_cleanup(void);
 
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index bfbda04af1ffda..9b837efbddb6db 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -766,7 +766,7 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
 static const struct file_operations vfio_device_fops;
 
 /* true if the vfio_device has open_device() called but not close_device() */
-bool vfio_assert_device_open(struct vfio_device *device)
+static bool vfio_assert_device_open(struct vfio_device *device)
 {
 	return !WARN_ON_ONCE(!READ_ONCE(device->open_count));
 }
@@ -1861,6 +1861,114 @@ int vfio_set_irqs_validate_and_prepare(struct vfio_irq_set *hdr, int num_irqs,
 }
 EXPORT_SYMBOL(vfio_set_irqs_validate_and_prepare);
 
+/*
+ * Pin contiguous user pages and return their associated host pages for local
+ * domain only.
+ * @device [in]  : device
+ * @iova [in]    : starting IOVA of user pages to be pinned.
+ * @npage [in]   : count of pages to be pinned.  This count should not
+ *		   be greater than VFIO_PIN_PAGES_MAX_ENTRIES.
+ * @prot [in]    : protection flags
+ * @pages[out]   : array of host pages
+ * Return error or number of pages pinned.
+ *
+ * A driver may only call this function if the vfio_device was created
+ * by vfio_register_emulated_iommu_dev() due to vfio_container_pin_pages().
+ */
+int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova,
+		   int npage, int prot, struct page **pages)
+{
+	if (!pages || !npage || !vfio_assert_device_open(device))
+		return -EINVAL;
+	if (device->group->container)
+		return vfio_container_pin_pages(device->group->container,
+						device->group->iommu_group,
+						iova, npage, prot, pages);
+	if (device->iommufd_access) {
+		int ret;
+
+		if (iova > ULONG_MAX)
+			return -EINVAL;
+		ret = iommufd_access_pin_pages(
+			device->iommufd_access, iova, npage * PAGE_SIZE, pages,
+			(prot & IOMMU_WRITE) ? IOMMUFD_ACCESS_RW_WRITE : 0);
+		if (ret)
+			return ret;
+		return npage;
+	}
+	return -EINVAL;
+}
+EXPORT_SYMBOL(vfio_pin_pages);
+
+/*
+ * Unpin contiguous host pages for local domain only.
+ * @device [in]  : device
+ * @iova [in]    : starting address of user pages to be unpinned.
+ * @npage [in]   : count of pages to be unpinned.  This count should not
+ *                 be greater than VFIO_PIN_PAGES_MAX_ENTRIES.
+ */
+void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova, int npage)
+{
+	if (WARN_ON(!vfio_assert_device_open(device)))
+		return;
+
+	if (device->group->container) {
+		vfio_container_unpin_pages(device->group->container, iova,
+					   npage);
+	} else if (device->iommufd_access) {
+		if (WARN_ON(iova > ULONG_MAX))
+			return;
+		iommufd_access_unpin_pages(device->iommufd_access, iova,
+					   npage * PAGE_SIZE);
+	}
+}
+EXPORT_SYMBOL(vfio_unpin_pages);
+
+/*
+ * This interface allows the CPUs to perform some sort of virtual DMA on
+ * behalf of the device.
+ *
+ * CPUs read/write from/into a range of IOVAs pointing to user space memory
+ * into/from a kernel buffer.
+ *
+ * As the read/write of user space memory is conducted via the CPUs and is
+ * not a real device DMA, it is not necessary to pin the user space memory.
+ *
+ * @device [in]		: VFIO device
+ * @iova [in]		: base IOVA of a user space buffer
+ * @data [in]		: pointer to kernel buffer
+ * @len [in]		: kernel buffer length
+ * @write		: indicate read or write
+ * Return error code on failure or 0 on success.
+ */
+int vfio_dma_rw(struct vfio_device *device, dma_addr_t iova, void *data,
+		size_t len, bool write)
+{
+	if (!data || len <= 0 || !vfio_assert_device_open(device))
+		return -EINVAL;
+
+	if (device->group->container)
+		return vfio_container_dma_rw(device->group->container, iova,
+					     data, len, write);
+
+	if (device->iommufd_access) {
+		unsigned int flags = 0;
+
+		if (iova > ULONG_MAX)
+			return -EINVAL;
+
+		/* VFIO historically tries to auto-detect a kthread */
+		if (!current->mm)
+			flags |= IOMMUFD_ACCESS_RW_KTHREAD;
+		if (write)
+			flags |= IOMMUFD_ACCESS_RW_WRITE;
+		return iommufd_access_rw(device->iommufd_access, iova, data,
+					 len, flags);
+	}
+	return -EINVAL;
+}
+EXPORT_SYMBOL(vfio_dma_rw);
+
 /*
  * Module/class support
  */
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index a7fc4d747dc226..d5f84f98c0fa8f 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -19,6 +19,7 @@
 struct kvm;
 struct iommufd_ctx;
 struct iommufd_device;
+struct iommufd_access;
 
 /*
  * VFIO devices can be placed in a set, this allows all devices to share this
@@ -56,8 +57,10 @@ struct vfio_device {
 	struct completion comp;
 	struct list_head group_next;
 	struct list_head iommu_entry;
+	struct iommufd_access *iommufd_access;
 #if IS_ENABLED(CONFIG_IOMMUFD)
 	struct iommufd_device *iommufd_device;
+	struct iommufd_ctx *iommufd_ictx;
 	bool iommufd_attached;
 #endif
 };
@@ -111,6 +114,10 @@ int vfio_iommufd_physical_bind(struct vfio_device *vdev,
 			       struct iommufd_ctx *ictx, u32 *out_device_id);
 void vfio_iommufd_physical_unbind(struct vfio_device *vdev);
 int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
+int vfio_iommufd_emulated_bind(struct vfio_device *vdev,
+			       struct iommufd_ctx *ictx, u32 *out_device_id);
+void vfio_iommufd_emulated_unbind(struct vfio_device *vdev);
+int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
 #else
 #define vfio_iommufd_physical_bind                                      \
 	((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx,   \
@@ -119,6 +126,13 @@ int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
 	((void (*)(struct vfio_device *vdev)) NULL)
 #define vfio_iommufd_physical_attach_ioas \
 	((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL)
+#define vfio_iommufd_emulated_bind                                      \
+	((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx,   \
+		  u32 *out_device_id)) NULL)
+#define vfio_iommufd_emulated_unbind \
+	((void (*)(struct vfio_device *vdev)) NULL)
+#define vfio_iommufd_emulated_attach_ioas \
+	((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL)
 #endif
 
 /**
-- 
2.38.0

[PATCH 09/10] vfio: Make vfio_container optionally compiled

[Jason Gunthorpe]

Add a kconfig CONFIG_VFIO_CONTAINER that controls compiling the container
code. If 'n' then only iommufd will provide the container service. All the
support for vfio iommu drivers, including type1, will not be built.

This allows a compilation check that no inappropriate dependencies between
the device/group and container have been created.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/Kconfig  | 37 ++++++++++++++++--------
 drivers/vfio/Makefile |  4 +--
 drivers/vfio/vfio.h   | 65 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 92 insertions(+), 14 deletions(-)

diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
index 1118d322eec97d..d384419d151dda 100644
--- a/drivers/vfio/Kconfig
+++ b/drivers/vfio/Kconfig
@@ -3,8 +3,8 @@ menuconfig VFIO
 	tristate "VFIO Non-Privileged userspace driver framework"
 	select IOMMU_API
 	depends on IOMMUFD || !IOMMUFD
-	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM || ARM64)
 	select INTERVAL_TREE
+	select VFIO_CONTAINER if IOMMUFD=n
 	help
 	  VFIO provides a framework for secure userspace device drivers.
 	  See Documentation/driver-api/vfio.rst for more details.
@@ -12,25 +12,27 @@ menuconfig VFIO
 	  If you don't know what to do here, say N.
 
 if VFIO
+config VFIO_CONTAINER
+	bool "Support for the VFIO container /dev/vfio/vfio"
+	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM || ARM64)
+	default y
+	help
+	  The VFIO container is the classic interface to VFIO for establishing
+	  mappings. If N is selected here then IOMMUFD must be used the manage
+	  the mappings.
+
+	  Unless testing IOMMUFD say Y here.
+
+if VFIO_CONTAINER
 config VFIO_IOMMU_TYPE1
 	tristate
-	default n
+	default MMU && (X86 || S390 || ARM || ARM64)
 
 config VFIO_IOMMU_SPAPR_TCE
 	tristate
 	depends on SPAPR_TCE_IOMMU
 	default VFIO
 
-config VFIO_SPAPR_EEH
-	tristate
-	depends on EEH && VFIO_IOMMU_SPAPR_TCE
-	default VFIO
-
-config VFIO_VIRQFD
-	tristate
-	select EVENTFD
-	default n
-
 config VFIO_NOIOMMU
 	bool "VFIO No-IOMMU support"
 	help
@@ -44,6 +46,17 @@ config VFIO_NOIOMMU
 	  this mode since there is no IOMMU to provide DMA translation.
 
 	  If you don't know what to do here, say N.
+endif
+
+config VFIO_SPAPR_EEH
+	tristate
+	depends on EEH && VFIO_IOMMU_SPAPR_TCE
+	default VFIO
+
+config VFIO_VIRQFD
+	tristate
+	select EVENTFD
+	default n
 
 source "drivers/vfio/pci/Kconfig"
 source "drivers/vfio/platform/Kconfig"
diff --git a/drivers/vfio/Makefile b/drivers/vfio/Makefile
index 3863922529ef20..b953517dc70f99 100644
--- a/drivers/vfio/Makefile
+++ b/drivers/vfio/Makefile
@@ -4,9 +4,9 @@ vfio_virqfd-y := virqfd.o
 obj-$(CONFIG_VFIO) += vfio.o
 
 vfio-y += vfio_main.o \
-	  iova_bitmap.o \
-	  container.o
+	  iova_bitmap.o
 vfio-$(CONFIG_IOMMUFD) += iommufd.o
+vfio-$(CONFIG_VFIO_CONTAINER) += container.o
 
 obj-$(CONFIG_VFIO_VIRQFD) += vfio_virqfd.o
 obj-$(CONFIG_VFIO_IOMMU_TYPE1) += vfio_iommu_type1.o
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index d57a08afb5cf5c..3378714a746274 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -55,7 +55,9 @@ struct vfio_group {
 	struct list_head		device_list;
 	struct mutex			device_lock;
 	struct list_head		vfio_next;
+#if IS_ENABLED(CONFIG_VFIO_CONTAINER)
 	struct list_head		container_next;
+#endif
 	enum vfio_group_type		type;
 	struct mutex			group_lock;
 	struct kvm			*kvm;
@@ -64,6 +66,7 @@ struct vfio_group {
 	struct iommufd_ctx		*iommufd;
 };
 
+#if IS_ENABLED(CONFIG_VFIO_CONTAINER)
 /* events for the backend driver notify callback */
 enum vfio_iommu_notify_type {
 	VFIO_IOMMU_CONTAINER_CLOSE = 0,
@@ -129,6 +132,68 @@ int vfio_container_dma_rw(struct vfio_container *container, dma_addr_t iova,
 
 int __init vfio_container_init(void);
 void vfio_container_cleanup(void);
+#else
+static inline struct vfio_container *
+vfio_container_from_file(struct file *filep)
+{
+	return NULL;
+}
+
+static inline int vfio_group_use_container(struct vfio_group *group)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void vfio_group_unuse_container(struct vfio_group *group)
+{
+}
+
+static inline int vfio_container_attach_group(struct vfio_container *container,
+					      struct vfio_group *group)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void vfio_group_detach_container(struct vfio_group *group)
+{
+}
+
+static inline void vfio_device_container_register(struct vfio_device *device)
+{
+}
+
+static inline void vfio_device_container_unregister(struct vfio_device *device)
+{
+}
+
+static inline int vfio_container_pin_pages(struct vfio_container *container,
+					   struct iommu_group *iommu_group,
+					   dma_addr_t iova, int npage, int prot,
+					   struct page **pages)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void vfio_container_unpin_pages(struct vfio_container *container,
+					      dma_addr_t iova, int npage)
+{
+}
+
+static inline int vfio_container_dma_rw(struct vfio_container *container,
+					dma_addr_t iova, void *data, size_t len,
+					bool write)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline int vfio_container_init(void)
+{
+	return 0;
+}
+static inline void vfio_container_cleanup(void)
+{
+}
+#endif
 
 #if IS_ENABLED(CONFIG_IOMMUFD)
 int vfio_iommufd_bind(struct vfio_device *device, struct iommufd_ctx *ictx);
-- 
2.38.0

[PATCH 09/10] vfio: Make vfio_container optionally compiled

[Jason Gunthorpe]

Add a kconfig CONFIG_VFIO_CONTAINER that controls compiling the container
code. If 'n' then only iommufd will provide the container service. All the
support for vfio iommu drivers, including type1, will not be built.

This allows a compilation check that no inappropriate dependencies between
the device/group and container have been created.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/Kconfig  | 37 ++++++++++++++++--------
 drivers/vfio/Makefile |  4 +--
 drivers/vfio/vfio.h   | 65 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 92 insertions(+), 14 deletions(-)

diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
index 1118d322eec97d..d384419d151dda 100644
--- a/drivers/vfio/Kconfig
+++ b/drivers/vfio/Kconfig
@@ -3,8 +3,8 @@ menuconfig VFIO
 	tristate "VFIO Non-Privileged userspace driver framework"
 	select IOMMU_API
 	depends on IOMMUFD || !IOMMUFD
-	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM || ARM64)
 	select INTERVAL_TREE
+	select VFIO_CONTAINER if IOMMUFD=n
 	help
 	  VFIO provides a framework for secure userspace device drivers.
 	  See Documentation/driver-api/vfio.rst for more details.
@@ -12,25 +12,27 @@ menuconfig VFIO
 	  If you don't know what to do here, say N.
 
 if VFIO
+config VFIO_CONTAINER
+	bool "Support for the VFIO container /dev/vfio/vfio"
+	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM || ARM64)
+	default y
+	help
+	  The VFIO container is the classic interface to VFIO for establishing
+	  mappings. If N is selected here then IOMMUFD must be used the manage
+	  the mappings.
+
+	  Unless testing IOMMUFD say Y here.
+
+if VFIO_CONTAINER
 config VFIO_IOMMU_TYPE1
 	tristate
-	default n
+	default MMU && (X86 || S390 || ARM || ARM64)
 
 config VFIO_IOMMU_SPAPR_TCE
 	tristate
 	depends on SPAPR_TCE_IOMMU
 	default VFIO
 
-config VFIO_SPAPR_EEH
-	tristate
-	depends on EEH && VFIO_IOMMU_SPAPR_TCE
-	default VFIO
-
-config VFIO_VIRQFD
-	tristate
-	select EVENTFD
-	default n
-
 config VFIO_NOIOMMU
 	bool "VFIO No-IOMMU support"
 	help
@@ -44,6 +46,17 @@ config VFIO_NOIOMMU
 	  this mode since there is no IOMMU to provide DMA translation.
 
 	  If you don't know what to do here, say N.
+endif
+
+config VFIO_SPAPR_EEH
+	tristate
+	depends on EEH && VFIO_IOMMU_SPAPR_TCE
+	default VFIO
+
+config VFIO_VIRQFD
+	tristate
+	select EVENTFD
+	default n
 
 source "drivers/vfio/pci/Kconfig"
 source "drivers/vfio/platform/Kconfig"
diff --git a/drivers/vfio/Makefile b/drivers/vfio/Makefile
index 3863922529ef20..b953517dc70f99 100644
--- a/drivers/vfio/Makefile
+++ b/drivers/vfio/Makefile
@@ -4,9 +4,9 @@ vfio_virqfd-y := virqfd.o
 obj-$(CONFIG_VFIO) += vfio.o
 
 vfio-y += vfio_main.o \
-	  iova_bitmap.o \
-	  container.o
+	  iova_bitmap.o
 vfio-$(CONFIG_IOMMUFD) += iommufd.o
+vfio-$(CONFIG_VFIO_CONTAINER) += container.o
 
 obj-$(CONFIG_VFIO_VIRQFD) += vfio_virqfd.o
 obj-$(CONFIG_VFIO_IOMMU_TYPE1) += vfio_iommu_type1.o
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index d57a08afb5cf5c..3378714a746274 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -55,7 +55,9 @@ struct vfio_group {
 	struct list_head		device_list;
 	struct mutex			device_lock;
 	struct list_head		vfio_next;
+#if IS_ENABLED(CONFIG_VFIO_CONTAINER)
 	struct list_head		container_next;
+#endif
 	enum vfio_group_type		type;
 	struct mutex			group_lock;
 	struct kvm			*kvm;
@@ -64,6 +66,7 @@ struct vfio_group {
 	struct iommufd_ctx		*iommufd;
 };
 
+#if IS_ENABLED(CONFIG_VFIO_CONTAINER)
 /* events for the backend driver notify callback */
 enum vfio_iommu_notify_type {
 	VFIO_IOMMU_CONTAINER_CLOSE = 0,
@@ -129,6 +132,68 @@ int vfio_container_dma_rw(struct vfio_container *container, dma_addr_t iova,
 
 int __init vfio_container_init(void);
 void vfio_container_cleanup(void);
+#else
+static inline struct vfio_container *
+vfio_container_from_file(struct file *filep)
+{
+	return NULL;
+}
+
+static inline int vfio_group_use_container(struct vfio_group *group)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void vfio_group_unuse_container(struct vfio_group *group)
+{
+}
+
+static inline int vfio_container_attach_group(struct vfio_container *container,
+					      struct vfio_group *group)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void vfio_group_detach_container(struct vfio_group *group)
+{
+}
+
+static inline void vfio_device_container_register(struct vfio_device *device)
+{
+}
+
+static inline void vfio_device_container_unregister(struct vfio_device *device)
+{
+}
+
+static inline int vfio_container_pin_pages(struct vfio_container *container,
+					   struct iommu_group *iommu_group,
+					   dma_addr_t iova, int npage, int prot,
+					   struct page **pages)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void vfio_container_unpin_pages(struct vfio_container *container,
+					      dma_addr_t iova, int npage)
+{
+}
+
+static inline int vfio_container_dma_rw(struct vfio_container *container,
+					dma_addr_t iova, void *data, size_t len,
+					bool write)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline int vfio_container_init(void)
+{
+	return 0;
+}
+static inline void vfio_container_cleanup(void)
+{
+}
+#endif
 
 #if IS_ENABLED(CONFIG_IOMMUFD)
 int vfio_iommufd_bind(struct vfio_device *device, struct iommufd_ctx *ictx);
-- 
2.38.0

[Intel-gfx] [PATCH 09/10] vfio: Make vfio_container optionally compiled

[Jason Gunthorpe]

Add a kconfig CONFIG_VFIO_CONTAINER that controls compiling the container
code. If 'n' then only iommufd will provide the container service. All the
support for vfio iommu drivers, including type1, will not be built.

This allows a compilation check that no inappropriate dependencies between
the device/group and container have been created.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/vfio/Kconfig  | 37 ++++++++++++++++--------
 drivers/vfio/Makefile |  4 +--
 drivers/vfio/vfio.h   | 65 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 92 insertions(+), 14 deletions(-)

diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
index 1118d322eec97d..d384419d151dda 100644
--- a/drivers/vfio/Kconfig
+++ b/drivers/vfio/Kconfig
@@ -3,8 +3,8 @@ menuconfig VFIO
 	tristate "VFIO Non-Privileged userspace driver framework"
 	select IOMMU_API
 	depends on IOMMUFD || !IOMMUFD
-	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM || ARM64)
 	select INTERVAL_TREE
+	select VFIO_CONTAINER if IOMMUFD=n
 	help
 	  VFIO provides a framework for secure userspace device drivers.
 	  See Documentation/driver-api/vfio.rst for more details.
@@ -12,25 +12,27 @@ menuconfig VFIO
 	  If you don't know what to do here, say N.
 
 if VFIO
+config VFIO_CONTAINER
+	bool "Support for the VFIO container /dev/vfio/vfio"
+	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM || ARM64)
+	default y
+	help
+	  The VFIO container is the classic interface to VFIO for establishing
+	  mappings. If N is selected here then IOMMUFD must be used the manage
+	  the mappings.
+
+	  Unless testing IOMMUFD say Y here.
+
+if VFIO_CONTAINER
 config VFIO_IOMMU_TYPE1
 	tristate
-	default n
+	default MMU && (X86 || S390 || ARM || ARM64)
 
 config VFIO_IOMMU_SPAPR_TCE
 	tristate
 	depends on SPAPR_TCE_IOMMU
 	default VFIO
 
-config VFIO_SPAPR_EEH
-	tristate
-	depends on EEH && VFIO_IOMMU_SPAPR_TCE
-	default VFIO
-
-config VFIO_VIRQFD
-	tristate
-	select EVENTFD
-	default n
-
 config VFIO_NOIOMMU
 	bool "VFIO No-IOMMU support"
 	help
@@ -44,6 +46,17 @@ config VFIO_NOIOMMU
 	  this mode since there is no IOMMU to provide DMA translation.
 
 	  If you don't know what to do here, say N.
+endif
+
+config VFIO_SPAPR_EEH
+	tristate
+	depends on EEH && VFIO_IOMMU_SPAPR_TCE
+	default VFIO
+
+config VFIO_VIRQFD
+	tristate
+	select EVENTFD
+	default n
 
 source "drivers/vfio/pci/Kconfig"
 source "drivers/vfio/platform/Kconfig"
diff --git a/drivers/vfio/Makefile b/drivers/vfio/Makefile
index 3863922529ef20..b953517dc70f99 100644
--- a/drivers/vfio/Makefile
+++ b/drivers/vfio/Makefile
@@ -4,9 +4,9 @@ vfio_virqfd-y := virqfd.o
 obj-$(CONFIG_VFIO) += vfio.o
 
 vfio-y += vfio_main.o \
-	  iova_bitmap.o \
-	  container.o
+	  iova_bitmap.o
 vfio-$(CONFIG_IOMMUFD) += iommufd.o
+vfio-$(CONFIG_VFIO_CONTAINER) += container.o
 
 obj-$(CONFIG_VFIO_VIRQFD) += vfio_virqfd.o
 obj-$(CONFIG_VFIO_IOMMU_TYPE1) += vfio_iommu_type1.o
diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
index d57a08afb5cf5c..3378714a746274 100644
--- a/drivers/vfio/vfio.h
+++ b/drivers/vfio/vfio.h
@@ -55,7 +55,9 @@ struct vfio_group {
 	struct list_head		device_list;
 	struct mutex			device_lock;
 	struct list_head		vfio_next;
+#if IS_ENABLED(CONFIG_VFIO_CONTAINER)
 	struct list_head		container_next;
+#endif
 	enum vfio_group_type		type;
 	struct mutex			group_lock;
 	struct kvm			*kvm;
@@ -64,6 +66,7 @@ struct vfio_group {
 	struct iommufd_ctx		*iommufd;
 };
 
+#if IS_ENABLED(CONFIG_VFIO_CONTAINER)
 /* events for the backend driver notify callback */
 enum vfio_iommu_notify_type {
 	VFIO_IOMMU_CONTAINER_CLOSE = 0,
@@ -129,6 +132,68 @@ int vfio_container_dma_rw(struct vfio_container *container, dma_addr_t iova,
 
 int __init vfio_container_init(void);
 void vfio_container_cleanup(void);
+#else
+static inline struct vfio_container *
+vfio_container_from_file(struct file *filep)
+{
+	return NULL;
+}
+
+static inline int vfio_group_use_container(struct vfio_group *group)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void vfio_group_unuse_container(struct vfio_group *group)
+{
+}
+
+static inline int vfio_container_attach_group(struct vfio_container *container,
+					      struct vfio_group *group)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void vfio_group_detach_container(struct vfio_group *group)
+{
+}
+
+static inline void vfio_device_container_register(struct vfio_device *device)
+{
+}
+
+static inline void vfio_device_container_unregister(struct vfio_device *device)
+{
+}
+
+static inline int vfio_container_pin_pages(struct vfio_container *container,
+					   struct iommu_group *iommu_group,
+					   dma_addr_t iova, int npage, int prot,
+					   struct page **pages)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void vfio_container_unpin_pages(struct vfio_container *container,
+					      dma_addr_t iova, int npage)
+{
+}
+
+static inline int vfio_container_dma_rw(struct vfio_container *container,
+					dma_addr_t iova, void *data, size_t len,
+					bool write)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline int vfio_container_init(void)
+{
+	return 0;
+}
+static inline void vfio_container_cleanup(void)
+{
+}
+#endif
 
 #if IS_ENABLED(CONFIG_IOMMUFD)
 int vfio_iommufd_bind(struct vfio_device *device, struct iommufd_ctx *ictx);
-- 
2.38.0

[PATCH 10/10] iommufd: Allow iommufd to supply /dev/vfio/vfio

[Jason Gunthorpe]

If the VFIO container is compiled out, give a kconfig option for iommufd
to provide the miscdev node with the same name and permissions as vfio
uses.

The compatibility node supports the same ioctls as VFIO and automatically
enables the VFIO compatible pinned page accounting mode.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/iommu/iommufd/Kconfig | 12 ++++++++++++
 drivers/iommu/iommufd/main.c  | 35 ++++++++++++++++++++++++++++++++---
 2 files changed, 44 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/iommufd/Kconfig b/drivers/iommu/iommufd/Kconfig
index f0a2012234fa09..afc83b7575cce6 100644
--- a/drivers/iommu/iommufd/Kconfig
+++ b/drivers/iommu/iommufd/Kconfig
@@ -14,6 +14,18 @@ config IOMMUFD
 	  If you don't know what to do here, say N.
 
 if IOMMUFD
+config IOMMUFD_VFIO_CONTAINER
+	bool "IOMMUFD provides the VFIO container /dev/vfio/vfio"
+	depends on VFIO && !VFIO_CONTAINER
+	default VFIO && !VFIO_CONTAINER
+	help
+	  IOMMUFD will provide /dev/vfio/vfio instead of VFIO. This relies on
+	  IOMMUFD providing compatibility emulation to give the same ioctls.
+	  It provides an option to build a kernel with legacy VFIO components
+	  removed.
+
+	  Unless testing IOMMUFD say N here.
+
 config IOMMUFD_TEST
 	bool "IOMMU Userspace API Test support"
 	depends on RUNTIME_TESTING_MENU
diff --git a/drivers/iommu/iommufd/main.c b/drivers/iommu/iommufd/main.c
index 8a31c1a14cdd53..19db81fbf7f08f 100644
--- a/drivers/iommu/iommufd/main.c
+++ b/drivers/iommu/iommufd/main.c
@@ -24,6 +24,7 @@
 #include <uapi/linux/iommufd.h>
 #include <linux/iommufd.h>
 
+#include "io_pagetable.h"
 #include "iommufd_private.h"
 #include "iommufd_test.h"
 
@@ -31,6 +32,7 @@ struct iommufd_object_ops {
 	void (*destroy)(struct iommufd_object *obj);
 };
 static struct iommufd_object_ops iommufd_object_ops[];
+static struct miscdevice vfio_misc_dev;
 
 struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx,
 					     size_t size,
@@ -167,6 +169,13 @@ static int iommufd_fops_open(struct inode *inode, struct file *filp)
 	if (!ictx)
 		return -ENOMEM;
 
+	/*
+	 * For compatibility with VFIO when /dev/vfio/vfio is opened we default
+	 * to the same rlimit accounting as vfio uses.
+	 */
+	if (filp->private_data == &vfio_misc_dev)
+		ictx->account_mode = IOPT_PAGES_ACCOUNT_MM;
+
 	xa_init_flags(&ictx->objects, XA_FLAGS_ALLOC1 | XA_FLAGS_ACCOUNT);
 	ictx->file = filp;
 	filp->private_data = ictx;
@@ -392,26 +401,46 @@ static struct miscdevice iommu_misc_dev = {
 	.mode = 0660,
 };
 
+
+static struct miscdevice vfio_misc_dev = {
+	.minor = VFIO_MINOR,
+	.name = "vfio",
+	.fops = &iommufd_fops,
+	.nodename = "vfio/vfio",
+	.mode = 0666,
+};
+
 static int __init iommufd_init(void)
 {
 	int ret;
 
 	ret = misc_register(&iommu_misc_dev);
-	if (ret) {
-		pr_err("Failed to register misc device\n");
+	if (ret)
 		return ret;
-	}
 
+	if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER)) {
+		ret = misc_register(&vfio_misc_dev);
+		if (ret)
+			goto err_misc;
+	}
 	return 0;
+err_misc:
+	misc_deregister(&iommu_misc_dev);
+	return ret;
 }
 
 static void __exit iommufd_exit(void)
 {
+	if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER))
+		misc_deregister(&vfio_misc_dev);
 	misc_deregister(&iommu_misc_dev);
 }
 
 module_init(iommufd_init);
 module_exit(iommufd_exit);
 
+#if IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER)
+MODULE_ALIAS_MISCDEV(VFIO_MINOR);
+#endif
 MODULE_DESCRIPTION("I/O Address Space Management for passthrough devices");
 MODULE_LICENSE("GPL");
-- 
2.38.0

[PATCH 10/10] iommufd: Allow iommufd to supply /dev/vfio/vfio

[Jason Gunthorpe]

If the VFIO container is compiled out, give a kconfig option for iommufd
to provide the miscdev node with the same name and permissions as vfio
uses.

The compatibility node supports the same ioctls as VFIO and automatically
enables the VFIO compatible pinned page accounting mode.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/iommu/iommufd/Kconfig | 12 ++++++++++++
 drivers/iommu/iommufd/main.c  | 35 ++++++++++++++++++++++++++++++++---
 2 files changed, 44 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/iommufd/Kconfig b/drivers/iommu/iommufd/Kconfig
index f0a2012234fa09..afc83b7575cce6 100644
--- a/drivers/iommu/iommufd/Kconfig
+++ b/drivers/iommu/iommufd/Kconfig
@@ -14,6 +14,18 @@ config IOMMUFD
 	  If you don't know what to do here, say N.
 
 if IOMMUFD
+config IOMMUFD_VFIO_CONTAINER
+	bool "IOMMUFD provides the VFIO container /dev/vfio/vfio"
+	depends on VFIO && !VFIO_CONTAINER
+	default VFIO && !VFIO_CONTAINER
+	help
+	  IOMMUFD will provide /dev/vfio/vfio instead of VFIO. This relies on
+	  IOMMUFD providing compatibility emulation to give the same ioctls.
+	  It provides an option to build a kernel with legacy VFIO components
+	  removed.
+
+	  Unless testing IOMMUFD say N here.
+
 config IOMMUFD_TEST
 	bool "IOMMU Userspace API Test support"
 	depends on RUNTIME_TESTING_MENU
diff --git a/drivers/iommu/iommufd/main.c b/drivers/iommu/iommufd/main.c
index 8a31c1a14cdd53..19db81fbf7f08f 100644
--- a/drivers/iommu/iommufd/main.c
+++ b/drivers/iommu/iommufd/main.c
@@ -24,6 +24,7 @@
 #include <uapi/linux/iommufd.h>
 #include <linux/iommufd.h>
 
+#include "io_pagetable.h"
 #include "iommufd_private.h"
 #include "iommufd_test.h"
 
@@ -31,6 +32,7 @@ struct iommufd_object_ops {
 	void (*destroy)(struct iommufd_object *obj);
 };
 static struct iommufd_object_ops iommufd_object_ops[];
+static struct miscdevice vfio_misc_dev;
 
 struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx,
 					     size_t size,
@@ -167,6 +169,13 @@ static int iommufd_fops_open(struct inode *inode, struct file *filp)
 	if (!ictx)
 		return -ENOMEM;
 
+	/*
+	 * For compatibility with VFIO when /dev/vfio/vfio is opened we default
+	 * to the same rlimit accounting as vfio uses.
+	 */
+	if (filp->private_data == &vfio_misc_dev)
+		ictx->account_mode = IOPT_PAGES_ACCOUNT_MM;
+
 	xa_init_flags(&ictx->objects, XA_FLAGS_ALLOC1 | XA_FLAGS_ACCOUNT);
 	ictx->file = filp;
 	filp->private_data = ictx;
@@ -392,26 +401,46 @@ static struct miscdevice iommu_misc_dev = {
 	.mode = 0660,
 };
 
+
+static struct miscdevice vfio_misc_dev = {
+	.minor = VFIO_MINOR,
+	.name = "vfio",
+	.fops = &iommufd_fops,
+	.nodename = "vfio/vfio",
+	.mode = 0666,
+};
+
 static int __init iommufd_init(void)
 {
 	int ret;
 
 	ret = misc_register(&iommu_misc_dev);
-	if (ret) {
-		pr_err("Failed to register misc device\n");
+	if (ret)
 		return ret;
-	}
 
+	if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER)) {
+		ret = misc_register(&vfio_misc_dev);
+		if (ret)
+			goto err_misc;
+	}
 	return 0;
+err_misc:
+	misc_deregister(&iommu_misc_dev);
+	return ret;
 }
 
 static void __exit iommufd_exit(void)
 {
+	if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER))
+		misc_deregister(&vfio_misc_dev);
 	misc_deregister(&iommu_misc_dev);
 }
 
 module_init(iommufd_init);
 module_exit(iommufd_exit);
 
+#if IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER)
+MODULE_ALIAS_MISCDEV(VFIO_MINOR);
+#endif
 MODULE_DESCRIPTION("I/O Address Space Management for passthrough devices");
 MODULE_LICENSE("GPL");
-- 
2.38.0

[Intel-gfx] [PATCH 10/10] iommufd: Allow iommufd to supply /dev/vfio/vfio

[Jason Gunthorpe]

If the VFIO container is compiled out, give a kconfig option for iommufd
to provide the miscdev node with the same name and permissions as vfio
uses.

The compatibility node supports the same ioctls as VFIO and automatically
enables the VFIO compatible pinned page accounting mode.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/iommu/iommufd/Kconfig | 12 ++++++++++++
 drivers/iommu/iommufd/main.c  | 35 ++++++++++++++++++++++++++++++++---
 2 files changed, 44 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/iommufd/Kconfig b/drivers/iommu/iommufd/Kconfig
index f0a2012234fa09..afc83b7575cce6 100644
--- a/drivers/iommu/iommufd/Kconfig
+++ b/drivers/iommu/iommufd/Kconfig
@@ -14,6 +14,18 @@ config IOMMUFD
 	  If you don't know what to do here, say N.
 
 if IOMMUFD
+config IOMMUFD_VFIO_CONTAINER
+	bool "IOMMUFD provides the VFIO container /dev/vfio/vfio"
+	depends on VFIO && !VFIO_CONTAINER
+	default VFIO && !VFIO_CONTAINER
+	help
+	  IOMMUFD will provide /dev/vfio/vfio instead of VFIO. This relies on
+	  IOMMUFD providing compatibility emulation to give the same ioctls.
+	  It provides an option to build a kernel with legacy VFIO components
+	  removed.
+
+	  Unless testing IOMMUFD say N here.
+
 config IOMMUFD_TEST
 	bool "IOMMU Userspace API Test support"
 	depends on RUNTIME_TESTING_MENU
diff --git a/drivers/iommu/iommufd/main.c b/drivers/iommu/iommufd/main.c
index 8a31c1a14cdd53..19db81fbf7f08f 100644
--- a/drivers/iommu/iommufd/main.c
+++ b/drivers/iommu/iommufd/main.c
@@ -24,6 +24,7 @@
 #include <uapi/linux/iommufd.h>
 #include <linux/iommufd.h>
 
+#include "io_pagetable.h"
 #include "iommufd_private.h"
 #include "iommufd_test.h"
 
@@ -31,6 +32,7 @@ struct iommufd_object_ops {
 	void (*destroy)(struct iommufd_object *obj);
 };
 static struct iommufd_object_ops iommufd_object_ops[];
+static struct miscdevice vfio_misc_dev;
 
 struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx,
 					     size_t size,
@@ -167,6 +169,13 @@ static int iommufd_fops_open(struct inode *inode, struct file *filp)
 	if (!ictx)
 		return -ENOMEM;
 
+	/*
+	 * For compatibility with VFIO when /dev/vfio/vfio is opened we default
+	 * to the same rlimit accounting as vfio uses.
+	 */
+	if (filp->private_data == &vfio_misc_dev)
+		ictx->account_mode = IOPT_PAGES_ACCOUNT_MM;
+
 	xa_init_flags(&ictx->objects, XA_FLAGS_ALLOC1 | XA_FLAGS_ACCOUNT);
 	ictx->file = filp;
 	filp->private_data = ictx;
@@ -392,26 +401,46 @@ static struct miscdevice iommu_misc_dev = {
 	.mode = 0660,
 };
 
+
+static struct miscdevice vfio_misc_dev = {
+	.minor = VFIO_MINOR,
+	.name = "vfio",
+	.fops = &iommufd_fops,
+	.nodename = "vfio/vfio",
+	.mode = 0666,
+};
+
 static int __init iommufd_init(void)
 {
 	int ret;
 
 	ret = misc_register(&iommu_misc_dev);
-	if (ret) {
-		pr_err("Failed to register misc device\n");
+	if (ret)
 		return ret;
-	}
 
+	if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER)) {
+		ret = misc_register(&vfio_misc_dev);
+		if (ret)
+			goto err_misc;
+	}
 	return 0;
+err_misc:
+	misc_deregister(&iommu_misc_dev);
+	return ret;
 }
 
 static void __exit iommufd_exit(void)
 {
+	if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER))
+		misc_deregister(&vfio_misc_dev);
 	misc_deregister(&iommu_misc_dev);
 }
 
 module_init(iommufd_init);
 module_exit(iommufd_exit);
 
+#if IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER)
+MODULE_ALIAS_MISCDEV(VFIO_MINOR);
+#endif
 MODULE_DESCRIPTION("I/O Address Space Management for passthrough devices");
 MODULE_LICENSE("GPL");
-- 
2.38.0

[Intel-gfx] ✗ Fi.CI.BUILD: failure for Connect VFIO to IOMMUFD

[Patchwork]

== Series Details ==

Series: Connect VFIO to IOMMUFD
URL   : https://patchwork.freedesktop.org/series/110133/
State : failure

== Summary ==

Error: patch https://patchwork.freedesktop.org/api/1.0/series/110133/revisions/1/mbox/ not applied
Applying: vfio: Move vfio_device driver open/close code to a function
Applying: vfio: Move vfio_device_assign_container() into vfio_device_first_open()
Applying: vfio: Rename vfio_device_assign/unassign_container()
Applying: vfio: Move storage of allow_unsafe_interrupts to vfio_main.c
Applying: vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()
Applying: vfio-iommufd: Allow iommufd to be used in place of a container fd
Applying: vfio-iommufd: Support iommufd for physical VFIO devices
Applying: vfio-iommufd: Support iommufd for emulated VFIO devices
Applying: vfio: Make vfio_container optionally compiled
Applying: iommufd: Allow iommufd to supply /dev/vfio/vfio
error: sha1 information is lacking or useless (drivers/iommu/iommufd/Kconfig).
error: could not build fake ancestor
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0010 iommufd: Allow iommufd to supply /dev/vfio/vfio
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Tue, 25 Oct 2022 15:17:10 -0300
Jason Gunthorpe <jgg@nvidia.com> wrote:

> This legacy module knob has become uAPI, when set on the vfio_iommu_type1
> it disables some security protections in the iommu drivers. Move the
> storage for this knob to vfio_main.c so that iommufd can access it too.

I don't really understand this, we're changing the behavior of the
iommufd_device_attach() operation based on the modules options of
vfio_iommu_type1, which may not be loaded or even compiled into the
kernel.  Our compatibility story falls apart when VFIO_CONTAINER is not
set, iommufd sneaks in to usurp /dev/vfio/vfio, and the user's module
options for type1 go unprocessed.

I hate to suggest that type1 becomes a module that does nothing more
than maintain consistency of this variable when the full type1 isn't
available, but is that what we need to do?  Thanks,

Alex

> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/vfio/vfio.h             | 2 ++
>  drivers/vfio/vfio_iommu_type1.c | 5 ++---
>  drivers/vfio/vfio_main.c        | 3 +++
>  3 files changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
> index f95f4925b83bbd..54e5a8e0834ccb 100644
> --- a/drivers/vfio/vfio.h
> +++ b/drivers/vfio/vfio.h
> @@ -130,4 +130,6 @@ extern bool vfio_noiommu __read_mostly;
>  enum { vfio_noiommu = false };
>  #endif
>  
> +extern bool vfio_allow_unsafe_interrupts;
> +
>  #endif
> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> index 23c24fe98c00d4..186e33a006d314 100644
> --- a/drivers/vfio/vfio_iommu_type1.c
> +++ b/drivers/vfio/vfio_iommu_type1.c
> @@ -44,9 +44,8 @@
>  #define DRIVER_AUTHOR   "Alex Williamson <alex.williamson@redhat.com>"
>  #define DRIVER_DESC     "Type1 IOMMU driver for VFIO"
>  
> -static bool allow_unsafe_interrupts;
>  module_param_named(allow_unsafe_interrupts,
> -		   allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
> +		   vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
>  MODULE_PARM_DESC(allow_unsafe_interrupts,
>  		 "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
>  
> @@ -2282,7 +2281,7 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
>  		    iommu_group_for_each_dev(iommu_group, (void *)IOMMU_CAP_INTR_REMAP,
>  					     vfio_iommu_device_capable);
>  
> -	if (!allow_unsafe_interrupts && !msi_remap) {
> +	if (!vfio_allow_unsafe_interrupts && !msi_remap) {
>  		pr_warn("%s: No interrupt remapping support.  Use the module param \"allow_unsafe_interrupts\" to enable VFIO IOMMU support on this platform\n",
>  		       __func__);
>  		ret = -EPERM;
> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index 8d809ecd982b39..1e414b2c48a511 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -51,6 +51,9 @@ static struct vfio {
>  	struct ida			device_ida;
>  } vfio;
>  
> +bool vfio_allow_unsafe_interrupts;
> +EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
> +
>  static DEFINE_XARRAY(vfio_device_set_xa);
>  static const struct file_operations vfio_group_fops;
>  

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Tue, 25 Oct 2022 15:17:10 -0300
Jason Gunthorpe <jgg@nvidia.com> wrote:

> This legacy module knob has become uAPI, when set on the vfio_iommu_type1
> it disables some security protections in the iommu drivers. Move the
> storage for this knob to vfio_main.c so that iommufd can access it too.

I don't really understand this, we're changing the behavior of the
iommufd_device_attach() operation based on the modules options of
vfio_iommu_type1, which may not be loaded or even compiled into the
kernel.  Our compatibility story falls apart when VFIO_CONTAINER is not
set, iommufd sneaks in to usurp /dev/vfio/vfio, and the user's module
options for type1 go unprocessed.

I hate to suggest that type1 becomes a module that does nothing more
than maintain consistency of this variable when the full type1 isn't
available, but is that what we need to do?  Thanks,

Alex

> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/vfio/vfio.h             | 2 ++
>  drivers/vfio/vfio_iommu_type1.c | 5 ++---
>  drivers/vfio/vfio_main.c        | 3 +++
>  3 files changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
> index f95f4925b83bbd..54e5a8e0834ccb 100644
> --- a/drivers/vfio/vfio.h
> +++ b/drivers/vfio/vfio.h
> @@ -130,4 +130,6 @@ extern bool vfio_noiommu __read_mostly;
>  enum { vfio_noiommu = false };
>  #endif
>  
> +extern bool vfio_allow_unsafe_interrupts;
> +
>  #endif
> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> index 23c24fe98c00d4..186e33a006d314 100644
> --- a/drivers/vfio/vfio_iommu_type1.c
> +++ b/drivers/vfio/vfio_iommu_type1.c
> @@ -44,9 +44,8 @@
>  #define DRIVER_AUTHOR   "Alex Williamson <alex.williamson@redhat.com>"
>  #define DRIVER_DESC     "Type1 IOMMU driver for VFIO"
>  
> -static bool allow_unsafe_interrupts;
>  module_param_named(allow_unsafe_interrupts,
> -		   allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
> +		   vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
>  MODULE_PARM_DESC(allow_unsafe_interrupts,
>  		 "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
>  
> @@ -2282,7 +2281,7 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
>  		    iommu_group_for_each_dev(iommu_group, (void *)IOMMU_CAP_INTR_REMAP,
>  					     vfio_iommu_device_capable);
>  
> -	if (!allow_unsafe_interrupts && !msi_remap) {
> +	if (!vfio_allow_unsafe_interrupts && !msi_remap) {
>  		pr_warn("%s: No interrupt remapping support.  Use the module param \"allow_unsafe_interrupts\" to enable VFIO IOMMU support on this platform\n",
>  		       __func__);
>  		ret = -EPERM;
> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index 8d809ecd982b39..1e414b2c48a511 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -51,6 +51,9 @@ static struct vfio {
>  	struct ida			device_ida;
>  } vfio;
>  
> +bool vfio_allow_unsafe_interrupts;
> +EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
> +
>  static DEFINE_XARRAY(vfio_device_set_xa);
>  static const struct file_operations vfio_group_fops;
>  

Re: [Intel-gfx] [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Tue, 25 Oct 2022 15:17:10 -0300
Jason Gunthorpe <jgg@nvidia.com> wrote:

> This legacy module knob has become uAPI, when set on the vfio_iommu_type1
> it disables some security protections in the iommu drivers. Move the
> storage for this knob to vfio_main.c so that iommufd can access it too.

I don't really understand this, we're changing the behavior of the
iommufd_device_attach() operation based on the modules options of
vfio_iommu_type1, which may not be loaded or even compiled into the
kernel.  Our compatibility story falls apart when VFIO_CONTAINER is not
set, iommufd sneaks in to usurp /dev/vfio/vfio, and the user's module
options for type1 go unprocessed.

I hate to suggest that type1 becomes a module that does nothing more
than maintain consistency of this variable when the full type1 isn't
available, but is that what we need to do?  Thanks,

Alex

> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/vfio/vfio.h             | 2 ++
>  drivers/vfio/vfio_iommu_type1.c | 5 ++---
>  drivers/vfio/vfio_main.c        | 3 +++
>  3 files changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
> index f95f4925b83bbd..54e5a8e0834ccb 100644
> --- a/drivers/vfio/vfio.h
> +++ b/drivers/vfio/vfio.h
> @@ -130,4 +130,6 @@ extern bool vfio_noiommu __read_mostly;
>  enum { vfio_noiommu = false };
>  #endif
>  
> +extern bool vfio_allow_unsafe_interrupts;
> +
>  #endif
> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> index 23c24fe98c00d4..186e33a006d314 100644
> --- a/drivers/vfio/vfio_iommu_type1.c
> +++ b/drivers/vfio/vfio_iommu_type1.c
> @@ -44,9 +44,8 @@
>  #define DRIVER_AUTHOR   "Alex Williamson <alex.williamson@redhat.com>"
>  #define DRIVER_DESC     "Type1 IOMMU driver for VFIO"
>  
> -static bool allow_unsafe_interrupts;
>  module_param_named(allow_unsafe_interrupts,
> -		   allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
> +		   vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
>  MODULE_PARM_DESC(allow_unsafe_interrupts,
>  		 "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
>  
> @@ -2282,7 +2281,7 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
>  		    iommu_group_for_each_dev(iommu_group, (void *)IOMMU_CAP_INTR_REMAP,
>  					     vfio_iommu_device_capable);
>  
> -	if (!allow_unsafe_interrupts && !msi_remap) {
> +	if (!vfio_allow_unsafe_interrupts && !msi_remap) {
>  		pr_warn("%s: No interrupt remapping support.  Use the module param \"allow_unsafe_interrupts\" to enable VFIO IOMMU support on this platform\n",
>  		       __func__);
>  		ret = -EPERM;
> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index 8d809ecd982b39..1e414b2c48a511 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -51,6 +51,9 @@ static struct vfio {
>  	struct ida			device_ida;
>  } vfio;
>  
> +bool vfio_allow_unsafe_interrupts;
> +EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
> +
>  static DEFINE_XARRAY(vfio_device_set_xa);
>  static const struct file_operations vfio_group_fops;
>  

Re: [PATCH 10/10] iommufd: Allow iommufd to supply /dev/vfio/vfio

[Alex Williamson]

On Tue, 25 Oct 2022 15:50:45 -0300
Jason Gunthorpe <jgg@nvidia.com> wrote:

> If the VFIO container is compiled out, give a kconfig option for iommufd
> to provide the miscdev node with the same name and permissions as vfio
> uses.
> 
> The compatibility node supports the same ioctls as VFIO and automatically
> enables the VFIO compatible pinned page accounting mode.

I think I'd like to see some sort of breadcrumb when /dev/vfio/vfio is
provided by something other than the vfio container code.  If we intend
to include this before P2P is resolved, that breadcrumb (dmesg I'm
guessing) might also list any known limitations of the compatibility to
save time with debugging.  Thanks,

Alex

> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/iommu/iommufd/Kconfig | 12 ++++++++++++
>  drivers/iommu/iommufd/main.c  | 35 ++++++++++++++++++++++++++++++++---
>  2 files changed, 44 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/iommu/iommufd/Kconfig b/drivers/iommu/iommufd/Kconfig
> index f0a2012234fa09..afc83b7575cce6 100644
> --- a/drivers/iommu/iommufd/Kconfig
> +++ b/drivers/iommu/iommufd/Kconfig
> @@ -14,6 +14,18 @@ config IOMMUFD
>  	  If you don't know what to do here, say N.
>  
>  if IOMMUFD
> +config IOMMUFD_VFIO_CONTAINER
> +	bool "IOMMUFD provides the VFIO container /dev/vfio/vfio"
> +	depends on VFIO && !VFIO_CONTAINER
> +	default VFIO && !VFIO_CONTAINER
> +	help
> +	  IOMMUFD will provide /dev/vfio/vfio instead of VFIO. This relies on
> +	  IOMMUFD providing compatibility emulation to give the same ioctls.
> +	  It provides an option to build a kernel with legacy VFIO components
> +	  removed.
> +
> +	  Unless testing IOMMUFD say N here.
> +
>  config IOMMUFD_TEST
>  	bool "IOMMU Userspace API Test support"
>  	depends on RUNTIME_TESTING_MENU
> diff --git a/drivers/iommu/iommufd/main.c b/drivers/iommu/iommufd/main.c
> index 8a31c1a14cdd53..19db81fbf7f08f 100644
> --- a/drivers/iommu/iommufd/main.c
> +++ b/drivers/iommu/iommufd/main.c
> @@ -24,6 +24,7 @@
>  #include <uapi/linux/iommufd.h>
>  #include <linux/iommufd.h>
>  
> +#include "io_pagetable.h"
>  #include "iommufd_private.h"
>  #include "iommufd_test.h"
>  
> @@ -31,6 +32,7 @@ struct iommufd_object_ops {
>  	void (*destroy)(struct iommufd_object *obj);
>  };
>  static struct iommufd_object_ops iommufd_object_ops[];
> +static struct miscdevice vfio_misc_dev;
>  
>  struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx,
>  					     size_t size,
> @@ -167,6 +169,13 @@ static int iommufd_fops_open(struct inode *inode, struct file *filp)
>  	if (!ictx)
>  		return -ENOMEM;
>  
> +	/*
> +	 * For compatibility with VFIO when /dev/vfio/vfio is opened we default
> +	 * to the same rlimit accounting as vfio uses.
> +	 */
> +	if (filp->private_data == &vfio_misc_dev)
> +		ictx->account_mode = IOPT_PAGES_ACCOUNT_MM;
> +
>  	xa_init_flags(&ictx->objects, XA_FLAGS_ALLOC1 | XA_FLAGS_ACCOUNT);
>  	ictx->file = filp;
>  	filp->private_data = ictx;
> @@ -392,26 +401,46 @@ static struct miscdevice iommu_misc_dev = {
>  	.mode = 0660,
>  };
>  
> +
> +static struct miscdevice vfio_misc_dev = {
> +	.minor = VFIO_MINOR,
> +	.name = "vfio",
> +	.fops = &iommufd_fops,
> +	.nodename = "vfio/vfio",
> +	.mode = 0666,
> +};
> +
>  static int __init iommufd_init(void)
>  {
>  	int ret;
>  
>  	ret = misc_register(&iommu_misc_dev);
> -	if (ret) {
> -		pr_err("Failed to register misc device\n");
> +	if (ret)
>  		return ret;
> -	}
>  
> +	if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER)) {
> +		ret = misc_register(&vfio_misc_dev);
> +		if (ret)
> +			goto err_misc;
> +	}
>  	return 0;
> +err_misc:
> +	misc_deregister(&iommu_misc_dev);
> +	return ret;
>  }
>  
>  static void __exit iommufd_exit(void)
>  {
> +	if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER))
> +		misc_deregister(&vfio_misc_dev);
>  	misc_deregister(&iommu_misc_dev);
>  }
>  
>  module_init(iommufd_init);
>  module_exit(iommufd_exit);
>  
> +#if IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER)
> +MODULE_ALIAS_MISCDEV(VFIO_MINOR);
> +#endif
>  MODULE_DESCRIPTION("I/O Address Space Management for passthrough devices");
>  MODULE_LICENSE("GPL");

Re: [Intel-gfx] [PATCH 10/10] iommufd: Allow iommufd to supply /dev/vfio/vfio

[Alex Williamson]

On Tue, 25 Oct 2022 15:50:45 -0300
Jason Gunthorpe <jgg@nvidia.com> wrote:

> If the VFIO container is compiled out, give a kconfig option for iommufd
> to provide the miscdev node with the same name and permissions as vfio
> uses.
> 
> The compatibility node supports the same ioctls as VFIO and automatically
> enables the VFIO compatible pinned page accounting mode.

I think I'd like to see some sort of breadcrumb when /dev/vfio/vfio is
provided by something other than the vfio container code.  If we intend
to include this before P2P is resolved, that breadcrumb (dmesg I'm
guessing) might also list any known limitations of the compatibility to
save time with debugging.  Thanks,

Alex

> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/iommu/iommufd/Kconfig | 12 ++++++++++++
>  drivers/iommu/iommufd/main.c  | 35 ++++++++++++++++++++++++++++++++---
>  2 files changed, 44 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/iommu/iommufd/Kconfig b/drivers/iommu/iommufd/Kconfig
> index f0a2012234fa09..afc83b7575cce6 100644
> --- a/drivers/iommu/iommufd/Kconfig
> +++ b/drivers/iommu/iommufd/Kconfig
> @@ -14,6 +14,18 @@ config IOMMUFD
>  	  If you don't know what to do here, say N.
>  
>  if IOMMUFD
> +config IOMMUFD_VFIO_CONTAINER
> +	bool "IOMMUFD provides the VFIO container /dev/vfio/vfio"
> +	depends on VFIO && !VFIO_CONTAINER
> +	default VFIO && !VFIO_CONTAINER
> +	help
> +	  IOMMUFD will provide /dev/vfio/vfio instead of VFIO. This relies on
> +	  IOMMUFD providing compatibility emulation to give the same ioctls.
> +	  It provides an option to build a kernel with legacy VFIO components
> +	  removed.
> +
> +	  Unless testing IOMMUFD say N here.
> +
>  config IOMMUFD_TEST
>  	bool "IOMMU Userspace API Test support"
>  	depends on RUNTIME_TESTING_MENU
> diff --git a/drivers/iommu/iommufd/main.c b/drivers/iommu/iommufd/main.c
> index 8a31c1a14cdd53..19db81fbf7f08f 100644
> --- a/drivers/iommu/iommufd/main.c
> +++ b/drivers/iommu/iommufd/main.c
> @@ -24,6 +24,7 @@
>  #include <uapi/linux/iommufd.h>
>  #include <linux/iommufd.h>
>  
> +#include "io_pagetable.h"
>  #include "iommufd_private.h"
>  #include "iommufd_test.h"
>  
> @@ -31,6 +32,7 @@ struct iommufd_object_ops {
>  	void (*destroy)(struct iommufd_object *obj);
>  };
>  static struct iommufd_object_ops iommufd_object_ops[];
> +static struct miscdevice vfio_misc_dev;
>  
>  struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx,
>  					     size_t size,
> @@ -167,6 +169,13 @@ static int iommufd_fops_open(struct inode *inode, struct file *filp)
>  	if (!ictx)
>  		return -ENOMEM;
>  
> +	/*
> +	 * For compatibility with VFIO when /dev/vfio/vfio is opened we default
> +	 * to the same rlimit accounting as vfio uses.
> +	 */
> +	if (filp->private_data == &vfio_misc_dev)
> +		ictx->account_mode = IOPT_PAGES_ACCOUNT_MM;
> +
>  	xa_init_flags(&ictx->objects, XA_FLAGS_ALLOC1 | XA_FLAGS_ACCOUNT);
>  	ictx->file = filp;
>  	filp->private_data = ictx;
> @@ -392,26 +401,46 @@ static struct miscdevice iommu_misc_dev = {
>  	.mode = 0660,
>  };
>  
> +
> +static struct miscdevice vfio_misc_dev = {
> +	.minor = VFIO_MINOR,
> +	.name = "vfio",
> +	.fops = &iommufd_fops,
> +	.nodename = "vfio/vfio",
> +	.mode = 0666,
> +};
> +
>  static int __init iommufd_init(void)
>  {
>  	int ret;
>  
>  	ret = misc_register(&iommu_misc_dev);
> -	if (ret) {
> -		pr_err("Failed to register misc device\n");
> +	if (ret)
>  		return ret;
> -	}
>  
> +	if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER)) {
> +		ret = misc_register(&vfio_misc_dev);
> +		if (ret)
> +			goto err_misc;
> +	}
>  	return 0;
> +err_misc:
> +	misc_deregister(&iommu_misc_dev);
> +	return ret;
>  }
>  
>  static void __exit iommufd_exit(void)
>  {
> +	if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER))
> +		misc_deregister(&vfio_misc_dev);
>  	misc_deregister(&iommu_misc_dev);
>  }
>  
>  module_init(iommufd_init);
>  module_exit(iommufd_exit);
>  
> +#if IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER)
> +MODULE_ALIAS_MISCDEV(VFIO_MINOR);
> +#endif
>  MODULE_DESCRIPTION("I/O Address Space Management for passthrough devices");
>  MODULE_LICENSE("GPL");

Re: [PATCH 10/10] iommufd: Allow iommufd to supply /dev/vfio/vfio

[Alex Williamson]

On Tue, 25 Oct 2022 15:50:45 -0300
Jason Gunthorpe <jgg@nvidia.com> wrote:

> If the VFIO container is compiled out, give a kconfig option for iommufd
> to provide the miscdev node with the same name and permissions as vfio
> uses.
> 
> The compatibility node supports the same ioctls as VFIO and automatically
> enables the VFIO compatible pinned page accounting mode.

I think I'd like to see some sort of breadcrumb when /dev/vfio/vfio is
provided by something other than the vfio container code.  If we intend
to include this before P2P is resolved, that breadcrumb (dmesg I'm
guessing) might also list any known limitations of the compatibility to
save time with debugging.  Thanks,

Alex

> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/iommu/iommufd/Kconfig | 12 ++++++++++++
>  drivers/iommu/iommufd/main.c  | 35 ++++++++++++++++++++++++++++++++---
>  2 files changed, 44 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/iommu/iommufd/Kconfig b/drivers/iommu/iommufd/Kconfig
> index f0a2012234fa09..afc83b7575cce6 100644
> --- a/drivers/iommu/iommufd/Kconfig
> +++ b/drivers/iommu/iommufd/Kconfig
> @@ -14,6 +14,18 @@ config IOMMUFD
>  	  If you don't know what to do here, say N.
>  
>  if IOMMUFD
> +config IOMMUFD_VFIO_CONTAINER
> +	bool "IOMMUFD provides the VFIO container /dev/vfio/vfio"
> +	depends on VFIO && !VFIO_CONTAINER
> +	default VFIO && !VFIO_CONTAINER
> +	help
> +	  IOMMUFD will provide /dev/vfio/vfio instead of VFIO. This relies on
> +	  IOMMUFD providing compatibility emulation to give the same ioctls.
> +	  It provides an option to build a kernel with legacy VFIO components
> +	  removed.
> +
> +	  Unless testing IOMMUFD say N here.
> +
>  config IOMMUFD_TEST
>  	bool "IOMMU Userspace API Test support"
>  	depends on RUNTIME_TESTING_MENU
> diff --git a/drivers/iommu/iommufd/main.c b/drivers/iommu/iommufd/main.c
> index 8a31c1a14cdd53..19db81fbf7f08f 100644
> --- a/drivers/iommu/iommufd/main.c
> +++ b/drivers/iommu/iommufd/main.c
> @@ -24,6 +24,7 @@
>  #include <uapi/linux/iommufd.h>
>  #include <linux/iommufd.h>
>  
> +#include "io_pagetable.h"
>  #include "iommufd_private.h"
>  #include "iommufd_test.h"
>  
> @@ -31,6 +32,7 @@ struct iommufd_object_ops {
>  	void (*destroy)(struct iommufd_object *obj);
>  };
>  static struct iommufd_object_ops iommufd_object_ops[];
> +static struct miscdevice vfio_misc_dev;
>  
>  struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx,
>  					     size_t size,
> @@ -167,6 +169,13 @@ static int iommufd_fops_open(struct inode *inode, struct file *filp)
>  	if (!ictx)
>  		return -ENOMEM;
>  
> +	/*
> +	 * For compatibility with VFIO when /dev/vfio/vfio is opened we default
> +	 * to the same rlimit accounting as vfio uses.
> +	 */
> +	if (filp->private_data == &vfio_misc_dev)
> +		ictx->account_mode = IOPT_PAGES_ACCOUNT_MM;
> +
>  	xa_init_flags(&ictx->objects, XA_FLAGS_ALLOC1 | XA_FLAGS_ACCOUNT);
>  	ictx->file = filp;
>  	filp->private_data = ictx;
> @@ -392,26 +401,46 @@ static struct miscdevice iommu_misc_dev = {
>  	.mode = 0660,
>  };
>  
> +
> +static struct miscdevice vfio_misc_dev = {
> +	.minor = VFIO_MINOR,
> +	.name = "vfio",
> +	.fops = &iommufd_fops,
> +	.nodename = "vfio/vfio",
> +	.mode = 0666,
> +};
> +
>  static int __init iommufd_init(void)
>  {
>  	int ret;
>  
>  	ret = misc_register(&iommu_misc_dev);
> -	if (ret) {
> -		pr_err("Failed to register misc device\n");
> +	if (ret)
>  		return ret;
> -	}
>  
> +	if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER)) {
> +		ret = misc_register(&vfio_misc_dev);
> +		if (ret)
> +			goto err_misc;
> +	}
>  	return 0;
> +err_misc:
> +	misc_deregister(&iommu_misc_dev);
> +	return ret;
>  }
>  
>  static void __exit iommufd_exit(void)
>  {
> +	if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER))
> +		misc_deregister(&vfio_misc_dev);
>  	misc_deregister(&iommu_misc_dev);
>  }
>  
>  module_init(iommufd_init);
>  module_exit(iommufd_exit);
>  
> +#if IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER)
> +MODULE_ALIAS_MISCDEV(VFIO_MINOR);
> +#endif
>  MODULE_DESCRIPTION("I/O Address Space Management for passthrough devices");
>  MODULE_LICENSE("GPL");

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Wed, Oct 26, 2022 at 03:24:42PM -0600, Alex Williamson wrote:
> On Tue, 25 Oct 2022 15:17:10 -0300
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > This legacy module knob has become uAPI, when set on the vfio_iommu_type1
> > it disables some security protections in the iommu drivers. Move the
> > storage for this knob to vfio_main.c so that iommufd can access it too.
> 
> I don't really understand this, we're changing the behavior of the
> iommufd_device_attach() operation based on the modules options of
> vfio_iommu_type1, 

The specific reason it was done is that we had a misconfigured test VM
in the farm that needed it, and that VM has since been fixed. But it
did highlight we should try to preserve this in some way.

> which may not be loaded or even compiled into the
> kernel.  Our compatibility story falls apart when VFIO_CONTAINER is not
> set, iommufd sneaks in to usurp /dev/vfio/vfio, and the user's module
> options for type1 go unprocessed.

There are two aspects here, trying to preseve the
allow_unsafe_interrupts knob as it is already as some ABI in the best
way we can.

And the second is how do we make this work in the new world where
there may be no type 1 module at all. This patch is not trying to
address that topic. I am expecting a range of small adjustments before
VFIO_CONTAINER=n becomes really fully viable.

> I hate to suggest that type1 becomes a module that does nothing more
> than maintain consistency of this variable when the full type1 isn't
> available, but is that what we need to do?

It is one idea, it depends how literal you want to be on "module
parameters are ABI". IMHO it is a weak form of ABI and the need of
this paramter in particular is not that common in modern times, AFAIK.

So perhaps we just also expose it through vfio.ko and expect people to
migrate. That would give a window were both options are available.

Jason

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Wed, Oct 26, 2022 at 03:24:42PM -0600, Alex Williamson wrote:
> On Tue, 25 Oct 2022 15:17:10 -0300
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > This legacy module knob has become uAPI, when set on the vfio_iommu_type1
> > it disables some security protections in the iommu drivers. Move the
> > storage for this knob to vfio_main.c so that iommufd can access it too.
> 
> I don't really understand this, we're changing the behavior of the
> iommufd_device_attach() operation based on the modules options of
> vfio_iommu_type1, 

The specific reason it was done is that we had a misconfigured test VM
in the farm that needed it, and that VM has since been fixed. But it
did highlight we should try to preserve this in some way.

> which may not be loaded or even compiled into the
> kernel.  Our compatibility story falls apart when VFIO_CONTAINER is not
> set, iommufd sneaks in to usurp /dev/vfio/vfio, and the user's module
> options for type1 go unprocessed.

There are two aspects here, trying to preseve the
allow_unsafe_interrupts knob as it is already as some ABI in the best
way we can.

And the second is how do we make this work in the new world where
there may be no type 1 module at all. This patch is not trying to
address that topic. I am expecting a range of small adjustments before
VFIO_CONTAINER=n becomes really fully viable.

> I hate to suggest that type1 becomes a module that does nothing more
> than maintain consistency of this variable when the full type1 isn't
> available, but is that what we need to do?

It is one idea, it depends how literal you want to be on "module
parameters are ABI". IMHO it is a weak form of ABI and the need of
this paramter in particular is not that common in modern times, AFAIK.

So perhaps we just also expose it through vfio.ko and expect people to
migrate. That would give a window were both options are available.

Jason

Re: [Intel-gfx] [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Wed, Oct 26, 2022 at 03:24:42PM -0600, Alex Williamson wrote:
> On Tue, 25 Oct 2022 15:17:10 -0300
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > This legacy module knob has become uAPI, when set on the vfio_iommu_type1
> > it disables some security protections in the iommu drivers. Move the
> > storage for this knob to vfio_main.c so that iommufd can access it too.
> 
> I don't really understand this, we're changing the behavior of the
> iommufd_device_attach() operation based on the modules options of
> vfio_iommu_type1, 

The specific reason it was done is that we had a misconfigured test VM
in the farm that needed it, and that VM has since been fixed. But it
did highlight we should try to preserve this in some way.

> which may not be loaded or even compiled into the
> kernel.  Our compatibility story falls apart when VFIO_CONTAINER is not
> set, iommufd sneaks in to usurp /dev/vfio/vfio, and the user's module
> options for type1 go unprocessed.

There are two aspects here, trying to preseve the
allow_unsafe_interrupts knob as it is already as some ABI in the best
way we can.

And the second is how do we make this work in the new world where
there may be no type 1 module at all. This patch is not trying to
address that topic. I am expecting a range of small adjustments before
VFIO_CONTAINER=n becomes really fully viable.

> I hate to suggest that type1 becomes a module that does nothing more
> than maintain consistency of this variable when the full type1 isn't
> available, but is that what we need to do?

It is one idea, it depends how literal you want to be on "module
parameters are ABI". IMHO it is a weak form of ABI and the need of
this paramter in particular is not that common in modern times, AFAIK.

So perhaps we just also expose it through vfio.ko and expect people to
migrate. That would give a window were both options are available.

Jason

Re: [PATCH 10/10] iommufd: Allow iommufd to supply /dev/vfio/vfio

[Jason Gunthorpe]

On Wed, Oct 26, 2022 at 03:31:33PM -0600, Alex Williamson wrote:
> On Tue, 25 Oct 2022 15:50:45 -0300
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > If the VFIO container is compiled out, give a kconfig option for iommufd
> > to provide the miscdev node with the same name and permissions as vfio
> > uses.
> > 
> > The compatibility node supports the same ioctls as VFIO and automatically
> > enables the VFIO compatible pinned page accounting mode.
> 
> I think I'd like to see some sort of breadcrumb when /dev/vfio/vfio is
> provided by something other than the vfio container code.  If we intend
> to include this before P2P is resolved, that breadcrumb 

I don't belive I can get P2P done soon enough. I plan to do it after
this is merged. Right now these two series are taking all my time.

> (dmesg I'm guessing) might also list any known limitations of the
> compatibility to save time with debugging.  Thanks,

Yes, that makes sense.

Do you want a dmesg at module load time, on every open, or a sysfs
something? What seems like it would make it into a bug report?

Thanks,
Jason

Re: [PATCH 10/10] iommufd: Allow iommufd to supply /dev/vfio/vfio

[Jason Gunthorpe]

On Wed, Oct 26, 2022 at 03:31:33PM -0600, Alex Williamson wrote:
> On Tue, 25 Oct 2022 15:50:45 -0300
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > If the VFIO container is compiled out, give a kconfig option for iommufd
> > to provide the miscdev node with the same name and permissions as vfio
> > uses.
> > 
> > The compatibility node supports the same ioctls as VFIO and automatically
> > enables the VFIO compatible pinned page accounting mode.
> 
> I think I'd like to see some sort of breadcrumb when /dev/vfio/vfio is
> provided by something other than the vfio container code.  If we intend
> to include this before P2P is resolved, that breadcrumb 

I don't belive I can get P2P done soon enough. I plan to do it after
this is merged. Right now these two series are taking all my time.

> (dmesg I'm guessing) might also list any known limitations of the
> compatibility to save time with debugging.  Thanks,

Yes, that makes sense.

Do you want a dmesg at module load time, on every open, or a sysfs
something? What seems like it would make it into a bug report?

Thanks,
Jason

Re: [Intel-gfx] [PATCH 10/10] iommufd: Allow iommufd to supply /dev/vfio/vfio

[Jason Gunthorpe]

On Wed, Oct 26, 2022 at 03:31:33PM -0600, Alex Williamson wrote:
> On Tue, 25 Oct 2022 15:50:45 -0300
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > If the VFIO container is compiled out, give a kconfig option for iommufd
> > to provide the miscdev node with the same name and permissions as vfio
> > uses.
> > 
> > The compatibility node supports the same ioctls as VFIO and automatically
> > enables the VFIO compatible pinned page accounting mode.
> 
> I think I'd like to see some sort of breadcrumb when /dev/vfio/vfio is
> provided by something other than the vfio container code.  If we intend
> to include this before P2P is resolved, that breadcrumb 

I don't belive I can get P2P done soon enough. I plan to do it after
this is merged. Right now these two series are taking all my time.

> (dmesg I'm guessing) might also list any known limitations of the
> compatibility to save time with debugging.  Thanks,

Yes, that makes sense.

Do you want a dmesg at module load time, on every open, or a sysfs
something? What seems like it would make it into a bug report?

Thanks,
Jason

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Nicolin Chen]

On Tue, Oct 25, 2022 at 03:17:06PM -0300, Jason Gunthorpe wrote:
> This series provides an alternative container layer for VFIO implemented
> using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
> not be compiled in.
> 
> At this point iommufd can be injected by passing in a iommfd FD to
> VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
> to obtain the compat IOAS and then connect up all the VFIO drivers as
> appropriate.
> 
> This is temporary stopping point, a following series will provide a way to
> directly open a VFIO device FD and directly connect it to IOMMUFD using
> native ioctls that can expose the IOMMUFD features like hwpt, future
> vPASID and dynamic attachment.
> 
> This series, in compat mode, has passed all the qemu tests we have
> available, including the test suites for the Intel GVT mdev. Aside from
> the temporary limitation with P2P memory this is belived to be fully
> compatible with VFIO.
> 
> This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd

Tested-by: Nicolin Chen <nicoleotsuka@nvidia.com>

Tested this branch on ARM64+SMMUv3 with the iommufd selftest and
QEMU passthrough sanity using noiommu and virtio-iommu setups by
combining with both CONFIG_VFIO_CONTAINER=y and =n.

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Nicolin Chen]

On Tue, Oct 25, 2022 at 03:17:06PM -0300, Jason Gunthorpe wrote:
> This series provides an alternative container layer for VFIO implemented
> using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
> not be compiled in.
> 
> At this point iommufd can be injected by passing in a iommfd FD to
> VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
> to obtain the compat IOAS and then connect up all the VFIO drivers as
> appropriate.
> 
> This is temporary stopping point, a following series will provide a way to
> directly open a VFIO device FD and directly connect it to IOMMUFD using
> native ioctls that can expose the IOMMUFD features like hwpt, future
> vPASID and dynamic attachment.
> 
> This series, in compat mode, has passed all the qemu tests we have
> available, including the test suites for the Intel GVT mdev. Aside from
> the temporary limitation with P2P memory this is belived to be fully
> compatible with VFIO.
> 
> This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd

Tested-by: Nicolin Chen <nicoleotsuka@nvidia.com>

Tested this branch on ARM64+SMMUv3 with the iommufd selftest and
QEMU passthrough sanity using noiommu and virtio-iommu setups by
combining with both CONFIG_VFIO_CONTAINER=y and =n.

Re: [Intel-gfx] [PATCH 00/10] Connect VFIO to IOMMUFD

[Nicolin Chen]

On Tue, Oct 25, 2022 at 03:17:06PM -0300, Jason Gunthorpe wrote:
> This series provides an alternative container layer for VFIO implemented
> using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
> not be compiled in.
> 
> At this point iommufd can be injected by passing in a iommfd FD to
> VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
> to obtain the compat IOAS and then connect up all the VFIO drivers as
> appropriate.
> 
> This is temporary stopping point, a following series will provide a way to
> directly open a VFIO device FD and directly connect it to IOMMUFD using
> native ioctls that can expose the IOMMUFD features like hwpt, future
> vPASID and dynamic attachment.
> 
> This series, in compat mode, has passed all the qemu tests we have
> available, including the test suites for the Intel GVT mdev. Aside from
> the temporary limitation with P2P memory this is belived to be fully
> compatible with VFIO.
> 
> This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd

Tested-by: Nicolin Chen <nicoleotsuka@nvidia.com>

Tested this branch on ARM64+SMMUv3 with the iommufd selftest and
QEMU passthrough sanity using noiommu and virtio-iommu setups by
combining with both CONFIG_VFIO_CONTAINER=y and =n.

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Nicolin Chen]

On Fri, Oct 28, 2022 at 04:53:21PM -0700, Nicolin Chen wrote:
> On Tue, Oct 25, 2022 at 03:17:06PM -0300, Jason Gunthorpe wrote:
> > This series provides an alternative container layer for VFIO implemented
> > using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
> > not be compiled in.
> > 
> > At this point iommufd can be injected by passing in a iommfd FD to
> > VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
> > to obtain the compat IOAS and then connect up all the VFIO drivers as
> > appropriate.
> > 
> > This is temporary stopping point, a following series will provide a way to
> > directly open a VFIO device FD and directly connect it to IOMMUFD using
> > native ioctls that can expose the IOMMUFD features like hwpt, future
> > vPASID and dynamic attachment.
> > 
> > This series, in compat mode, has passed all the qemu tests we have
> > available, including the test suites for the Intel GVT mdev. Aside from
> > the temporary limitation with P2P memory this is belived to be fully
> > compatible with VFIO.
> > 
> > This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd
> 
> Tested-by: Nicolin Chen <nicoleotsuka@nvidia.com>

Sorry, wrong email -- should be:
Tested-by: Nicolin Chen <nicolinc@nvidia.com>

> Tested this branch on ARM64+SMMUv3 with the iommufd selftest and
> QEMU passthrough sanity using noiommu and virtio-iommu setups by
> combining with both CONFIG_VFIO_CONTAINER=y and =n.

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Nicolin Chen]

On Fri, Oct 28, 2022 at 04:53:21PM -0700, Nicolin Chen wrote:
> On Tue, Oct 25, 2022 at 03:17:06PM -0300, Jason Gunthorpe wrote:
> > This series provides an alternative container layer for VFIO implemented
> > using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
> > not be compiled in.
> > 
> > At this point iommufd can be injected by passing in a iommfd FD to
> > VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
> > to obtain the compat IOAS and then connect up all the VFIO drivers as
> > appropriate.
> > 
> > This is temporary stopping point, a following series will provide a way to
> > directly open a VFIO device FD and directly connect it to IOMMUFD using
> > native ioctls that can expose the IOMMUFD features like hwpt, future
> > vPASID and dynamic attachment.
> > 
> > This series, in compat mode, has passed all the qemu tests we have
> > available, including the test suites for the Intel GVT mdev. Aside from
> > the temporary limitation with P2P memory this is belived to be fully
> > compatible with VFIO.
> > 
> > This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd
> 
> Tested-by: Nicolin Chen <nicoleotsuka@nvidia.com>

Sorry, wrong email -- should be:
Tested-by: Nicolin Chen <nicolinc@nvidia.com>

> Tested this branch on ARM64+SMMUv3 with the iommufd selftest and
> QEMU passthrough sanity using noiommu and virtio-iommu setups by
> combining with both CONFIG_VFIO_CONTAINER=y and =n.

Re: [Intel-gfx] [PATCH 00/10] Connect VFIO to IOMMUFD

[Nicolin Chen]

On Fri, Oct 28, 2022 at 04:53:21PM -0700, Nicolin Chen wrote:
> On Tue, Oct 25, 2022 at 03:17:06PM -0300, Jason Gunthorpe wrote:
> > This series provides an alternative container layer for VFIO implemented
> > using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
> > not be compiled in.
> > 
> > At this point iommufd can be injected by passing in a iommfd FD to
> > VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
> > to obtain the compat IOAS and then connect up all the VFIO drivers as
> > appropriate.
> > 
> > This is temporary stopping point, a following series will provide a way to
> > directly open a VFIO device FD and directly connect it to IOMMUFD using
> > native ioctls that can expose the IOMMUFD features like hwpt, future
> > vPASID and dynamic attachment.
> > 
> > This series, in compat mode, has passed all the qemu tests we have
> > available, including the test suites for the Intel GVT mdev. Aside from
> > the temporary limitation with P2P memory this is belived to be fully
> > compatible with VFIO.
> > 
> > This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd
> 
> Tested-by: Nicolin Chen <nicoleotsuka@nvidia.com>

Sorry, wrong email -- should be:
Tested-by: Nicolin Chen <nicolinc@nvidia.com>

> Tested this branch on ARM64+SMMUv3 with the iommufd selftest and
> QEMU passthrough sanity using noiommu and virtio-iommu setups by
> combining with both CONFIG_VFIO_CONTAINER=y and =n.

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Yi Liu]

Hi Jason,

On 2022/10/26 02:17, Jason Gunthorpe wrote:
> This series provides an alternative container layer for VFIO implemented
> using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
> not be compiled in.
> 
> At this point iommufd can be injected by passing in a iommfd FD to
> VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
> to obtain the compat IOAS and then connect up all the VFIO drivers as
> appropriate.
> 
> This is temporary stopping point, a following series will provide a way to
> directly open a VFIO device FD and directly connect it to IOMMUFD using
> native ioctls that can expose the IOMMUFD features like hwpt, future
> vPASID and dynamic attachment.
> 
> This series, in compat mode, has passed all the qemu tests we have
> available, including the test suites for the Intel GVT mdev. Aside from
> the temporary limitation with P2P memory this is belived to be fully
> compatible with VFIO.
> 
> This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd

In our side, we found the gvt-g test failed. Guest i915 driver stuck at
init phase. While with your former version  (commit ID 
a249441ba6fd9d658f4a1b568453e3a742d12686), gvt-g test is passing. I
noticed there a quite a few change in iommufd/pages.c from last version.
We are internally tracing in the gvt-g side, may also good to have your
attention.

> It requires the iommufd series:
> 
> https://lore.kernel.org/r/0-v3-402a7d6459de+24b-iommufd_jgg@nvidia.com
> 
> Jason Gunthorpe (10):
>    vfio: Move vfio_device driver open/close code to a function
>    vfio: Move vfio_device_assign_container() into
>      vfio_device_first_open()
>    vfio: Rename vfio_device_assign/unassign_container()
>    vfio: Move storage of allow_unsafe_interrupts to vfio_main.c
>    vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for
>      vfio_file_enforced_coherent()
>    vfio-iommufd: Allow iommufd to be used in place of a container fd
>    vfio-iommufd: Support iommufd for physical VFIO devices
>    vfio-iommufd: Support iommufd for emulated VFIO devices
>    vfio: Make vfio_container optionally compiled
>    iommufd: Allow iommufd to supply /dev/vfio/vfio
> 
>   drivers/gpu/drm/i915/gvt/kvmgt.c              |   3 +
>   drivers/iommu/iommufd/Kconfig                 |  12 +
>   drivers/iommu/iommufd/main.c                  |  35 +-
>   drivers/s390/cio/vfio_ccw_ops.c               |   3 +
>   drivers/s390/crypto/vfio_ap_ops.c             |   3 +
>   drivers/vfio/Kconfig                          |  38 ++-
>   drivers/vfio/Makefile                         |   5 +-
>   drivers/vfio/container.c                      | 136 ++------
>   drivers/vfio/fsl-mc/vfio_fsl_mc.c             |   3 +
>   drivers/vfio/iommufd.c                        | 161 +++++++++
>   .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    |   6 +
>   drivers/vfio/pci/mlx5/main.c                  |   3 +
>   drivers/vfio/pci/vfio_pci.c                   |   3 +
>   drivers/vfio/platform/vfio_amba.c             |   3 +
>   drivers/vfio/platform/vfio_platform.c         |   3 +
>   drivers/vfio/vfio.h                           | 100 +++++-
>   drivers/vfio/vfio_iommu_type1.c               |   5 +-
>   drivers/vfio/vfio_main.c                      | 318 ++++++++++++++----
>   include/linux/vfio.h                          |  39 +++
>   19 files changed, 681 insertions(+), 198 deletions(-)
>   create mode 100644 drivers/vfio/iommufd.c
> 
> 
> base-commit: 3bec937e94942a6aee8854be1c1f5cc2b92d15e2

-- 
Regards,
Yi Liu

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Yi Liu]

Hi Jason,

On 2022/10/26 02:17, Jason Gunthorpe wrote:
> This series provides an alternative container layer for VFIO implemented
> using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
> not be compiled in.
> 
> At this point iommufd can be injected by passing in a iommfd FD to
> VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
> to obtain the compat IOAS and then connect up all the VFIO drivers as
> appropriate.
> 
> This is temporary stopping point, a following series will provide a way to
> directly open a VFIO device FD and directly connect it to IOMMUFD using
> native ioctls that can expose the IOMMUFD features like hwpt, future
> vPASID and dynamic attachment.
> 
> This series, in compat mode, has passed all the qemu tests we have
> available, including the test suites for the Intel GVT mdev. Aside from
> the temporary limitation with P2P memory this is belived to be fully
> compatible with VFIO.
> 
> This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd

In our side, we found the gvt-g test failed. Guest i915 driver stuck at
init phase. While with your former version  (commit ID 
a249441ba6fd9d658f4a1b568453e3a742d12686), gvt-g test is passing. I
noticed there a quite a few change in iommufd/pages.c from last version.
We are internally tracing in the gvt-g side, may also good to have your
attention.

> It requires the iommufd series:
> 
> https://lore.kernel.org/r/0-v3-402a7d6459de+24b-iommufd_jgg@nvidia.com
> 
> Jason Gunthorpe (10):
>    vfio: Move vfio_device driver open/close code to a function
>    vfio: Move vfio_device_assign_container() into
>      vfio_device_first_open()
>    vfio: Rename vfio_device_assign/unassign_container()
>    vfio: Move storage of allow_unsafe_interrupts to vfio_main.c
>    vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for
>      vfio_file_enforced_coherent()
>    vfio-iommufd: Allow iommufd to be used in place of a container fd
>    vfio-iommufd: Support iommufd for physical VFIO devices
>    vfio-iommufd: Support iommufd for emulated VFIO devices
>    vfio: Make vfio_container optionally compiled
>    iommufd: Allow iommufd to supply /dev/vfio/vfio
> 
>   drivers/gpu/drm/i915/gvt/kvmgt.c              |   3 +
>   drivers/iommu/iommufd/Kconfig                 |  12 +
>   drivers/iommu/iommufd/main.c                  |  35 +-
>   drivers/s390/cio/vfio_ccw_ops.c               |   3 +
>   drivers/s390/crypto/vfio_ap_ops.c             |   3 +
>   drivers/vfio/Kconfig                          |  38 ++-
>   drivers/vfio/Makefile                         |   5 +-
>   drivers/vfio/container.c                      | 136 ++------
>   drivers/vfio/fsl-mc/vfio_fsl_mc.c             |   3 +
>   drivers/vfio/iommufd.c                        | 161 +++++++++
>   .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    |   6 +
>   drivers/vfio/pci/mlx5/main.c                  |   3 +
>   drivers/vfio/pci/vfio_pci.c                   |   3 +
>   drivers/vfio/platform/vfio_amba.c             |   3 +
>   drivers/vfio/platform/vfio_platform.c         |   3 +
>   drivers/vfio/vfio.h                           | 100 +++++-
>   drivers/vfio/vfio_iommu_type1.c               |   5 +-
>   drivers/vfio/vfio_main.c                      | 318 ++++++++++++++----
>   include/linux/vfio.h                          |  39 +++
>   19 files changed, 681 insertions(+), 198 deletions(-)
>   create mode 100644 drivers/vfio/iommufd.c
> 
> 
> base-commit: 3bec937e94942a6aee8854be1c1f5cc2b92d15e2

-- 
Regards,
Yi Liu

Re: [Intel-gfx] [PATCH 00/10] Connect VFIO to IOMMUFD

[Yi Liu]

Hi Jason,

On 2022/10/26 02:17, Jason Gunthorpe wrote:
> This series provides an alternative container layer for VFIO implemented
> using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
> not be compiled in.
> 
> At this point iommufd can be injected by passing in a iommfd FD to
> VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
> to obtain the compat IOAS and then connect up all the VFIO drivers as
> appropriate.
> 
> This is temporary stopping point, a following series will provide a way to
> directly open a VFIO device FD and directly connect it to IOMMUFD using
> native ioctls that can expose the IOMMUFD features like hwpt, future
> vPASID and dynamic attachment.
> 
> This series, in compat mode, has passed all the qemu tests we have
> available, including the test suites for the Intel GVT mdev. Aside from
> the temporary limitation with P2P memory this is belived to be fully
> compatible with VFIO.
> 
> This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd

In our side, we found the gvt-g test failed. Guest i915 driver stuck at
init phase. While with your former version  (commit ID 
a249441ba6fd9d658f4a1b568453e3a742d12686), gvt-g test is passing. I
noticed there a quite a few change in iommufd/pages.c from last version.
We are internally tracing in the gvt-g side, may also good to have your
attention.

> It requires the iommufd series:
> 
> https://lore.kernel.org/r/0-v3-402a7d6459de+24b-iommufd_jgg@nvidia.com
> 
> Jason Gunthorpe (10):
>    vfio: Move vfio_device driver open/close code to a function
>    vfio: Move vfio_device_assign_container() into
>      vfio_device_first_open()
>    vfio: Rename vfio_device_assign/unassign_container()
>    vfio: Move storage of allow_unsafe_interrupts to vfio_main.c
>    vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for
>      vfio_file_enforced_coherent()
>    vfio-iommufd: Allow iommufd to be used in place of a container fd
>    vfio-iommufd: Support iommufd for physical VFIO devices
>    vfio-iommufd: Support iommufd for emulated VFIO devices
>    vfio: Make vfio_container optionally compiled
>    iommufd: Allow iommufd to supply /dev/vfio/vfio
> 
>   drivers/gpu/drm/i915/gvt/kvmgt.c              |   3 +
>   drivers/iommu/iommufd/Kconfig                 |  12 +
>   drivers/iommu/iommufd/main.c                  |  35 +-
>   drivers/s390/cio/vfio_ccw_ops.c               |   3 +
>   drivers/s390/crypto/vfio_ap_ops.c             |   3 +
>   drivers/vfio/Kconfig                          |  38 ++-
>   drivers/vfio/Makefile                         |   5 +-
>   drivers/vfio/container.c                      | 136 ++------
>   drivers/vfio/fsl-mc/vfio_fsl_mc.c             |   3 +
>   drivers/vfio/iommufd.c                        | 161 +++++++++
>   .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    |   6 +
>   drivers/vfio/pci/mlx5/main.c                  |   3 +
>   drivers/vfio/pci/vfio_pci.c                   |   3 +
>   drivers/vfio/platform/vfio_amba.c             |   3 +
>   drivers/vfio/platform/vfio_platform.c         |   3 +
>   drivers/vfio/vfio.h                           | 100 +++++-
>   drivers/vfio/vfio_iommu_type1.c               |   5 +-
>   drivers/vfio/vfio_main.c                      | 318 ++++++++++++++----
>   include/linux/vfio.h                          |  39 +++
>   19 files changed, 681 insertions(+), 198 deletions(-)
>   create mode 100644 drivers/vfio/iommufd.c
> 
> 
> base-commit: 3bec937e94942a6aee8854be1c1f5cc2b92d15e2

-- 
Regards,
Yi Liu

Re: [Intel-gfx] [PATCH 00/10] Connect VFIO to IOMMUFD

[Jason Gunthorpe]

On Mon, Oct 31, 2022 at 06:38:45PM +0800, Yi Liu wrote:
> Hi Jason,
> 
> On 2022/10/26 02:17, Jason Gunthorpe wrote:
> > This series provides an alternative container layer for VFIO implemented
> > using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
> > not be compiled in.
> > 
> > At this point iommufd can be injected by passing in a iommfd FD to
> > VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
> > to obtain the compat IOAS and then connect up all the VFIO drivers as
> > appropriate.
> > 
> > This is temporary stopping point, a following series will provide a way to
> > directly open a VFIO device FD and directly connect it to IOMMUFD using
> > native ioctls that can expose the IOMMUFD features like hwpt, future
> > vPASID and dynamic attachment.
> > 
> > This series, in compat mode, has passed all the qemu tests we have
> > available, including the test suites for the Intel GVT mdev. Aside from
> > the temporary limitation with P2P memory this is belived to be fully
> > compatible with VFIO.
> > 
> > This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd
> 
> In our side, we found the gvt-g test failed. Guest i915 driver stuck at
> init phase. While with your former version  (commit ID
> a249441ba6fd9d658f4a1b568453e3a742d12686), gvt-g test is passing. 

Oh, I didn't realize you grabbed such an older version for this testing..

> noticed there a quite a few change in iommufd/pages.c from last version.
> We are internally tracing in the gvt-g side, may also good to have your
> attention.

syzkaller just ran into this that I was starting to investigate:

@@ -1505,7 +1505,7 @@ int iopt_pages_fill_xarray(struct iopt_pages *pages, unsigned long start_index,
        int rc;
 
        pfn_reader_user_init(&user, pages);
-       user.upages_len = last_index - start_index + 1;
+       user.upages_len = (last_index - start_index + 1) * sizeof(*out_pages);
        interval_tree_for_each_double_span(&span, &pages->access_itree,

It would certainly hit gvt - but you should get WARN_ON's not hangs

There is something wrong with the test suite that it isn't covering
the above, I'm going to look into that today.

Jason

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Jason Gunthorpe]

On Mon, Oct 31, 2022 at 06:38:45PM +0800, Yi Liu wrote:
> Hi Jason,
> 
> On 2022/10/26 02:17, Jason Gunthorpe wrote:
> > This series provides an alternative container layer for VFIO implemented
> > using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
> > not be compiled in.
> > 
> > At this point iommufd can be injected by passing in a iommfd FD to
> > VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
> > to obtain the compat IOAS and then connect up all the VFIO drivers as
> > appropriate.
> > 
> > This is temporary stopping point, a following series will provide a way to
> > directly open a VFIO device FD and directly connect it to IOMMUFD using
> > native ioctls that can expose the IOMMUFD features like hwpt, future
> > vPASID and dynamic attachment.
> > 
> > This series, in compat mode, has passed all the qemu tests we have
> > available, including the test suites for the Intel GVT mdev. Aside from
> > the temporary limitation with P2P memory this is belived to be fully
> > compatible with VFIO.
> > 
> > This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd
> 
> In our side, we found the gvt-g test failed. Guest i915 driver stuck at
> init phase. While with your former version  (commit ID
> a249441ba6fd9d658f4a1b568453e3a742d12686), gvt-g test is passing. 

Oh, I didn't realize you grabbed such an older version for this testing..

> noticed there a quite a few change in iommufd/pages.c from last version.
> We are internally tracing in the gvt-g side, may also good to have your
> attention.

syzkaller just ran into this that I was starting to investigate:

@@ -1505,7 +1505,7 @@ int iopt_pages_fill_xarray(struct iopt_pages *pages, unsigned long start_index,
        int rc;
 
        pfn_reader_user_init(&user, pages);
-       user.upages_len = last_index - start_index + 1;
+       user.upages_len = (last_index - start_index + 1) * sizeof(*out_pages);
        interval_tree_for_each_double_span(&span, &pages->access_itree,

It would certainly hit gvt - but you should get WARN_ON's not hangs

There is something wrong with the test suite that it isn't covering
the above, I'm going to look into that today.

Jason

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Jason Gunthorpe]

On Mon, Oct 31, 2022 at 06:38:45PM +0800, Yi Liu wrote:
> Hi Jason,
> 
> On 2022/10/26 02:17, Jason Gunthorpe wrote:
> > This series provides an alternative container layer for VFIO implemented
> > using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
> > not be compiled in.
> > 
> > At this point iommufd can be injected by passing in a iommfd FD to
> > VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
> > to obtain the compat IOAS and then connect up all the VFIO drivers as
> > appropriate.
> > 
> > This is temporary stopping point, a following series will provide a way to
> > directly open a VFIO device FD and directly connect it to IOMMUFD using
> > native ioctls that can expose the IOMMUFD features like hwpt, future
> > vPASID and dynamic attachment.
> > 
> > This series, in compat mode, has passed all the qemu tests we have
> > available, including the test suites for the Intel GVT mdev. Aside from
> > the temporary limitation with P2P memory this is belived to be fully
> > compatible with VFIO.
> > 
> > This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd
> 
> In our side, we found the gvt-g test failed. Guest i915 driver stuck at
> init phase. While with your former version  (commit ID
> a249441ba6fd9d658f4a1b568453e3a742d12686), gvt-g test is passing. 

Oh, I didn't realize you grabbed such an older version for this testing..

> noticed there a quite a few change in iommufd/pages.c from last version.
> We are internally tracing in the gvt-g side, may also good to have your
> attention.

syzkaller just ran into this that I was starting to investigate:

@@ -1505,7 +1505,7 @@ int iopt_pages_fill_xarray(struct iopt_pages *pages, unsigned long start_index,
        int rc;
 
        pfn_reader_user_init(&user, pages);
-       user.upages_len = last_index - start_index + 1;
+       user.upages_len = (last_index - start_index + 1) * sizeof(*out_pages);
        interval_tree_for_each_double_span(&span, &pages->access_itree,

It would certainly hit gvt - but you should get WARN_ON's not hangs

There is something wrong with the test suite that it isn't covering
the above, I'm going to look into that today.

Jason

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Yi Liu]

On 2022/10/31 20:18, Jason Gunthorpe wrote:
> On Mon, Oct 31, 2022 at 06:38:45PM +0800, Yi Liu wrote:
>> Hi Jason,
>>
>> On 2022/10/26 02:17, Jason Gunthorpe wrote:
>>> This series provides an alternative container layer for VFIO implemented
>>> using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
>>> not be compiled in.
>>>
>>> At this point iommufd can be injected by passing in a iommfd FD to
>>> VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
>>> to obtain the compat IOAS and then connect up all the VFIO drivers as
>>> appropriate.
>>>
>>> This is temporary stopping point, a following series will provide a way to
>>> directly open a VFIO device FD and directly connect it to IOMMUFD using
>>> native ioctls that can expose the IOMMUFD features like hwpt, future
>>> vPASID and dynamic attachment.
>>>
>>> This series, in compat mode, has passed all the qemu tests we have
>>> available, including the test suites for the Intel GVT mdev. Aside from
>>> the temporary limitation with P2P memory this is belived to be fully
>>> compatible with VFIO.
>>>
>>> This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd
>>
>> In our side, we found the gvt-g test failed. Guest i915 driver stuck at
>> init phase. While with your former version  (commit ID
>> a249441ba6fd9d658f4a1b568453e3a742d12686), gvt-g test is passing.
> 
> Oh, I didn't realize you grabbed such an older version for this testing..

yeah, this was grabbed before your posting. :-)

>> noticed there a quite a few change in iommufd/pages.c from last version.
>> We are internally tracing in the gvt-g side, may also good to have your
>> attention.
> 
> syzkaller just ran into this that I was starting to investigate:
> 
> @@ -1505,7 +1505,7 @@ int iopt_pages_fill_xarray(struct iopt_pages *pages, unsigned long start_index,
>          int rc;
>   
>          pfn_reader_user_init(&user, pages);
> -       user.upages_len = last_index - start_index + 1;
> +       user.upages_len = (last_index - start_index + 1) * sizeof(*out_pages);
>          interval_tree_for_each_double_span(&span, &pages->access_itree,
> 
> It would certainly hit gvt - but you should get WARN_ON's not hangs
> 
> There is something wrong with the test suite that it isn't covering
> the above, I'm going to look into that today.

sounds to be the cause. I didn't see any significant change in vfio_main.c
that may fail gvt. So should the iommufd changes. Then we will re-run the
test after your update.:-)

-- 
Regards,
Yi Liu

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Yi Liu]

On 2022/10/31 20:18, Jason Gunthorpe wrote:
> On Mon, Oct 31, 2022 at 06:38:45PM +0800, Yi Liu wrote:
>> Hi Jason,
>>
>> On 2022/10/26 02:17, Jason Gunthorpe wrote:
>>> This series provides an alternative container layer for VFIO implemented
>>> using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
>>> not be compiled in.
>>>
>>> At this point iommufd can be injected by passing in a iommfd FD to
>>> VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
>>> to obtain the compat IOAS and then connect up all the VFIO drivers as
>>> appropriate.
>>>
>>> This is temporary stopping point, a following series will provide a way to
>>> directly open a VFIO device FD and directly connect it to IOMMUFD using
>>> native ioctls that can expose the IOMMUFD features like hwpt, future
>>> vPASID and dynamic attachment.
>>>
>>> This series, in compat mode, has passed all the qemu tests we have
>>> available, including the test suites for the Intel GVT mdev. Aside from
>>> the temporary limitation with P2P memory this is belived to be fully
>>> compatible with VFIO.
>>>
>>> This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd
>>
>> In our side, we found the gvt-g test failed. Guest i915 driver stuck at
>> init phase. While with your former version  (commit ID
>> a249441ba6fd9d658f4a1b568453e3a742d12686), gvt-g test is passing.
> 
> Oh, I didn't realize you grabbed such an older version for this testing..

yeah, this was grabbed before your posting. :-)

>> noticed there a quite a few change in iommufd/pages.c from last version.
>> We are internally tracing in the gvt-g side, may also good to have your
>> attention.
> 
> syzkaller just ran into this that I was starting to investigate:
> 
> @@ -1505,7 +1505,7 @@ int iopt_pages_fill_xarray(struct iopt_pages *pages, unsigned long start_index,
>          int rc;
>   
>          pfn_reader_user_init(&user, pages);
> -       user.upages_len = last_index - start_index + 1;
> +       user.upages_len = (last_index - start_index + 1) * sizeof(*out_pages);
>          interval_tree_for_each_double_span(&span, &pages->access_itree,
> 
> It would certainly hit gvt - but you should get WARN_ON's not hangs
> 
> There is something wrong with the test suite that it isn't covering
> the above, I'm going to look into that today.

sounds to be the cause. I didn't see any significant change in vfio_main.c
that may fail gvt. So should the iommufd changes. Then we will re-run the
test after your update.:-)

-- 
Regards,
Yi Liu

Re: [Intel-gfx] [PATCH 00/10] Connect VFIO to IOMMUFD

[Yi Liu]

On 2022/10/31 20:18, Jason Gunthorpe wrote:
> On Mon, Oct 31, 2022 at 06:38:45PM +0800, Yi Liu wrote:
>> Hi Jason,
>>
>> On 2022/10/26 02:17, Jason Gunthorpe wrote:
>>> This series provides an alternative container layer for VFIO implemented
>>> using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will
>>> not be compiled in.
>>>
>>> At this point iommufd can be injected by passing in a iommfd FD to
>>> VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd
>>> to obtain the compat IOAS and then connect up all the VFIO drivers as
>>> appropriate.
>>>
>>> This is temporary stopping point, a following series will provide a way to
>>> directly open a VFIO device FD and directly connect it to IOMMUFD using
>>> native ioctls that can expose the IOMMUFD features like hwpt, future
>>> vPASID and dynamic attachment.
>>>
>>> This series, in compat mode, has passed all the qemu tests we have
>>> available, including the test suites for the Intel GVT mdev. Aside from
>>> the temporary limitation with P2P memory this is belived to be fully
>>> compatible with VFIO.
>>>
>>> This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd
>>
>> In our side, we found the gvt-g test failed. Guest i915 driver stuck at
>> init phase. While with your former version  (commit ID
>> a249441ba6fd9d658f4a1b568453e3a742d12686), gvt-g test is passing.
> 
> Oh, I didn't realize you grabbed such an older version for this testing..

yeah, this was grabbed before your posting. :-)

>> noticed there a quite a few change in iommufd/pages.c from last version.
>> We are internally tracing in the gvt-g side, may also good to have your
>> attention.
> 
> syzkaller just ran into this that I was starting to investigate:
> 
> @@ -1505,7 +1505,7 @@ int iopt_pages_fill_xarray(struct iopt_pages *pages, unsigned long start_index,
>          int rc;
>   
>          pfn_reader_user_init(&user, pages);
> -       user.upages_len = last_index - start_index + 1;
> +       user.upages_len = (last_index - start_index + 1) * sizeof(*out_pages);
>          interval_tree_for_each_double_span(&span, &pages->access_itree,
> 
> It would certainly hit gvt - but you should get WARN_ON's not hangs
> 
> There is something wrong with the test suite that it isn't covering
> the above, I'm going to look into that today.

sounds to be the cause. I didn't see any significant change in vfio_main.c
that may fail gvt. So should the iommufd changes. Then we will re-run the
test after your update.:-)

-- 
Regards,
Yi Liu

Re: [Intel-gfx] [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Fri, 28 Oct 2022 15:40:09 -0300
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Wed, Oct 26, 2022 at 03:24:42PM -0600, Alex Williamson wrote:
> > On Tue, 25 Oct 2022 15:17:10 -0300
> > Jason Gunthorpe <jgg@nvidia.com> wrote:
> >   
> > > This legacy module knob has become uAPI, when set on the vfio_iommu_type1
> > > it disables some security protections in the iommu drivers. Move the
> > > storage for this knob to vfio_main.c so that iommufd can access it too.  
> > 
> > I don't really understand this, we're changing the behavior of the
> > iommufd_device_attach() operation based on the modules options of
> > vfio_iommu_type1,   
> 
> The specific reason it was done is that we had a misconfigured test VM
> in the farm that needed it, and that VM has since been fixed. But it
> did highlight we should try to preserve this in some way.
> 
> > which may not be loaded or even compiled into the
> > kernel.  Our compatibility story falls apart when VFIO_CONTAINER is not
> > set, iommufd sneaks in to usurp /dev/vfio/vfio, and the user's module
> > options for type1 go unprocessed.  
> 
> There are two aspects here, trying to preseve the
> allow_unsafe_interrupts knob as it is already as some ABI in the best
> way we can.
> 
> And the second is how do we make this work in the new world where
> there may be no type 1 module at all. This patch is not trying to
> address that topic. I am expecting a range of small adjustments before
> VFIO_CONTAINER=n becomes really fully viable.
> 
> > I hate to suggest that type1 becomes a module that does nothing more
> > than maintain consistency of this variable when the full type1 isn't
> > available, but is that what we need to do?  
> 
> It is one idea, it depends how literal you want to be on "module
> parameters are ABI". IMHO it is a weak form of ABI and the need of
> this paramter in particular is not that common in modern times, AFAIK.
> 
> So perhaps we just also expose it through vfio.ko and expect people to
> migrate. That would give a window were both options are available.

That might be best.  Ultimately this is an opt-out of a feature that
has security implications, so I'd rather error on the side of requiring
the user to re-assert that opt-out.  It seems the potential good in
eliminating stale or unnecessary options outweighs any weak claims of
preserving an ABI for a module that's no longer in service.

However, I'd question whether vfio is the right place for that new
module option.  As proposed, vfio is only passing it through to
iommufd, where an error related to lack of the hardware feature is
masked behind an -EPERM by the time it gets back to vfio, making any
sort of advisory to the user about the module option convoluted.  It
seems like iommufd should own the option to opt-out universally, not
just through the vfio use case.  Thanks,

Alex

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Fri, 28 Oct 2022 15:40:09 -0300
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Wed, Oct 26, 2022 at 03:24:42PM -0600, Alex Williamson wrote:
> > On Tue, 25 Oct 2022 15:17:10 -0300
> > Jason Gunthorpe <jgg@nvidia.com> wrote:
> >   
> > > This legacy module knob has become uAPI, when set on the vfio_iommu_type1
> > > it disables some security protections in the iommu drivers. Move the
> > > storage for this knob to vfio_main.c so that iommufd can access it too.  
> > 
> > I don't really understand this, we're changing the behavior of the
> > iommufd_device_attach() operation based on the modules options of
> > vfio_iommu_type1,   
> 
> The specific reason it was done is that we had a misconfigured test VM
> in the farm that needed it, and that VM has since been fixed. But it
> did highlight we should try to preserve this in some way.
> 
> > which may not be loaded or even compiled into the
> > kernel.  Our compatibility story falls apart when VFIO_CONTAINER is not
> > set, iommufd sneaks in to usurp /dev/vfio/vfio, and the user's module
> > options for type1 go unprocessed.  
> 
> There are two aspects here, trying to preseve the
> allow_unsafe_interrupts knob as it is already as some ABI in the best
> way we can.
> 
> And the second is how do we make this work in the new world where
> there may be no type 1 module at all. This patch is not trying to
> address that topic. I am expecting a range of small adjustments before
> VFIO_CONTAINER=n becomes really fully viable.
> 
> > I hate to suggest that type1 becomes a module that does nothing more
> > than maintain consistency of this variable when the full type1 isn't
> > available, but is that what we need to do?  
> 
> It is one idea, it depends how literal you want to be on "module
> parameters are ABI". IMHO it is a weak form of ABI and the need of
> this paramter in particular is not that common in modern times, AFAIK.
> 
> So perhaps we just also expose it through vfio.ko and expect people to
> migrate. That would give a window were both options are available.

That might be best.  Ultimately this is an opt-out of a feature that
has security implications, so I'd rather error on the side of requiring
the user to re-assert that opt-out.  It seems the potential good in
eliminating stale or unnecessary options outweighs any weak claims of
preserving an ABI for a module that's no longer in service.

However, I'd question whether vfio is the right place for that new
module option.  As proposed, vfio is only passing it through to
iommufd, where an error related to lack of the hardware feature is
masked behind an -EPERM by the time it gets back to vfio, making any
sort of advisory to the user about the module option convoluted.  It
seems like iommufd should own the option to opt-out universally, not
just through the vfio use case.  Thanks,

Alex

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Fri, 28 Oct 2022 15:40:09 -0300
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Wed, Oct 26, 2022 at 03:24:42PM -0600, Alex Williamson wrote:
> > On Tue, 25 Oct 2022 15:17:10 -0300
> > Jason Gunthorpe <jgg@nvidia.com> wrote:
> >   
> > > This legacy module knob has become uAPI, when set on the vfio_iommu_type1
> > > it disables some security protections in the iommu drivers. Move the
> > > storage for this knob to vfio_main.c so that iommufd can access it too.  
> > 
> > I don't really understand this, we're changing the behavior of the
> > iommufd_device_attach() operation based on the modules options of
> > vfio_iommu_type1,   
> 
> The specific reason it was done is that we had a misconfigured test VM
> in the farm that needed it, and that VM has since been fixed. But it
> did highlight we should try to preserve this in some way.
> 
> > which may not be loaded or even compiled into the
> > kernel.  Our compatibility story falls apart when VFIO_CONTAINER is not
> > set, iommufd sneaks in to usurp /dev/vfio/vfio, and the user's module
> > options for type1 go unprocessed.  
> 
> There are two aspects here, trying to preseve the
> allow_unsafe_interrupts knob as it is already as some ABI in the best
> way we can.
> 
> And the second is how do we make this work in the new world where
> there may be no type 1 module at all. This patch is not trying to
> address that topic. I am expecting a range of small adjustments before
> VFIO_CONTAINER=n becomes really fully viable.
> 
> > I hate to suggest that type1 becomes a module that does nothing more
> > than maintain consistency of this variable when the full type1 isn't
> > available, but is that what we need to do?  
> 
> It is one idea, it depends how literal you want to be on "module
> parameters are ABI". IMHO it is a weak form of ABI and the need of
> this paramter in particular is not that common in modern times, AFAIK.
> 
> So perhaps we just also expose it through vfio.ko and expect people to
> migrate. That would give a window were both options are available.

That might be best.  Ultimately this is an opt-out of a feature that
has security implications, so I'd rather error on the side of requiring
the user to re-assert that opt-out.  It seems the potential good in
eliminating stale or unnecessary options outweighs any weak claims of
preserving an ABI for a module that's no longer in service.

However, I'd question whether vfio is the right place for that new
module option.  As proposed, vfio is only passing it through to
iommufd, where an error related to lack of the hardware feature is
masked behind an -EPERM by the time it gets back to vfio, making any
sort of advisory to the user about the module option convoluted.  It
seems like iommufd should own the option to opt-out universally, not
just through the vfio use case.  Thanks,

Alex

Re: [PATCH 10/10] iommufd: Allow iommufd to supply /dev/vfio/vfio

[Alex Williamson]

On Fri, 28 Oct 2022 15:44:36 -0300
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Wed, Oct 26, 2022 at 03:31:33PM -0600, Alex Williamson wrote:
> > On Tue, 25 Oct 2022 15:50:45 -0300
> > Jason Gunthorpe <jgg@nvidia.com> wrote:
> >   
> > > If the VFIO container is compiled out, give a kconfig option for iommufd
> > > to provide the miscdev node with the same name and permissions as vfio
> > > uses.
> > > 
> > > The compatibility node supports the same ioctls as VFIO and automatically
> > > enables the VFIO compatible pinned page accounting mode.  
> > 
> > I think I'd like to see some sort of breadcrumb when /dev/vfio/vfio is
> > provided by something other than the vfio container code.  If we intend
> > to include this before P2P is resolved, that breadcrumb   
> 
> I don't belive I can get P2P done soon enough. I plan to do it after
> this is merged. Right now these two series are taking all my time.
> 
> > (dmesg I'm guessing) might also list any known limitations of the
> > compatibility to save time with debugging.  Thanks,  
> 
> Yes, that makes sense.
> 
> Do you want a dmesg at module load time, on every open, or a sysfs
> something? What seems like it would make it into a bug report?

I think dmesg at module load time should probably be ok, every open
seems like harassment and sysfs would require updated support in
various bug reporting tools.  Users are often terrible about reporting
full dmesg in bugs, but they do often filter it for "IOMMU" or "VFIO",
so keep that in mind when crafting the log message.  Thanks,

Alex

Re: [PATCH 10/10] iommufd: Allow iommufd to supply /dev/vfio/vfio

[Alex Williamson]

On Fri, 28 Oct 2022 15:44:36 -0300
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Wed, Oct 26, 2022 at 03:31:33PM -0600, Alex Williamson wrote:
> > On Tue, 25 Oct 2022 15:50:45 -0300
> > Jason Gunthorpe <jgg@nvidia.com> wrote:
> >   
> > > If the VFIO container is compiled out, give a kconfig option for iommufd
> > > to provide the miscdev node with the same name and permissions as vfio
> > > uses.
> > > 
> > > The compatibility node supports the same ioctls as VFIO and automatically
> > > enables the VFIO compatible pinned page accounting mode.  
> > 
> > I think I'd like to see some sort of breadcrumb when /dev/vfio/vfio is
> > provided by something other than the vfio container code.  If we intend
> > to include this before P2P is resolved, that breadcrumb   
> 
> I don't belive I can get P2P done soon enough. I plan to do it after
> this is merged. Right now these two series are taking all my time.
> 
> > (dmesg I'm guessing) might also list any known limitations of the
> > compatibility to save time with debugging.  Thanks,  
> 
> Yes, that makes sense.
> 
> Do you want a dmesg at module load time, on every open, or a sysfs
> something? What seems like it would make it into a bug report?

I think dmesg at module load time should probably be ok, every open
seems like harassment and sysfs would require updated support in
various bug reporting tools.  Users are often terrible about reporting
full dmesg in bugs, but they do often filter it for "IOMMU" or "VFIO",
so keep that in mind when crafting the log message.  Thanks,

Alex

Re: [Intel-gfx] [PATCH 10/10] iommufd: Allow iommufd to supply /dev/vfio/vfio

[Alex Williamson]

On Fri, 28 Oct 2022 15:44:36 -0300
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Wed, Oct 26, 2022 at 03:31:33PM -0600, Alex Williamson wrote:
> > On Tue, 25 Oct 2022 15:50:45 -0300
> > Jason Gunthorpe <jgg@nvidia.com> wrote:
> >   
> > > If the VFIO container is compiled out, give a kconfig option for iommufd
> > > to provide the miscdev node with the same name and permissions as vfio
> > > uses.
> > > 
> > > The compatibility node supports the same ioctls as VFIO and automatically
> > > enables the VFIO compatible pinned page accounting mode.  
> > 
> > I think I'd like to see some sort of breadcrumb when /dev/vfio/vfio is
> > provided by something other than the vfio container code.  If we intend
> > to include this before P2P is resolved, that breadcrumb   
> 
> I don't belive I can get P2P done soon enough. I plan to do it after
> this is merged. Right now these two series are taking all my time.
> 
> > (dmesg I'm guessing) might also list any known limitations of the
> > compatibility to save time with debugging.  Thanks,  
> 
> Yes, that makes sense.
> 
> Do you want a dmesg at module load time, on every open, or a sysfs
> something? What seems like it would make it into a bug report?

I think dmesg at module load time should probably be ok, every open
seems like harassment and sysfs would require updated support in
various bug reporting tools.  Users are often terrible about reporting
full dmesg in bugs, but they do often filter it for "IOMMU" or "VFIO",
so keep that in mind when crafting the log message.  Thanks,

Alex

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Jason Gunthorpe]

On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
> > There is something wrong with the test suite that it isn't covering
> > the above, I'm going to look into that today.
> 
> sounds to be the cause. I didn't see any significant change in vfio_main.c
> that may fail gvt. So should the iommufd changes. Then we will re-run the
> test after your update.:-)

I updated the github with all the changes made so far, it is worth
trying again!

Thanks,
Jason

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Jason Gunthorpe]

On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
> > There is something wrong with the test suite that it isn't covering
> > the above, I'm going to look into that today.
> 
> sounds to be the cause. I didn't see any significant change in vfio_main.c
> that may fail gvt. So should the iommufd changes. Then we will re-run the
> test after your update.:-)

I updated the github with all the changes made so far, it is worth
trying again!

Thanks,
Jason

Re: [Intel-gfx] [PATCH 00/10] Connect VFIO to IOMMUFD

[Jason Gunthorpe]

On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
> > There is something wrong with the test suite that it isn't covering
> > the above, I'm going to look into that today.
> 
> sounds to be the cause. I didn't see any significant change in vfio_main.c
> that may fail gvt. So should the iommufd changes. Then we will re-run the
> test after your update.:-)

I updated the github with all the changes made so far, it is worth
trying again!

Thanks,
Jason

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Yi Liu]

On 2022/11/1 07:24, Jason Gunthorpe wrote:
> On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
>>> There is something wrong with the test suite that it isn't covering
>>> the above, I'm going to look into that today.
>>
>> sounds to be the cause. I didn't see any significant change in vfio_main.c
>> that may fail gvt. So should the iommufd changes. Then we will re-run the
>> test after your update.:-)
> 
> I updated the github with all the changes made so far, it is worth
> trying again!

gvt is still failing with below call trace in host side. vfio_unpin_pages()
is still in problem. Any idea on it?


[  206.464107] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.464167] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.464168] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.464171] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.464172] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.464173] Call Trace:
[  206.464174]  <TASK>
[  206.464175]  vfio_unpin_pages+0x49/0x60
[  206.464177]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.464187]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.464196]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.464229]  vfs_write+0xc6/0x3b0
[  206.464296] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.464301] R10: 00000000008b66a8 R11: 0000000000000293 R12: 
00000000008b66a8
[  206.464318] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.464357]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.464377] RSP: 0018:ffffc0b1c301fb90 EFLAGS: 00010283
[  206.464381] R10: 00000000000b66b4 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.464382] R13: 000000010d0f6000 R14: ffffc0b1c301fc58 R15: 
ffff9d7a8d400000
[  206.464384] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.464385] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.464386] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.464387] Call Trace:
[  206.464388]  <TASK>
[  206.464389]  ? x86_emulate_instruction+0x4cd/0x890 [kvm]
[  206.464415]  vfio_pin_pages+0x5e/0xb0
[  206.464417]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.464434]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.464505] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.464509] R10: 00000000008b66b0 R11: 0000000000000293 R12: 
00000000008b66b0
[  206.464513]  </TASK>
[  206.464519] ------------[ cut here ]------------
[  206.464558]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.464572] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.464578] RSP: 0018:ffffc0b1c301fbc0 EFLAGS: 00010246
[  206.464583] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.464584] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.464586] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.464588] Call Trace:
[  206.464588]  <TASK>
[  206.464590]  vfio_unpin_pages+0x49/0x60
[  206.464591]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.464610]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.464640]  vfio_device_fops_write+0x1f/0x40
[  206.464681] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.464690]  </TASK>
[  206.464702] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.464742]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.464756] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.464767] R13: 000000010d0f6000 R14: ffffc0b1c301fbd0 R15: 
ffff9d7a8d400000
[  206.464771] Call Trace:
[  206.464772]  <TASK>
[  206.464773]  ? writeback_registers+0x56/0xb0 [kvm]
[  206.464802]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.464811]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.464819]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.464826]  ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
[  206.464854]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.464879]  ? intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.464902]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.464910] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.464912] R10: 00000000008b66b8 R11: 0000000000000293 R12: 
00000000008b66b8
[  206.464922] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.464960]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.464974] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.464985] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.464987] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.464988] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.464989] Call Trace:
[  206.464990]  <TASK>
[  206.464992]  vfio_unpin_pages+0x49/0x60
[  206.464993]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.465003]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.465012]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.465019]  ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
[  206.465039]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.465047]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.465054]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.465061]  vfio_device_fops_write+0x1f/0x40
[  206.465063]  vfs_write+0xc6/0x3b0
[  206.465065]  ? intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.465081]  __x64_sys_pwrite64+0x96/0xc0
[  206.465100] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.465106] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.465159]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.465173] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.465175] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.465184] R13: 000000010d0f6000 R14: ffffc0b1c301fb68 R15: 
ffff9d7a8d400000
[  206.465186] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.465188] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.465189] Call Trace:
[  206.465191]  ? segmented_write.isra.0+0x5e/0x90 [kvm]
[  206.465218]  vfio_pin_pages+0x5e/0xb0
[  206.465220]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.465230]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.465239]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.465247]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.465255]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.465261]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.465269]  vfio_device_fops_write+0x1f/0x40
[  206.465270]  vfs_write+0xc6/0x3b0
[  206.465272]  ? intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.465280]  ? __fget_light+0xa7/0x130
[  206.465285]  __x64_sys_pwrite64+0x96/0xc0
[  206.465320]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.465336] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.465338] R10: 00000000008b66c0 R11: 0000000000000293 R12: 
00000000008b66c0
[  206.465347] ------------[ cut here ]------------
[  206.465386]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.465400] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.465405] RSP: 0018:ffffc0b1c301fad0 EFLAGS: 00010246
[  206.465410] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.465412] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.465413] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.465414] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.465415] Call Trace:
[  206.465416]  <TASK>
[  206.465417]  vfio_unpin_pages+0x49/0x60
[  206.465419]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.465429]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.465438]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.465446]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.465461]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.465481]  __x64_sys_pwrite64+0x96/0xc0
[  206.465522]  ? do_syscall_64+0x69/0x90
[  206.465530] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.465538] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.465590]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.465604] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.465613] R10: 00000000000b66cc R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.465615] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.465617] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.465618] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.465619] Call Trace:
[  206.465620]  <TASK>
[  206.465621]  ? x86_emulate_instruction+0x4cd/0x890 [kvm]
[  206.465649]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.465672]  ? vmx_vcpu_put+0x122/0x210 [kvm_intel]
[  206.465702]  vfs_write+0xc6/0x3b0
[  206.465753]  ? do_syscall_64+0x69/0x90
[  206.465760] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.465762] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.465769]  </TASK>
[  206.465775] ------------[ cut here ]------------
[  206.465814]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.465828] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.465834] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.465835] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.465836] RBP: ffffc0b1c301fb10 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.465837] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.465838] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.465839] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.465841] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.465842] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.465843] Call Trace:
[  206.465844]  <TASK>
[  206.465845]  vfio_unpin_pages+0x49/0x60
[  206.465847]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.465857]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.465872]  ? vmx_vcpu_put+0x122/0x210 [kvm_intel]
[  206.465902]  vfio_device_fops_write+0x1f/0x40
[  206.465913]  ? __fget_light+0xa7/0x130
[  206.465957]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.465965] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.465982] ------------[ cut here ]------------
[  206.465986] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.466051] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.466052] Call Trace:
[  206.466052]  <TASK>
[  206.466083]  vfio_pin_pages+0x5e/0xb0
[  206.466085]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.466094]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.466102]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.466129]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.466165]  __x64_sys_pwrite64+0x96/0xc0
[  206.466176] RIP: 0033:0x7fcbbebcacbf
[  206.466187]  </TASK>
[  206.466193] ------------[ cut here ]------------
[  206.466194] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.466236]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.466250] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.466261] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.466268]  vfio_unpin_pages+0x49/0x60
[  206.466279]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.466288]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.466296]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.466304]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.466311]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.466322]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.466342]  ? __fget_light+0xa7/0x130
[  206.466344]  __x64_sys_pwrite64+0x96/0xc0
[  206.466347]  do_syscall_64+0x59/0x90
[  206.466351]  ? do_syscall_64+0x69/0x90
[  206.466358] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.466387] ------------[ cut here ]------------
[  206.466427]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.466441] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.466443] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.466456] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.466504]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.466510]  ? restore_fpregs_from_fpstate+0x47/0xc0
[  206.466514]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.466521]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.466528]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.466537]  vfs_write+0xc6/0x3b0
[  206.466539]  ? vfio_device_fops_write+0x1f/0x40
[  206.466567]  __x64_sys_pwrite64+0x96/0xc0
[  206.466570]  do_syscall_64+0x59/0x90
[  206.466572]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.466574]  ? do_syscall_64+0x69/0x90
[  206.466575]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.466585] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.466599] ------------[ cut here ]------------
[  206.466600] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.466639]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.466653] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.466666] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.466672]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.466682]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.466691]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.466697]  ? restore_fpregs_from_fpstate+0x47/0xc0
[  206.466701]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.466709]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.466716]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.466723]  vfio_device_fops_write+0x1f/0x40
[  206.466724]  vfs_write+0xc6/0x3b0
[  206.466727]  ? vfio_device_fops_write+0x1f/0x40
[  206.466728]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.466752]  ? __fget_light+0xa7/0x130
[  206.466755]  __x64_sys_pwrite64+0x96/0xc0
[  206.466758]  do_syscall_64+0x59/0x90
[  206.466765]  ? do_syscall_64+0x69/0x90
[  206.466772] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.466794] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.466833]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.466847] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.466862] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.466883]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.466891]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.466899]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.466912]  vfio_device_fops_write+0x1f/0x40
[  206.466914]  vfs_write+0xc6/0x3b0
[  206.466918]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.466942]  ? __fget_light+0xa7/0x130
[  206.466945]  __x64_sys_pwrite64+0x96/0xc0
[  206.466947]  do_syscall_64+0x59/0x90
[  206.466949]  ? do_syscall_64+0x69/0x90
[  206.466958] RIP: 0033:0x7fcbbebcacbf
[  206.466963] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.466977] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.467057]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.467066]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.467074]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.467082]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.467089]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467096]  vfio_device_fops_write+0x1f/0x40
[  206.467097]  vfs_write+0xc6/0x3b0
[  206.467099]  ? vfio_device_fops_write+0x1f/0x40
[  206.467101]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.467125]  ? __fget_light+0xa7/0x130
[  206.467128]  __x64_sys_pwrite64+0x96/0xc0
[  206.467131]  do_syscall_64+0x59/0x90
[  206.467132]  ? do_syscall_64+0x69/0x90
[  206.467134]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.467136]  ? do_syscall_64+0x69/0x90
[  206.467137]  ? do_syscall_64+0x69/0x90
[  206.467138]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.467142] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.467204]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.467218] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.467219] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.467220] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.467231] R10: 00000000000b66ec R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.467265]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.467273]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.467280]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467287]  vfio_device_fops_write+0x1f/0x40
[  206.467288]  vfs_write+0xc6/0x3b0
[  206.467322]  ? __fget_light+0xa7/0x130
[  206.467324]  __x64_sys_pwrite64+0x96/0xc0
[  206.467327]  do_syscall_64+0x59/0x90
[  206.467329]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.467331]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.467333]  ? do_syscall_64+0x69/0x90
[  206.467335]  ? do_syscall_64+0x69/0x90
[  206.467336]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.467338]  ? do_syscall_64+0x69/0x90
[  206.467340]  ? do_syscall_64+0x69/0x90
[  206.467345] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.467363] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.467442]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.467450]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.467458]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.467466]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.467473]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467480]  vfio_device_fops_write+0x1f/0x40
[  206.467482]  vfs_write+0xc6/0x3b0
[  206.467484]  ? intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467491]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.467515]  ? __fget_light+0xa7/0x130
[  206.467518]  __x64_sys_pwrite64+0x96/0xc0
[  206.467520]  do_syscall_64+0x59/0x90
[  206.467522]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.467525]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.467533]  ? do_syscall_64+0x69/0x90
[  206.467542] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.467563] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.467678]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.467686]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.467693]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.467700]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467707]  vfio_device_fops_write+0x1f/0x40
[  206.467709]  vfs_write+0xc6/0x3b0
[  206.467731]  ? __fget_light+0xa7/0x130
[  206.467733]  __x64_sys_pwrite64+0x96/0xc0
[  206.467736]  do_syscall_64+0x59/0x90
[  206.467738]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.467740]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.467742]  ? do_syscall_64+0x69/0x90
[  206.467744]  ? do_syscall_64+0x69/0x90
[  206.467748] RIP: 0033:0x7fcbbebcacbf
[  206.467751] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.467765] ------------[ cut here ]------------
[  206.467768] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.467847]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.467856]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.467864]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.467872]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.467879]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467886]  vfio_device_fops_write+0x1f/0x40
[  206.467887]  vfs_write+0xc6/0x3b0
[  206.467890]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.467910]  ? __fget_light+0xa7/0x130
[  206.467916]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.467922]  ? do_syscall_64+0x69/0x90
[  206.467925]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.467927] RIP: 0033:0x7fcbbebcacbf
[  206.467928] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.467933] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.467990]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.468003] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.468007] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.468098]  vfio_device_fops_write+0x1f/0x40
[  206.468122]  ? __fget_light+0xa7/0x130
[  206.468124]  ? __fget_light+0xa7/0x130
[  206.468126]  __x64_sys_pwrite64+0x96/0xc0
[  206.468129]  do_syscall_64+0x59/0x90
[  206.468130]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.468133]  ? do_syscall_64+0x69/0x90
[  206.468137]  ? do_syscall_64+0x69/0x90
[  206.468141] RIP: 0033:0x7fcbbebcacbf
[  206.468143] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.468145] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.468146] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.468147] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.468197]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.468211] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.468212] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.468215] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.468229] Call Trace:
[  206.468283]  vfs_write+0xc6/0x3b0
[  206.468312]  do_syscall_64+0x59/0x90
[  206.468313]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.468315]  ? do_syscall_64+0x69/0x90
[  206.468318]  ? do_syscall_64+0x69/0x90
[  206.468321]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.468323] RIP: 0033:0x7fcbbebcacbf
[  206.468324] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.468327] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.468328] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.468329] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.468336] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.468387]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.468404] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.468407] RSP: 0018:ffffc0b1c301fb50 EFLAGS: 00010283
[  206.468410] RBP: ffffc0b1c301fbb8 R08: 0000000000000001 R09: 
ffffc0b1c301fc18
[  206.468416] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.468532]  ? do_syscall_64+0x69/0x90
[  206.468538] RIP: 0033:0x7fcbbebcacbf
[  206.468539] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.468540] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.468542] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.468544] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.468545] R10: 00000000008b6700 R11: 0000000000000293 R12: 
00000000008b6700
[  206.468546] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.468549]  </TASK>
[  206.468549] ---[ end trace 0000000000000000 ]---
[  206.468557] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.468676]  vfio_device_fops_write+0x1f/0x40
[  206.468723] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.468724] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.468725] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.468726] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.468727] R10: 00000000008b6700 R11: 0000000000000293 R12: 
00000000008b6700
[  206.468728] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.468731]  </TASK>
[  206.468731] ---[ end trace 0000000000000000 ]---
[  206.468732] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.468742] ------------[ cut here ]------------
[  206.468743] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.468746] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.468808] R13: 000000010d0f6000 R14: ffffc0b1c301fba0 R15: 
ffff9d7a8d400000
[  206.468813] Call Trace:
[  206.468825]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.468921]  do_syscall_64+0x59/0x90
[  206.468938] RIP: 0033:0x7fcbbebcacbf
[  206.468942] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.468944] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.468945] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.468946] R10: 00000000008b6708 R11: 0000000000000293 R12: 
00000000008b6708
[  206.468947] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.468949]  </TASK>
[  206.468950] ---[ end trace 0000000000000000 ]---
[  206.468951] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.468955] ------------[ cut here ]------------
[  206.468956] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.468958] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.469023] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.469046]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.469140]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.469157] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.469158] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.469159] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.469161] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.469164]  </TASK>
[  206.469165] ---[ end trace 0000000000000000 ]---
[  206.469166] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.469176] ------------[ cut here ]------------
[  206.469179] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.469311]  ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
[  206.469360]  __x64_sys_pwrite64+0x96/0xc0
[  206.469387] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.469391] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.469392] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.469394] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.469397]  </TASK>
[  206.469398] ---[ end trace 0000000000000000 ]---
[  206.469399] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.469403] ------------[ cut here ]------------
[  206.469404] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.469443]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.469457] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.469463] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.469466] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.469468] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.469486]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.469537]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.469569]  ? do_syscall_64+0x69/0x90
[  206.469579] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.469581] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.469582] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.469583] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.469584] R10: 00000000008b6710 R11: 0000000000000293 R12: 
00000000008b6710
[  206.469585] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.469587]  </TASK>
[  206.469588] ---[ end trace 0000000000000000 ]---
[  206.469589] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.469601] ------------[ cut here ]------------
[  206.469602] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.469605] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.469664] RBP: ffffc0b1c301fb10 R08: 0000000000000001 R09: 
ffffc0b1c301fb70
[  206.469669] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.469673]  ? x86_emulate_instruction+0x4cd/0x890 [kvm]
[  206.469772]  ? do_syscall_64+0x69/0x90
[  206.469782] RIP: 0033:0x7fcbbebcacbf
[  206.469785] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.469787] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.469788] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.469789] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.469790] R10: 00000000008b6718 R11: 0000000000000293 R12: 
00000000008b6718
[  206.469791] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.469793]  </TASK>
[  206.469794] ---[ end trace 0000000000000000 ]---
[  206.469795] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.469799] ------------[ cut here ]------------
[  206.469839]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.469853] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.469854] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.469858] RSP: 0018:ffffc0b1c301fad8 EFLAGS: 00010246
[  206.469861] RBP: ffffc0b1c301fb18 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.469897]  ? restore_fpregs_from_fpstate+0x47/0xc0
[  206.469939]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.469941]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.469943]  ? do_syscall_64+0x69/0x90
[  206.469945]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.469947]  ? do_syscall_64+0x69/0x90
[  206.469948]  ? do_syscall_64+0x69/0x90
[  206.469949]  ? do_syscall_64+0x69/0x90
[  206.469950]  ? do_syscall_64+0x69/0x90
[  206.469952]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.469954] RIP: 0033:0x7fcbbebcacbf
[  206.469955] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.469957] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.469958] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.469959] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.469960] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.469961] R10: 00000000008b6718 R11: 0000000000000293 R12: 
00000000008b6718
[  206.469962] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.469965]  </TASK>
[  206.469966] ---[ end trace 0000000000000000 ]---
[  206.469967] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.469977] ------------[ cut here ]------------
[  206.469978] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.470017]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.470035] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.470043] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.470161]  do_syscall_64+0x59/0x90
[  206.470172]  ? do_syscall_64+0x69/0x90
[  206.470175]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.470177] RIP: 0033:0x7fcbbebcacbf
[  206.470179] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.470180] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.470182] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.470183] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.470184] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.470185] R10: 00000000008b6720 R11: 0000000000000293 R12: 
00000000008b6720
[  206.470186] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.470188]  </TASK>
[  206.470189] ---[ end trace 0000000000000000 ]---
[  206.470190] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.470194] ------------[ cut here ]------------
[  206.470237]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.470251] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.470257] RSP: 0018:ffffc0b1c301fb30 EFLAGS: 00010246
[  206.470258] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.470262] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.470281]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.470359]  ? do_syscall_64+0x69/0x90
[  206.470367] RIP: 0033:0x7fcbbebcacbf
[  206.470368] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.470369] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.470371] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.470372] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.470373] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.470374] R10: 00000000008b6720 R11: 0000000000000293 R12: 
00000000008b6720
[  206.470375] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.470398] ------------[ cut here ]------------
[  206.470438]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.470457] RSP: 0018:ffffc0b1c301fb38 EFLAGS: 00010283
[  206.470458] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.470460] RBP: ffffc0b1c301fba0 R08: 0000000000000001 R09: 
ffffc0b1c301fc00
[  206.470465] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.470470]  vfio_pin_pages+0x5e/0xb0
[  206.470526]  ? __fget_light+0xa7/0x130
[  206.470547] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.470549] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.470550] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.470551] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.470552] R10: 00000000008b6728 R11: 0000000000000293 R12: 
00000000008b6728
[  206.470553] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.470556]  </TASK>
[  206.470556] ---[ end trace 0000000000000000 ]---
[  206.470557] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.470561] ------------[ cut here ]------------
[  206.470562] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.470601]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.470616] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.470619] RSP: 0018:ffffc0b1c301fb68 EFLAGS: 00010246
[  206.470623] RBP: ffffc0b1c301fba8 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.470627] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.470629] Call Trace:
[  206.470652]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.470698]  ? do_syscall_64+0x69/0x90
[  206.470711] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.470715] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.470716] R10: 00000000008b6728 R11: 0000000000000293 R12: 
00000000008b6728
[  206.470717] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.470719]  </TASK>
[  206.470720] ---[ end trace 0000000000000000 ]---
[  206.470721] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.470731] ------------[ cut here ]------------
[  206.470732] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.470734] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.470843]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.470878]  ? do_syscall_64+0x69/0x90
[  206.470886] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.470887] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.470888] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.470889] R10: 00000000008b6730 R11: 0000000000000293 R12: 
00000000008b6730
[  206.470890] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.470893]  </TASK>
[  206.470893] ---[ end trace 0000000000000000 ]---
[  206.470894] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.470898] ------------[ cut here ]------------
[  206.470899] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.470901] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.470967] Call Trace:
[  206.470996]  ? vmx_vcpu_put+0x122/0x210 [kvm_intel]
[  206.471040]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.471052] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.471054] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.471055] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.471056] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.471057] R10: 00000000008b6730 R11: 0000000000000293 R12: 
00000000008b6730
[  206.471058] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.471060]  </TASK>
[  206.471061] ---[ end trace 0000000000000000 ]---
[  206.471062] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.471072] ------------[ cut here ]------------
[  206.471073] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.471075] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.471136] R10: 00000000000b673c R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.471141] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.471207]  ? do_syscall_64+0x69/0x90
[  206.471216] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.471218] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.471219] R10: 00000000008b6738 R11: 0000000000000293 R12: 
00000000008b6738
[  206.471220] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.471224]  </TASK>
[  206.471225] ---[ end trace 0000000000000000 ]---
[  206.471227] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.471231] ------------[ cut here ]------------
[  206.471232] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.471234] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.471293] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.471296] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.471299] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.471314]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.471367]  ? do_syscall_64+0x69/0x90
[  206.471376] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.471379] R10: 00000000008b6738 R11: 0000000000000293 R12: 
00000000008b6738
[  206.471380] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.471383]  </TASK>
[  206.471384] ---[ end trace 0000000000000000 ]---
[  206.471385] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.471396] ------------[ cut here ]------------
[  206.471397] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.471399] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.471464] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.471500]  vfio_pin_pages+0x5e/0xb0
[  206.471519]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.471568]  vfio_device_fops_write+0x1f/0x40
[  206.471594] RIP: 0033:0x7fcbbebcacbf
[  206.471600] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.471605]  </TASK>
[  206.471606] ---[ end trace 0000000000000000 ]---
[  206.471606] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.471611] ------------[ cut here ]------------
[  206.471614] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.471671] RSP: 0018:ffffc0b1c301fb60 EFLAGS: 00010246
[  206.471673] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.471677] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.471681] Call Trace:
[  206.471684]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.471730]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.471754]  vfs_write+0xc6/0x3b0
[  206.471762]  ? __fget_light+0xa7/0x130
[  206.471781] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.471791] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.471801] ------------[ cut here ]------------
[  206.471804] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.471867] R13: 000000010d0f6000 R14: ffffc0b1c301fb70 R15: 
ffff9d7a8d400000
[  206.471903]  vfio_pin_pages+0x5e/0xb0
[  206.471921]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.471956]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.471978]  __x64_sys_pwrite64+0x96/0xc0
[  206.471988]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.472004] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.472019] ------------[ cut here ]------------
[  206.472022] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.472080] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.472081] RBP: ffffc0b1c301fb18 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.472085] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.472091]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.472116]  ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
[  206.472152]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.472166]  __x64_sys_pwrite64+0x96/0xc0
[  206.472173]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.472179]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.472194] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.472213] ------------[ cut here ]------------
[  206.472256]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.472270] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.472272] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.472273] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.472275] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.472279] RBP: ffffc0b1c301fb68 R08: 0000000000000001 R09: 
ffffc0b1c301fbc8
[  206.472285] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.472315]  vfio_pin_pages+0x5e/0xb0
[  206.472333]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.472356]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.472370]  __x64_sys_pwrite64+0x96/0xc0
[  206.472379]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.472392] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.472405] ------------[ cut here ]------------
[  206.472406] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.472408] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.472468] RBP: ffffc0b1c301fb70 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.472472] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.472475] Call Trace:
[  206.472478]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.472513]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.472531]  ? __fget_light+0xa7/0x130
[  206.472537]  ? fire_user_return_notifiers+0x42/0x70
[  206.472554] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.472564] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.472575] ------------[ cut here ]------------
[  206.472576] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.472616]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.472630] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.472632] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.472632] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.472634] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.472642] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.472648]  ? kvm_pmu_trigger_event+0x59/0x250 [kvm]
[  206.472696]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.472738]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.472749]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.472788]  ? do_syscall_64+0x69/0x90
[  206.472797] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.472804]  </TASK>
[  206.472804] ---[ end trace 0000000000000000 ]---
[  206.472805] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.472809] ------------[ cut here ]------------
[  206.472810] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.472812] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.472875] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.472882]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.472907]  ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
[  206.472983]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.473001] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.473002] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.473003] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.473004] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.473005] R10: 00000000008b6758 R11: 0000000000000293 R12: 
00000000008b6758
[  206.473006] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.473009]  </TASK>
[  206.473010] ---[ end trace 0000000000000000 ]---
[  206.473011] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.473021] ------------[ cut here ]------------
[  206.473022] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.473024] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.473087] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.473091] Call Trace:
[  206.473118]  ? kvm_arch_vcpu_ioctl_run+0x3c4/0x1850 [kvm]
[  206.473171]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.473191] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.473196]  </TASK>
[  206.473197] ---[ end trace 0000000000000000 ]---
[  206.473198] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.473202] ------------[ cut here ]------------
[  206.473203] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.473205] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.473265] RSP: 0018:ffffc0b1c301fbc0 EFLAGS: 00010246
[  206.473269] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.473273] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.473278]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.473344]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.473365]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.473382] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.473401] ------------[ cut here ]------------
[  206.473402] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.473405] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.473442]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.473461] RSP: 0018:ffffc0b1c301fb20 EFLAGS: 00010283
[  206.473463] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.473468] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.473476]  vfio_pin_pages+0x5e/0xb0
[  206.473494]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.473536]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.473551]  vfs_write+0xc6/0x3b0
[  206.473583]  __x64_sys_pwrite64+0x96/0xc0
[  206.473592]  ? do_syscall_64+0x69/0x90
[  206.473603] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.473616] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.473653]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.473674] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.473675] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.473680] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.473686]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.473711]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.473762]  vfs_write+0xc6/0x3b0
[  206.473793]  __x64_sys_pwrite64+0x96/0xc0
[  206.473800]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.473812] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.473833] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.473870]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.473884] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.473886] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.473887] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.473888] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.473894] R10: 00000000000b6774 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.473900] Call Trace:
[  206.473928]  vfio_pin_pages+0x5e/0xb0
[  206.473947]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.473965]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.473982]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.474011]  __x64_sys_pwrite64+0x96/0xc0
[  206.474022] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.474031] ---[ end trace 0000000000000000 ]---
[  206.474075]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.474089] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.474091] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.474092] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.474093] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.474095] RSP: 0018:ffffc0b1c301fbc0 EFLAGS: 00010246
[  206.474096] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.474097] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.474098] RBP: ffffc0b1c301fc00 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.474099] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.474100] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.474103] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.474104] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.474120]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.474152]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.474167]  vfs_write+0xc6/0x3b0
[  206.474196]  ? __fget_light+0xa7/0x130
[  206.474205]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.474209] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.474212] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.474281]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.474295] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.474296] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.474297] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.474299] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.474306] R13: 000000010d0f6000 R14: ffffc0b1c301fbd0 R15: 
ffff9d7a8d400000
[  206.474323]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.474349]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.474386]  ? intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.474400]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.474406]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.474413] RIP: 0033:0x7fcbbebcacbf
[  206.474416] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.474421] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.474470]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.474485] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.474486] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.474487] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.474489] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.474493] RBP: ffffc0b1c301fb78 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.474498] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.474499] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.474522]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.474554]  vfio_device_fops_write+0x1f/0x40
[  206.474577]  ? intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.474590]  do_syscall_64+0x59/0x90
[  206.474597]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.474604] RIP: 0033:0x7fcbbebcacbf
[  206.474607] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.474611] R10: 00000000008b6778 R11: 0000000000000293 R12: 
00000000008b6778
[  206.474668]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.474682] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.474683] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.474684] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.474686] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.474687] RSP: 0018:ffffc0b1c301fb20 EFLAGS: 00010283
[  206.474689] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.474690] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.474691] RBP: ffffc0b1c301fb88 R08: 0000000000000001 R09: 
ffffc0b1c301fbe8
[  206.474696] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.474700]  ? writeback_registers+0x56/0xb0 [kvm]
[  206.474774]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.474824]  vfs_write+0xc6/0x3b0
[  206.474834]  ? __fget_light+0xa7/0x130
[  206.474841]  ? fire_user_return_notifiers+0x42/0x70
[  206.474849]  ? do_syscall_64+0x69/0x90
[  206.474854] RIP: 0033:0x7fcbbebcacbf
[  206.474856] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.474864]  </TASK>
[  206.474873] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.474925] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.474935] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.474936] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.474939] Call Trace:
[  206.474943]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.474989]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.475012]  vfs_write+0xc6/0x3b0
[  206.475021]  ? __fget_light+0xa7/0x130
[  206.475028]  ? fire_user_return_notifiers+0x42/0x70
[  206.475036]  ? do_syscall_64+0x69/0x90
[  206.475042] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.475104]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.475118] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.475120] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.475121] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.475123] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.475124] RSP: 0018:ffffc0b1c301fb90 EFLAGS: 00010283
[  206.475125] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.475126] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.475127] RBP: ffffc0b1c301fbf8 R08: 0000000000000001 R09: 
ffffc0b1c301fc58
[  206.475128] R10: 00000000000b678c R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.475129] R13: 000000010d0f6000 R14: ffffc0b1c301fc58 R15: 
ffff9d7a8d400000
[  206.475134] Call Trace:
[  206.475158]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.475182]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.475204]  vfs_write+0xc6/0x3b0
[  206.475215]  do_syscall_64+0x59/0x90
[  206.475220] RIP: 0033:0x7fcbbebcacbf
[  206.475224] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.475239] ------------[ cut here ]------------
[  206.475279]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.475294] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.475295] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.475296] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.475298] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.475299] RSP: 0018:ffffc0b1c301fbc0 EFLAGS: 00010246
[  206.475300] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.475301] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.475302] RBP: ffffc0b1c301fc00 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.475304] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.475307] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.475323]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.475354]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.475368]  ? __fget_light+0xa7/0x130
[  206.475376]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.475381] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.475386] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.475442]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.475456] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.475458] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.475458] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.475460] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.475467] R13: 000000010d0f6000 R14: ffffc0b1c301fbb0 R15: 
ffff9d7a8d400000
[  206.475474]  vfio_pin_pages+0x5e/0xb0
[  206.475533]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.475549]  vfs_write+0xc6/0x3b0
[  206.475581]  do_syscall_64+0x59/0x90
[  206.475587]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.475594] RIP: 0033:0x7fcbbebcacbf
[  206.475597] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.475599] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.475650]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.475664] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.475665] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.475666] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.475668] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.475669] RSP: 0018:ffffc0b1c301fb18 EFLAGS: 00010246
[  206.475672] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.475676] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.475679] Call Trace:
[  206.475693]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.475743]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.475762]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.475795]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.475802]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.475807] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.475827] ------------[ cut here ]------------
[  206.475867]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.475881] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.475883] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.475884] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.475885] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.475887] RSP: 0018:ffffc0b1c301fad0 EFLAGS: 00010283
[  206.475888] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.475889] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.475890] RBP: ffffc0b1c301fb38 R08: 0000000000000001 R09: 
ffffc0b1c301fb98
[  206.475894] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.475918]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.475942]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.475957]  vfs_write+0xc6/0x3b0
[  206.476003]  ? __fget_light+0xa7/0x130
[  206.476012]  ? do_syscall_64+0x69/0x90
[  206.476020]  ? do_syscall_64+0x69/0x90
[  206.476026] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476034]  </TASK>
[  206.476043] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.476079]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476103] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.476106] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.476131]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.476162]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.476193]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.476233]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.476242] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476303]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476317] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.476319] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.476320] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.476321] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.476323] RSP: 0018:ffffc0b1c301fb90 EFLAGS: 00010283
[  206.476324] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.476325] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.476326] RBP: ffffc0b1c301fbf8 R08: 0000000000000001 R09: 
ffffc0b1c301fc58
[  206.476327] R10: 00000000000b67a4 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.476333] Call Trace:
[  206.476346]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.476376]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.476389]  ? do_syscall_64+0x69/0x90
[  206.476395]  __x64_sys_pwrite64+0x96/0xc0
[  206.476403] RIP: 0033:0x7fcbbebcacbf
[  206.476405] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476409] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.476458]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476472] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.476474] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.476475] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.476476] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.476478] RSP: 0018:ffffc0b1c301fbc0 EFLAGS: 00010246
[  206.476479] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.476480] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.476481] RBP: ffffc0b1c301fc00 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.476484] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.476490]  vfio_unpin_pages+0x49/0x60
[  206.476491]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.476533]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.476547]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.476555]  ? do_syscall_64+0x69/0x90
[  206.476561] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476563] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.476566] R10: 00000000008b67a0 R11: 0000000000000293 R12: 
00000000008b67a0
[  206.476582] ------------[ cut here ]------------
[  206.476621]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476635] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.476636] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.476637] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.476639] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.476640] RSP: 0018:ffffc0b1c301faf0 EFLAGS: 00010283
[  206.476645] R13: 000000010d0f6000 R14: ffffc0b1c301fbb8 R15: 
ffff9d7a8d400000
[  206.476670]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.476708]  vfs_write+0xc6/0x3b0
[  206.476723]  ? do_syscall_64+0x69/0x90
[  206.476730]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.476734]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.476739] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476742] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.476748] ---[ end trace 0000000000000000 ]---
[  206.476792]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476806] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.476807] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.476808] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.476810] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.476811] RSP: 0018:ffffc0b1c301fb20 EFLAGS: 00010246
[  206.476814] RBP: ffffc0b1c301fb60 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.476825]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.476835]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.476850]  ? vmx_vcpu_put+0x122/0x210 [kvm_intel]
[  206.476890]  ? __fget_light+0xa7/0x130
[  206.476900]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.476907]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.476912] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476913] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.476916] R10: 00000000008b67a8 R11: 0000000000000293 R12: 
00000000008b67a8
[  206.476971]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476985] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.476986] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.476987] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.476989] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.476990] RSP: 0018:ffffc0b1c301fac0 EFLAGS: 00010283
[  206.476992] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.476993] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.476994] RBP: ffffc0b1c301fb28 R08: 0000000000000001 R09: 
ffffc0b1c301fb88
[  206.476995] R10: 00000000000b67b4 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.476996] R13: 000000010d0f6000 R14: ffffc0b1c301fb88 R15: 
ffff9d7a8d400000
[  206.476997] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.476998] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.476999] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.477000] Call Trace:
[  206.477048]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.477078]  vfio_device_fops_write+0x1f/0x40
[  206.477117]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.477126]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.477131] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.477133] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.477137] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.477148] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.477199] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.477200] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.477201] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.477204] RSP: 0018:ffffc0b1c301faf0 EFLAGS: 00010246
[  206.477207] RBP: ffffc0b1c301fb30 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.477247]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.477255]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.477276]  __x64_sys_pwrite64+0x96/0xc0
[  206.477313]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.477322]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.477327] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.477330] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.477337] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.477389]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.477403] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.477404] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.477405] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.477408] RSP: 0018:ffffc0b1c301fb10 EFLAGS: 00010283
[  206.477410] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.477415] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.477471]  vfs_write+0xc6/0x3b0
[  206.477505]  ? fire_user_return_notifiers+0x42/0x70
[  206.477518] RIP: 0033:0x7fcbbebcacbf
[  206.477522] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.477525] R10: 00000000008b67b8 R11: 0000000000000293 R12: 
00000000008b67b8
[  206.477531] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.477574]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.477589] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.477590] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.477592] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.477593] RSP: 0018:ffffc0b1c301fb40 EFLAGS: 00010246
[  206.477595] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.477598] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.477603] Call Trace:
[  206.477607]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.477697]  ? do_syscall_64+0x69/0x90
[  206.477701]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.477704] RIP: 0033:0x7fcbbebcacbf
[  206.477708] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.477711] R10: 00000000008b67b8 R11: 0000000000000293 R12: 
00000000008b67b8
[  206.477716] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.477730] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.477786] RSP: 0018:ffffc0b1c301fae0 EFLAGS: 00010283
[  206.477787] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.477788] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.477789] RBP: ffffc0b1c301fb48 R08: 0000000000000001 R09: 
ffffc0b1c301fba8
[  206.477790] R10: 00000000000b67c4 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.477791] R13: 000000010d0f6000 R14: ffffc0b1c301fba8 R15: 
ffff9d7a8d400000
[  206.477792] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.477794] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.477795] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.477824]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.477873]  vfio_device_fops_write+0x1f/0x40
[  206.477926] RIP: 0033:0x7fcbbebcacbf
[  206.477929] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.477937]  </TASK>
[  206.477943] ------------[ cut here ]------------
[  206.477946] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.478007] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.478008] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.478010] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.478012] Call Trace:
[  206.478013]  <TASK>
[  206.478026]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.478076]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.478094]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.478152] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.478154] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.478155] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.478162] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.478183] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.478250] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.478251] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.478252] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.478254] Call Trace:
[  206.478254]  <TASK>
[  206.478256]  vfio_pin_pages+0x5e/0xb0
[  206.478275]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.478374] RIP: 0033:0x7fcbbebcacbf
[  206.478379] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.478391] ------------[ cut here ]------------
[  206.478430]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.478443] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.478449] RSP: 0018:ffffc0b1c301fb68 EFLAGS: 00010246
[  206.478453] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.478454] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.478455] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.478456] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.478457] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.478461]  vfio_unpin_pages+0x49/0x60
[  206.478462]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.478473]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.478488]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.478539]  vfs_write+0xc6/0x3b0
[  206.478584] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.478587] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.478592]  </TASK>
[  206.478608] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.478674] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.478675] Call Trace:
[  206.478676]  <TASK>
[  206.478678]  vfio_pin_pages+0x5e/0xb0
[  206.478679]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.478696]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.478713]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.478751]  ? vfs_write+0xc6/0x3b0
[  206.478797] RIP: 0033:0x7fcbbebcacbf
[  206.478802] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.478808]  </TASK>
[  206.478814] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.478853]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.478871] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.478878] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.478879] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.478880] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.478881] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.478882] Call Trace:
[  206.478883]  <TASK>
[  206.478885]  vfio_unpin_pages+0x49/0x60
[  206.478905]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.478930]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.478993]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.479009] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.479010] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.479011] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.479014] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.479032] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.479095] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.479096] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.479097] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.479099] Call Trace:
[  206.479132]  vfio_pin_pages+0x5e/0xb0
[  206.479142]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.479201]  vfs_write+0xc6/0x3b0
[  206.479229] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.479234] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.479246] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.479310] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.479312] Call Trace:
[  206.479312]  <TASK>
[  206.479314]  vfio_unpin_pages+0x49/0x60
[  206.479325]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.479340]  ? vcpu_put+0x22/0x50 [kvm]
[  206.479384]  vfs_write+0xc6/0x3b0
[  206.479396]  do_syscall_64+0x59/0x90
[  206.479409] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.479414] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.479430] ------------[ cut here ]------------
[  206.479470]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.479484] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.479490] RSP: 0018:ffffc0b1c301fb48 EFLAGS: 00010283
[  206.479498] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.479499] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.479500] Call Trace:
[  206.479501]  <TASK>
[  206.479503]  vfio_pin_pages+0x5e/0xb0
[  206.479504]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.479513]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.479520]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.479527]  ? kvm_arch_vcpu_ioctl_run+0x3c4/0x1850 [kvm]
[  206.479553]  ? vcpu_put+0x22/0x50 [kvm]
[  206.479573]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.479581]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.479588]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.479595]  vfio_device_fops_write+0x1f/0x40
[  206.479597]  vfs_write+0xc6/0x3b0
[  206.479604]  ? __fget_light+0xa7/0x130
[  206.479615]  ? do_syscall_64+0x69/0x90
[  206.479627] R10: 00000000008b67e0 R11: 0000000000000293 R12: 
00000000008b67e0
[  206.479638] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.479677]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.479691] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.479705] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.479707]  <TASK>
[  206.479710]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.479720]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.479728]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.479735]  ? kvm_arch_vcpu_ioctl_run+0x3c4/0x1850 [kvm]
[  206.479789]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.479796]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.479803]  vfio_device_fops_write+0x1f/0x40
[  206.479804]  vfs_write+0xc6/0x3b0
[  206.479811]  ? __fget_light+0xa7/0x130
[  206.479821]  ? do_syscall_64+0x69/0x90
[  206.479833] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.479840] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.479850] ------------[ cut here ]------------
[  206.479890]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.479904] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.479915] R13: 000000010d0f6000 R14: ffffc0b1c301fc60 R15: 
ffff9d7a8d400000
[  206.479921]  <TASK>
[  206.479924]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.479933]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.479948]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.479963]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.479970]  vfio_device_fops_write+0x1f/0x40
[  206.479971]  vfs_write+0xc6/0x3b0
[  206.479976]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.479978]  ? __fget_light+0xa7/0x130
[  206.479980]  __x64_sys_pwrite64+0x96/0xc0
[  206.479983]  do_syscall_64+0x59/0x90
[  206.479988] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480043]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.480057] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.480058] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.480062] RSP: 0018:ffffc0b1c301fbc8 EFLAGS: 00010246
[  206.480071] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.480130]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480132]  ? __fget_light+0xa7/0x130
[  206.480134]  __x64_sys_pwrite64+0x96/0xc0
[  206.480139]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.480142] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480144] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.480145] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.480146] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.480148] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.480152]  </TASK>
[  206.480165] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.480204]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.480224] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.480232] R13: 000000010d0f6000 R14: ffffc0b1c301fbf8 R15: 
ffff9d7a8d400000
[  206.480270]  vfio_pin_pages+0x5e/0xb0
[  206.480359]  do_syscall_64+0x59/0x90
[  206.480370] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480371] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.480373] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.480374] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.480375] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.480376] R10: 00000000008b67f0 R11: 0000000000000293 R12: 
00000000008b67f0
[  206.480377] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.480379]  </TASK>
[  206.480388] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.480552]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480555]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.480559] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480560] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.480562] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.480563] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.480564] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.480565] R10: 00000000008b67f0 R11: 0000000000000293 R12: 
00000000008b67f0
[  206.480585] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.480731]  vfs_write+0xc6/0x3b0
[  206.480764]  ? __fget_light+0xa7/0x130
[  206.480766]  __x64_sys_pwrite64+0x96/0xc0
[  206.480769]  do_syscall_64+0x59/0x90
[  206.480771]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.480773]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480775]  ? do_syscall_64+0x69/0x90
[  206.480777]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.480779]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480781]  ? do_syscall_64+0x69/0x90
[  206.480783]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.480785] RIP: 0033:0x7fcbbebcacbf
[  206.480786] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480788] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.480789] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.480790] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.480797] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.480841]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.480855] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.480857] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.480860] RSP: 0018:ffffc0b1c301faf8 EFLAGS: 00010246
[  206.480868] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.480910]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.480936]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.480960]  ? __fget_light+0xa7/0x130
[  206.480965]  do_syscall_64+0x59/0x90
[  206.480967]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.480969]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480971]  ? do_syscall_64+0x69/0x90
[  206.480973]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.480975]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480977]  ? do_syscall_64+0x69/0x90
[  206.480979]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.480981] RIP: 0033:0x7fcbbebcacbf
[  206.480982] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480992]  </TASK>
[  206.481045]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.481059] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.481060] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.481061] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.481064] RSP: 0018:ffffc0b1c301fac8 EFLAGS: 00010283
[  206.481065] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.481104]  vfio_pin_pages+0x5e/0xb0
[  206.481145]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.481153]  vfio_device_fops_write+0x1f/0x40
[  206.481154]  vfs_write+0xc6/0x3b0
[  206.481157]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.481176]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.481200]  ? __fget_light+0xa7/0x130
[  206.481202]  __x64_sys_pwrite64+0x96/0xc0
[  206.481205]  do_syscall_64+0x59/0x90
[  206.481207]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481209]  ? do_syscall_64+0x69/0x90
[  206.481210]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481212]  ? do_syscall_64+0x69/0x90
[  206.481214]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.481218]  ? do_syscall_64+0x69/0x90
[  206.481220]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.481223] RIP: 0033:0x7fcbbebcacbf
[  206.481225] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.481230] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.481281]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.481299] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.481301] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.481305] RSP: 0018:ffffc0b1c301faf8 EFLAGS: 00010246
[  206.481311] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.481368]  vfio_device_fops_write+0x1f/0x40
[  206.481369]  vfs_write+0xc6/0x3b0
[  206.481392]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.481415]  ? __fget_light+0xa7/0x130
[  206.481418]  __x64_sys_pwrite64+0x96/0xc0
[  206.481420]  do_syscall_64+0x59/0x90
[  206.481422]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481424]  ? do_syscall_64+0x69/0x90
[  206.481425]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481427]  ? do_syscall_64+0x69/0x90
[  206.481429]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.481432]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481434]  ? do_syscall_64+0x69/0x90
[  206.481435]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.481437] RIP: 0033:0x7fcbbebcacbf
[  206.481438] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.481440] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.481441] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.481442] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.481448]  </TASK>
[  206.481502]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.481515] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.481517] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.481518] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.481519] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.481521] RSP: 0018:ffffc0b1c301fb58 EFLAGS: 00010283
[  206.481522] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.481529] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.481565]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.481580]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.481587]  vfio_device_fops_write+0x1f/0x40
[  206.481589]  vfs_write+0xc6/0x3b0
[  206.481591]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.481594]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481596]  ? __fget_light+0xa7/0x130
[  206.481598]  __x64_sys_pwrite64+0x96/0xc0
[  206.481601]  do_syscall_64+0x59/0x90
[  206.481603]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481604]  ? do_syscall_64+0x69/0x90
[  206.481606]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.481608] RIP: 0033:0x7fcbbebcacbf
[  206.481609] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.481611] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.481612] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.481613] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.481614] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.481616] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.481664]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.481678] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.481683] RSP: 0018:ffffc0b1c301fb88 EFLAGS: 00010246
[  206.481684] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.481687] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.481729]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.481768]  ? do_syscall_64+0x69/0x90
[  206.481769]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.481771] RIP: 0033:0x7fcbbebcacbf
[  206.481773] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.481774] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.481775] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.481777] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.481778] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.481779] R10: 00000000008b6808 R11: 0000000000000293 R12: 
00000000008b6808
[  206.481780] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.481782]  </TASK>
[  206.481783] ---[ end trace 0000000000000000 ]---
[  206.481797] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.481946]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.481992]  ? do_syscall_64+0x69/0x90
[  206.481996]  ? do_syscall_64+0x69/0x90
[  206.481997]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.481999] RIP: 0033:0x7fcbbebcacbf
[  206.482001] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.482002] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.482004] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.482005] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.482006] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.482007] R10: 00000000008b6810 R11: 0000000000000293 R12: 
00000000008b6810
[  206.482008] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.482010]  </TASK>
[  206.482011] ---[ end trace 0000000000000000 ]---
[  206.482012] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.482016] ------------[ cut here ]------------
[  206.482017] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.482055]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.482073] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.482079] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.482084] Call Trace:
[  206.482150]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.482194]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482198]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482199]  ? do_syscall_64+0x69/0x90
[  206.482201]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.482203] RIP: 0033:0x7fcbbebcacbf
[  206.482205] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.482206] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.482207] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.482209] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.482210] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.482211] R10: 00000000008b6810 R11: 0000000000000293 R12: 
00000000008b6810
[  206.482212] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.482278]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.482292] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.482294] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.482295] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.482298] RSP: 0018:ffffc0b1c301faa8 EFLAGS: 00010283
[  206.482299] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.482304] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.482388]  vfs_write+0xc6/0x3b0
[  206.482422]  ? do_syscall_64+0x69/0x90
[  206.482423]  ? fire_user_return_notifiers+0x42/0x70
[  206.482425]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.482428]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482430]  ? do_syscall_64+0x69/0x90
[  206.482431]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482433]  ? do_syscall_64+0x69/0x90
[  206.482434]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.482437] RIP: 0033:0x7fcbbebcacbf
[  206.482438] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.482439] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.482441] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.482444] R10: 00000000008b6818 R11: 0000000000000293 R12: 
00000000008b6818
[  206.482493]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.482507] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.482510] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.482513] RSP: 0018:ffffc0b1c301fad8 EFLAGS: 00010246
[  206.482515] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.482568]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.482607]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.482609]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482612]  ? fire_user_return_notifiers+0x42/0x70
[  206.482615]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.482617]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482619]  ? do_syscall_64+0x69/0x90
[  206.482621]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482623]  ? do_syscall_64+0x69/0x90
[  206.482624]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.482626] RIP: 0033:0x7fcbbebcacbf
[  206.482628] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.482690]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.482704] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.482705] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.482706] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.482709] RSP: 0018:ffffc0b1c301fb08 EFLAGS: 00010283
[  206.482713] R10: 00000000000b6824 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.482774]  vfs_write+0xc6/0x3b0
[  206.482776]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.482796]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.482798]  ? __fget_light+0xa7/0x130
[  206.482801]  __x64_sys_pwrite64+0x96/0xc0
[  206.482804]  do_syscall_64+0x59/0x90
[  206.482805]  ? do_syscall_64+0x69/0x90
[  206.482807]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482809]  ? do_syscall_64+0x69/0x90
[  206.482812]  ? do_syscall_64+0x69/0x90
[  206.482813]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.482816] RIP: 0033:0x7fcbbebcacbf
[  206.482818] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.482820] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.482823] R10: 00000000008b6820 R11: 0000000000000293 R12: 
00000000008b6820
[  206.482872]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.482886] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.482890] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.482931]  ? restore_fpregs_from_fpstate+0x47/0xc0
[  206.482985]  __x64_sys_pwrite64+0x96/0xc0
[  206.482987]  do_syscall_64+0x59/0x90
[  206.482989]  ? do_syscall_64+0x69/0x90
[  206.482991]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482993]  ? do_syscall_64+0x69/0x90
[  206.482996]  ? do_syscall_64+0x69/0x90
[  206.482997]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.483000] RIP: 0033:0x7fcbbebcacbf
[  206.483001] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.483007] R10: 00000000008b6820 R11: 0000000000000293 R12: 
00000000008b6820
[  206.483062]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.483076] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.483077] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.483081] RSP: 0018:ffffc0b1c301fb28 EFLAGS: 00010283
[  206.483085] RBP: ffffc0b1c301fb90 R08: 0000000000000001 R09: 
ffffc0b1c301fbf0
[  206.483166]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.483198]  __x64_sys_pwrite64+0x96/0xc0
[  206.483200]  do_syscall_64+0x59/0x90
[  206.483202]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.483204]  ? do_syscall_64+0x69/0x90
[  206.483205]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.483207]  ? do_syscall_64+0x69/0x90
[  206.483209]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.483211] RIP: 0033:0x7fcbbebcacbf
[  206.483212] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.483214] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.483215] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.483216] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.483217] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.483223]  </TASK>
[  206.483230] ------------[ cut here ]------------
[  206.483270]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.483287] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.483290] RSP: 0018:ffffc0b1c301fb58 EFLAGS: 00010246
[  206.483294] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.483322]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.483414]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.483419] RIP: 0033:0x7fcbbebcacbf
[  206.483421] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.483422] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.483423] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.483425] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.483426] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.483427] R10: 00000000008b6828 R11: 0000000000000293 R12: 
00000000008b6828
[  206.483428] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.483444] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.483484]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.483502] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.483510] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.483559]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.483589]  ? do_syscall_64+0x69/0x90
[  206.483591]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.483593] RIP: 0033:0x7fcbbebcacbf
[  206.483594] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.483596] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.483597] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.483598] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.483599] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.483600] R10: 00000000008b6830 R11: 0000000000000293 R12: 
00000000008b6830
[  206.483601] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.483604]  </TASK>
[  206.483605] ---[ end trace 0000000000000000 ]---
[  206.483610] ------------[ cut here ]------------
[  206.483650]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.483667] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.483674] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.483679] Call Trace:
[  206.483701]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.483752]  ? do_syscall_64+0x69/0x90
[  206.483762] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.483764] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.483765] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.483766] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.483767] R10: 00000000008b6830 R11: 0000000000000293 R12: 
00000000008b6830
[  206.483768] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.483770]  </TASK>
[  206.483771] ---[ end trace 0000000000000000 ]---
[  206.483772] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.483782] ------------[ cut here ]------------
[  206.483783] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.483785] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.483878]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.483970] RIP: 0033:0x7fcbbebcacbf
[  206.483974] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.483975] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.483977] R10: 00000000008b6838 R11: 0000000000000293 R12: 
00000000008b6838
[  206.483978] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.483981]  </TASK>
[  206.483982] ---[ end trace 0000000000000000 ]---
[  206.483983] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.483987] ------------[ cut here ]------------
[  206.483988] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.483990] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.484085]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.484162]  ? fire_user_return_notifiers+0x42/0x70
[  206.484181] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.484189] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.484200] ------------[ cut here ]------------
[  206.484200] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.484243]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.484257] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.484259] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.484259] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.484261] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.484262] RSP: 0018:ffffc0b1c301fb30 EFLAGS: 00010283
[  206.484265] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.484267] R10: 00000000000b6844 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.484277]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.484301]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.484327]  ? __fget_light+0xa7/0x130
[  206.484338]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.484345] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.484401]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.484414] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.484416] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.484417] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.484418] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.484425] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.484443]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.484482]  vfio_device_fops_write+0x1f/0x40
[  206.484488]  __x64_sys_pwrite64+0x96/0xc0
[  206.484498]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.484504] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.484567]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.484582] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.484583] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.484584] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.484585] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.484587] RSP: 0018:ffffc0b1c301fb60 EFLAGS: 00010283
[  206.484595] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.484597] Call Trace:
[  206.484598]  <TASK>
[  206.484653]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.484679]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.484708]  ? do_syscall_64+0x69/0x90
[  206.484712] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.484722]  </TASK>
[  206.484768]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.484782] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.484783] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.484784] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.484786] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.484787] RSP: 0018:ffffc0b1c301fb90 EFLAGS: 00010246
[  206.484788] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.484789] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.484790] RBP: ffffc0b1c301fbd0 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.484791] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.484792] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.484800]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.484842]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.484883]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.484888] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.484891] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.484896]  </TASK>
[  206.484908] ------------[ cut here ]------------
[  206.484949]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.484963] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.484965] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.484966] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.484967] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.484969] RSP: 0018:ffffc0b1c301fac0 EFLAGS: 00010283
[  206.484970] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.484971] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.485007]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.485021]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.485038]  do_syscall_64+0x59/0x90
[  206.485053]  ? do_syscall_64+0x69/0x90
[  206.485059] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.485061] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.485064] R10: 00000000008b6850 R11: 0000000000000293 R12: 
00000000008b6850
[  206.485076] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.485136] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.485172]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.485198]  ? __fget_light+0xa7/0x130
[  206.485211]  ? do_syscall_64+0x69/0x90
[  206.485216]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.485223] RIP: 0033:0x7fcbbebcacbf
[  206.485226] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.485229] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.485232] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.485292]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.485307] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.485308] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.485309] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.485311] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.485312] RSP: 0018:ffffc0b1c301fb70 EFLAGS: 00010283
[  206.485322] Call Trace:
[  206.485342]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.485349]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.485376]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.485426]  ? __fget_light+0xa7/0x130
[  206.485436]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.485441] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.485442] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.485444] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.485449] ---[ end trace 0000000000000000 ]---
[  206.485494]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas vi

-- 
Regards,
Yi Liu

Re: [Intel-gfx] [PATCH 00/10] Connect VFIO to IOMMUFD

[Yi Liu]

On 2022/11/1 07:24, Jason Gunthorpe wrote:
> On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
>>> There is something wrong with the test suite that it isn't covering
>>> the above, I'm going to look into that today.
>>
>> sounds to be the cause. I didn't see any significant change in vfio_main.c
>> that may fail gvt. So should the iommufd changes. Then we will re-run the
>> test after your update.:-)
> 
> I updated the github with all the changes made so far, it is worth
> trying again!

gvt is still failing with below call trace in host side. vfio_unpin_pages()
is still in problem. Any idea on it?


[  206.464107] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.464167] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.464168] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.464171] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.464172] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.464173] Call Trace:
[  206.464174]  <TASK>
[  206.464175]  vfio_unpin_pages+0x49/0x60
[  206.464177]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.464187]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.464196]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.464229]  vfs_write+0xc6/0x3b0
[  206.464296] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.464301] R10: 00000000008b66a8 R11: 0000000000000293 R12: 
00000000008b66a8
[  206.464318] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.464357]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.464377] RSP: 0018:ffffc0b1c301fb90 EFLAGS: 00010283
[  206.464381] R10: 00000000000b66b4 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.464382] R13: 000000010d0f6000 R14: ffffc0b1c301fc58 R15: 
ffff9d7a8d400000
[  206.464384] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.464385] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.464386] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.464387] Call Trace:
[  206.464388]  <TASK>
[  206.464389]  ? x86_emulate_instruction+0x4cd/0x890 [kvm]
[  206.464415]  vfio_pin_pages+0x5e/0xb0
[  206.464417]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.464434]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.464505] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.464509] R10: 00000000008b66b0 R11: 0000000000000293 R12: 
00000000008b66b0
[  206.464513]  </TASK>
[  206.464519] ------------[ cut here ]------------
[  206.464558]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.464572] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.464578] RSP: 0018:ffffc0b1c301fbc0 EFLAGS: 00010246
[  206.464583] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.464584] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.464586] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.464588] Call Trace:
[  206.464588]  <TASK>
[  206.464590]  vfio_unpin_pages+0x49/0x60
[  206.464591]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.464610]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.464640]  vfio_device_fops_write+0x1f/0x40
[  206.464681] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.464690]  </TASK>
[  206.464702] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.464742]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.464756] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.464767] R13: 000000010d0f6000 R14: ffffc0b1c301fbd0 R15: 
ffff9d7a8d400000
[  206.464771] Call Trace:
[  206.464772]  <TASK>
[  206.464773]  ? writeback_registers+0x56/0xb0 [kvm]
[  206.464802]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.464811]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.464819]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.464826]  ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
[  206.464854]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.464879]  ? intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.464902]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.464910] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.464912] R10: 00000000008b66b8 R11: 0000000000000293 R12: 
00000000008b66b8
[  206.464922] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.464960]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.464974] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.464985] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.464987] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.464988] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.464989] Call Trace:
[  206.464990]  <TASK>
[  206.464992]  vfio_unpin_pages+0x49/0x60
[  206.464993]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.465003]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.465012]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.465019]  ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
[  206.465039]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.465047]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.465054]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.465061]  vfio_device_fops_write+0x1f/0x40
[  206.465063]  vfs_write+0xc6/0x3b0
[  206.465065]  ? intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.465081]  __x64_sys_pwrite64+0x96/0xc0
[  206.465100] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.465106] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.465159]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.465173] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.465175] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.465184] R13: 000000010d0f6000 R14: ffffc0b1c301fb68 R15: 
ffff9d7a8d400000
[  206.465186] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.465188] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.465189] Call Trace:
[  206.465191]  ? segmented_write.isra.0+0x5e/0x90 [kvm]
[  206.465218]  vfio_pin_pages+0x5e/0xb0
[  206.465220]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.465230]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.465239]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.465247]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.465255]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.465261]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.465269]  vfio_device_fops_write+0x1f/0x40
[  206.465270]  vfs_write+0xc6/0x3b0
[  206.465272]  ? intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.465280]  ? __fget_light+0xa7/0x130
[  206.465285]  __x64_sys_pwrite64+0x96/0xc0
[  206.465320]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.465336] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.465338] R10: 00000000008b66c0 R11: 0000000000000293 R12: 
00000000008b66c0
[  206.465347] ------------[ cut here ]------------
[  206.465386]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.465400] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.465405] RSP: 0018:ffffc0b1c301fad0 EFLAGS: 00010246
[  206.465410] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.465412] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.465413] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.465414] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.465415] Call Trace:
[  206.465416]  <TASK>
[  206.465417]  vfio_unpin_pages+0x49/0x60
[  206.465419]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.465429]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.465438]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.465446]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.465461]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.465481]  __x64_sys_pwrite64+0x96/0xc0
[  206.465522]  ? do_syscall_64+0x69/0x90
[  206.465530] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.465538] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.465590]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.465604] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.465613] R10: 00000000000b66cc R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.465615] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.465617] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.465618] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.465619] Call Trace:
[  206.465620]  <TASK>
[  206.465621]  ? x86_emulate_instruction+0x4cd/0x890 [kvm]
[  206.465649]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.465672]  ? vmx_vcpu_put+0x122/0x210 [kvm_intel]
[  206.465702]  vfs_write+0xc6/0x3b0
[  206.465753]  ? do_syscall_64+0x69/0x90
[  206.465760] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.465762] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.465769]  </TASK>
[  206.465775] ------------[ cut here ]------------
[  206.465814]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.465828] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.465834] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.465835] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.465836] RBP: ffffc0b1c301fb10 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.465837] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.465838] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.465839] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.465841] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.465842] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.465843] Call Trace:
[  206.465844]  <TASK>
[  206.465845]  vfio_unpin_pages+0x49/0x60
[  206.465847]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.465857]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.465872]  ? vmx_vcpu_put+0x122/0x210 [kvm_intel]
[  206.465902]  vfio_device_fops_write+0x1f/0x40
[  206.465913]  ? __fget_light+0xa7/0x130
[  206.465957]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.465965] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.465982] ------------[ cut here ]------------
[  206.465986] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.466051] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.466052] Call Trace:
[  206.466052]  <TASK>
[  206.466083]  vfio_pin_pages+0x5e/0xb0
[  206.466085]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.466094]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.466102]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.466129]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.466165]  __x64_sys_pwrite64+0x96/0xc0
[  206.466176] RIP: 0033:0x7fcbbebcacbf
[  206.466187]  </TASK>
[  206.466193] ------------[ cut here ]------------
[  206.466194] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.466236]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.466250] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.466261] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.466268]  vfio_unpin_pages+0x49/0x60
[  206.466279]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.466288]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.466296]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.466304]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.466311]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.466322]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.466342]  ? __fget_light+0xa7/0x130
[  206.466344]  __x64_sys_pwrite64+0x96/0xc0
[  206.466347]  do_syscall_64+0x59/0x90
[  206.466351]  ? do_syscall_64+0x69/0x90
[  206.466358] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.466387] ------------[ cut here ]------------
[  206.466427]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.466441] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.466443] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.466456] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.466504]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.466510]  ? restore_fpregs_from_fpstate+0x47/0xc0
[  206.466514]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.466521]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.466528]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.466537]  vfs_write+0xc6/0x3b0
[  206.466539]  ? vfio_device_fops_write+0x1f/0x40
[  206.466567]  __x64_sys_pwrite64+0x96/0xc0
[  206.466570]  do_syscall_64+0x59/0x90
[  206.466572]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.466574]  ? do_syscall_64+0x69/0x90
[  206.466575]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.466585] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.466599] ------------[ cut here ]------------
[  206.466600] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.466639]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.466653] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.466666] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.466672]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.466682]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.466691]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.466697]  ? restore_fpregs_from_fpstate+0x47/0xc0
[  206.466701]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.466709]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.466716]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.466723]  vfio_device_fops_write+0x1f/0x40
[  206.466724]  vfs_write+0xc6/0x3b0
[  206.466727]  ? vfio_device_fops_write+0x1f/0x40
[  206.466728]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.466752]  ? __fget_light+0xa7/0x130
[  206.466755]  __x64_sys_pwrite64+0x96/0xc0
[  206.466758]  do_syscall_64+0x59/0x90
[  206.466765]  ? do_syscall_64+0x69/0x90
[  206.466772] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.466794] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.466833]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.466847] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.466862] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.466883]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.466891]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.466899]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.466912]  vfio_device_fops_write+0x1f/0x40
[  206.466914]  vfs_write+0xc6/0x3b0
[  206.466918]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.466942]  ? __fget_light+0xa7/0x130
[  206.466945]  __x64_sys_pwrite64+0x96/0xc0
[  206.466947]  do_syscall_64+0x59/0x90
[  206.466949]  ? do_syscall_64+0x69/0x90
[  206.466958] RIP: 0033:0x7fcbbebcacbf
[  206.466963] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.466977] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.467057]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.467066]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.467074]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.467082]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.467089]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467096]  vfio_device_fops_write+0x1f/0x40
[  206.467097]  vfs_write+0xc6/0x3b0
[  206.467099]  ? vfio_device_fops_write+0x1f/0x40
[  206.467101]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.467125]  ? __fget_light+0xa7/0x130
[  206.467128]  __x64_sys_pwrite64+0x96/0xc0
[  206.467131]  do_syscall_64+0x59/0x90
[  206.467132]  ? do_syscall_64+0x69/0x90
[  206.467134]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.467136]  ? do_syscall_64+0x69/0x90
[  206.467137]  ? do_syscall_64+0x69/0x90
[  206.467138]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.467142] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.467204]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.467218] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.467219] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.467220] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.467231] R10: 00000000000b66ec R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.467265]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.467273]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.467280]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467287]  vfio_device_fops_write+0x1f/0x40
[  206.467288]  vfs_write+0xc6/0x3b0
[  206.467322]  ? __fget_light+0xa7/0x130
[  206.467324]  __x64_sys_pwrite64+0x96/0xc0
[  206.467327]  do_syscall_64+0x59/0x90
[  206.467329]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.467331]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.467333]  ? do_syscall_64+0x69/0x90
[  206.467335]  ? do_syscall_64+0x69/0x90
[  206.467336]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.467338]  ? do_syscall_64+0x69/0x90
[  206.467340]  ? do_syscall_64+0x69/0x90
[  206.467345] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.467363] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.467442]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.467450]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.467458]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.467466]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.467473]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467480]  vfio_device_fops_write+0x1f/0x40
[  206.467482]  vfs_write+0xc6/0x3b0
[  206.467484]  ? intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467491]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.467515]  ? __fget_light+0xa7/0x130
[  206.467518]  __x64_sys_pwrite64+0x96/0xc0
[  206.467520]  do_syscall_64+0x59/0x90
[  206.467522]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.467525]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.467533]  ? do_syscall_64+0x69/0x90
[  206.467542] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.467563] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.467678]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.467686]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.467693]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.467700]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467707]  vfio_device_fops_write+0x1f/0x40
[  206.467709]  vfs_write+0xc6/0x3b0
[  206.467731]  ? __fget_light+0xa7/0x130
[  206.467733]  __x64_sys_pwrite64+0x96/0xc0
[  206.467736]  do_syscall_64+0x59/0x90
[  206.467738]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.467740]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.467742]  ? do_syscall_64+0x69/0x90
[  206.467744]  ? do_syscall_64+0x69/0x90
[  206.467748] RIP: 0033:0x7fcbbebcacbf
[  206.467751] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.467765] ------------[ cut here ]------------
[  206.467768] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.467847]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.467856]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.467864]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.467872]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.467879]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467886]  vfio_device_fops_write+0x1f/0x40
[  206.467887]  vfs_write+0xc6/0x3b0
[  206.467890]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.467910]  ? __fget_light+0xa7/0x130
[  206.467916]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.467922]  ? do_syscall_64+0x69/0x90
[  206.467925]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.467927] RIP: 0033:0x7fcbbebcacbf
[  206.467928] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.467933] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.467990]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.468003] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.468007] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.468098]  vfio_device_fops_write+0x1f/0x40
[  206.468122]  ? __fget_light+0xa7/0x130
[  206.468124]  ? __fget_light+0xa7/0x130
[  206.468126]  __x64_sys_pwrite64+0x96/0xc0
[  206.468129]  do_syscall_64+0x59/0x90
[  206.468130]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.468133]  ? do_syscall_64+0x69/0x90
[  206.468137]  ? do_syscall_64+0x69/0x90
[  206.468141] RIP: 0033:0x7fcbbebcacbf
[  206.468143] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.468145] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.468146] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.468147] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.468197]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.468211] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.468212] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.468215] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.468229] Call Trace:
[  206.468283]  vfs_write+0xc6/0x3b0
[  206.468312]  do_syscall_64+0x59/0x90
[  206.468313]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.468315]  ? do_syscall_64+0x69/0x90
[  206.468318]  ? do_syscall_64+0x69/0x90
[  206.468321]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.468323] RIP: 0033:0x7fcbbebcacbf
[  206.468324] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.468327] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.468328] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.468329] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.468336] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.468387]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.468404] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.468407] RSP: 0018:ffffc0b1c301fb50 EFLAGS: 00010283
[  206.468410] RBP: ffffc0b1c301fbb8 R08: 0000000000000001 R09: 
ffffc0b1c301fc18
[  206.468416] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.468532]  ? do_syscall_64+0x69/0x90
[  206.468538] RIP: 0033:0x7fcbbebcacbf
[  206.468539] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.468540] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.468542] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.468544] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.468545] R10: 00000000008b6700 R11: 0000000000000293 R12: 
00000000008b6700
[  206.468546] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.468549]  </TASK>
[  206.468549] ---[ end trace 0000000000000000 ]---
[  206.468557] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.468676]  vfio_device_fops_write+0x1f/0x40
[  206.468723] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.468724] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.468725] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.468726] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.468727] R10: 00000000008b6700 R11: 0000000000000293 R12: 
00000000008b6700
[  206.468728] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.468731]  </TASK>
[  206.468731] ---[ end trace 0000000000000000 ]---
[  206.468732] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.468742] ------------[ cut here ]------------
[  206.468743] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.468746] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.468808] R13: 000000010d0f6000 R14: ffffc0b1c301fba0 R15: 
ffff9d7a8d400000
[  206.468813] Call Trace:
[  206.468825]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.468921]  do_syscall_64+0x59/0x90
[  206.468938] RIP: 0033:0x7fcbbebcacbf
[  206.468942] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.468944] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.468945] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.468946] R10: 00000000008b6708 R11: 0000000000000293 R12: 
00000000008b6708
[  206.468947] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.468949]  </TASK>
[  206.468950] ---[ end trace 0000000000000000 ]---
[  206.468951] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.468955] ------------[ cut here ]------------
[  206.468956] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.468958] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.469023] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.469046]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.469140]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.469157] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.469158] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.469159] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.469161] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.469164]  </TASK>
[  206.469165] ---[ end trace 0000000000000000 ]---
[  206.469166] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.469176] ------------[ cut here ]------------
[  206.469179] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.469311]  ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
[  206.469360]  __x64_sys_pwrite64+0x96/0xc0
[  206.469387] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.469391] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.469392] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.469394] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.469397]  </TASK>
[  206.469398] ---[ end trace 0000000000000000 ]---
[  206.469399] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.469403] ------------[ cut here ]------------
[  206.469404] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.469443]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.469457] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.469463] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.469466] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.469468] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.469486]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.469537]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.469569]  ? do_syscall_64+0x69/0x90
[  206.469579] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.469581] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.469582] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.469583] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.469584] R10: 00000000008b6710 R11: 0000000000000293 R12: 
00000000008b6710
[  206.469585] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.469587]  </TASK>
[  206.469588] ---[ end trace 0000000000000000 ]---
[  206.469589] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.469601] ------------[ cut here ]------------
[  206.469602] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.469605] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.469664] RBP: ffffc0b1c301fb10 R08: 0000000000000001 R09: 
ffffc0b1c301fb70
[  206.469669] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.469673]  ? x86_emulate_instruction+0x4cd/0x890 [kvm]
[  206.469772]  ? do_syscall_64+0x69/0x90
[  206.469782] RIP: 0033:0x7fcbbebcacbf
[  206.469785] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.469787] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.469788] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.469789] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.469790] R10: 00000000008b6718 R11: 0000000000000293 R12: 
00000000008b6718
[  206.469791] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.469793]  </TASK>
[  206.469794] ---[ end trace 0000000000000000 ]---
[  206.469795] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.469799] ------------[ cut here ]------------
[  206.469839]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.469853] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.469854] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.469858] RSP: 0018:ffffc0b1c301fad8 EFLAGS: 00010246
[  206.469861] RBP: ffffc0b1c301fb18 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.469897]  ? restore_fpregs_from_fpstate+0x47/0xc0
[  206.469939]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.469941]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.469943]  ? do_syscall_64+0x69/0x90
[  206.469945]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.469947]  ? do_syscall_64+0x69/0x90
[  206.469948]  ? do_syscall_64+0x69/0x90
[  206.469949]  ? do_syscall_64+0x69/0x90
[  206.469950]  ? do_syscall_64+0x69/0x90
[  206.469952]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.469954] RIP: 0033:0x7fcbbebcacbf
[  206.469955] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.469957] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.469958] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.469959] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.469960] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.469961] R10: 00000000008b6718 R11: 0000000000000293 R12: 
00000000008b6718
[  206.469962] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.469965]  </TASK>
[  206.469966] ---[ end trace 0000000000000000 ]---
[  206.469967] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.469977] ------------[ cut here ]------------
[  206.469978] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.470017]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.470035] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.470043] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.470161]  do_syscall_64+0x59/0x90
[  206.470172]  ? do_syscall_64+0x69/0x90
[  206.470175]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.470177] RIP: 0033:0x7fcbbebcacbf
[  206.470179] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.470180] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.470182] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.470183] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.470184] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.470185] R10: 00000000008b6720 R11: 0000000000000293 R12: 
00000000008b6720
[  206.470186] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.470188]  </TASK>
[  206.470189] ---[ end trace 0000000000000000 ]---
[  206.470190] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.470194] ------------[ cut here ]------------
[  206.470237]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.470251] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.470257] RSP: 0018:ffffc0b1c301fb30 EFLAGS: 00010246
[  206.470258] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.470262] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.470281]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.470359]  ? do_syscall_64+0x69/0x90
[  206.470367] RIP: 0033:0x7fcbbebcacbf
[  206.470368] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.470369] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.470371] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.470372] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.470373] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.470374] R10: 00000000008b6720 R11: 0000000000000293 R12: 
00000000008b6720
[  206.470375] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.470398] ------------[ cut here ]------------
[  206.470438]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.470457] RSP: 0018:ffffc0b1c301fb38 EFLAGS: 00010283
[  206.470458] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.470460] RBP: ffffc0b1c301fba0 R08: 0000000000000001 R09: 
ffffc0b1c301fc00
[  206.470465] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.470470]  vfio_pin_pages+0x5e/0xb0
[  206.470526]  ? __fget_light+0xa7/0x130
[  206.470547] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.470549] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.470550] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.470551] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.470552] R10: 00000000008b6728 R11: 0000000000000293 R12: 
00000000008b6728
[  206.470553] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.470556]  </TASK>
[  206.470556] ---[ end trace 0000000000000000 ]---
[  206.470557] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.470561] ------------[ cut here ]------------
[  206.470562] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.470601]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.470616] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.470619] RSP: 0018:ffffc0b1c301fb68 EFLAGS: 00010246
[  206.470623] RBP: ffffc0b1c301fba8 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.470627] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.470629] Call Trace:
[  206.470652]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.470698]  ? do_syscall_64+0x69/0x90
[  206.470711] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.470715] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.470716] R10: 00000000008b6728 R11: 0000000000000293 R12: 
00000000008b6728
[  206.470717] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.470719]  </TASK>
[  206.470720] ---[ end trace 0000000000000000 ]---
[  206.470721] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.470731] ------------[ cut here ]------------
[  206.470732] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.470734] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.470843]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.470878]  ? do_syscall_64+0x69/0x90
[  206.470886] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.470887] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.470888] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.470889] R10: 00000000008b6730 R11: 0000000000000293 R12: 
00000000008b6730
[  206.470890] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.470893]  </TASK>
[  206.470893] ---[ end trace 0000000000000000 ]---
[  206.470894] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.470898] ------------[ cut here ]------------
[  206.470899] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.470901] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.470967] Call Trace:
[  206.470996]  ? vmx_vcpu_put+0x122/0x210 [kvm_intel]
[  206.471040]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.471052] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.471054] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.471055] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.471056] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.471057] R10: 00000000008b6730 R11: 0000000000000293 R12: 
00000000008b6730
[  206.471058] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.471060]  </TASK>
[  206.471061] ---[ end trace 0000000000000000 ]---
[  206.471062] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.471072] ------------[ cut here ]------------
[  206.471073] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.471075] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.471136] R10: 00000000000b673c R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.471141] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.471207]  ? do_syscall_64+0x69/0x90
[  206.471216] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.471218] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.471219] R10: 00000000008b6738 R11: 0000000000000293 R12: 
00000000008b6738
[  206.471220] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.471224]  </TASK>
[  206.471225] ---[ end trace 0000000000000000 ]---
[  206.471227] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.471231] ------------[ cut here ]------------
[  206.471232] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.471234] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.471293] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.471296] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.471299] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.471314]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.471367]  ? do_syscall_64+0x69/0x90
[  206.471376] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.471379] R10: 00000000008b6738 R11: 0000000000000293 R12: 
00000000008b6738
[  206.471380] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.471383]  </TASK>
[  206.471384] ---[ end trace 0000000000000000 ]---
[  206.471385] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.471396] ------------[ cut here ]------------
[  206.471397] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.471399] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.471464] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.471500]  vfio_pin_pages+0x5e/0xb0
[  206.471519]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.471568]  vfio_device_fops_write+0x1f/0x40
[  206.471594] RIP: 0033:0x7fcbbebcacbf
[  206.471600] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.471605]  </TASK>
[  206.471606] ---[ end trace 0000000000000000 ]---
[  206.471606] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.471611] ------------[ cut here ]------------
[  206.471614] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.471671] RSP: 0018:ffffc0b1c301fb60 EFLAGS: 00010246
[  206.471673] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.471677] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.471681] Call Trace:
[  206.471684]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.471730]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.471754]  vfs_write+0xc6/0x3b0
[  206.471762]  ? __fget_light+0xa7/0x130
[  206.471781] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.471791] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.471801] ------------[ cut here ]------------
[  206.471804] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.471867] R13: 000000010d0f6000 R14: ffffc0b1c301fb70 R15: 
ffff9d7a8d400000
[  206.471903]  vfio_pin_pages+0x5e/0xb0
[  206.471921]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.471956]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.471978]  __x64_sys_pwrite64+0x96/0xc0
[  206.471988]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.472004] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.472019] ------------[ cut here ]------------
[  206.472022] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.472080] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.472081] RBP: ffffc0b1c301fb18 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.472085] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.472091]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.472116]  ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
[  206.472152]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.472166]  __x64_sys_pwrite64+0x96/0xc0
[  206.472173]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.472179]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.472194] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.472213] ------------[ cut here ]------------
[  206.472256]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.472270] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.472272] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.472273] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.472275] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.472279] RBP: ffffc0b1c301fb68 R08: 0000000000000001 R09: 
ffffc0b1c301fbc8
[  206.472285] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.472315]  vfio_pin_pages+0x5e/0xb0
[  206.472333]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.472356]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.472370]  __x64_sys_pwrite64+0x96/0xc0
[  206.472379]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.472392] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.472405] ------------[ cut here ]------------
[  206.472406] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.472408] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.472468] RBP: ffffc0b1c301fb70 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.472472] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.472475] Call Trace:
[  206.472478]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.472513]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.472531]  ? __fget_light+0xa7/0x130
[  206.472537]  ? fire_user_return_notifiers+0x42/0x70
[  206.472554] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.472564] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.472575] ------------[ cut here ]------------
[  206.472576] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.472616]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.472630] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.472632] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.472632] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.472634] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.472642] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.472648]  ? kvm_pmu_trigger_event+0x59/0x250 [kvm]
[  206.472696]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.472738]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.472749]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.472788]  ? do_syscall_64+0x69/0x90
[  206.472797] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.472804]  </TASK>
[  206.472804] ---[ end trace 0000000000000000 ]---
[  206.472805] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.472809] ------------[ cut here ]------------
[  206.472810] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.472812] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.472875] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.472882]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.472907]  ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
[  206.472983]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.473001] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.473002] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.473003] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.473004] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.473005] R10: 00000000008b6758 R11: 0000000000000293 R12: 
00000000008b6758
[  206.473006] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.473009]  </TASK>
[  206.473010] ---[ end trace 0000000000000000 ]---
[  206.473011] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.473021] ------------[ cut here ]------------
[  206.473022] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.473024] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.473087] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.473091] Call Trace:
[  206.473118]  ? kvm_arch_vcpu_ioctl_run+0x3c4/0x1850 [kvm]
[  206.473171]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.473191] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.473196]  </TASK>
[  206.473197] ---[ end trace 0000000000000000 ]---
[  206.473198] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.473202] ------------[ cut here ]------------
[  206.473203] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.473205] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.473265] RSP: 0018:ffffc0b1c301fbc0 EFLAGS: 00010246
[  206.473269] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.473273] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.473278]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.473344]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.473365]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.473382] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.473401] ------------[ cut here ]------------
[  206.473402] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.473405] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.473442]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.473461] RSP: 0018:ffffc0b1c301fb20 EFLAGS: 00010283
[  206.473463] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.473468] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.473476]  vfio_pin_pages+0x5e/0xb0
[  206.473494]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.473536]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.473551]  vfs_write+0xc6/0x3b0
[  206.473583]  __x64_sys_pwrite64+0x96/0xc0
[  206.473592]  ? do_syscall_64+0x69/0x90
[  206.473603] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.473616] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.473653]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.473674] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.473675] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.473680] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.473686]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.473711]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.473762]  vfs_write+0xc6/0x3b0
[  206.473793]  __x64_sys_pwrite64+0x96/0xc0
[  206.473800]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.473812] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.473833] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.473870]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.473884] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.473886] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.473887] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.473888] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.473894] R10: 00000000000b6774 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.473900] Call Trace:
[  206.473928]  vfio_pin_pages+0x5e/0xb0
[  206.473947]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.473965]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.473982]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.474011]  __x64_sys_pwrite64+0x96/0xc0
[  206.474022] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.474031] ---[ end trace 0000000000000000 ]---
[  206.474075]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.474089] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.474091] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.474092] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.474093] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.474095] RSP: 0018:ffffc0b1c301fbc0 EFLAGS: 00010246
[  206.474096] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.474097] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.474098] RBP: ffffc0b1c301fc00 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.474099] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.474100] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.474103] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.474104] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.474120]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.474152]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.474167]  vfs_write+0xc6/0x3b0
[  206.474196]  ? __fget_light+0xa7/0x130
[  206.474205]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.474209] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.474212] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.474281]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.474295] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.474296] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.474297] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.474299] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.474306] R13: 000000010d0f6000 R14: ffffc0b1c301fbd0 R15: 
ffff9d7a8d400000
[  206.474323]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.474349]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.474386]  ? intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.474400]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.474406]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.474413] RIP: 0033:0x7fcbbebcacbf
[  206.474416] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.474421] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.474470]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.474485] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.474486] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.474487] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.474489] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.474493] RBP: ffffc0b1c301fb78 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.474498] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.474499] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.474522]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.474554]  vfio_device_fops_write+0x1f/0x40
[  206.474577]  ? intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.474590]  do_syscall_64+0x59/0x90
[  206.474597]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.474604] RIP: 0033:0x7fcbbebcacbf
[  206.474607] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.474611] R10: 00000000008b6778 R11: 0000000000000293 R12: 
00000000008b6778
[  206.474668]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.474682] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.474683] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.474684] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.474686] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.474687] RSP: 0018:ffffc0b1c301fb20 EFLAGS: 00010283
[  206.474689] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.474690] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.474691] RBP: ffffc0b1c301fb88 R08: 0000000000000001 R09: 
ffffc0b1c301fbe8
[  206.474696] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.474700]  ? writeback_registers+0x56/0xb0 [kvm]
[  206.474774]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.474824]  vfs_write+0xc6/0x3b0
[  206.474834]  ? __fget_light+0xa7/0x130
[  206.474841]  ? fire_user_return_notifiers+0x42/0x70
[  206.474849]  ? do_syscall_64+0x69/0x90
[  206.474854] RIP: 0033:0x7fcbbebcacbf
[  206.474856] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.474864]  </TASK>
[  206.474873] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.474925] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.474935] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.474936] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.474939] Call Trace:
[  206.474943]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.474989]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.475012]  vfs_write+0xc6/0x3b0
[  206.475021]  ? __fget_light+0xa7/0x130
[  206.475028]  ? fire_user_return_notifiers+0x42/0x70
[  206.475036]  ? do_syscall_64+0x69/0x90
[  206.475042] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.475104]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.475118] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.475120] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.475121] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.475123] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.475124] RSP: 0018:ffffc0b1c301fb90 EFLAGS: 00010283
[  206.475125] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.475126] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.475127] RBP: ffffc0b1c301fbf8 R08: 0000000000000001 R09: 
ffffc0b1c301fc58
[  206.475128] R10: 00000000000b678c R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.475129] R13: 000000010d0f6000 R14: ffffc0b1c301fc58 R15: 
ffff9d7a8d400000
[  206.475134] Call Trace:
[  206.475158]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.475182]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.475204]  vfs_write+0xc6/0x3b0
[  206.475215]  do_syscall_64+0x59/0x90
[  206.475220] RIP: 0033:0x7fcbbebcacbf
[  206.475224] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.475239] ------------[ cut here ]------------
[  206.475279]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.475294] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.475295] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.475296] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.475298] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.475299] RSP: 0018:ffffc0b1c301fbc0 EFLAGS: 00010246
[  206.475300] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.475301] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.475302] RBP: ffffc0b1c301fc00 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.475304] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.475307] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.475323]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.475354]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.475368]  ? __fget_light+0xa7/0x130
[  206.475376]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.475381] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.475386] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.475442]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.475456] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.475458] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.475458] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.475460] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.475467] R13: 000000010d0f6000 R14: ffffc0b1c301fbb0 R15: 
ffff9d7a8d400000
[  206.475474]  vfio_pin_pages+0x5e/0xb0
[  206.475533]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.475549]  vfs_write+0xc6/0x3b0
[  206.475581]  do_syscall_64+0x59/0x90
[  206.475587]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.475594] RIP: 0033:0x7fcbbebcacbf
[  206.475597] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.475599] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.475650]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.475664] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.475665] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.475666] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.475668] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.475669] RSP: 0018:ffffc0b1c301fb18 EFLAGS: 00010246
[  206.475672] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.475676] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.475679] Call Trace:
[  206.475693]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.475743]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.475762]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.475795]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.475802]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.475807] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.475827] ------------[ cut here ]------------
[  206.475867]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.475881] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.475883] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.475884] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.475885] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.475887] RSP: 0018:ffffc0b1c301fad0 EFLAGS: 00010283
[  206.475888] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.475889] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.475890] RBP: ffffc0b1c301fb38 R08: 0000000000000001 R09: 
ffffc0b1c301fb98
[  206.475894] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.475918]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.475942]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.475957]  vfs_write+0xc6/0x3b0
[  206.476003]  ? __fget_light+0xa7/0x130
[  206.476012]  ? do_syscall_64+0x69/0x90
[  206.476020]  ? do_syscall_64+0x69/0x90
[  206.476026] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476034]  </TASK>
[  206.476043] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.476079]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476103] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.476106] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.476131]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.476162]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.476193]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.476233]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.476242] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476303]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476317] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.476319] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.476320] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.476321] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.476323] RSP: 0018:ffffc0b1c301fb90 EFLAGS: 00010283
[  206.476324] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.476325] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.476326] RBP: ffffc0b1c301fbf8 R08: 0000000000000001 R09: 
ffffc0b1c301fc58
[  206.476327] R10: 00000000000b67a4 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.476333] Call Trace:
[  206.476346]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.476376]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.476389]  ? do_syscall_64+0x69/0x90
[  206.476395]  __x64_sys_pwrite64+0x96/0xc0
[  206.476403] RIP: 0033:0x7fcbbebcacbf
[  206.476405] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476409] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.476458]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476472] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.476474] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.476475] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.476476] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.476478] RSP: 0018:ffffc0b1c301fbc0 EFLAGS: 00010246
[  206.476479] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.476480] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.476481] RBP: ffffc0b1c301fc00 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.476484] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.476490]  vfio_unpin_pages+0x49/0x60
[  206.476491]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.476533]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.476547]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.476555]  ? do_syscall_64+0x69/0x90
[  206.476561] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476563] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.476566] R10: 00000000008b67a0 R11: 0000000000000293 R12: 
00000000008b67a0
[  206.476582] ------------[ cut here ]------------
[  206.476621]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476635] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.476636] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.476637] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.476639] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.476640] RSP: 0018:ffffc0b1c301faf0 EFLAGS: 00010283
[  206.476645] R13: 000000010d0f6000 R14: ffffc0b1c301fbb8 R15: 
ffff9d7a8d400000
[  206.476670]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.476708]  vfs_write+0xc6/0x3b0
[  206.476723]  ? do_syscall_64+0x69/0x90
[  206.476730]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.476734]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.476739] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476742] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.476748] ---[ end trace 0000000000000000 ]---
[  206.476792]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476806] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.476807] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.476808] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.476810] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.476811] RSP: 0018:ffffc0b1c301fb20 EFLAGS: 00010246
[  206.476814] RBP: ffffc0b1c301fb60 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.476825]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.476835]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.476850]  ? vmx_vcpu_put+0x122/0x210 [kvm_intel]
[  206.476890]  ? __fget_light+0xa7/0x130
[  206.476900]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.476907]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.476912] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476913] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.476916] R10: 00000000008b67a8 R11: 0000000000000293 R12: 
00000000008b67a8
[  206.476971]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476985] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.476986] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.476987] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.476989] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.476990] RSP: 0018:ffffc0b1c301fac0 EFLAGS: 00010283
[  206.476992] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.476993] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.476994] RBP: ffffc0b1c301fb28 R08: 0000000000000001 R09: 
ffffc0b1c301fb88
[  206.476995] R10: 00000000000b67b4 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.476996] R13: 000000010d0f6000 R14: ffffc0b1c301fb88 R15: 
ffff9d7a8d400000
[  206.476997] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.476998] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.476999] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.477000] Call Trace:
[  206.477048]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.477078]  vfio_device_fops_write+0x1f/0x40
[  206.477117]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.477126]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.477131] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.477133] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.477137] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.477148] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.477199] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.477200] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.477201] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.477204] RSP: 0018:ffffc0b1c301faf0 EFLAGS: 00010246
[  206.477207] RBP: ffffc0b1c301fb30 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.477247]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.477255]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.477276]  __x64_sys_pwrite64+0x96/0xc0
[  206.477313]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.477322]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.477327] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.477330] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.477337] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.477389]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.477403] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.477404] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.477405] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.477408] RSP: 0018:ffffc0b1c301fb10 EFLAGS: 00010283
[  206.477410] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.477415] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.477471]  vfs_write+0xc6/0x3b0
[  206.477505]  ? fire_user_return_notifiers+0x42/0x70
[  206.477518] RIP: 0033:0x7fcbbebcacbf
[  206.477522] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.477525] R10: 00000000008b67b8 R11: 0000000000000293 R12: 
00000000008b67b8
[  206.477531] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.477574]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.477589] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.477590] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.477592] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.477593] RSP: 0018:ffffc0b1c301fb40 EFLAGS: 00010246
[  206.477595] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.477598] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.477603] Call Trace:
[  206.477607]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.477697]  ? do_syscall_64+0x69/0x90
[  206.477701]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.477704] RIP: 0033:0x7fcbbebcacbf
[  206.477708] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.477711] R10: 00000000008b67b8 R11: 0000000000000293 R12: 
00000000008b67b8
[  206.477716] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.477730] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.477786] RSP: 0018:ffffc0b1c301fae0 EFLAGS: 00010283
[  206.477787] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.477788] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.477789] RBP: ffffc0b1c301fb48 R08: 0000000000000001 R09: 
ffffc0b1c301fba8
[  206.477790] R10: 00000000000b67c4 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.477791] R13: 000000010d0f6000 R14: ffffc0b1c301fba8 R15: 
ffff9d7a8d400000
[  206.477792] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.477794] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.477795] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.477824]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.477873]  vfio_device_fops_write+0x1f/0x40
[  206.477926] RIP: 0033:0x7fcbbebcacbf
[  206.477929] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.477937]  </TASK>
[  206.477943] ------------[ cut here ]------------
[  206.477946] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.478007] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.478008] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.478010] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.478012] Call Trace:
[  206.478013]  <TASK>
[  206.478026]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.478076]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.478094]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.478152] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.478154] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.478155] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.478162] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.478183] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.478250] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.478251] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.478252] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.478254] Call Trace:
[  206.478254]  <TASK>
[  206.478256]  vfio_pin_pages+0x5e/0xb0
[  206.478275]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.478374] RIP: 0033:0x7fcbbebcacbf
[  206.478379] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.478391] ------------[ cut here ]------------
[  206.478430]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.478443] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.478449] RSP: 0018:ffffc0b1c301fb68 EFLAGS: 00010246
[  206.478453] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.478454] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.478455] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.478456] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.478457] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.478461]  vfio_unpin_pages+0x49/0x60
[  206.478462]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.478473]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.478488]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.478539]  vfs_write+0xc6/0x3b0
[  206.478584] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.478587] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.478592]  </TASK>
[  206.478608] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.478674] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.478675] Call Trace:
[  206.478676]  <TASK>
[  206.478678]  vfio_pin_pages+0x5e/0xb0
[  206.478679]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.478696]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.478713]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.478751]  ? vfs_write+0xc6/0x3b0
[  206.478797] RIP: 0033:0x7fcbbebcacbf
[  206.478802] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.478808]  </TASK>
[  206.478814] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.478853]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.478871] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.478878] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.478879] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.478880] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.478881] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.478882] Call Trace:
[  206.478883]  <TASK>
[  206.478885]  vfio_unpin_pages+0x49/0x60
[  206.478905]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.478930]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.478993]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.479009] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.479010] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.479011] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.479014] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.479032] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.479095] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.479096] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.479097] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.479099] Call Trace:
[  206.479132]  vfio_pin_pages+0x5e/0xb0
[  206.479142]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.479201]  vfs_write+0xc6/0x3b0
[  206.479229] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.479234] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.479246] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.479310] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.479312] Call Trace:
[  206.479312]  <TASK>
[  206.479314]  vfio_unpin_pages+0x49/0x60
[  206.479325]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.479340]  ? vcpu_put+0x22/0x50 [kvm]
[  206.479384]  vfs_write+0xc6/0x3b0
[  206.479396]  do_syscall_64+0x59/0x90
[  206.479409] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.479414] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.479430] ------------[ cut here ]------------
[  206.479470]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.479484] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.479490] RSP: 0018:ffffc0b1c301fb48 EFLAGS: 00010283
[  206.479498] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.479499] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.479500] Call Trace:
[  206.479501]  <TASK>
[  206.479503]  vfio_pin_pages+0x5e/0xb0
[  206.479504]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.479513]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.479520]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.479527]  ? kvm_arch_vcpu_ioctl_run+0x3c4/0x1850 [kvm]
[  206.479553]  ? vcpu_put+0x22/0x50 [kvm]
[  206.479573]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.479581]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.479588]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.479595]  vfio_device_fops_write+0x1f/0x40
[  206.479597]  vfs_write+0xc6/0x3b0
[  206.479604]  ? __fget_light+0xa7/0x130
[  206.479615]  ? do_syscall_64+0x69/0x90
[  206.479627] R10: 00000000008b67e0 R11: 0000000000000293 R12: 
00000000008b67e0
[  206.479638] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.479677]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.479691] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.479705] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.479707]  <TASK>
[  206.479710]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.479720]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.479728]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.479735]  ? kvm_arch_vcpu_ioctl_run+0x3c4/0x1850 [kvm]
[  206.479789]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.479796]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.479803]  vfio_device_fops_write+0x1f/0x40
[  206.479804]  vfs_write+0xc6/0x3b0
[  206.479811]  ? __fget_light+0xa7/0x130
[  206.479821]  ? do_syscall_64+0x69/0x90
[  206.479833] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.479840] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.479850] ------------[ cut here ]------------
[  206.479890]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.479904] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.479915] R13: 000000010d0f6000 R14: ffffc0b1c301fc60 R15: 
ffff9d7a8d400000
[  206.479921]  <TASK>
[  206.479924]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.479933]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.479948]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.479963]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.479970]  vfio_device_fops_write+0x1f/0x40
[  206.479971]  vfs_write+0xc6/0x3b0
[  206.479976]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.479978]  ? __fget_light+0xa7/0x130
[  206.479980]  __x64_sys_pwrite64+0x96/0xc0
[  206.479983]  do_syscall_64+0x59/0x90
[  206.479988] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480043]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.480057] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.480058] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.480062] RSP: 0018:ffffc0b1c301fbc8 EFLAGS: 00010246
[  206.480071] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.480130]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480132]  ? __fget_light+0xa7/0x130
[  206.480134]  __x64_sys_pwrite64+0x96/0xc0
[  206.480139]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.480142] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480144] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.480145] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.480146] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.480148] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.480152]  </TASK>
[  206.480165] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.480204]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.480224] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.480232] R13: 000000010d0f6000 R14: ffffc0b1c301fbf8 R15: 
ffff9d7a8d400000
[  206.480270]  vfio_pin_pages+0x5e/0xb0
[  206.480359]  do_syscall_64+0x59/0x90
[  206.480370] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480371] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.480373] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.480374] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.480375] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.480376] R10: 00000000008b67f0 R11: 0000000000000293 R12: 
00000000008b67f0
[  206.480377] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.480379]  </TASK>
[  206.480388] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.480552]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480555]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.480559] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480560] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.480562] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.480563] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.480564] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.480565] R10: 00000000008b67f0 R11: 0000000000000293 R12: 
00000000008b67f0
[  206.480585] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.480731]  vfs_write+0xc6/0x3b0
[  206.480764]  ? __fget_light+0xa7/0x130
[  206.480766]  __x64_sys_pwrite64+0x96/0xc0
[  206.480769]  do_syscall_64+0x59/0x90
[  206.480771]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.480773]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480775]  ? do_syscall_64+0x69/0x90
[  206.480777]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.480779]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480781]  ? do_syscall_64+0x69/0x90
[  206.480783]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.480785] RIP: 0033:0x7fcbbebcacbf
[  206.480786] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480788] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.480789] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.480790] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.480797] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.480841]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.480855] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.480857] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.480860] RSP: 0018:ffffc0b1c301faf8 EFLAGS: 00010246
[  206.480868] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.480910]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.480936]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.480960]  ? __fget_light+0xa7/0x130
[  206.480965]  do_syscall_64+0x59/0x90
[  206.480967]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.480969]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480971]  ? do_syscall_64+0x69/0x90
[  206.480973]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.480975]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480977]  ? do_syscall_64+0x69/0x90
[  206.480979]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.480981] RIP: 0033:0x7fcbbebcacbf
[  206.480982] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480992]  </TASK>
[  206.481045]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.481059] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.481060] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.481061] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.481064] RSP: 0018:ffffc0b1c301fac8 EFLAGS: 00010283
[  206.481065] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.481104]  vfio_pin_pages+0x5e/0xb0
[  206.481145]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.481153]  vfio_device_fops_write+0x1f/0x40
[  206.481154]  vfs_write+0xc6/0x3b0
[  206.481157]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.481176]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.481200]  ? __fget_light+0xa7/0x130
[  206.481202]  __x64_sys_pwrite64+0x96/0xc0
[  206.481205]  do_syscall_64+0x59/0x90
[  206.481207]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481209]  ? do_syscall_64+0x69/0x90
[  206.481210]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481212]  ? do_syscall_64+0x69/0x90
[  206.481214]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.481218]  ? do_syscall_64+0x69/0x90
[  206.481220]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.481223] RIP: 0033:0x7fcbbebcacbf
[  206.481225] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.481230] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.481281]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.481299] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.481301] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.481305] RSP: 0018:ffffc0b1c301faf8 EFLAGS: 00010246
[  206.481311] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.481368]  vfio_device_fops_write+0x1f/0x40
[  206.481369]  vfs_write+0xc6/0x3b0
[  206.481392]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.481415]  ? __fget_light+0xa7/0x130
[  206.481418]  __x64_sys_pwrite64+0x96/0xc0
[  206.481420]  do_syscall_64+0x59/0x90
[  206.481422]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481424]  ? do_syscall_64+0x69/0x90
[  206.481425]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481427]  ? do_syscall_64+0x69/0x90
[  206.481429]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.481432]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481434]  ? do_syscall_64+0x69/0x90
[  206.481435]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.481437] RIP: 0033:0x7fcbbebcacbf
[  206.481438] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.481440] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.481441] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.481442] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.481448]  </TASK>
[  206.481502]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.481515] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.481517] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.481518] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.481519] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.481521] RSP: 0018:ffffc0b1c301fb58 EFLAGS: 00010283
[  206.481522] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.481529] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.481565]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.481580]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.481587]  vfio_device_fops_write+0x1f/0x40
[  206.481589]  vfs_write+0xc6/0x3b0
[  206.481591]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.481594]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481596]  ? __fget_light+0xa7/0x130
[  206.481598]  __x64_sys_pwrite64+0x96/0xc0
[  206.481601]  do_syscall_64+0x59/0x90
[  206.481603]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481604]  ? do_syscall_64+0x69/0x90
[  206.481606]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.481608] RIP: 0033:0x7fcbbebcacbf
[  206.481609] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.481611] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.481612] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.481613] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.481614] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.481616] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.481664]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.481678] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.481683] RSP: 0018:ffffc0b1c301fb88 EFLAGS: 00010246
[  206.481684] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.481687] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.481729]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.481768]  ? do_syscall_64+0x69/0x90
[  206.481769]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.481771] RIP: 0033:0x7fcbbebcacbf
[  206.481773] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.481774] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.481775] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.481777] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.481778] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.481779] R10: 00000000008b6808 R11: 0000000000000293 R12: 
00000000008b6808
[  206.481780] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.481782]  </TASK>
[  206.481783] ---[ end trace 0000000000000000 ]---
[  206.481797] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.481946]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.481992]  ? do_syscall_64+0x69/0x90
[  206.481996]  ? do_syscall_64+0x69/0x90
[  206.481997]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.481999] RIP: 0033:0x7fcbbebcacbf
[  206.482001] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.482002] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.482004] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.482005] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.482006] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.482007] R10: 00000000008b6810 R11: 0000000000000293 R12: 
00000000008b6810
[  206.482008] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.482010]  </TASK>
[  206.482011] ---[ end trace 0000000000000000 ]---
[  206.482012] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.482016] ------------[ cut here ]------------
[  206.482017] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.482055]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.482073] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.482079] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.482084] Call Trace:
[  206.482150]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.482194]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482198]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482199]  ? do_syscall_64+0x69/0x90
[  206.482201]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.482203] RIP: 0033:0x7fcbbebcacbf
[  206.482205] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.482206] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.482207] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.482209] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.482210] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.482211] R10: 00000000008b6810 R11: 0000000000000293 R12: 
00000000008b6810
[  206.482212] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.482278]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.482292] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.482294] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.482295] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.482298] RSP: 0018:ffffc0b1c301faa8 EFLAGS: 00010283
[  206.482299] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.482304] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.482388]  vfs_write+0xc6/0x3b0
[  206.482422]  ? do_syscall_64+0x69/0x90
[  206.482423]  ? fire_user_return_notifiers+0x42/0x70
[  206.482425]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.482428]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482430]  ? do_syscall_64+0x69/0x90
[  206.482431]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482433]  ? do_syscall_64+0x69/0x90
[  206.482434]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.482437] RIP: 0033:0x7fcbbebcacbf
[  206.482438] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.482439] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.482441] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.482444] R10: 00000000008b6818 R11: 0000000000000293 R12: 
00000000008b6818
[  206.482493]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.482507] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.482510] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.482513] RSP: 0018:ffffc0b1c301fad8 EFLAGS: 00010246
[  206.482515] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.482568]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.482607]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.482609]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482612]  ? fire_user_return_notifiers+0x42/0x70
[  206.482615]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.482617]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482619]  ? do_syscall_64+0x69/0x90
[  206.482621]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482623]  ? do_syscall_64+0x69/0x90
[  206.482624]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.482626] RIP: 0033:0x7fcbbebcacbf
[  206.482628] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.482690]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.482704] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.482705] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.482706] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.482709] RSP: 0018:ffffc0b1c301fb08 EFLAGS: 00010283
[  206.482713] R10: 00000000000b6824 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.482774]  vfs_write+0xc6/0x3b0
[  206.482776]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.482796]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.482798]  ? __fget_light+0xa7/0x130
[  206.482801]  __x64_sys_pwrite64+0x96/0xc0
[  206.482804]  do_syscall_64+0x59/0x90
[  206.482805]  ? do_syscall_64+0x69/0x90
[  206.482807]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482809]  ? do_syscall_64+0x69/0x90
[  206.482812]  ? do_syscall_64+0x69/0x90
[  206.482813]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.482816] RIP: 0033:0x7fcbbebcacbf
[  206.482818] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.482820] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.482823] R10: 00000000008b6820 R11: 0000000000000293 R12: 
00000000008b6820
[  206.482872]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.482886] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.482890] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.482931]  ? restore_fpregs_from_fpstate+0x47/0xc0
[  206.482985]  __x64_sys_pwrite64+0x96/0xc0
[  206.482987]  do_syscall_64+0x59/0x90
[  206.482989]  ? do_syscall_64+0x69/0x90
[  206.482991]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482993]  ? do_syscall_64+0x69/0x90
[  206.482996]  ? do_syscall_64+0x69/0x90
[  206.482997]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.483000] RIP: 0033:0x7fcbbebcacbf
[  206.483001] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.483007] R10: 00000000008b6820 R11: 0000000000000293 R12: 
00000000008b6820
[  206.483062]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.483076] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.483077] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.483081] RSP: 0018:ffffc0b1c301fb28 EFLAGS: 00010283
[  206.483085] RBP: ffffc0b1c301fb90 R08: 0000000000000001 R09: 
ffffc0b1c301fbf0
[  206.483166]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.483198]  __x64_sys_pwrite64+0x96/0xc0
[  206.483200]  do_syscall_64+0x59/0x90
[  206.483202]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.483204]  ? do_syscall_64+0x69/0x90
[  206.483205]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.483207]  ? do_syscall_64+0x69/0x90
[  206.483209]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.483211] RIP: 0033:0x7fcbbebcacbf
[  206.483212] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.483214] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.483215] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.483216] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.483217] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.483223]  </TASK>
[  206.483230] ------------[ cut here ]------------
[  206.483270]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.483287] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.483290] RSP: 0018:ffffc0b1c301fb58 EFLAGS: 00010246
[  206.483294] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.483322]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.483414]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.483419] RIP: 0033:0x7fcbbebcacbf
[  206.483421] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.483422] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.483423] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.483425] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.483426] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.483427] R10: 00000000008b6828 R11: 0000000000000293 R12: 
00000000008b6828
[  206.483428] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.483444] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.483484]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.483502] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.483510] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.483559]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.483589]  ? do_syscall_64+0x69/0x90
[  206.483591]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.483593] RIP: 0033:0x7fcbbebcacbf
[  206.483594] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.483596] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.483597] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.483598] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.483599] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.483600] R10: 00000000008b6830 R11: 0000000000000293 R12: 
00000000008b6830
[  206.483601] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.483604]  </TASK>
[  206.483605] ---[ end trace 0000000000000000 ]---
[  206.483610] ------------[ cut here ]------------
[  206.483650]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.483667] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.483674] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.483679] Call Trace:
[  206.483701]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.483752]  ? do_syscall_64+0x69/0x90
[  206.483762] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.483764] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.483765] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.483766] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.483767] R10: 00000000008b6830 R11: 0000000000000293 R12: 
00000000008b6830
[  206.483768] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.483770]  </TASK>
[  206.483771] ---[ end trace 0000000000000000 ]---
[  206.483772] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.483782] ------------[ cut here ]------------
[  206.483783] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.483785] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.483878]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.483970] RIP: 0033:0x7fcbbebcacbf
[  206.483974] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.483975] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.483977] R10: 00000000008b6838 R11: 0000000000000293 R12: 
00000000008b6838
[  206.483978] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.483981]  </TASK>
[  206.483982] ---[ end trace 0000000000000000 ]---
[  206.483983] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.483987] ------------[ cut here ]------------
[  206.483988] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.483990] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.484085]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.484162]  ? fire_user_return_notifiers+0x42/0x70
[  206.484181] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.484189] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.484200] ------------[ cut here ]------------
[  206.484200] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.484243]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.484257] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.484259] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.484259] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.484261] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.484262] RSP: 0018:ffffc0b1c301fb30 EFLAGS: 00010283
[  206.484265] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.484267] R10: 00000000000b6844 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.484277]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.484301]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.484327]  ? __fget_light+0xa7/0x130
[  206.484338]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.484345] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.484401]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.484414] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.484416] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.484417] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.484418] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.484425] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.484443]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.484482]  vfio_device_fops_write+0x1f/0x40
[  206.484488]  __x64_sys_pwrite64+0x96/0xc0
[  206.484498]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.484504] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.484567]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.484582] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.484583] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.484584] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.484585] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.484587] RSP: 0018:ffffc0b1c301fb60 EFLAGS: 00010283
[  206.484595] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.484597] Call Trace:
[  206.484598]  <TASK>
[  206.484653]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.484679]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.484708]  ? do_syscall_64+0x69/0x90
[  206.484712] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.484722]  </TASK>
[  206.484768]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.484782] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.484783] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.484784] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.484786] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.484787] RSP: 0018:ffffc0b1c301fb90 EFLAGS: 00010246
[  206.484788] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.484789] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.484790] RBP: ffffc0b1c301fbd0 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.484791] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.484792] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.484800]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.484842]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.484883]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.484888] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.484891] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.484896]  </TASK>
[  206.484908] ------------[ cut here ]------------
[  206.484949]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.484963] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.484965] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.484966] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.484967] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.484969] RSP: 0018:ffffc0b1c301fac0 EFLAGS: 00010283
[  206.484970] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.484971] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.485007]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.485021]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.485038]  do_syscall_64+0x59/0x90
[  206.485053]  ? do_syscall_64+0x69/0x90
[  206.485059] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.485061] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.485064] R10: 00000000008b6850 R11: 0000000000000293 R12: 
00000000008b6850
[  206.485076] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.485136] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.485172]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.485198]  ? __fget_light+0xa7/0x130
[  206.485211]  ? do_syscall_64+0x69/0x90
[  206.485216]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.485223] RIP: 0033:0x7fcbbebcacbf
[  206.485226] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.485229] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.485232] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.485292]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.485307] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.485308] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.485309] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.485311] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.485312] RSP: 0018:ffffc0b1c301fb70 EFLAGS: 00010283
[  206.485322] Call Trace:
[  206.485342]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.485349]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.485376]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.485426]  ? __fget_light+0xa7/0x130
[  206.485436]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.485441] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.485442] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.485444] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.485449] ---[ end trace 0000000000000000 ]---
[  206.485494]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas vi

-- 
Regards,
Yi Liu

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Yi Liu]

On 2022/11/1 07:24, Jason Gunthorpe wrote:
> On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
>>> There is something wrong with the test suite that it isn't covering
>>> the above, I'm going to look into that today.
>>
>> sounds to be the cause. I didn't see any significant change in vfio_main.c
>> that may fail gvt. So should the iommufd changes. Then we will re-run the
>> test after your update.:-)
> 
> I updated the github with all the changes made so far, it is worth
> trying again!

gvt is still failing with below call trace in host side. vfio_unpin_pages()
is still in problem. Any idea on it?


[  206.464107] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.464167] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.464168] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.464171] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.464172] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.464173] Call Trace:
[  206.464174]  <TASK>
[  206.464175]  vfio_unpin_pages+0x49/0x60
[  206.464177]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.464187]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.464196]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.464229]  vfs_write+0xc6/0x3b0
[  206.464296] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.464301] R10: 00000000008b66a8 R11: 0000000000000293 R12: 
00000000008b66a8
[  206.464318] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.464357]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.464377] RSP: 0018:ffffc0b1c301fb90 EFLAGS: 00010283
[  206.464381] R10: 00000000000b66b4 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.464382] R13: 000000010d0f6000 R14: ffffc0b1c301fc58 R15: 
ffff9d7a8d400000
[  206.464384] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.464385] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.464386] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.464387] Call Trace:
[  206.464388]  <TASK>
[  206.464389]  ? x86_emulate_instruction+0x4cd/0x890 [kvm]
[  206.464415]  vfio_pin_pages+0x5e/0xb0
[  206.464417]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.464434]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.464505] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.464509] R10: 00000000008b66b0 R11: 0000000000000293 R12: 
00000000008b66b0
[  206.464513]  </TASK>
[  206.464519] ------------[ cut here ]------------
[  206.464558]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.464572] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.464578] RSP: 0018:ffffc0b1c301fbc0 EFLAGS: 00010246
[  206.464583] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.464584] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.464586] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.464588] Call Trace:
[  206.464588]  <TASK>
[  206.464590]  vfio_unpin_pages+0x49/0x60
[  206.464591]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.464610]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.464640]  vfio_device_fops_write+0x1f/0x40
[  206.464681] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.464690]  </TASK>
[  206.464702] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.464742]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.464756] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.464767] R13: 000000010d0f6000 R14: ffffc0b1c301fbd0 R15: 
ffff9d7a8d400000
[  206.464771] Call Trace:
[  206.464772]  <TASK>
[  206.464773]  ? writeback_registers+0x56/0xb0 [kvm]
[  206.464802]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.464811]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.464819]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.464826]  ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
[  206.464854]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.464879]  ? intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.464902]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.464910] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.464912] R10: 00000000008b66b8 R11: 0000000000000293 R12: 
00000000008b66b8
[  206.464922] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.464960]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.464974] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.464985] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.464987] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.464988] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.464989] Call Trace:
[  206.464990]  <TASK>
[  206.464992]  vfio_unpin_pages+0x49/0x60
[  206.464993]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.465003]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.465012]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.465019]  ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
[  206.465039]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.465047]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.465054]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.465061]  vfio_device_fops_write+0x1f/0x40
[  206.465063]  vfs_write+0xc6/0x3b0
[  206.465065]  ? intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.465081]  __x64_sys_pwrite64+0x96/0xc0
[  206.465100] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.465106] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.465159]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.465173] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.465175] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.465184] R13: 000000010d0f6000 R14: ffffc0b1c301fb68 R15: 
ffff9d7a8d400000
[  206.465186] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.465188] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.465189] Call Trace:
[  206.465191]  ? segmented_write.isra.0+0x5e/0x90 [kvm]
[  206.465218]  vfio_pin_pages+0x5e/0xb0
[  206.465220]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.465230]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.465239]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.465247]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.465255]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.465261]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.465269]  vfio_device_fops_write+0x1f/0x40
[  206.465270]  vfs_write+0xc6/0x3b0
[  206.465272]  ? intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.465280]  ? __fget_light+0xa7/0x130
[  206.465285]  __x64_sys_pwrite64+0x96/0xc0
[  206.465320]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.465336] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.465338] R10: 00000000008b66c0 R11: 0000000000000293 R12: 
00000000008b66c0
[  206.465347] ------------[ cut here ]------------
[  206.465386]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.465400] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.465405] RSP: 0018:ffffc0b1c301fad0 EFLAGS: 00010246
[  206.465410] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.465412] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.465413] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.465414] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.465415] Call Trace:
[  206.465416]  <TASK>
[  206.465417]  vfio_unpin_pages+0x49/0x60
[  206.465419]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.465429]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.465438]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.465446]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.465461]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.465481]  __x64_sys_pwrite64+0x96/0xc0
[  206.465522]  ? do_syscall_64+0x69/0x90
[  206.465530] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.465538] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.465590]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.465604] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.465613] R10: 00000000000b66cc R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.465615] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.465617] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.465618] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.465619] Call Trace:
[  206.465620]  <TASK>
[  206.465621]  ? x86_emulate_instruction+0x4cd/0x890 [kvm]
[  206.465649]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.465672]  ? vmx_vcpu_put+0x122/0x210 [kvm_intel]
[  206.465702]  vfs_write+0xc6/0x3b0
[  206.465753]  ? do_syscall_64+0x69/0x90
[  206.465760] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.465762] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.465769]  </TASK>
[  206.465775] ------------[ cut here ]------------
[  206.465814]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.465828] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.465834] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.465835] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.465836] RBP: ffffc0b1c301fb10 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.465837] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.465838] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.465839] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.465841] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.465842] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.465843] Call Trace:
[  206.465844]  <TASK>
[  206.465845]  vfio_unpin_pages+0x49/0x60
[  206.465847]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.465857]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.465872]  ? vmx_vcpu_put+0x122/0x210 [kvm_intel]
[  206.465902]  vfio_device_fops_write+0x1f/0x40
[  206.465913]  ? __fget_light+0xa7/0x130
[  206.465957]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.465965] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.465982] ------------[ cut here ]------------
[  206.465986] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.466051] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.466052] Call Trace:
[  206.466052]  <TASK>
[  206.466083]  vfio_pin_pages+0x5e/0xb0
[  206.466085]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.466094]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.466102]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.466129]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.466165]  __x64_sys_pwrite64+0x96/0xc0
[  206.466176] RIP: 0033:0x7fcbbebcacbf
[  206.466187]  </TASK>
[  206.466193] ------------[ cut here ]------------
[  206.466194] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.466236]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.466250] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.466261] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.466268]  vfio_unpin_pages+0x49/0x60
[  206.466279]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.466288]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.466296]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.466304]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.466311]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.466322]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.466342]  ? __fget_light+0xa7/0x130
[  206.466344]  __x64_sys_pwrite64+0x96/0xc0
[  206.466347]  do_syscall_64+0x59/0x90
[  206.466351]  ? do_syscall_64+0x69/0x90
[  206.466358] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.466387] ------------[ cut here ]------------
[  206.466427]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.466441] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.466443] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.466456] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.466504]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.466510]  ? restore_fpregs_from_fpstate+0x47/0xc0
[  206.466514]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.466521]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.466528]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.466537]  vfs_write+0xc6/0x3b0
[  206.466539]  ? vfio_device_fops_write+0x1f/0x40
[  206.466567]  __x64_sys_pwrite64+0x96/0xc0
[  206.466570]  do_syscall_64+0x59/0x90
[  206.466572]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.466574]  ? do_syscall_64+0x69/0x90
[  206.466575]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.466585] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.466599] ------------[ cut here ]------------
[  206.466600] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.466639]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.466653] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.466666] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.466672]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.466682]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.466691]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.466697]  ? restore_fpregs_from_fpstate+0x47/0xc0
[  206.466701]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.466709]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.466716]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.466723]  vfio_device_fops_write+0x1f/0x40
[  206.466724]  vfs_write+0xc6/0x3b0
[  206.466727]  ? vfio_device_fops_write+0x1f/0x40
[  206.466728]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.466752]  ? __fget_light+0xa7/0x130
[  206.466755]  __x64_sys_pwrite64+0x96/0xc0
[  206.466758]  do_syscall_64+0x59/0x90
[  206.466765]  ? do_syscall_64+0x69/0x90
[  206.466772] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.466794] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.466833]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.466847] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.466862] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.466883]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.466891]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.466899]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.466912]  vfio_device_fops_write+0x1f/0x40
[  206.466914]  vfs_write+0xc6/0x3b0
[  206.466918]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.466942]  ? __fget_light+0xa7/0x130
[  206.466945]  __x64_sys_pwrite64+0x96/0xc0
[  206.466947]  do_syscall_64+0x59/0x90
[  206.466949]  ? do_syscall_64+0x69/0x90
[  206.466958] RIP: 0033:0x7fcbbebcacbf
[  206.466963] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.466977] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.467057]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.467066]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.467074]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.467082]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.467089]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467096]  vfio_device_fops_write+0x1f/0x40
[  206.467097]  vfs_write+0xc6/0x3b0
[  206.467099]  ? vfio_device_fops_write+0x1f/0x40
[  206.467101]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.467125]  ? __fget_light+0xa7/0x130
[  206.467128]  __x64_sys_pwrite64+0x96/0xc0
[  206.467131]  do_syscall_64+0x59/0x90
[  206.467132]  ? do_syscall_64+0x69/0x90
[  206.467134]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.467136]  ? do_syscall_64+0x69/0x90
[  206.467137]  ? do_syscall_64+0x69/0x90
[  206.467138]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.467142] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.467204]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.467218] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.467219] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.467220] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.467231] R10: 00000000000b66ec R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.467265]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.467273]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.467280]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467287]  vfio_device_fops_write+0x1f/0x40
[  206.467288]  vfs_write+0xc6/0x3b0
[  206.467322]  ? __fget_light+0xa7/0x130
[  206.467324]  __x64_sys_pwrite64+0x96/0xc0
[  206.467327]  do_syscall_64+0x59/0x90
[  206.467329]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.467331]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.467333]  ? do_syscall_64+0x69/0x90
[  206.467335]  ? do_syscall_64+0x69/0x90
[  206.467336]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.467338]  ? do_syscall_64+0x69/0x90
[  206.467340]  ? do_syscall_64+0x69/0x90
[  206.467345] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.467363] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.467442]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.467450]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.467458]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.467466]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.467473]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467480]  vfio_device_fops_write+0x1f/0x40
[  206.467482]  vfs_write+0xc6/0x3b0
[  206.467484]  ? intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467491]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.467515]  ? __fget_light+0xa7/0x130
[  206.467518]  __x64_sys_pwrite64+0x96/0xc0
[  206.467520]  do_syscall_64+0x59/0x90
[  206.467522]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.467525]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.467533]  ? do_syscall_64+0x69/0x90
[  206.467542] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.467563] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.467678]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.467686]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.467693]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.467700]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467707]  vfio_device_fops_write+0x1f/0x40
[  206.467709]  vfs_write+0xc6/0x3b0
[  206.467731]  ? __fget_light+0xa7/0x130
[  206.467733]  __x64_sys_pwrite64+0x96/0xc0
[  206.467736]  do_syscall_64+0x59/0x90
[  206.467738]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.467740]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.467742]  ? do_syscall_64+0x69/0x90
[  206.467744]  ? do_syscall_64+0x69/0x90
[  206.467748] RIP: 0033:0x7fcbbebcacbf
[  206.467751] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.467765] ------------[ cut here ]------------
[  206.467768] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.467847]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.467856]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.467864]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.467872]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.467879]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.467886]  vfio_device_fops_write+0x1f/0x40
[  206.467887]  vfs_write+0xc6/0x3b0
[  206.467890]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.467910]  ? __fget_light+0xa7/0x130
[  206.467916]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.467922]  ? do_syscall_64+0x69/0x90
[  206.467925]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.467927] RIP: 0033:0x7fcbbebcacbf
[  206.467928] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.467933] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.467990]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.468003] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.468007] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.468098]  vfio_device_fops_write+0x1f/0x40
[  206.468122]  ? __fget_light+0xa7/0x130
[  206.468124]  ? __fget_light+0xa7/0x130
[  206.468126]  __x64_sys_pwrite64+0x96/0xc0
[  206.468129]  do_syscall_64+0x59/0x90
[  206.468130]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.468133]  ? do_syscall_64+0x69/0x90
[  206.468137]  ? do_syscall_64+0x69/0x90
[  206.468141] RIP: 0033:0x7fcbbebcacbf
[  206.468143] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.468145] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.468146] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.468147] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.468197]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.468211] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.468212] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.468215] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.468229] Call Trace:
[  206.468283]  vfs_write+0xc6/0x3b0
[  206.468312]  do_syscall_64+0x59/0x90
[  206.468313]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.468315]  ? do_syscall_64+0x69/0x90
[  206.468318]  ? do_syscall_64+0x69/0x90
[  206.468321]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.468323] RIP: 0033:0x7fcbbebcacbf
[  206.468324] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.468327] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.468328] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.468329] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.468336] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.468387]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.468404] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.468407] RSP: 0018:ffffc0b1c301fb50 EFLAGS: 00010283
[  206.468410] RBP: ffffc0b1c301fbb8 R08: 0000000000000001 R09: 
ffffc0b1c301fc18
[  206.468416] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.468532]  ? do_syscall_64+0x69/0x90
[  206.468538] RIP: 0033:0x7fcbbebcacbf
[  206.468539] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.468540] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.468542] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.468544] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.468545] R10: 00000000008b6700 R11: 0000000000000293 R12: 
00000000008b6700
[  206.468546] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.468549]  </TASK>
[  206.468549] ---[ end trace 0000000000000000 ]---
[  206.468557] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.468676]  vfio_device_fops_write+0x1f/0x40
[  206.468723] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.468724] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.468725] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.468726] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.468727] R10: 00000000008b6700 R11: 0000000000000293 R12: 
00000000008b6700
[  206.468728] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.468731]  </TASK>
[  206.468731] ---[ end trace 0000000000000000 ]---
[  206.468732] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.468742] ------------[ cut here ]------------
[  206.468743] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.468746] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.468808] R13: 000000010d0f6000 R14: ffffc0b1c301fba0 R15: 
ffff9d7a8d400000
[  206.468813] Call Trace:
[  206.468825]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.468921]  do_syscall_64+0x59/0x90
[  206.468938] RIP: 0033:0x7fcbbebcacbf
[  206.468942] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.468944] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.468945] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.468946] R10: 00000000008b6708 R11: 0000000000000293 R12: 
00000000008b6708
[  206.468947] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.468949]  </TASK>
[  206.468950] ---[ end trace 0000000000000000 ]---
[  206.468951] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.468955] ------------[ cut here ]------------
[  206.468956] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.468958] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.469023] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.469046]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.469140]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.469157] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.469158] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.469159] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.469161] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.469164]  </TASK>
[  206.469165] ---[ end trace 0000000000000000 ]---
[  206.469166] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.469176] ------------[ cut here ]------------
[  206.469179] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.469311]  ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
[  206.469360]  __x64_sys_pwrite64+0x96/0xc0
[  206.469387] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.469391] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.469392] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.469394] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.469397]  </TASK>
[  206.469398] ---[ end trace 0000000000000000 ]---
[  206.469399] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.469403] ------------[ cut here ]------------
[  206.469404] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.469443]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.469457] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.469463] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.469466] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.469468] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.469486]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.469537]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.469569]  ? do_syscall_64+0x69/0x90
[  206.469579] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.469581] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.469582] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.469583] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.469584] R10: 00000000008b6710 R11: 0000000000000293 R12: 
00000000008b6710
[  206.469585] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.469587]  </TASK>
[  206.469588] ---[ end trace 0000000000000000 ]---
[  206.469589] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.469601] ------------[ cut here ]------------
[  206.469602] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.469605] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.469664] RBP: ffffc0b1c301fb10 R08: 0000000000000001 R09: 
ffffc0b1c301fb70
[  206.469669] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.469673]  ? x86_emulate_instruction+0x4cd/0x890 [kvm]
[  206.469772]  ? do_syscall_64+0x69/0x90
[  206.469782] RIP: 0033:0x7fcbbebcacbf
[  206.469785] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.469787] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.469788] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.469789] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.469790] R10: 00000000008b6718 R11: 0000000000000293 R12: 
00000000008b6718
[  206.469791] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.469793]  </TASK>
[  206.469794] ---[ end trace 0000000000000000 ]---
[  206.469795] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.469799] ------------[ cut here ]------------
[  206.469839]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.469853] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.469854] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.469858] RSP: 0018:ffffc0b1c301fad8 EFLAGS: 00010246
[  206.469861] RBP: ffffc0b1c301fb18 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.469897]  ? restore_fpregs_from_fpstate+0x47/0xc0
[  206.469939]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.469941]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.469943]  ? do_syscall_64+0x69/0x90
[  206.469945]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.469947]  ? do_syscall_64+0x69/0x90
[  206.469948]  ? do_syscall_64+0x69/0x90
[  206.469949]  ? do_syscall_64+0x69/0x90
[  206.469950]  ? do_syscall_64+0x69/0x90
[  206.469952]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.469954] RIP: 0033:0x7fcbbebcacbf
[  206.469955] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.469957] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.469958] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.469959] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.469960] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.469961] R10: 00000000008b6718 R11: 0000000000000293 R12: 
00000000008b6718
[  206.469962] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.469965]  </TASK>
[  206.469966] ---[ end trace 0000000000000000 ]---
[  206.469967] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.469977] ------------[ cut here ]------------
[  206.469978] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.470017]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.470035] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.470043] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.470161]  do_syscall_64+0x59/0x90
[  206.470172]  ? do_syscall_64+0x69/0x90
[  206.470175]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.470177] RIP: 0033:0x7fcbbebcacbf
[  206.470179] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.470180] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.470182] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.470183] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.470184] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.470185] R10: 00000000008b6720 R11: 0000000000000293 R12: 
00000000008b6720
[  206.470186] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.470188]  </TASK>
[  206.470189] ---[ end trace 0000000000000000 ]---
[  206.470190] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.470194] ------------[ cut here ]------------
[  206.470237]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.470251] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.470257] RSP: 0018:ffffc0b1c301fb30 EFLAGS: 00010246
[  206.470258] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.470262] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.470281]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.470359]  ? do_syscall_64+0x69/0x90
[  206.470367] RIP: 0033:0x7fcbbebcacbf
[  206.470368] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.470369] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.470371] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.470372] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.470373] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.470374] R10: 00000000008b6720 R11: 0000000000000293 R12: 
00000000008b6720
[  206.470375] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.470398] ------------[ cut here ]------------
[  206.470438]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.470457] RSP: 0018:ffffc0b1c301fb38 EFLAGS: 00010283
[  206.470458] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.470460] RBP: ffffc0b1c301fba0 R08: 0000000000000001 R09: 
ffffc0b1c301fc00
[  206.470465] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.470470]  vfio_pin_pages+0x5e/0xb0
[  206.470526]  ? __fget_light+0xa7/0x130
[  206.470547] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.470549] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.470550] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.470551] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.470552] R10: 00000000008b6728 R11: 0000000000000293 R12: 
00000000008b6728
[  206.470553] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.470556]  </TASK>
[  206.470556] ---[ end trace 0000000000000000 ]---
[  206.470557] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.470561] ------------[ cut here ]------------
[  206.470562] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.470601]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.470616] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.470619] RSP: 0018:ffffc0b1c301fb68 EFLAGS: 00010246
[  206.470623] RBP: ffffc0b1c301fba8 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.470627] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.470629] Call Trace:
[  206.470652]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.470698]  ? do_syscall_64+0x69/0x90
[  206.470711] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.470715] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.470716] R10: 00000000008b6728 R11: 0000000000000293 R12: 
00000000008b6728
[  206.470717] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.470719]  </TASK>
[  206.470720] ---[ end trace 0000000000000000 ]---
[  206.470721] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.470731] ------------[ cut here ]------------
[  206.470732] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.470734] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.470843]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.470878]  ? do_syscall_64+0x69/0x90
[  206.470886] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.470887] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.470888] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.470889] R10: 00000000008b6730 R11: 0000000000000293 R12: 
00000000008b6730
[  206.470890] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.470893]  </TASK>
[  206.470893] ---[ end trace 0000000000000000 ]---
[  206.470894] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.470898] ------------[ cut here ]------------
[  206.470899] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.470901] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.470967] Call Trace:
[  206.470996]  ? vmx_vcpu_put+0x122/0x210 [kvm_intel]
[  206.471040]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.471052] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.471054] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.471055] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.471056] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.471057] R10: 00000000008b6730 R11: 0000000000000293 R12: 
00000000008b6730
[  206.471058] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.471060]  </TASK>
[  206.471061] ---[ end trace 0000000000000000 ]---
[  206.471062] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.471072] ------------[ cut here ]------------
[  206.471073] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.471075] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.471136] R10: 00000000000b673c R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.471141] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.471207]  ? do_syscall_64+0x69/0x90
[  206.471216] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.471218] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.471219] R10: 00000000008b6738 R11: 0000000000000293 R12: 
00000000008b6738
[  206.471220] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.471224]  </TASK>
[  206.471225] ---[ end trace 0000000000000000 ]---
[  206.471227] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.471231] ------------[ cut here ]------------
[  206.471232] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.471234] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.471293] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.471296] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.471299] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.471314]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.471367]  ? do_syscall_64+0x69/0x90
[  206.471376] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.471379] R10: 00000000008b6738 R11: 0000000000000293 R12: 
00000000008b6738
[  206.471380] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.471383]  </TASK>
[  206.471384] ---[ end trace 0000000000000000 ]---
[  206.471385] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.471396] ------------[ cut here ]------------
[  206.471397] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.471399] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.471464] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.471500]  vfio_pin_pages+0x5e/0xb0
[  206.471519]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.471568]  vfio_device_fops_write+0x1f/0x40
[  206.471594] RIP: 0033:0x7fcbbebcacbf
[  206.471600] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.471605]  </TASK>
[  206.471606] ---[ end trace 0000000000000000 ]---
[  206.471606] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.471611] ------------[ cut here ]------------
[  206.471614] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.471671] RSP: 0018:ffffc0b1c301fb60 EFLAGS: 00010246
[  206.471673] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.471677] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.471681] Call Trace:
[  206.471684]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.471730]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.471754]  vfs_write+0xc6/0x3b0
[  206.471762]  ? __fget_light+0xa7/0x130
[  206.471781] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.471791] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.471801] ------------[ cut here ]------------
[  206.471804] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.471867] R13: 000000010d0f6000 R14: ffffc0b1c301fb70 R15: 
ffff9d7a8d400000
[  206.471903]  vfio_pin_pages+0x5e/0xb0
[  206.471921]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.471956]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.471978]  __x64_sys_pwrite64+0x96/0xc0
[  206.471988]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.472004] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.472019] ------------[ cut here ]------------
[  206.472022] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.472080] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.472081] RBP: ffffc0b1c301fb18 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.472085] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.472091]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.472116]  ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
[  206.472152]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.472166]  __x64_sys_pwrite64+0x96/0xc0
[  206.472173]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.472179]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.472194] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.472213] ------------[ cut here ]------------
[  206.472256]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.472270] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.472272] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.472273] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.472275] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.472279] RBP: ffffc0b1c301fb68 R08: 0000000000000001 R09: 
ffffc0b1c301fbc8
[  206.472285] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.472315]  vfio_pin_pages+0x5e/0xb0
[  206.472333]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.472356]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.472370]  __x64_sys_pwrite64+0x96/0xc0
[  206.472379]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.472392] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.472405] ------------[ cut here ]------------
[  206.472406] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.472408] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.472468] RBP: ffffc0b1c301fb70 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.472472] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.472475] Call Trace:
[  206.472478]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.472513]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.472531]  ? __fget_light+0xa7/0x130
[  206.472537]  ? fire_user_return_notifiers+0x42/0x70
[  206.472554] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.472564] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.472575] ------------[ cut here ]------------
[  206.472576] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.472616]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.472630] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.472632] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.472632] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.472634] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.472642] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.472648]  ? kvm_pmu_trigger_event+0x59/0x250 [kvm]
[  206.472696]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.472738]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.472749]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.472788]  ? do_syscall_64+0x69/0x90
[  206.472797] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.472804]  </TASK>
[  206.472804] ---[ end trace 0000000000000000 ]---
[  206.472805] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.472809] ------------[ cut here ]------------
[  206.472810] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.472812] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.472875] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.472882]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.472907]  ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
[  206.472983]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.473001] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.473002] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.473003] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.473004] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.473005] R10: 00000000008b6758 R11: 0000000000000293 R12: 
00000000008b6758
[  206.473006] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.473009]  </TASK>
[  206.473010] ---[ end trace 0000000000000000 ]---
[  206.473011] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.473021] ------------[ cut here ]------------
[  206.473022] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.473024] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.473087] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.473091] Call Trace:
[  206.473118]  ? kvm_arch_vcpu_ioctl_run+0x3c4/0x1850 [kvm]
[  206.473171]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.473191] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.473196]  </TASK>
[  206.473197] ---[ end trace 0000000000000000 ]---
[  206.473198] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.473202] ------------[ cut here ]------------
[  206.473203] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.473205] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.473265] RSP: 0018:ffffc0b1c301fbc0 EFLAGS: 00010246
[  206.473269] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.473273] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.473278]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.473344]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.473365]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.473382] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.473401] ------------[ cut here ]------------
[  206.473402] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.473405] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.473442]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.473461] RSP: 0018:ffffc0b1c301fb20 EFLAGS: 00010283
[  206.473463] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.473468] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.473476]  vfio_pin_pages+0x5e/0xb0
[  206.473494]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.473536]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.473551]  vfs_write+0xc6/0x3b0
[  206.473583]  __x64_sys_pwrite64+0x96/0xc0
[  206.473592]  ? do_syscall_64+0x69/0x90
[  206.473603] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.473616] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.473653]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.473674] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.473675] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.473680] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.473686]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.473711]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.473762]  vfs_write+0xc6/0x3b0
[  206.473793]  __x64_sys_pwrite64+0x96/0xc0
[  206.473800]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.473812] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.473833] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.473870]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.473884] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.473886] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.473887] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.473888] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.473894] R10: 00000000000b6774 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.473900] Call Trace:
[  206.473928]  vfio_pin_pages+0x5e/0xb0
[  206.473947]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.473965]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.473982]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.474011]  __x64_sys_pwrite64+0x96/0xc0
[  206.474022] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.474031] ---[ end trace 0000000000000000 ]---
[  206.474075]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.474089] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.474091] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.474092] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.474093] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.474095] RSP: 0018:ffffc0b1c301fbc0 EFLAGS: 00010246
[  206.474096] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.474097] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.474098] RBP: ffffc0b1c301fc00 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.474099] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.474100] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.474103] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.474104] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.474120]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.474152]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.474167]  vfs_write+0xc6/0x3b0
[  206.474196]  ? __fget_light+0xa7/0x130
[  206.474205]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.474209] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.474212] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.474281]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.474295] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.474296] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.474297] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.474299] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.474306] R13: 000000010d0f6000 R14: ffffc0b1c301fbd0 R15: 
ffff9d7a8d400000
[  206.474323]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.474349]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.474386]  ? intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.474400]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.474406]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.474413] RIP: 0033:0x7fcbbebcacbf
[  206.474416] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.474421] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.474470]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.474485] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.474486] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.474487] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.474489] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.474493] RBP: ffffc0b1c301fb78 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.474498] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.474499] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.474522]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.474554]  vfio_device_fops_write+0x1f/0x40
[  206.474577]  ? intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.474590]  do_syscall_64+0x59/0x90
[  206.474597]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.474604] RIP: 0033:0x7fcbbebcacbf
[  206.474607] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.474611] R10: 00000000008b6778 R11: 0000000000000293 R12: 
00000000008b6778
[  206.474668]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.474682] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.474683] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.474684] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.474686] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.474687] RSP: 0018:ffffc0b1c301fb20 EFLAGS: 00010283
[  206.474689] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.474690] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.474691] RBP: ffffc0b1c301fb88 R08: 0000000000000001 R09: 
ffffc0b1c301fbe8
[  206.474696] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.474700]  ? writeback_registers+0x56/0xb0 [kvm]
[  206.474774]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.474824]  vfs_write+0xc6/0x3b0
[  206.474834]  ? __fget_light+0xa7/0x130
[  206.474841]  ? fire_user_return_notifiers+0x42/0x70
[  206.474849]  ? do_syscall_64+0x69/0x90
[  206.474854] RIP: 0033:0x7fcbbebcacbf
[  206.474856] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.474864]  </TASK>
[  206.474873] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.474925] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.474935] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.474936] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.474939] Call Trace:
[  206.474943]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.474989]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.475012]  vfs_write+0xc6/0x3b0
[  206.475021]  ? __fget_light+0xa7/0x130
[  206.475028]  ? fire_user_return_notifiers+0x42/0x70
[  206.475036]  ? do_syscall_64+0x69/0x90
[  206.475042] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.475104]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.475118] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.475120] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.475121] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.475123] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.475124] RSP: 0018:ffffc0b1c301fb90 EFLAGS: 00010283
[  206.475125] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.475126] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.475127] RBP: ffffc0b1c301fbf8 R08: 0000000000000001 R09: 
ffffc0b1c301fc58
[  206.475128] R10: 00000000000b678c R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.475129] R13: 000000010d0f6000 R14: ffffc0b1c301fc58 R15: 
ffff9d7a8d400000
[  206.475134] Call Trace:
[  206.475158]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.475182]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.475204]  vfs_write+0xc6/0x3b0
[  206.475215]  do_syscall_64+0x59/0x90
[  206.475220] RIP: 0033:0x7fcbbebcacbf
[  206.475224] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.475239] ------------[ cut here ]------------
[  206.475279]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.475294] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.475295] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.475296] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.475298] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.475299] RSP: 0018:ffffc0b1c301fbc0 EFLAGS: 00010246
[  206.475300] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.475301] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.475302] RBP: ffffc0b1c301fc00 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.475304] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.475307] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.475323]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.475354]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.475368]  ? __fget_light+0xa7/0x130
[  206.475376]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.475381] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.475386] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.475442]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.475456] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.475458] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.475458] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.475460] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.475467] R13: 000000010d0f6000 R14: ffffc0b1c301fbb0 R15: 
ffff9d7a8d400000
[  206.475474]  vfio_pin_pages+0x5e/0xb0
[  206.475533]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.475549]  vfs_write+0xc6/0x3b0
[  206.475581]  do_syscall_64+0x59/0x90
[  206.475587]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.475594] RIP: 0033:0x7fcbbebcacbf
[  206.475597] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.475599] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.475650]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.475664] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.475665] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.475666] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.475668] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.475669] RSP: 0018:ffffc0b1c301fb18 EFLAGS: 00010246
[  206.475672] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.475676] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.475679] Call Trace:
[  206.475693]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.475743]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.475762]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.475795]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.475802]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.475807] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.475827] ------------[ cut here ]------------
[  206.475867]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.475881] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.475883] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.475884] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.475885] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.475887] RSP: 0018:ffffc0b1c301fad0 EFLAGS: 00010283
[  206.475888] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.475889] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.475890] RBP: ffffc0b1c301fb38 R08: 0000000000000001 R09: 
ffffc0b1c301fb98
[  206.475894] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.475918]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.475942]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.475957]  vfs_write+0xc6/0x3b0
[  206.476003]  ? __fget_light+0xa7/0x130
[  206.476012]  ? do_syscall_64+0x69/0x90
[  206.476020]  ? do_syscall_64+0x69/0x90
[  206.476026] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476034]  </TASK>
[  206.476043] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.476079]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476103] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.476106] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.476131]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.476162]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.476193]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.476233]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.476242] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476303]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476317] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.476319] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.476320] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.476321] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.476323] RSP: 0018:ffffc0b1c301fb90 EFLAGS: 00010283
[  206.476324] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.476325] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.476326] RBP: ffffc0b1c301fbf8 R08: 0000000000000001 R09: 
ffffc0b1c301fc58
[  206.476327] R10: 00000000000b67a4 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.476333] Call Trace:
[  206.476346]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.476376]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.476389]  ? do_syscall_64+0x69/0x90
[  206.476395]  __x64_sys_pwrite64+0x96/0xc0
[  206.476403] RIP: 0033:0x7fcbbebcacbf
[  206.476405] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476409] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.476458]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476472] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.476474] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.476475] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.476476] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.476478] RSP: 0018:ffffc0b1c301fbc0 EFLAGS: 00010246
[  206.476479] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.476480] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.476481] RBP: ffffc0b1c301fc00 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.476484] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.476490]  vfio_unpin_pages+0x49/0x60
[  206.476491]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.476533]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.476547]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.476555]  ? do_syscall_64+0x69/0x90
[  206.476561] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476563] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.476566] R10: 00000000008b67a0 R11: 0000000000000293 R12: 
00000000008b67a0
[  206.476582] ------------[ cut here ]------------
[  206.476621]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476635] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.476636] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.476637] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.476639] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.476640] RSP: 0018:ffffc0b1c301faf0 EFLAGS: 00010283
[  206.476645] R13: 000000010d0f6000 R14: ffffc0b1c301fbb8 R15: 
ffff9d7a8d400000
[  206.476670]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.476708]  vfs_write+0xc6/0x3b0
[  206.476723]  ? do_syscall_64+0x69/0x90
[  206.476730]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.476734]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.476739] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476742] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.476748] ---[ end trace 0000000000000000 ]---
[  206.476792]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476806] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.476807] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.476808] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.476810] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.476811] RSP: 0018:ffffc0b1c301fb20 EFLAGS: 00010246
[  206.476814] RBP: ffffc0b1c301fb60 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.476825]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.476835]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.476850]  ? vmx_vcpu_put+0x122/0x210 [kvm_intel]
[  206.476890]  ? __fget_light+0xa7/0x130
[  206.476900]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.476907]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.476912] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.476913] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.476916] R10: 00000000008b67a8 R11: 0000000000000293 R12: 
00000000008b67a8
[  206.476971]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.476985] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.476986] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.476987] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.476989] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.476990] RSP: 0018:ffffc0b1c301fac0 EFLAGS: 00010283
[  206.476992] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.476993] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.476994] RBP: ffffc0b1c301fb28 R08: 0000000000000001 R09: 
ffffc0b1c301fb88
[  206.476995] R10: 00000000000b67b4 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.476996] R13: 000000010d0f6000 R14: ffffc0b1c301fb88 R15: 
ffff9d7a8d400000
[  206.476997] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.476998] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.476999] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.477000] Call Trace:
[  206.477048]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.477078]  vfio_device_fops_write+0x1f/0x40
[  206.477117]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.477126]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.477131] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.477133] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.477137] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.477148] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.477199] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.477200] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.477201] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.477204] RSP: 0018:ffffc0b1c301faf0 EFLAGS: 00010246
[  206.477207] RBP: ffffc0b1c301fb30 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.477247]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.477255]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.477276]  __x64_sys_pwrite64+0x96/0xc0
[  206.477313]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.477322]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.477327] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.477330] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.477337] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.477389]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.477403] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.477404] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.477405] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.477408] RSP: 0018:ffffc0b1c301fb10 EFLAGS: 00010283
[  206.477410] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.477415] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.477471]  vfs_write+0xc6/0x3b0
[  206.477505]  ? fire_user_return_notifiers+0x42/0x70
[  206.477518] RIP: 0033:0x7fcbbebcacbf
[  206.477522] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.477525] R10: 00000000008b67b8 R11: 0000000000000293 R12: 
00000000008b67b8
[  206.477531] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.477574]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.477589] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.477590] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.477592] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.477593] RSP: 0018:ffffc0b1c301fb40 EFLAGS: 00010246
[  206.477595] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.477598] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.477603] Call Trace:
[  206.477607]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.477697]  ? do_syscall_64+0x69/0x90
[  206.477701]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.477704] RIP: 0033:0x7fcbbebcacbf
[  206.477708] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.477711] R10: 00000000008b67b8 R11: 0000000000000293 R12: 
00000000008b67b8
[  206.477716] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.477730] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.477786] RSP: 0018:ffffc0b1c301fae0 EFLAGS: 00010283
[  206.477787] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.477788] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.477789] RBP: ffffc0b1c301fb48 R08: 0000000000000001 R09: 
ffffc0b1c301fba8
[  206.477790] R10: 00000000000b67c4 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.477791] R13: 000000010d0f6000 R14: ffffc0b1c301fba8 R15: 
ffff9d7a8d400000
[  206.477792] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.477794] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.477795] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.477824]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.477873]  vfio_device_fops_write+0x1f/0x40
[  206.477926] RIP: 0033:0x7fcbbebcacbf
[  206.477929] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.477937]  </TASK>
[  206.477943] ------------[ cut here ]------------
[  206.477946] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.478007] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.478008] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.478010] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.478012] Call Trace:
[  206.478013]  <TASK>
[  206.478026]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.478076]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.478094]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.478152] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.478154] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.478155] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.478162] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.478183] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.478250] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.478251] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.478252] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.478254] Call Trace:
[  206.478254]  <TASK>
[  206.478256]  vfio_pin_pages+0x5e/0xb0
[  206.478275]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.478374] RIP: 0033:0x7fcbbebcacbf
[  206.478379] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.478391] ------------[ cut here ]------------
[  206.478430]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.478443] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.478449] RSP: 0018:ffffc0b1c301fb68 EFLAGS: 00010246
[  206.478453] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.478454] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.478455] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.478456] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.478457] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.478461]  vfio_unpin_pages+0x49/0x60
[  206.478462]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.478473]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.478488]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.478539]  vfs_write+0xc6/0x3b0
[  206.478584] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.478587] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.478592]  </TASK>
[  206.478608] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.478674] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.478675] Call Trace:
[  206.478676]  <TASK>
[  206.478678]  vfio_pin_pages+0x5e/0xb0
[  206.478679]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.478696]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.478713]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.478751]  ? vfs_write+0xc6/0x3b0
[  206.478797] RIP: 0033:0x7fcbbebcacbf
[  206.478802] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.478808]  </TASK>
[  206.478814] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.478853]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.478871] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.478878] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.478879] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.478880] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.478881] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.478882] Call Trace:
[  206.478883]  <TASK>
[  206.478885]  vfio_unpin_pages+0x49/0x60
[  206.478905]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.478930]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.478993]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.479009] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.479010] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.479011] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.479014] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.479032] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.479095] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.479096] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.479097] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.479099] Call Trace:
[  206.479132]  vfio_pin_pages+0x5e/0xb0
[  206.479142]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.479201]  vfs_write+0xc6/0x3b0
[  206.479229] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.479234] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.479246] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.479310] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.479312] Call Trace:
[  206.479312]  <TASK>
[  206.479314]  vfio_unpin_pages+0x49/0x60
[  206.479325]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.479340]  ? vcpu_put+0x22/0x50 [kvm]
[  206.479384]  vfs_write+0xc6/0x3b0
[  206.479396]  do_syscall_64+0x59/0x90
[  206.479409] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.479414] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.479430] ------------[ cut here ]------------
[  206.479470]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.479484] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.479490] RSP: 0018:ffffc0b1c301fb48 EFLAGS: 00010283
[  206.479498] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.479499] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.479500] Call Trace:
[  206.479501]  <TASK>
[  206.479503]  vfio_pin_pages+0x5e/0xb0
[  206.479504]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.479513]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.479520]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.479527]  ? kvm_arch_vcpu_ioctl_run+0x3c4/0x1850 [kvm]
[  206.479553]  ? vcpu_put+0x22/0x50 [kvm]
[  206.479573]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.479581]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.479588]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.479595]  vfio_device_fops_write+0x1f/0x40
[  206.479597]  vfs_write+0xc6/0x3b0
[  206.479604]  ? __fget_light+0xa7/0x130
[  206.479615]  ? do_syscall_64+0x69/0x90
[  206.479627] R10: 00000000008b67e0 R11: 0000000000000293 R12: 
00000000008b67e0
[  206.479638] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.479677]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.479691] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.479705] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.479707]  <TASK>
[  206.479710]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.479720]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.479728]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.479735]  ? kvm_arch_vcpu_ioctl_run+0x3c4/0x1850 [kvm]
[  206.479789]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.479796]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.479803]  vfio_device_fops_write+0x1f/0x40
[  206.479804]  vfs_write+0xc6/0x3b0
[  206.479811]  ? __fget_light+0xa7/0x130
[  206.479821]  ? do_syscall_64+0x69/0x90
[  206.479833] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.479840] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.479850] ------------[ cut here ]------------
[  206.479890]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.479904] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.479915] R13: 000000010d0f6000 R14: ffffc0b1c301fc60 R15: 
ffff9d7a8d400000
[  206.479921]  <TASK>
[  206.479924]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.479933]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.479948]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.479963]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.479970]  vfio_device_fops_write+0x1f/0x40
[  206.479971]  vfs_write+0xc6/0x3b0
[  206.479976]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.479978]  ? __fget_light+0xa7/0x130
[  206.479980]  __x64_sys_pwrite64+0x96/0xc0
[  206.479983]  do_syscall_64+0x59/0x90
[  206.479988] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480043]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.480057] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.480058] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.480062] RSP: 0018:ffffc0b1c301fbc8 EFLAGS: 00010246
[  206.480071] CR2: 000056165df16008 CR3: 000000011447c006 CR4: 
00000000003726e0
[  206.480130]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480132]  ? __fget_light+0xa7/0x130
[  206.480134]  __x64_sys_pwrite64+0x96/0xc0
[  206.480139]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.480142] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480144] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.480145] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.480146] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.480148] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.480152]  </TASK>
[  206.480165] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.480204]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.480224] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.480232] R13: 000000010d0f6000 R14: ffffc0b1c301fbf8 R15: 
ffff9d7a8d400000
[  206.480270]  vfio_pin_pages+0x5e/0xb0
[  206.480359]  do_syscall_64+0x59/0x90
[  206.480370] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480371] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.480373] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.480374] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.480375] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.480376] R10: 00000000008b67f0 R11: 0000000000000293 R12: 
00000000008b67f0
[  206.480377] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.480379]  </TASK>
[  206.480388] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.480552]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480555]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.480559] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480560] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.480562] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.480563] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.480564] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.480565] R10: 00000000008b67f0 R11: 0000000000000293 R12: 
00000000008b67f0
[  206.480585] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.480731]  vfs_write+0xc6/0x3b0
[  206.480764]  ? __fget_light+0xa7/0x130
[  206.480766]  __x64_sys_pwrite64+0x96/0xc0
[  206.480769]  do_syscall_64+0x59/0x90
[  206.480771]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.480773]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480775]  ? do_syscall_64+0x69/0x90
[  206.480777]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.480779]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480781]  ? do_syscall_64+0x69/0x90
[  206.480783]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.480785] RIP: 0033:0x7fcbbebcacbf
[  206.480786] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480788] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.480789] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.480790] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.480797] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.480841]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.480855] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.480857] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.480860] RSP: 0018:ffffc0b1c301faf8 EFLAGS: 00010246
[  206.480868] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.480910]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.480936]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.480960]  ? __fget_light+0xa7/0x130
[  206.480965]  do_syscall_64+0x59/0x90
[  206.480967]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.480969]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480971]  ? do_syscall_64+0x69/0x90
[  206.480973]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.480975]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.480977]  ? do_syscall_64+0x69/0x90
[  206.480979]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.480981] RIP: 0033:0x7fcbbebcacbf
[  206.480982] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.480992]  </TASK>
[  206.481045]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.481059] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.481060] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.481061] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.481064] RSP: 0018:ffffc0b1c301fac8 EFLAGS: 00010283
[  206.481065] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.481104]  vfio_pin_pages+0x5e/0xb0
[  206.481145]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.481153]  vfio_device_fops_write+0x1f/0x40
[  206.481154]  vfs_write+0xc6/0x3b0
[  206.481157]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.481176]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.481200]  ? __fget_light+0xa7/0x130
[  206.481202]  __x64_sys_pwrite64+0x96/0xc0
[  206.481205]  do_syscall_64+0x59/0x90
[  206.481207]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481209]  ? do_syscall_64+0x69/0x90
[  206.481210]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481212]  ? do_syscall_64+0x69/0x90
[  206.481214]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.481218]  ? do_syscall_64+0x69/0x90
[  206.481220]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.481223] RIP: 0033:0x7fcbbebcacbf
[  206.481225] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.481230] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.481281]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.481299] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.481301] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.481305] RSP: 0018:ffffc0b1c301faf8 EFLAGS: 00010246
[  206.481311] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.481368]  vfio_device_fops_write+0x1f/0x40
[  206.481369]  vfs_write+0xc6/0x3b0
[  206.481392]  ? kvm_on_user_return+0x84/0xe0 [kvm]
[  206.481415]  ? __fget_light+0xa7/0x130
[  206.481418]  __x64_sys_pwrite64+0x96/0xc0
[  206.481420]  do_syscall_64+0x59/0x90
[  206.481422]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481424]  ? do_syscall_64+0x69/0x90
[  206.481425]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481427]  ? do_syscall_64+0x69/0x90
[  206.481429]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.481432]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481434]  ? do_syscall_64+0x69/0x90
[  206.481435]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.481437] RIP: 0033:0x7fcbbebcacbf
[  206.481438] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.481440] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.481441] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.481442] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.481448]  </TASK>
[  206.481502]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.481515] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.481517] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.481518] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.481519] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.481521] RSP: 0018:ffffc0b1c301fb58 EFLAGS: 00010283
[  206.481522] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.481529] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.481565]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.481580]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.481587]  vfio_device_fops_write+0x1f/0x40
[  206.481589]  vfs_write+0xc6/0x3b0
[  206.481591]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.481594]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481596]  ? __fget_light+0xa7/0x130
[  206.481598]  __x64_sys_pwrite64+0x96/0xc0
[  206.481601]  do_syscall_64+0x59/0x90
[  206.481603]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.481604]  ? do_syscall_64+0x69/0x90
[  206.481606]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.481608] RIP: 0033:0x7fcbbebcacbf
[  206.481609] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.481611] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.481612] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.481613] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.481614] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.481616] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.481664]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.481678] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.481683] RSP: 0018:ffffc0b1c301fb88 EFLAGS: 00010246
[  206.481684] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.481687] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.481729]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.481768]  ? do_syscall_64+0x69/0x90
[  206.481769]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.481771] RIP: 0033:0x7fcbbebcacbf
[  206.481773] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.481774] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.481775] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.481777] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.481778] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.481779] R10: 00000000008b6808 R11: 0000000000000293 R12: 
00000000008b6808
[  206.481780] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.481782]  </TASK>
[  206.481783] ---[ end trace 0000000000000000 ]---
[  206.481797] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.481946]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.481992]  ? do_syscall_64+0x69/0x90
[  206.481996]  ? do_syscall_64+0x69/0x90
[  206.481997]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.481999] RIP: 0033:0x7fcbbebcacbf
[  206.482001] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.482002] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.482004] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.482005] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.482006] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.482007] R10: 00000000008b6810 R11: 0000000000000293 R12: 
00000000008b6810
[  206.482008] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.482010]  </TASK>
[  206.482011] ---[ end trace 0000000000000000 ]---
[  206.482012] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.482016] ------------[ cut here ]------------
[  206.482017] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.482055]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.482073] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.482079] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.482084] Call Trace:
[  206.482150]  intel_vgpu_rw+0xa6/0x1e0 [kvmgt]
[  206.482194]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482198]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482199]  ? do_syscall_64+0x69/0x90
[  206.482201]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.482203] RIP: 0033:0x7fcbbebcacbf
[  206.482205] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.482206] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.482207] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.482209] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.482210] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.482211] R10: 00000000008b6810 R11: 0000000000000293 R12: 
00000000008b6810
[  206.482212] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.482278]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.482292] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.482294] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.482295] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.482298] RSP: 0018:ffffc0b1c301faa8 EFLAGS: 00010283
[  206.482299] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.482304] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.482388]  vfs_write+0xc6/0x3b0
[  206.482422]  ? do_syscall_64+0x69/0x90
[  206.482423]  ? fire_user_return_notifiers+0x42/0x70
[  206.482425]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.482428]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482430]  ? do_syscall_64+0x69/0x90
[  206.482431]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482433]  ? do_syscall_64+0x69/0x90
[  206.482434]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.482437] RIP: 0033:0x7fcbbebcacbf
[  206.482438] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.482439] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.482441] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.482444] R10: 00000000008b6818 R11: 0000000000000293 R12: 
00000000008b6818
[  206.482493]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.482507] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.482510] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.482513] RSP: 0018:ffffc0b1c301fad8 EFLAGS: 00010246
[  206.482515] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.482568]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.482607]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.482609]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482612]  ? fire_user_return_notifiers+0x42/0x70
[  206.482615]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.482617]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482619]  ? do_syscall_64+0x69/0x90
[  206.482621]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482623]  ? do_syscall_64+0x69/0x90
[  206.482624]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.482626] RIP: 0033:0x7fcbbebcacbf
[  206.482628] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.482690]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.482704] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.482705] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.482706] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.482709] RSP: 0018:ffffc0b1c301fb08 EFLAGS: 00010283
[  206.482713] R10: 00000000000b6824 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.482774]  vfs_write+0xc6/0x3b0
[  206.482776]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.482796]  ? exit_to_user_mode_prepare+0x35/0x170
[  206.482798]  ? __fget_light+0xa7/0x130
[  206.482801]  __x64_sys_pwrite64+0x96/0xc0
[  206.482804]  do_syscall_64+0x59/0x90
[  206.482805]  ? do_syscall_64+0x69/0x90
[  206.482807]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482809]  ? do_syscall_64+0x69/0x90
[  206.482812]  ? do_syscall_64+0x69/0x90
[  206.482813]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.482816] RIP: 0033:0x7fcbbebcacbf
[  206.482818] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.482820] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.482823] R10: 00000000008b6820 R11: 0000000000000293 R12: 
00000000008b6820
[  206.482872]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.482886] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.482890] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.482931]  ? restore_fpregs_from_fpstate+0x47/0xc0
[  206.482985]  __x64_sys_pwrite64+0x96/0xc0
[  206.482987]  do_syscall_64+0x59/0x90
[  206.482989]  ? do_syscall_64+0x69/0x90
[  206.482991]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.482993]  ? do_syscall_64+0x69/0x90
[  206.482996]  ? do_syscall_64+0x69/0x90
[  206.482997]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.483000] RIP: 0033:0x7fcbbebcacbf
[  206.483001] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.483007] R10: 00000000008b6820 R11: 0000000000000293 R12: 
00000000008b6820
[  206.483062]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.483076] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.483077] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.483081] RSP: 0018:ffffc0b1c301fb28 EFLAGS: 00010283
[  206.483085] RBP: ffffc0b1c301fb90 R08: 0000000000000001 R09: 
ffffc0b1c301fbf0
[  206.483166]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.483198]  __x64_sys_pwrite64+0x96/0xc0
[  206.483200]  do_syscall_64+0x59/0x90
[  206.483202]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.483204]  ? do_syscall_64+0x69/0x90
[  206.483205]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.483207]  ? do_syscall_64+0x69/0x90
[  206.483209]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.483211] RIP: 0033:0x7fcbbebcacbf
[  206.483212] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.483214] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.483215] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.483216] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.483217] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.483223]  </TASK>
[  206.483230] ------------[ cut here ]------------
[  206.483270]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.483287] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.483290] RSP: 0018:ffffc0b1c301fb58 EFLAGS: 00010246
[  206.483294] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.483322]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.483414]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.483419] RIP: 0033:0x7fcbbebcacbf
[  206.483421] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.483422] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.483423] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.483425] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.483426] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.483427] R10: 00000000008b6828 R11: 0000000000000293 R12: 
00000000008b6828
[  206.483428] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.483444] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.483484]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.483502] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.483510] FS:  00007fcbb9200700(0000) GS:ffff9d81dac40000(0000) 
knlGS:0000000000000000
[  206.483559]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.483589]  ? do_syscall_64+0x69/0x90
[  206.483591]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.483593] RIP: 0033:0x7fcbbebcacbf
[  206.483594] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.483596] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.483597] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.483598] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.483599] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.483600] R10: 00000000008b6830 R11: 0000000000000293 R12: 
00000000008b6830
[  206.483601] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.483604]  </TASK>
[  206.483605] ---[ end trace 0000000000000000 ]---
[  206.483610] ------------[ cut here ]------------
[  206.483650]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.483667] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.483674] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.483679] Call Trace:
[  206.483701]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.483752]  ? do_syscall_64+0x69/0x90
[  206.483762] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.483764] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.483765] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.483766] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.483767] R10: 00000000008b6830 R11: 0000000000000293 R12: 
00000000008b6830
[  206.483768] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.483770]  </TASK>
[  206.483771] ---[ end trace 0000000000000000 ]---
[  206.483772] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.483782] ------------[ cut here ]------------
[  206.483783] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.483785] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.483878]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.483970] RIP: 0033:0x7fcbbebcacbf
[  206.483974] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.483975] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.483977] R10: 00000000008b6838 R11: 0000000000000293 R12: 
00000000008b6838
[  206.483978] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.483981]  </TASK>
[  206.483982] ---[ end trace 0000000000000000 ]---
[  206.483983] gvt: vgpu 2: vfio_pin_pages failed for iova 
0x000000010d0f6000, ret -22
[  206.483987] ------------[ cut here ]------------
[  206.483988] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:524 iommufd_access_unpin_pages+0x117/0x130
[  206.483990] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.484085]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.484162]  ? fire_user_return_notifiers+0x42/0x70
[  206.484181] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.484189] gvt: vgpu 2: fail to populate guest ggtt entry
[  206.484200] ------------[ cut here ]------------
[  206.484200] WARNING: CPU: 9 PID: 3362 at 
drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
[  206.484243]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.484257] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.484259] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.484259] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.484261] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.484262] RSP: 0018:ffffc0b1c301fb30 EFLAGS: 00010283
[  206.484265] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.484267] R10: 00000000000b6844 R11: ffff9d7a97ce0000 R12: 
0000000000000001
[  206.484277]  gvt_dma_map_page+0xb7/0x130 [kvmgt]
[  206.484301]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.484327]  ? __fget_light+0xa7/0x130
[  206.484338]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.484345] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.484401]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.484414] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.484416] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.484417] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.484418] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.484425] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.484443]  intel_gvt_dma_map_guest_page+0x78/0x240 [kvmgt]
[  206.484482]  vfio_device_fops_write+0x1f/0x40
[  206.484488]  __x64_sys_pwrite64+0x96/0xc0
[  206.484498]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.484504] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.484567]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.484582] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.484583] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.484584] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.484585] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.484587] RSP: 0018:ffffc0b1c301fb60 EFLAGS: 00010283
[  206.484595] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  206.484597] Call Trace:
[  206.484598]  <TASK>
[  206.484653]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.484679]  ? kvm_vcpu_ioctl+0x120/0x6f0 [kvm]
[  206.484708]  ? do_syscall_64+0x69/0x90
[  206.484712] Code: 08 89 3c 24 48 89 4c 24 18 e8 3d f3 ff ff 4c 8b 54 24 
18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 12 00 00 00 0f 05 
<48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 6d f3 ff ff 48 8b
[  206.484722]  </TASK>
[  206.484768]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.484782] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.484783] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.484784] RIP: 0010:iommufd_access_unpin_pages+0x117/0x130
[  206.484786] Code: 48 39 45 c8 77 20 48 8b 45 d8 65 48 2b 04 25 28 00 00 
00 75 19 48 83 c4 20 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 0f 0b eb dc 
<0f> 0b eb d8 e8 70 1c 5a 00 0f 0b eb cf 66 66 2e 0f 1f 84 00 00 00
[  206.484787] RSP: 0018:ffffc0b1c301fb90 EFLAGS: 00010246
[  206.484788] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
0000000000000027
[  206.484789] RDX: 0000000000000000 RSI: 000000010d0f6000 RDI: 
ffff9d7a8477d780
[  206.484790] RBP: ffffc0b1c301fbd0 R08: 0000000000000000 R09: 
705f6e69705f6f69
[  206.484791] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.484792] R13: 000000010d0f6000 R14: 0000000000000000 R15: 
ffff9d7a8d400000
[  206.484800]  gvt_dma_map_page.cold+0x75/0x14e [kvmgt]
[  206.484842]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.484883]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.484888] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.484891] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.484896]  </TASK>
[  206.484908] ------------[ cut here ]------------
[  206.484949]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.484963] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.484965] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.484966] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.484967] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.484969] RSP: 0018:ffffc0b1c301fac0 EFLAGS: 00010283
[  206.484970] RAX: ffff9d7a8d400000 RBX: 0000000000000000 RCX: 
ffff9d7a84422800
[  206.484971] RDX: 0000000000001000 RSI: ffff9d7a84422838 RDI: 
ffff9d7a8477d780
[  206.485007]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.485021]  intel_vgpu_write+0x1ab/0x240 [kvmgt]
[  206.485038]  do_syscall_64+0x59/0x90
[  206.485053]  ? do_syscall_64+0x69/0x90
[  206.485059] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.485061] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.485064] R10: 00000000008b6850 R11: 0000000000000293 R12: 
00000000008b6850
[  206.485076] Modules linked in: nfnetlink bridge stp llc bnep 
snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic 
ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi 
snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_msr 
intel_rapl_common snd_seq_midi rtw88_8822be rtw88_8822b snd_seq_midi_event 
intel_tcc_cooling rtw88_pci x86_pkg_temp_thermal intel_powerclamp 
snd_rawmidi btusb rtw88_core btrtl coretemp btbcm snd_seq btintel 
snd_seq_device btmtk mac80211 mei_hdcp mei_pxp kvm_intel bluetooth 
snd_timer rapl ecdh_generic cfg80211 input_leds intel_cstate snd ecc 
serio_raw mei_me libarc4 ee1004 soundcore mei mac_hid acpi_pad sch_fq_codel 
kvmgt mdev kvm parport_pc nfsd ppdev lp auth_rpcgss parport nfs_acl lockd 
grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx 
xor raid6_pq libcrc32c raid1 raid0 multipath linear dm_mirror 
dm_region_hash dm_log hid_generic i915 drm_buddy
[  206.485136] R10: 000000006776203a R11: 000000003a747667 R12: 
ffff9d7a84422800
[  206.485172]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.485198]  ? __fget_light+0xa7/0x130
[  206.485211]  ? do_syscall_64+0x69/0x90
[  206.485216]  ? syscall_exit_to_user_mode+0x26/0x50
[  206.485223] RIP: 0033:0x7fcbbebcacbf
[  206.485226] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.485229] RDX: 0000000000000008 RSI: 00007fcbb91ff4d8 RDI: 
0000000000000017
[  206.485232] R13: 000000010d0f6001 R14: 000055f21fd146f0 R15: 
000055f21fd14600
[  206.485292]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas 
video wmi
[  206.485307] CPU: 9 PID: 3362 Comm: qemu-system-x86 Tainted: G        W 
         6.1.0-rc2-iommufd-d2b2119+ #1
[  206.485308] Hardware name: Supermicro C7Z370-CG-IW/C7Z370-CG-IW, BIOS 
1.1 02/08/2018
[  206.485309] RIP: 0010:iommufd_access_pin_pages+0x337/0x360
[  206.485311] Code: 7f 68 89 45 b4 48 83 c7 08 e8 05 de 5a 00 48 8b 75 b8 
44 8b 45 b4 e9 36 fe ff ff 48 8b 75 b8 41 b8 fe ff ff ff e9 27 fe ff ff 
<0f> 0b 41 b8 ea ff ff ff e9 35 fe ff ff 41 b8 ea ff ff ff e9 2a fe
[  206.485312] RSP: 0018:ffffc0b1c301fb70 EFLAGS: 00010283
[  206.485322] Call Trace:
[  206.485342]  intel_vgpu_emulate_ggtt_mmio_write+0x47e/0x550 [kvmgt]
[  206.485349]  ? kvm_arch_vcpu_put+0x1a9/0x200 [kvm]
[  206.485376]  intel_vgpu_emulate_mmio_write+0x2eb/0x360 [kvmgt]
[  206.485426]  ? __fget_light+0xa7/0x130
[  206.485436]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  206.485441] RSP: 002b:00007fcbb91ff490 EFLAGS: 00000293 ORIG_RAX: 
0000000000000012
[  206.485442] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 
00007fcbbebcacbf
[  206.485444] RBP: 0000000000000008 R08: 0000000000000000 R09: 
ffffffffffffffff
[  206.485449] ---[ end trace 0000000000000000 ]---
[  206.485494]  i2c_algo_bit ttm drm_display_helper cec rc_core 
drm_kms_helper crct10dif_pclmul syscopyarea usbhid crc32_pclmul 
polyval_clmulni sysfillrect hid polyval_generic ghash_clmulni_intel 
sysimgblt fb_sys_fops sha512_ssse3 aesni_intel crypto_simd mxm_wmi psmouse 
cryptd drm e1000e ahci i2c_i801 xhci_pci i2c_smbus libahci xhci_pci_renesas vi

-- 
Regards,
Yi Liu

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Nicolin Chen]

On Tue, Nov 01, 2022 at 11:04:38AM +0800, Yi Liu wrote:
> On 2022/11/1 07:24, Jason Gunthorpe wrote:
> > On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
> > > > There is something wrong with the test suite that it isn't covering
> > > > the above, I'm going to look into that today.
> > > 
> > > sounds to be the cause. I didn't see any significant change in vfio_main.c
> > > that may fail gvt. So should the iommufd changes. Then we will re-run the
> > > test after your update.:-)
> > 
> > I updated the github with all the changes made so far, it is worth
> > trying again!
> 
> gvt is still failing with below call trace in host side. vfio_unpin_pages()
> is still in problem. Any idea on it?

> [  206.464318] WARNING: CPU: 9 PID: 3362 at
> drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360

Judging from this WARNING, and since gvt (mdev) needs pin_pages(),
I assume this might be a fix, though Jason's latest change for the
iova_alignment seems to be added for CONFIG_IOMMUFD_TEST only.

------
diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
index 72a289c5f8c9..185075528d5e 100644
--- a/drivers/vfio/iommufd.c
+++ b/drivers/vfio/iommufd.c
@@ -120,6 +120,7 @@ static void vfio_emulated_unmap(void *data, unsigned long iova,
 }
 
 static const struct iommufd_access_ops vfio_user_ops = {
+	.needs_pin_pages = 1,
 	.unmap = vfio_emulated_unmap,
 };
 
------

Perhaps you can try it first to see if we can test the rest part of
the routine for now, till Jason acks tomorrow.

Thanks
Nic

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Nicolin Chen]

On Tue, Nov 01, 2022 at 11:04:38AM +0800, Yi Liu wrote:
> On 2022/11/1 07:24, Jason Gunthorpe wrote:
> > On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
> > > > There is something wrong with the test suite that it isn't covering
> > > > the above, I'm going to look into that today.
> > > 
> > > sounds to be the cause. I didn't see any significant change in vfio_main.c
> > > that may fail gvt. So should the iommufd changes. Then we will re-run the
> > > test after your update.:-)
> > 
> > I updated the github with all the changes made so far, it is worth
> > trying again!
> 
> gvt is still failing with below call trace in host side. vfio_unpin_pages()
> is still in problem. Any idea on it?

> [  206.464318] WARNING: CPU: 9 PID: 3362 at
> drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360

Judging from this WARNING, and since gvt (mdev) needs pin_pages(),
I assume this might be a fix, though Jason's latest change for the
iova_alignment seems to be added for CONFIG_IOMMUFD_TEST only.

------
diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
index 72a289c5f8c9..185075528d5e 100644
--- a/drivers/vfio/iommufd.c
+++ b/drivers/vfio/iommufd.c
@@ -120,6 +120,7 @@ static void vfio_emulated_unmap(void *data, unsigned long iova,
 }
 
 static const struct iommufd_access_ops vfio_user_ops = {
+	.needs_pin_pages = 1,
 	.unmap = vfio_emulated_unmap,
 };
 
------

Perhaps you can try it first to see if we can test the rest part of
the routine for now, till Jason acks tomorrow.

Thanks
Nic

Re: [Intel-gfx] [PATCH 00/10] Connect VFIO to IOMMUFD

[Nicolin Chen]

On Tue, Nov 01, 2022 at 11:04:38AM +0800, Yi Liu wrote:
> On 2022/11/1 07:24, Jason Gunthorpe wrote:
> > On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
> > > > There is something wrong with the test suite that it isn't covering
> > > > the above, I'm going to look into that today.
> > > 
> > > sounds to be the cause. I didn't see any significant change in vfio_main.c
> > > that may fail gvt. So should the iommufd changes. Then we will re-run the
> > > test after your update.:-)
> > 
> > I updated the github with all the changes made so far, it is worth
> > trying again!
> 
> gvt is still failing with below call trace in host side. vfio_unpin_pages()
> is still in problem. Any idea on it?

> [  206.464318] WARNING: CPU: 9 PID: 3362 at
> drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360

Judging from this WARNING, and since gvt (mdev) needs pin_pages(),
I assume this might be a fix, though Jason's latest change for the
iova_alignment seems to be added for CONFIG_IOMMUFD_TEST only.

------
diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
index 72a289c5f8c9..185075528d5e 100644
--- a/drivers/vfio/iommufd.c
+++ b/drivers/vfio/iommufd.c
@@ -120,6 +120,7 @@ static void vfio_emulated_unmap(void *data, unsigned long iova,
 }
 
 static const struct iommufd_access_ops vfio_user_ops = {
+	.needs_pin_pages = 1,
 	.unmap = vfio_emulated_unmap,
 };
 
------

Perhaps you can try it first to see if we can test the rest part of
the routine for now, till Jason acks tomorrow.

Thanks
Nic

RE: [PATCH 01/10] vfio: Move vfio_device driver open/close code to a function

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:17 AM
> 
> This error unwind is getting complicated. Move all the code into two
> pair'd function. The functions should be called when the open_count == 1
> after incrementing/before decrementing.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Reviewed-by: Kevin Tian <kevin.tian@intel.com>, with a nit

> +	/*
> +	 * Here we pass the KVM pointer with the group under the read lock.

Now the read lock is replaced by mutex. Let's correct it when moving this
piece of code.

RE: [PATCH 01/10] vfio: Move vfio_device driver open/close code to a function

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:17 AM
> 
> This error unwind is getting complicated. Move all the code into two
> pair'd function. The functions should be called when the open_count == 1
> after incrementing/before decrementing.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Reviewed-by: Kevin Tian <kevin.tian@intel.com>, with a nit

> +	/*
> +	 * Here we pass the KVM pointer with the group under the read lock.

Now the read lock is replaced by mutex. Let's correct it when moving this
piece of code.

Re: [Intel-gfx] [PATCH 01/10] vfio: Move vfio_device driver open/close code to a function

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:17 AM
> 
> This error unwind is getting complicated. Move all the code into two
> pair'd function. The functions should be called when the open_count == 1
> after incrementing/before decrementing.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Reviewed-by: Kevin Tian <kevin.tian@intel.com>, with a nit

> +	/*
> +	 * Here we pass the KVM pointer with the group under the read lock.

Now the read lock is replaced by mutex. Let's correct it when moving this
piece of code.

RE: [PATCH 02/10] vfio: Move vfio_device_assign_container() into vfio_device_first_open()

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:17 AM
> 
> +err_container:
> +	vfio_device_unassign_container(device);
>  err_module_put:
>  	device->kvm = NULL;

err_module_put should be moved after nullifying device->kvm.

otherwise it looks good to me:

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

RE: [PATCH 02/10] vfio: Move vfio_device_assign_container() into vfio_device_first_open()

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:17 AM
> 
> +err_container:
> +	vfio_device_unassign_container(device);
>  err_module_put:
>  	device->kvm = NULL;

err_module_put should be moved after nullifying device->kvm.

otherwise it looks good to me:

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

Re: [Intel-gfx] [PATCH 02/10] vfio: Move vfio_device_assign_container() into vfio_device_first_open()

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:17 AM
> 
> +err_container:
> +	vfio_device_unassign_container(device);
>  err_module_put:
>  	device->kvm = NULL;

err_module_put should be moved after nullifying device->kvm.

otherwise it looks good to me:

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

RE: [PATCH 03/10] vfio: Rename vfio_device_assign/unassign_container()

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:17 AM
> 
> These functions don't really assign anything anymore, they just increment
> some refcounts and do a sanity check. Call them
> vfio_group_[un]use_container()
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

RE: [PATCH 03/10] vfio: Rename vfio_device_assign/unassign_container()

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:17 AM
> 
> These functions don't really assign anything anymore, they just increment
> some refcounts and do a sanity check. Call them
> vfio_group_[un]use_container()
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

Re: [Intel-gfx] [PATCH 03/10] vfio: Rename vfio_device_assign/unassign_container()

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:17 AM
> 
> These functions don't really assign anything anymore, they just increment
> some refcounts and do a sanity check. Call them
> vfio_group_[un]use_container()
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

RE: [PATCH 05/10] vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:17 AM
> 
> iommufd doesn't establish the iommu_domains until after the device FD is
> opened, even if the container has been set. This design is part of moving
> away from the group centric iommu APIs.
> 
> This is fine, except that the normal sequence of establishing the kvm
> wbindv won't work:

wbindv -> wbinvd

> 
>    group = open("/dev/vfio/XX")
>    ioctl(group, VFIO_GROUP_SET_CONTAINER)
>    ioctl(kvm, KVM_DEV_VFIO_GROUP_ADD)
>    ioctl(group, VFIO_GROUP_GET_DEVICE_FD)
> 
> As the domains don't start existing until GET_DEVICE_FD. Further,
> GET_DEVICE_FD requires that KVM_DEV_VFIO_GROUP_ADD already be
> done as that
> is what sets the group->kvm and thus device->kvm for the driver to use
> during open.
> 
> Now that we have device centric cap ops and the new
> IOMMU_CAP_ENFORCE_CACHE_COHERENCY we know what the
> iommu_domain will be
> capable of without having to create it. Use this to compute

it's worth noting that the prerequisite is that vfio always enforces
cache coherency on a domain according to the iommu capability
of the devices attached to that domain. There is no mix of attaching
a device supporting the cap to a domain which doesn't enforce
coherency. With that we know what the domain will be w/o having
to create it.

> vfio_file_enforced_coherent() and resolve the ordering problems.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/vfio/container.c |  5 +++--
>  drivers/vfio/vfio.h      |  2 --
>  drivers/vfio/vfio_main.c | 27 ++++++++++++++-------------
>  3 files changed, 17 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
> index 499777930b08fa..d97747dfb05d02 100644
> --- a/drivers/vfio/container.c
> +++ b/drivers/vfio/container.c
> @@ -188,8 +188,9 @@ void vfio_device_container_unregister(struct
> vfio_device *device)
>  			device->group->container->iommu_data, device);
>  }
> 
> -long vfio_container_ioctl_check_extension(struct vfio_container *container,
> -					  unsigned long arg)
> +static long
> +vfio_container_ioctl_check_extension(struct vfio_container *container,
> +				     unsigned long arg)
>  {
>  	struct vfio_iommu_driver *driver;
>  	long ret = 0;
> diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
> index 54e5a8e0834ccb..247590334e14b0 100644
> --- a/drivers/vfio/vfio.h
> +++ b/drivers/vfio/vfio.h
> @@ -119,8 +119,6 @@ int vfio_container_attach_group(struct
> vfio_container *container,
>  void vfio_group_detach_container(struct vfio_group *group);
>  void vfio_device_container_register(struct vfio_device *device);
>  void vfio_device_container_unregister(struct vfio_device *device);
> -long vfio_container_ioctl_check_extension(struct vfio_container *container,
> -					  unsigned long arg);
>  int __init vfio_container_init(void);
>  void vfio_container_cleanup(void);
> 
> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index 1e414b2c48a511..a8d1fbfcc3ddad 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -1625,24 +1625,25 @@ EXPORT_SYMBOL_GPL(vfio_file_is_group);
>  bool vfio_file_enforced_coherent(struct file *file)
>  {
>  	struct vfio_group *group = file->private_data;
> -	bool ret;
> +	struct vfio_device *device;
> +	bool ret = true;
> 
>  	if (!vfio_file_is_group(file))
>  		return true;
> 
> -	mutex_lock(&group->group_lock);
> -	if (group->container) {
> -		ret = vfio_container_ioctl_check_extension(group->container,
> -
> VFIO_DMA_CC_IOMMU);
> -	} else {
> -		/*
> -		 * Since the coherency state is determined only once a
> container
> -		 * is attached the user must do so before they can prove they
> -		 * have permission.
> -		 */
> -		ret = true;
> +	/*
> +	 * If the device does not have
> IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
> +	 * any domain later attached to it will also not support it.
> +	 */

also add the other part i.e. if the device does have the cap then any domain
later attached to it will have the cap enabled. Only with both clarified
we can safely use the device cap here.

> +	mutex_lock(&group->device_lock);
> +	list_for_each_entry(device, &group->device_list, group_next) {
> +		if (!device_iommu_capable(device->dev,
> +
> IOMMU_CAP_ENFORCE_CACHE_COHERENCY)) {
> +			ret = false;
> +			break;
> +		}
>  	}
> -	mutex_unlock(&group->group_lock);
> +	mutex_unlock(&group->device_lock);
>  	return ret;
>  }
>  EXPORT_SYMBOL_GPL(vfio_file_enforced_coherent);
> --
> 2.38.0
> 

RE: [PATCH 05/10] vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:17 AM
> 
> iommufd doesn't establish the iommu_domains until after the device FD is
> opened, even if the container has been set. This design is part of moving
> away from the group centric iommu APIs.
> 
> This is fine, except that the normal sequence of establishing the kvm
> wbindv won't work:

wbindv -> wbinvd

> 
>    group = open("/dev/vfio/XX")
>    ioctl(group, VFIO_GROUP_SET_CONTAINER)
>    ioctl(kvm, KVM_DEV_VFIO_GROUP_ADD)
>    ioctl(group, VFIO_GROUP_GET_DEVICE_FD)
> 
> As the domains don't start existing until GET_DEVICE_FD. Further,
> GET_DEVICE_FD requires that KVM_DEV_VFIO_GROUP_ADD already be
> done as that
> is what sets the group->kvm and thus device->kvm for the driver to use
> during open.
> 
> Now that we have device centric cap ops and the new
> IOMMU_CAP_ENFORCE_CACHE_COHERENCY we know what the
> iommu_domain will be
> capable of without having to create it. Use this to compute

it's worth noting that the prerequisite is that vfio always enforces
cache coherency on a domain according to the iommu capability
of the devices attached to that domain. There is no mix of attaching
a device supporting the cap to a domain which doesn't enforce
coherency. With that we know what the domain will be w/o having
to create it.

> vfio_file_enforced_coherent() and resolve the ordering problems.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/vfio/container.c |  5 +++--
>  drivers/vfio/vfio.h      |  2 --
>  drivers/vfio/vfio_main.c | 27 ++++++++++++++-------------
>  3 files changed, 17 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
> index 499777930b08fa..d97747dfb05d02 100644
> --- a/drivers/vfio/container.c
> +++ b/drivers/vfio/container.c
> @@ -188,8 +188,9 @@ void vfio_device_container_unregister(struct
> vfio_device *device)
>  			device->group->container->iommu_data, device);
>  }
> 
> -long vfio_container_ioctl_check_extension(struct vfio_container *container,
> -					  unsigned long arg)
> +static long
> +vfio_container_ioctl_check_extension(struct vfio_container *container,
> +				     unsigned long arg)
>  {
>  	struct vfio_iommu_driver *driver;
>  	long ret = 0;
> diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
> index 54e5a8e0834ccb..247590334e14b0 100644
> --- a/drivers/vfio/vfio.h
> +++ b/drivers/vfio/vfio.h
> @@ -119,8 +119,6 @@ int vfio_container_attach_group(struct
> vfio_container *container,
>  void vfio_group_detach_container(struct vfio_group *group);
>  void vfio_device_container_register(struct vfio_device *device);
>  void vfio_device_container_unregister(struct vfio_device *device);
> -long vfio_container_ioctl_check_extension(struct vfio_container *container,
> -					  unsigned long arg);
>  int __init vfio_container_init(void);
>  void vfio_container_cleanup(void);
> 
> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index 1e414b2c48a511..a8d1fbfcc3ddad 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -1625,24 +1625,25 @@ EXPORT_SYMBOL_GPL(vfio_file_is_group);
>  bool vfio_file_enforced_coherent(struct file *file)
>  {
>  	struct vfio_group *group = file->private_data;
> -	bool ret;
> +	struct vfio_device *device;
> +	bool ret = true;
> 
>  	if (!vfio_file_is_group(file))
>  		return true;
> 
> -	mutex_lock(&group->group_lock);
> -	if (group->container) {
> -		ret = vfio_container_ioctl_check_extension(group->container,
> -
> VFIO_DMA_CC_IOMMU);
> -	} else {
> -		/*
> -		 * Since the coherency state is determined only once a
> container
> -		 * is attached the user must do so before they can prove they
> -		 * have permission.
> -		 */
> -		ret = true;
> +	/*
> +	 * If the device does not have
> IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
> +	 * any domain later attached to it will also not support it.
> +	 */

also add the other part i.e. if the device does have the cap then any domain
later attached to it will have the cap enabled. Only with both clarified
we can safely use the device cap here.

> +	mutex_lock(&group->device_lock);
> +	list_for_each_entry(device, &group->device_list, group_next) {
> +		if (!device_iommu_capable(device->dev,
> +
> IOMMU_CAP_ENFORCE_CACHE_COHERENCY)) {
> +			ret = false;
> +			break;
> +		}
>  	}
> -	mutex_unlock(&group->group_lock);
> +	mutex_unlock(&group->device_lock);
>  	return ret;
>  }
>  EXPORT_SYMBOL_GPL(vfio_file_enforced_coherent);
> --
> 2.38.0
> 

Re: [Intel-gfx] [PATCH 05/10] vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:17 AM
> 
> iommufd doesn't establish the iommu_domains until after the device FD is
> opened, even if the container has been set. This design is part of moving
> away from the group centric iommu APIs.
> 
> This is fine, except that the normal sequence of establishing the kvm
> wbindv won't work:

wbindv -> wbinvd

> 
>    group = open("/dev/vfio/XX")
>    ioctl(group, VFIO_GROUP_SET_CONTAINER)
>    ioctl(kvm, KVM_DEV_VFIO_GROUP_ADD)
>    ioctl(group, VFIO_GROUP_GET_DEVICE_FD)
> 
> As the domains don't start existing until GET_DEVICE_FD. Further,
> GET_DEVICE_FD requires that KVM_DEV_VFIO_GROUP_ADD already be
> done as that
> is what sets the group->kvm and thus device->kvm for the driver to use
> during open.
> 
> Now that we have device centric cap ops and the new
> IOMMU_CAP_ENFORCE_CACHE_COHERENCY we know what the
> iommu_domain will be
> capable of without having to create it. Use this to compute

it's worth noting that the prerequisite is that vfio always enforces
cache coherency on a domain according to the iommu capability
of the devices attached to that domain. There is no mix of attaching
a device supporting the cap to a domain which doesn't enforce
coherency. With that we know what the domain will be w/o having
to create it.

> vfio_file_enforced_coherent() and resolve the ordering problems.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/vfio/container.c |  5 +++--
>  drivers/vfio/vfio.h      |  2 --
>  drivers/vfio/vfio_main.c | 27 ++++++++++++++-------------
>  3 files changed, 17 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
> index 499777930b08fa..d97747dfb05d02 100644
> --- a/drivers/vfio/container.c
> +++ b/drivers/vfio/container.c
> @@ -188,8 +188,9 @@ void vfio_device_container_unregister(struct
> vfio_device *device)
>  			device->group->container->iommu_data, device);
>  }
> 
> -long vfio_container_ioctl_check_extension(struct vfio_container *container,
> -					  unsigned long arg)
> +static long
> +vfio_container_ioctl_check_extension(struct vfio_container *container,
> +				     unsigned long arg)
>  {
>  	struct vfio_iommu_driver *driver;
>  	long ret = 0;
> diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
> index 54e5a8e0834ccb..247590334e14b0 100644
> --- a/drivers/vfio/vfio.h
> +++ b/drivers/vfio/vfio.h
> @@ -119,8 +119,6 @@ int vfio_container_attach_group(struct
> vfio_container *container,
>  void vfio_group_detach_container(struct vfio_group *group);
>  void vfio_device_container_register(struct vfio_device *device);
>  void vfio_device_container_unregister(struct vfio_device *device);
> -long vfio_container_ioctl_check_extension(struct vfio_container *container,
> -					  unsigned long arg);
>  int __init vfio_container_init(void);
>  void vfio_container_cleanup(void);
> 
> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index 1e414b2c48a511..a8d1fbfcc3ddad 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -1625,24 +1625,25 @@ EXPORT_SYMBOL_GPL(vfio_file_is_group);
>  bool vfio_file_enforced_coherent(struct file *file)
>  {
>  	struct vfio_group *group = file->private_data;
> -	bool ret;
> +	struct vfio_device *device;
> +	bool ret = true;
> 
>  	if (!vfio_file_is_group(file))
>  		return true;
> 
> -	mutex_lock(&group->group_lock);
> -	if (group->container) {
> -		ret = vfio_container_ioctl_check_extension(group->container,
> -
> VFIO_DMA_CC_IOMMU);
> -	} else {
> -		/*
> -		 * Since the coherency state is determined only once a
> container
> -		 * is attached the user must do so before they can prove they
> -		 * have permission.
> -		 */
> -		ret = true;
> +	/*
> +	 * If the device does not have
> IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
> +	 * any domain later attached to it will also not support it.
> +	 */

also add the other part i.e. if the device does have the cap then any domain
later attached to it will have the cap enabled. Only with both clarified
we can safely use the device cap here.

> +	mutex_lock(&group->device_lock);
> +	list_for_each_entry(device, &group->device_list, group_next) {
> +		if (!device_iommu_capable(device->dev,
> +
> IOMMU_CAP_ENFORCE_CACHE_COHERENCY)) {
> +			ret = false;
> +			break;
> +		}
>  	}
> -	mutex_unlock(&group->group_lock);
> +	mutex_unlock(&group->device_lock);
>  	return ret;
>  }
>  EXPORT_SYMBOL_GPL(vfio_file_enforced_coherent);
> --
> 2.38.0
> 

RE: [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:51 AM
>
>  menuconfig VFIO
>  	tristate "VFIO Non-Privileged userspace driver framework"
>  	select IOMMU_API
> +	depends on IOMMUFD || !IOMMUFD

Out of curiosity. What is the meaning of this dependency claim?

> @@ -717,12 +735,23 @@ static int vfio_group_ioctl_set_container(struct
> vfio_group *group,
>  	}
> 
>  	container = vfio_container_from_file(f.file);
> -	ret = -EINVAL;

this changes the errno from -EINVAL to -EBADF for the original container
path. Is it desired?

>  	if (container) {
>  		ret = vfio_container_attach_group(container, group);
>  		goto out_unlock;
>  	}
> 
> +	iommufd = iommufd_ctx_from_file(f.file);
> +	if (!IS_ERR(iommufd)) {

The only errno which iommufd_ctx_from_file() may return is -EBADFD
which duplicates with -EBADF assignment in following line.

What about having it return NULL pointer similar as the container
helper does?

> +		u32 ioas_id;
> +
> +		group->iommufd = iommufd;
> +		ret = iommufd_vfio_compat_ioas_id(iommufd, &ioas_id);

exchange the order of above two lines and only assign group->iommufd
when the compat call succeeds.

> @@ -900,7 +940,7 @@ static int vfio_group_ioctl_get_status(struct
> vfio_group *group,
>  		return -ENODEV;
>  	}
> 
> -	if (group->container)
> +	if (group->container || group->iommufd)
>  		status.flags |= VFIO_GROUP_FLAGS_CONTAINER_SET |
>  				VFIO_GROUP_FLAGS_VIABLE;

Copy some explanation from commit msg to explain the subtle difference
between container and iommufd.

RE: [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:51 AM
>
>  menuconfig VFIO
>  	tristate "VFIO Non-Privileged userspace driver framework"
>  	select IOMMU_API
> +	depends on IOMMUFD || !IOMMUFD

Out of curiosity. What is the meaning of this dependency claim?

> @@ -717,12 +735,23 @@ static int vfio_group_ioctl_set_container(struct
> vfio_group *group,
>  	}
> 
>  	container = vfio_container_from_file(f.file);
> -	ret = -EINVAL;

this changes the errno from -EINVAL to -EBADF for the original container
path. Is it desired?

>  	if (container) {
>  		ret = vfio_container_attach_group(container, group);
>  		goto out_unlock;
>  	}
> 
> +	iommufd = iommufd_ctx_from_file(f.file);
> +	if (!IS_ERR(iommufd)) {

The only errno which iommufd_ctx_from_file() may return is -EBADFD
which duplicates with -EBADF assignment in following line.

What about having it return NULL pointer similar as the container
helper does?

> +		u32 ioas_id;
> +
> +		group->iommufd = iommufd;
> +		ret = iommufd_vfio_compat_ioas_id(iommufd, &ioas_id);

exchange the order of above two lines and only assign group->iommufd
when the compat call succeeds.

> @@ -900,7 +940,7 @@ static int vfio_group_ioctl_get_status(struct
> vfio_group *group,
>  		return -ENODEV;
>  	}
> 
> -	if (group->container)
> +	if (group->container || group->iommufd)
>  		status.flags |= VFIO_GROUP_FLAGS_CONTAINER_SET |
>  				VFIO_GROUP_FLAGS_VIABLE;

Copy some explanation from commit msg to explain the subtle difference
between container and iommufd.

Re: [Intel-gfx] [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:51 AM
>
>  menuconfig VFIO
>  	tristate "VFIO Non-Privileged userspace driver framework"
>  	select IOMMU_API
> +	depends on IOMMUFD || !IOMMUFD

Out of curiosity. What is the meaning of this dependency claim?

> @@ -717,12 +735,23 @@ static int vfio_group_ioctl_set_container(struct
> vfio_group *group,
>  	}
> 
>  	container = vfio_container_from_file(f.file);
> -	ret = -EINVAL;

this changes the errno from -EINVAL to -EBADF for the original container
path. Is it desired?

>  	if (container) {
>  		ret = vfio_container_attach_group(container, group);
>  		goto out_unlock;
>  	}
> 
> +	iommufd = iommufd_ctx_from_file(f.file);
> +	if (!IS_ERR(iommufd)) {

The only errno which iommufd_ctx_from_file() may return is -EBADFD
which duplicates with -EBADF assignment in following line.

What about having it return NULL pointer similar as the container
helper does?

> +		u32 ioas_id;
> +
> +		group->iommufd = iommufd;
> +		ret = iommufd_vfio_compat_ioas_id(iommufd, &ioas_id);

exchange the order of above two lines and only assign group->iommufd
when the compat call succeeds.

> @@ -900,7 +940,7 @@ static int vfio_group_ioctl_get_status(struct
> vfio_group *group,
>  		return -ENODEV;
>  	}
> 
> -	if (group->container)
> +	if (group->container || group->iommufd)
>  		status.flags |= VFIO_GROUP_FLAGS_CONTAINER_SET |
>  				VFIO_GROUP_FLAGS_VIABLE;

Copy some explanation from commit msg to explain the subtle difference
between container and iommufd.

RE: [PATCH 07/10] vfio-iommufd: Support iommufd for physical VFIO devices

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:51 AM
> 
> +int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx)
> +{
> +	u32 ioas_id;
> +	u32 device_id;
> +	int ret;
> +
> +	lockdep_assert_held(&vdev->dev_set->lock);
> +
> +	/*
> +	 * If the driver doesn't provide this op then it means the device does
> +	 * not do DMA at all. So nothing to do.
> +	 */
> +	if (!vdev->ops->bind_iommufd)
> +		return 0;

Nothing to do or return -EOPNOTSUPP?

> +
> +	ret = vdev->ops->bind_iommufd(vdev, ictx, &device_id);
> +	if (ret)
> +		return ret;
> +
> +	if (vdev->ops->attach_ioas) {

__vfio_register_dev() already verifies that all three callbacks must
co-exist. Then no need to check it again here and later.

> +void vfio_iommufd_unbind(struct vfio_device *vdev)
> +{
> +	lockdep_assert_held(&vdev->dev_set->lock);
> +
> +	if (!vdev->iommufd_device)
> +		return;

there is no iommufd_device in the emulated path...

RE: [PATCH 07/10] vfio-iommufd: Support iommufd for physical VFIO devices

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:51 AM
> 
> +int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx)
> +{
> +	u32 ioas_id;
> +	u32 device_id;
> +	int ret;
> +
> +	lockdep_assert_held(&vdev->dev_set->lock);
> +
> +	/*
> +	 * If the driver doesn't provide this op then it means the device does
> +	 * not do DMA at all. So nothing to do.
> +	 */
> +	if (!vdev->ops->bind_iommufd)
> +		return 0;

Nothing to do or return -EOPNOTSUPP?

> +
> +	ret = vdev->ops->bind_iommufd(vdev, ictx, &device_id);
> +	if (ret)
> +		return ret;
> +
> +	if (vdev->ops->attach_ioas) {

__vfio_register_dev() already verifies that all three callbacks must
co-exist. Then no need to check it again here and later.

> +void vfio_iommufd_unbind(struct vfio_device *vdev)
> +{
> +	lockdep_assert_held(&vdev->dev_set->lock);
> +
> +	if (!vdev->iommufd_device)
> +		return;

there is no iommufd_device in the emulated path...

Re: [Intel-gfx] [PATCH 07/10] vfio-iommufd: Support iommufd for physical VFIO devices

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:51 AM
> 
> +int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx)
> +{
> +	u32 ioas_id;
> +	u32 device_id;
> +	int ret;
> +
> +	lockdep_assert_held(&vdev->dev_set->lock);
> +
> +	/*
> +	 * If the driver doesn't provide this op then it means the device does
> +	 * not do DMA at all. So nothing to do.
> +	 */
> +	if (!vdev->ops->bind_iommufd)
> +		return 0;

Nothing to do or return -EOPNOTSUPP?

> +
> +	ret = vdev->ops->bind_iommufd(vdev, ictx, &device_id);
> +	if (ret)
> +		return ret;
> +
> +	if (vdev->ops->attach_ioas) {

__vfio_register_dev() already verifies that all three callbacks must
co-exist. Then no need to check it again here and later.

> +void vfio_iommufd_unbind(struct vfio_device *vdev)
> +{
> +	lockdep_assert_held(&vdev->dev_set->lock);
> +
> +	if (!vdev->iommufd_device)
> +		return;

there is no iommufd_device in the emulated path...

RE: [PATCH 08/10] vfio-iommufd: Support iommufd for emulated VFIO devices

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:51 AM
> 
> Emulated VFIO devices are calling vfio_register_emulated_iommu_dev() and
> consist of all the mdev drivers.
> 
> Like the physical drivers, support for iommufd is provided by the driver
> supplying the correct correct standard ops. Provide ops from the core that
> duplicate what vfio_register_emulated_iommu_dev() does.
> 
> Emulated drivers are where it is more likely to see variation in the
> iommfd support ops. For instance IDXD will probably need to setup both a
> iommfd_device context linked to a PASID and an iommufd_access context to
> support all their mdev operations.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/gpu/drm/i915/gvt/kvmgt.c  |   3 +
>  drivers/s390/cio/vfio_ccw_ops.c   |   3 +
>  drivers/s390/crypto/vfio_ap_ops.c |   3 +
>  drivers/vfio/container.c          | 108 ++++++-----------------------
>  drivers/vfio/iommufd.c            |  57 ++++++++++++++++
>  drivers/vfio/vfio.h               |  10 ++-
>  drivers/vfio/vfio_main.c          | 110 +++++++++++++++++++++++++++++-
>  include/linux/vfio.h              |  14 ++++
>  8 files changed, 217 insertions(+), 91 deletions(-)

mtty, mdpy and mbochs?

> diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
> index 8772dad6808539..0388f2e33447eb 100644
> --- a/drivers/vfio/container.c
> +++ b/drivers/vfio/container.c
> @@ -540,113 +540,45 @@ void vfio_group_unuse_container(struct
> vfio_group *group)
>  	fput(group->opened_file);
>  }
> 
> -/*
> - * Pin contiguous user pages and return their associated host pages for local
> - * domain only.
> - * @device [in]  : device
> - * @iova [in]    : starting IOVA of user pages to be pinned.
> - * @npage [in]   : count of pages to be pinned.  This count should not
> - *		   be greater than VFIO_PIN_PAGES_MAX_ENTRIES.
> - * @prot [in]    : protection flags
> - * @pages[out]   : array of host pages
> - * Return error or number of pages pinned.
> - *
> - * A driver may only call this function if the vfio_device was created
> - * by vfio_register_emulated_iommu_dev().
> - */
> -int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova,
> -		   int npage, int prot, struct page **pages)
> +int vfio_container_pin_pages(struct vfio_container *container,
> +			     struct iommu_group *iommu_group, dma_addr_t
> iova,
> +			     int npage, int prot, struct page **pages)
>  {
> -	struct vfio_container *container;
> -	struct vfio_group *group = device->group;
> -	struct vfio_iommu_driver *driver;
> -	int ret;
> -
> -	if (!pages || !npage || !vfio_assert_device_open(device))
> -		return -EINVAL;
> +	/* group->container cannot change while a vfio device is open */
> +	struct vfio_iommu_driver *driver = container->iommu_driver;
> 
>  	if (npage > VFIO_PIN_PAGES_MAX_ENTRIES)
>  		return -E2BIG;
> 
>  	/* group->container cannot change while a vfio device is open */
> -	container = group->container;
>  	driver = container->iommu_driver;

duplicated comment and assignment.

Actually, I'm not sure whether the comment should be put within this
container helper and other two. There is no group reference in these
helpers then it sounds like the comment makes more sense to be in the
caller side?

> +void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova, int
> npage)
> +{
> +	if (WARN_ON(!vfio_assert_device_open(device)))
> +		return;
> +
> +	if (device->group->container) {
> +		vfio_container_unpin_pages(device->group->container, iova,
> +					   npage);
> +	} else if (device->iommufd_access) {

be consistent with other two helpers i.e. if-if instead of if-else

> +		if (WARN_ON(iova > ULONG_MAX))
> +			return;

Is there a reason why this is a WARN_ON only in unpin but not in pin?

> +int vfio_dma_rw(struct vfio_device *device, dma_addr_t iova, void *data,
> +		size_t len, bool write)
> +{
> +	if (!data || len <= 0 || !vfio_assert_device_open(device))
> +		return -EINVAL;
> +
> +	if (device->group->container)
> +		return vfio_container_dma_rw(device->group->container,
> iova,
> +					     data, len, write);
> +
> +	if (device->iommufd_access) {
> +		unsigned int flags = 0;
> +
> +		if (iova > ULONG_MAX)
> +			return -EINVAL;
> +
> +		/* VFIO historically tries to auto-detect a kthread */
> +		if (!current->mm)
> +			flags |= IOMMUFD_ACCESS_RW_KTHREAD;

Can you elaborate why this cannot be put in iommufd as the default
policy similar to what vfio container does?

RE: [PATCH 08/10] vfio-iommufd: Support iommufd for emulated VFIO devices

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:51 AM
> 
> Emulated VFIO devices are calling vfio_register_emulated_iommu_dev() and
> consist of all the mdev drivers.
> 
> Like the physical drivers, support for iommufd is provided by the driver
> supplying the correct correct standard ops. Provide ops from the core that
> duplicate what vfio_register_emulated_iommu_dev() does.
> 
> Emulated drivers are where it is more likely to see variation in the
> iommfd support ops. For instance IDXD will probably need to setup both a
> iommfd_device context linked to a PASID and an iommufd_access context to
> support all their mdev operations.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/gpu/drm/i915/gvt/kvmgt.c  |   3 +
>  drivers/s390/cio/vfio_ccw_ops.c   |   3 +
>  drivers/s390/crypto/vfio_ap_ops.c |   3 +
>  drivers/vfio/container.c          | 108 ++++++-----------------------
>  drivers/vfio/iommufd.c            |  57 ++++++++++++++++
>  drivers/vfio/vfio.h               |  10 ++-
>  drivers/vfio/vfio_main.c          | 110 +++++++++++++++++++++++++++++-
>  include/linux/vfio.h              |  14 ++++
>  8 files changed, 217 insertions(+), 91 deletions(-)

mtty, mdpy and mbochs?

> diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
> index 8772dad6808539..0388f2e33447eb 100644
> --- a/drivers/vfio/container.c
> +++ b/drivers/vfio/container.c
> @@ -540,113 +540,45 @@ void vfio_group_unuse_container(struct
> vfio_group *group)
>  	fput(group->opened_file);
>  }
> 
> -/*
> - * Pin contiguous user pages and return their associated host pages for local
> - * domain only.
> - * @device [in]  : device
> - * @iova [in]    : starting IOVA of user pages to be pinned.
> - * @npage [in]   : count of pages to be pinned.  This count should not
> - *		   be greater than VFIO_PIN_PAGES_MAX_ENTRIES.
> - * @prot [in]    : protection flags
> - * @pages[out]   : array of host pages
> - * Return error or number of pages pinned.
> - *
> - * A driver may only call this function if the vfio_device was created
> - * by vfio_register_emulated_iommu_dev().
> - */
> -int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova,
> -		   int npage, int prot, struct page **pages)
> +int vfio_container_pin_pages(struct vfio_container *container,
> +			     struct iommu_group *iommu_group, dma_addr_t
> iova,
> +			     int npage, int prot, struct page **pages)
>  {
> -	struct vfio_container *container;
> -	struct vfio_group *group = device->group;
> -	struct vfio_iommu_driver *driver;
> -	int ret;
> -
> -	if (!pages || !npage || !vfio_assert_device_open(device))
> -		return -EINVAL;
> +	/* group->container cannot change while a vfio device is open */
> +	struct vfio_iommu_driver *driver = container->iommu_driver;
> 
>  	if (npage > VFIO_PIN_PAGES_MAX_ENTRIES)
>  		return -E2BIG;
> 
>  	/* group->container cannot change while a vfio device is open */
> -	container = group->container;
>  	driver = container->iommu_driver;

duplicated comment and assignment.

Actually, I'm not sure whether the comment should be put within this
container helper and other two. There is no group reference in these
helpers then it sounds like the comment makes more sense to be in the
caller side?

> +void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova, int
> npage)
> +{
> +	if (WARN_ON(!vfio_assert_device_open(device)))
> +		return;
> +
> +	if (device->group->container) {
> +		vfio_container_unpin_pages(device->group->container, iova,
> +					   npage);
> +	} else if (device->iommufd_access) {

be consistent with other two helpers i.e. if-if instead of if-else

> +		if (WARN_ON(iova > ULONG_MAX))
> +			return;

Is there a reason why this is a WARN_ON only in unpin but not in pin?

> +int vfio_dma_rw(struct vfio_device *device, dma_addr_t iova, void *data,
> +		size_t len, bool write)
> +{
> +	if (!data || len <= 0 || !vfio_assert_device_open(device))
> +		return -EINVAL;
> +
> +	if (device->group->container)
> +		return vfio_container_dma_rw(device->group->container,
> iova,
> +					     data, len, write);
> +
> +	if (device->iommufd_access) {
> +		unsigned int flags = 0;
> +
> +		if (iova > ULONG_MAX)
> +			return -EINVAL;
> +
> +		/* VFIO historically tries to auto-detect a kthread */
> +		if (!current->mm)
> +			flags |= IOMMUFD_ACCESS_RW_KTHREAD;

Can you elaborate why this cannot be put in iommufd as the default
policy similar to what vfio container does?

Re: [Intel-gfx] [PATCH 08/10] vfio-iommufd: Support iommufd for emulated VFIO devices

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:51 AM
> 
> Emulated VFIO devices are calling vfio_register_emulated_iommu_dev() and
> consist of all the mdev drivers.
> 
> Like the physical drivers, support for iommufd is provided by the driver
> supplying the correct correct standard ops. Provide ops from the core that
> duplicate what vfio_register_emulated_iommu_dev() does.
> 
> Emulated drivers are where it is more likely to see variation in the
> iommfd support ops. For instance IDXD will probably need to setup both a
> iommfd_device context linked to a PASID and an iommufd_access context to
> support all their mdev operations.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/gpu/drm/i915/gvt/kvmgt.c  |   3 +
>  drivers/s390/cio/vfio_ccw_ops.c   |   3 +
>  drivers/s390/crypto/vfio_ap_ops.c |   3 +
>  drivers/vfio/container.c          | 108 ++++++-----------------------
>  drivers/vfio/iommufd.c            |  57 ++++++++++++++++
>  drivers/vfio/vfio.h               |  10 ++-
>  drivers/vfio/vfio_main.c          | 110 +++++++++++++++++++++++++++++-
>  include/linux/vfio.h              |  14 ++++
>  8 files changed, 217 insertions(+), 91 deletions(-)

mtty, mdpy and mbochs?

> diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
> index 8772dad6808539..0388f2e33447eb 100644
> --- a/drivers/vfio/container.c
> +++ b/drivers/vfio/container.c
> @@ -540,113 +540,45 @@ void vfio_group_unuse_container(struct
> vfio_group *group)
>  	fput(group->opened_file);
>  }
> 
> -/*
> - * Pin contiguous user pages and return their associated host pages for local
> - * domain only.
> - * @device [in]  : device
> - * @iova [in]    : starting IOVA of user pages to be pinned.
> - * @npage [in]   : count of pages to be pinned.  This count should not
> - *		   be greater than VFIO_PIN_PAGES_MAX_ENTRIES.
> - * @prot [in]    : protection flags
> - * @pages[out]   : array of host pages
> - * Return error or number of pages pinned.
> - *
> - * A driver may only call this function if the vfio_device was created
> - * by vfio_register_emulated_iommu_dev().
> - */
> -int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova,
> -		   int npage, int prot, struct page **pages)
> +int vfio_container_pin_pages(struct vfio_container *container,
> +			     struct iommu_group *iommu_group, dma_addr_t
> iova,
> +			     int npage, int prot, struct page **pages)
>  {
> -	struct vfio_container *container;
> -	struct vfio_group *group = device->group;
> -	struct vfio_iommu_driver *driver;
> -	int ret;
> -
> -	if (!pages || !npage || !vfio_assert_device_open(device))
> -		return -EINVAL;
> +	/* group->container cannot change while a vfio device is open */
> +	struct vfio_iommu_driver *driver = container->iommu_driver;
> 
>  	if (npage > VFIO_PIN_PAGES_MAX_ENTRIES)
>  		return -E2BIG;
> 
>  	/* group->container cannot change while a vfio device is open */
> -	container = group->container;
>  	driver = container->iommu_driver;

duplicated comment and assignment.

Actually, I'm not sure whether the comment should be put within this
container helper and other two. There is no group reference in these
helpers then it sounds like the comment makes more sense to be in the
caller side?

> +void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova, int
> npage)
> +{
> +	if (WARN_ON(!vfio_assert_device_open(device)))
> +		return;
> +
> +	if (device->group->container) {
> +		vfio_container_unpin_pages(device->group->container, iova,
> +					   npage);
> +	} else if (device->iommufd_access) {

be consistent with other two helpers i.e. if-if instead of if-else

> +		if (WARN_ON(iova > ULONG_MAX))
> +			return;

Is there a reason why this is a WARN_ON only in unpin but not in pin?

> +int vfio_dma_rw(struct vfio_device *device, dma_addr_t iova, void *data,
> +		size_t len, bool write)
> +{
> +	if (!data || len <= 0 || !vfio_assert_device_open(device))
> +		return -EINVAL;
> +
> +	if (device->group->container)
> +		return vfio_container_dma_rw(device->group->container,
> iova,
> +					     data, len, write);
> +
> +	if (device->iommufd_access) {
> +		unsigned int flags = 0;
> +
> +		if (iova > ULONG_MAX)
> +			return -EINVAL;
> +
> +		/* VFIO historically tries to auto-detect a kthread */
> +		if (!current->mm)
> +			flags |= IOMMUFD_ACCESS_RW_KTHREAD;

Can you elaborate why this cannot be put in iommufd as the default
policy similar to what vfio container does?

RE: [PATCH 09/10] vfio: Make vfio_container optionally compiled

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:51 AM
> 
>  if VFIO
> +config VFIO_CONTAINER
> +	bool "Support for the VFIO container /dev/vfio/vfio"
> +	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM ||
> ARM64)
> +	default y
> +	help
> +	  The VFIO container is the classic interface to VFIO for establishing
> +	  mappings. If N is selected here then IOMMUFD must be used the

establishing IOMMU mappings.

s/used the manage/used to manage/

> manage
> +	  the mappings.
> +
> +	  Unless testing IOMMUFD say Y here.
> +
> +if VFIO_CONTAINER
>  config VFIO_IOMMU_TYPE1
>  	tristate
> -	default n
> +	default MMU && (X86 || S390 || ARM || ARM64)

there are already same conditions for selecting TYPE1 from
VFIO_CONTAINER. what does duplicating conditions here
bring to us?

RE: [PATCH 09/10] vfio: Make vfio_container optionally compiled

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:51 AM
> 
>  if VFIO
> +config VFIO_CONTAINER
> +	bool "Support for the VFIO container /dev/vfio/vfio"
> +	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM ||
> ARM64)
> +	default y
> +	help
> +	  The VFIO container is the classic interface to VFIO for establishing
> +	  mappings. If N is selected here then IOMMUFD must be used the

establishing IOMMU mappings.

s/used the manage/used to manage/

> manage
> +	  the mappings.
> +
> +	  Unless testing IOMMUFD say Y here.
> +
> +if VFIO_CONTAINER
>  config VFIO_IOMMU_TYPE1
>  	tristate
> -	default n
> +	default MMU && (X86 || S390 || ARM || ARM64)

there are already same conditions for selecting TYPE1 from
VFIO_CONTAINER. what does duplicating conditions here
bring to us?

Re: [Intel-gfx] [PATCH 09/10] vfio: Make vfio_container optionally compiled

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, October 26, 2022 2:51 AM
> 
>  if VFIO
> +config VFIO_CONTAINER
> +	bool "Support for the VFIO container /dev/vfio/vfio"
> +	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM ||
> ARM64)
> +	default y
> +	help
> +	  The VFIO container is the classic interface to VFIO for establishing
> +	  mappings. If N is selected here then IOMMUFD must be used the

establishing IOMMU mappings.

s/used the manage/used to manage/

> manage
> +	  the mappings.
> +
> +	  Unless testing IOMMUFD say Y here.
> +
> +if VFIO_CONTAINER
>  config VFIO_IOMMU_TYPE1
>  	tristate
> -	default n
> +	default MMU && (X86 || S390 || ARM || ARM64)

there are already same conditions for selecting TYPE1 from
VFIO_CONTAINER. what does duplicating conditions here
bring to us?

Re: [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Nicolin Chen]

On Tue, Nov 01, 2022 at 08:09:52AM +0000, Tian, Kevin wrote:

> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:51 AM
> >
> >  menuconfig VFIO
> >       tristate "VFIO Non-Privileged userspace driver framework"
> >       select IOMMU_API
> > +     depends on IOMMUFD || !IOMMUFD
> 
> Out of curiosity. What is the meaning of this dependency claim?

"is it a module or not" -- from https://lwn.net/Articles/683476/

Re: [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Nicolin Chen]

On Tue, Nov 01, 2022 at 08:09:52AM +0000, Tian, Kevin wrote:

> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:51 AM
> >
> >  menuconfig VFIO
> >       tristate "VFIO Non-Privileged userspace driver framework"
> >       select IOMMU_API
> > +     depends on IOMMUFD || !IOMMUFD
> 
> Out of curiosity. What is the meaning of this dependency claim?

"is it a module or not" -- from https://lwn.net/Articles/683476/

Re: [Intel-gfx] [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Nicolin Chen]

On Tue, Nov 01, 2022 at 08:09:52AM +0000, Tian, Kevin wrote:

> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:51 AM
> >
> >  menuconfig VFIO
> >       tristate "VFIO Non-Privileged userspace driver framework"
> >       select IOMMU_API
> > +     depends on IOMMUFD || !IOMMUFD
> 
> Out of curiosity. What is the meaning of this dependency claim?

"is it a module or not" -- from https://lwn.net/Articles/683476/

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 11:04:38AM +0800, Yi Liu wrote:
> On 2022/11/1 07:24, Jason Gunthorpe wrote:
> > On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
> > > > There is something wrong with the test suite that it isn't covering
> > > > the above, I'm going to look into that today.
> > > 
> > > sounds to be the cause. I didn't see any significant change in vfio_main.c
> > > that may fail gvt. So should the iommufd changes. Then we will re-run the
> > > test after your update.:-)
> > 
> > I updated the github with all the changes made so far, it is worth
> > trying again!
> 
> gvt is still failing with below call trace in host side. vfio_unpin_pages()
> is still in problem. Any idea on it?

Oh, this is my mistake, I rushed a bit getting updated branches:

diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
index 40eb6931ab2321..29e7b1fdd0cd4a 100644
--- a/drivers/vfio/iommufd.c
+++ b/drivers/vfio/iommufd.c
@@ -118,6 +118,7 @@ static void vfio_emulated_unmap(void *data, unsigned long iova,
 }
 
 static const struct iommufd_access_ops vfio_user_ops = {
+	.needs_pin_pages = 1,
 	.unmap = vfio_emulated_unmap,
 };

Thanks, 
Jason

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 11:04:38AM +0800, Yi Liu wrote:
> On 2022/11/1 07:24, Jason Gunthorpe wrote:
> > On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
> > > > There is something wrong with the test suite that it isn't covering
> > > > the above, I'm going to look into that today.
> > > 
> > > sounds to be the cause. I didn't see any significant change in vfio_main.c
> > > that may fail gvt. So should the iommufd changes. Then we will re-run the
> > > test after your update.:-)
> > 
> > I updated the github with all the changes made so far, it is worth
> > trying again!
> 
> gvt is still failing with below call trace in host side. vfio_unpin_pages()
> is still in problem. Any idea on it?

Oh, this is my mistake, I rushed a bit getting updated branches:

diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
index 40eb6931ab2321..29e7b1fdd0cd4a 100644
--- a/drivers/vfio/iommufd.c
+++ b/drivers/vfio/iommufd.c
@@ -118,6 +118,7 @@ static void vfio_emulated_unmap(void *data, unsigned long iova,
 }
 
 static const struct iommufd_access_ops vfio_user_ops = {
+	.needs_pin_pages = 1,
 	.unmap = vfio_emulated_unmap,
 };

Thanks, 
Jason

Re: [Intel-gfx] [PATCH 00/10] Connect VFIO to IOMMUFD

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 11:04:38AM +0800, Yi Liu wrote:
> On 2022/11/1 07:24, Jason Gunthorpe wrote:
> > On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
> > > > There is something wrong with the test suite that it isn't covering
> > > > the above, I'm going to look into that today.
> > > 
> > > sounds to be the cause. I didn't see any significant change in vfio_main.c
> > > that may fail gvt. So should the iommufd changes. Then we will re-run the
> > > test after your update.:-)
> > 
> > I updated the github with all the changes made so far, it is worth
> > trying again!
> 
> gvt is still failing with below call trace in host side. vfio_unpin_pages()
> is still in problem. Any idea on it?

Oh, this is my mistake, I rushed a bit getting updated branches:

diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
index 40eb6931ab2321..29e7b1fdd0cd4a 100644
--- a/drivers/vfio/iommufd.c
+++ b/drivers/vfio/iommufd.c
@@ -118,6 +118,7 @@ static void vfio_emulated_unmap(void *data, unsigned long iova,
 }
 
 static const struct iommufd_access_ops vfio_user_ops = {
+	.needs_pin_pages = 1,
 	.unmap = vfio_emulated_unmap,
 };

Thanks, 
Jason

Re: [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 02:19:04AM -0700, Nicolin Chen wrote:
> On Tue, Nov 01, 2022 at 08:09:52AM +0000, Tian, Kevin wrote:
> 
> > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > Sent: Wednesday, October 26, 2022 2:51 AM
> > >
> > >  menuconfig VFIO
> > >       tristate "VFIO Non-Privileged userspace driver framework"
> > >       select IOMMU_API
> > > +     depends on IOMMUFD || !IOMMUFD
> > 
> > Out of curiosity. What is the meaning of this dependency claim?
> 
> "is it a module or not" -- from https://lwn.net/Articles/683476/

Yes, it is the kconfig pattern for "This symbol optionally uses the
other symbol, and if the other symbol is turned on then it has to be
the right y/m value"

ie rejects vfio being built-in but iommufd being a module

Jason

Re: [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 02:19:04AM -0700, Nicolin Chen wrote:
> On Tue, Nov 01, 2022 at 08:09:52AM +0000, Tian, Kevin wrote:
> 
> > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > Sent: Wednesday, October 26, 2022 2:51 AM
> > >
> > >  menuconfig VFIO
> > >       tristate "VFIO Non-Privileged userspace driver framework"
> > >       select IOMMU_API
> > > +     depends on IOMMUFD || !IOMMUFD
> > 
> > Out of curiosity. What is the meaning of this dependency claim?
> 
> "is it a module or not" -- from https://lwn.net/Articles/683476/

Yes, it is the kconfig pattern for "This symbol optionally uses the
other symbol, and if the other symbol is turned on then it has to be
the right y/m value"

ie rejects vfio being built-in but iommufd being a module

Jason

Re: [Intel-gfx] [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 02:19:04AM -0700, Nicolin Chen wrote:
> On Tue, Nov 01, 2022 at 08:09:52AM +0000, Tian, Kevin wrote:
> 
> > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > Sent: Wednesday, October 26, 2022 2:51 AM
> > >
> > >  menuconfig VFIO
> > >       tristate "VFIO Non-Privileged userspace driver framework"
> > >       select IOMMU_API
> > > +     depends on IOMMUFD || !IOMMUFD
> > 
> > Out of curiosity. What is the meaning of this dependency claim?
> 
> "is it a module or not" -- from https://lwn.net/Articles/683476/

Yes, it is the kconfig pattern for "This symbol optionally uses the
other symbol, and if the other symbol is turned on then it has to be
the right y/m value"

ie rejects vfio being built-in but iommufd being a module

Jason

Re: [PATCH 01/10] vfio: Move vfio_device driver open/close code to a function

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 07:33:30AM +0000, Tian, Kevin wrote:

> > +	/*
> > +	 * Here we pass the KVM pointer with the group under the read lock.
> 
> Now the read lock is replaced by mutex. Let's correct it when moving this
> piece of code.

Done, thanks

Jason

Re: [PATCH 01/10] vfio: Move vfio_device driver open/close code to a function

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 07:33:30AM +0000, Tian, Kevin wrote:

> > +	/*
> > +	 * Here we pass the KVM pointer with the group under the read lock.
> 
> Now the read lock is replaced by mutex. Let's correct it when moving this
> piece of code.

Done, thanks

Jason

Re: [Intel-gfx] [PATCH 01/10] vfio: Move vfio_device driver open/close code to a function

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 07:33:30AM +0000, Tian, Kevin wrote:

> > +	/*
> > +	 * Here we pass the KVM pointer with the group under the read lock.
> 
> Now the read lock is replaced by mutex. Let's correct it when moving this
> piece of code.

Done, thanks

Jason

Re: [PATCH 02/10] vfio: Move vfio_device_assign_container() into vfio_device_first_open()

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 07:38:47AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:17 AM
> > 
> > +err_container:
> > +	vfio_device_unassign_container(device);
> >  err_module_put:
> >  	device->kvm = NULL;
> 
> err_module_put should be moved after nullifying device->kvm.
> 
> otherwise it looks good to me:
> 
> Reviewed-by: Kevin Tian <kevin.tian@intel.com>

Done, thanks

Jason

Re: [PATCH 02/10] vfio: Move vfio_device_assign_container() into vfio_device_first_open()

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 07:38:47AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:17 AM
> > 
> > +err_container:
> > +	vfio_device_unassign_container(device);
> >  err_module_put:
> >  	device->kvm = NULL;
> 
> err_module_put should be moved after nullifying device->kvm.
> 
> otherwise it looks good to me:
> 
> Reviewed-by: Kevin Tian <kevin.tian@intel.com>

Done, thanks

Jason

Re: [Intel-gfx] [PATCH 02/10] vfio: Move vfio_device_assign_container() into vfio_device_first_open()

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 07:38:47AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:17 AM
> > 
> > +err_container:
> > +	vfio_device_unassign_container(device);
> >  err_module_put:
> >  	device->kvm = NULL;
> 
> err_module_put should be moved after nullifying device->kvm.
> 
> otherwise it looks good to me:
> 
> Reviewed-by: Kevin Tian <kevin.tian@intel.com>

Done, thanks

Jason

Re: [PATCH 05/10] vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 07:52:23AM +0000, Tian, Kevin wrote:
> > IOMMU_CAP_ENFORCE_CACHE_COHERENCY we know what the
> > iommu_domain will be
> > capable of without having to create it. Use this to compute
> 
> it's worth noting that the prerequisite is that vfio always enforces
> cache coherency on a domain according to the iommu capability
> of the devices attached to that domain. There is no mix of attaching
> a device supporting the cap to a domain which doesn't enforce
> coherency. With that we know what the domain will be w/o having
> to create it.

OK, I added this:

VFIO always tries to upgrade domains to enforce cache coherency, it never
attaches a device that supports enforce cache coherency to a less capable
domain, so the cap test is a sufficient proxy for the ultimate
outcome. iommufd also ensures that devices that set the cap will be
connected to enforcing domains.

> > +	/*
> > +	 * If the device does not have
> > IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
> > +	 * any domain later attached to it will also not support it.
> > +	 */
> 
> also add the other part i.e. if the device does have the cap then any domain
> later attached to it will have the cap enabled. Only with both clarified
> we can safely use the device cap here.

And this:

	/*
	 * If the device does not have IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
	 * any domain later attached to it will also not support it. If the cap
	 * is set then the iommu_domain eventually attached to the device/group
	 * must must use a domain with enforce_cache_coherency().
	 */

Jason

Re: [PATCH 05/10] vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 07:52:23AM +0000, Tian, Kevin wrote:
> > IOMMU_CAP_ENFORCE_CACHE_COHERENCY we know what the
> > iommu_domain will be
> > capable of without having to create it. Use this to compute
> 
> it's worth noting that the prerequisite is that vfio always enforces
> cache coherency on a domain according to the iommu capability
> of the devices attached to that domain. There is no mix of attaching
> a device supporting the cap to a domain which doesn't enforce
> coherency. With that we know what the domain will be w/o having
> to create it.

OK, I added this:

VFIO always tries to upgrade domains to enforce cache coherency, it never
attaches a device that supports enforce cache coherency to a less capable
domain, so the cap test is a sufficient proxy for the ultimate
outcome. iommufd also ensures that devices that set the cap will be
connected to enforcing domains.

> > +	/*
> > +	 * If the device does not have
> > IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
> > +	 * any domain later attached to it will also not support it.
> > +	 */
> 
> also add the other part i.e. if the device does have the cap then any domain
> later attached to it will have the cap enabled. Only with both clarified
> we can safely use the device cap here.

And this:

	/*
	 * If the device does not have IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
	 * any domain later attached to it will also not support it. If the cap
	 * is set then the iommu_domain eventually attached to the device/group
	 * must must use a domain with enforce_cache_coherency().
	 */

Jason

Re: [Intel-gfx] [PATCH 05/10] vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 07:52:23AM +0000, Tian, Kevin wrote:
> > IOMMU_CAP_ENFORCE_CACHE_COHERENCY we know what the
> > iommu_domain will be
> > capable of without having to create it. Use this to compute
> 
> it's worth noting that the prerequisite is that vfio always enforces
> cache coherency on a domain according to the iommu capability
> of the devices attached to that domain. There is no mix of attaching
> a device supporting the cap to a domain which doesn't enforce
> coherency. With that we know what the domain will be w/o having
> to create it.

OK, I added this:

VFIO always tries to upgrade domains to enforce cache coherency, it never
attaches a device that supports enforce cache coherency to a less capable
domain, so the cap test is a sufficient proxy for the ultimate
outcome. iommufd also ensures that devices that set the cap will be
connected to enforcing domains.

> > +	/*
> > +	 * If the device does not have
> > IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
> > +	 * any domain later attached to it will also not support it.
> > +	 */
> 
> also add the other part i.e. if the device does have the cap then any domain
> later attached to it will have the cap enabled. Only with both clarified
> we can safely use the device cap here.

And this:

	/*
	 * If the device does not have IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
	 * any domain later attached to it will also not support it. If the cap
	 * is set then the iommu_domain eventually attached to the device/group
	 * must must use a domain with enforce_cache_coherency().
	 */

Jason

Re: [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 08:09:52AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:51 AM
> >
> >  menuconfig VFIO
> >  	tristate "VFIO Non-Privileged userspace driver framework"
> >  	select IOMMU_API
> > +	depends on IOMMUFD || !IOMMUFD
> 
> Out of curiosity. What is the meaning of this dependency claim?
> 
> > @@ -717,12 +735,23 @@ static int vfio_group_ioctl_set_container(struct
> > vfio_group *group,
> >  	}
> > 
> >  	container = vfio_container_from_file(f.file);
> > -	ret = -EINVAL;
> 
> this changes the errno from -EINVAL to -EBADF for the original container
> path. Is it desired?

Yes, EBADFD is the right error code (it is a typo it was EBADF)

> >  	if (container) {
> >  		ret = vfio_container_attach_group(container, group);
> >  		goto out_unlock;
> >  	}
> > 
> > +	iommufd = iommufd_ctx_from_file(f.file);
> > +	if (!IS_ERR(iommufd)) {
> 
> The only errno which iommufd_ctx_from_file() may return is -EBADFD
> which duplicates with -EBADF assignment in following line.

The concept is that other places using iommufd_ctx_from_file() should
forward the return code directly. vfio is probably the only thing that
is going to be multiplexing like this.

> > +		u32 ioas_id;
> > +
> > +		group->iommufd = iommufd;
> > +		ret = iommufd_vfio_compat_ioas_id(iommufd, &ioas_id);
> 
> exchange the order of above two lines and only assign group->iommufd
> when the compat call succeeds.

Yeah, that is probably a small bug:

-               group->iommufd = iommufd;
                ret = iommufd_vfio_compat_ioas_id(iommufd, &ioas_id);
+               if (ret) {
+                       iommufd_ctx_put(group->iommufd);
+                       goto out_unlock;
+               }
+
+               group->iommufd = iommufd;
                goto out_unlock;


> > @@ -900,7 +940,7 @@ static int vfio_group_ioctl_get_status(struct
> > vfio_group *group,
> >  		return -ENODEV;
> >  	}
> > 
> > -	if (group->container)
> > +	if (group->container || group->iommufd)
> >  		status.flags |= VFIO_GROUP_FLAGS_CONTAINER_SET |
> >  				VFIO_GROUP_FLAGS_VIABLE;
> 
> Copy some explanation from commit msg to explain the subtle difference
> between container and iommufd.

	/*
 	 * With the container FD the iommu_group_claim_dma_owner() is done
	 * during SET_CONTAINER but for IOMMFD this is done during
	 * VFIO_GROUP_GET_DEVICE_FD. Meaning that with iommufd
	 * VFIO_GROUP_FLAGS_VIABLE could be set but GET_DEVICE_FD will fail due
	 * to viability.
	 */

Thanks,
Jason

Re: [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 08:09:52AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:51 AM
> >
> >  menuconfig VFIO
> >  	tristate "VFIO Non-Privileged userspace driver framework"
> >  	select IOMMU_API
> > +	depends on IOMMUFD || !IOMMUFD
> 
> Out of curiosity. What is the meaning of this dependency claim?
> 
> > @@ -717,12 +735,23 @@ static int vfio_group_ioctl_set_container(struct
> > vfio_group *group,
> >  	}
> > 
> >  	container = vfio_container_from_file(f.file);
> > -	ret = -EINVAL;
> 
> this changes the errno from -EINVAL to -EBADF for the original container
> path. Is it desired?

Yes, EBADFD is the right error code (it is a typo it was EBADF)

> >  	if (container) {
> >  		ret = vfio_container_attach_group(container, group);
> >  		goto out_unlock;
> >  	}
> > 
> > +	iommufd = iommufd_ctx_from_file(f.file);
> > +	if (!IS_ERR(iommufd)) {
> 
> The only errno which iommufd_ctx_from_file() may return is -EBADFD
> which duplicates with -EBADF assignment in following line.

The concept is that other places using iommufd_ctx_from_file() should
forward the return code directly. vfio is probably the only thing that
is going to be multiplexing like this.

> > +		u32 ioas_id;
> > +
> > +		group->iommufd = iommufd;
> > +		ret = iommufd_vfio_compat_ioas_id(iommufd, &ioas_id);
> 
> exchange the order of above two lines and only assign group->iommufd
> when the compat call succeeds.

Yeah, that is probably a small bug:

-               group->iommufd = iommufd;
                ret = iommufd_vfio_compat_ioas_id(iommufd, &ioas_id);
+               if (ret) {
+                       iommufd_ctx_put(group->iommufd);
+                       goto out_unlock;
+               }
+
+               group->iommufd = iommufd;
                goto out_unlock;


> > @@ -900,7 +940,7 @@ static int vfio_group_ioctl_get_status(struct
> > vfio_group *group,
> >  		return -ENODEV;
> >  	}
> > 
> > -	if (group->container)
> > +	if (group->container || group->iommufd)
> >  		status.flags |= VFIO_GROUP_FLAGS_CONTAINER_SET |
> >  				VFIO_GROUP_FLAGS_VIABLE;
> 
> Copy some explanation from commit msg to explain the subtle difference
> between container and iommufd.

	/*
 	 * With the container FD the iommu_group_claim_dma_owner() is done
	 * during SET_CONTAINER but for IOMMFD this is done during
	 * VFIO_GROUP_GET_DEVICE_FD. Meaning that with iommufd
	 * VFIO_GROUP_FLAGS_VIABLE could be set but GET_DEVICE_FD will fail due
	 * to viability.
	 */

Thanks,
Jason

Re: [Intel-gfx] [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 08:09:52AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:51 AM
> >
> >  menuconfig VFIO
> >  	tristate "VFIO Non-Privileged userspace driver framework"
> >  	select IOMMU_API
> > +	depends on IOMMUFD || !IOMMUFD
> 
> Out of curiosity. What is the meaning of this dependency claim?
> 
> > @@ -717,12 +735,23 @@ static int vfio_group_ioctl_set_container(struct
> > vfio_group *group,
> >  	}
> > 
> >  	container = vfio_container_from_file(f.file);
> > -	ret = -EINVAL;
> 
> this changes the errno from -EINVAL to -EBADF for the original container
> path. Is it desired?

Yes, EBADFD is the right error code (it is a typo it was EBADF)

> >  	if (container) {
> >  		ret = vfio_container_attach_group(container, group);
> >  		goto out_unlock;
> >  	}
> > 
> > +	iommufd = iommufd_ctx_from_file(f.file);
> > +	if (!IS_ERR(iommufd)) {
> 
> The only errno which iommufd_ctx_from_file() may return is -EBADFD
> which duplicates with -EBADF assignment in following line.

The concept is that other places using iommufd_ctx_from_file() should
forward the return code directly. vfio is probably the only thing that
is going to be multiplexing like this.

> > +		u32 ioas_id;
> > +
> > +		group->iommufd = iommufd;
> > +		ret = iommufd_vfio_compat_ioas_id(iommufd, &ioas_id);
> 
> exchange the order of above two lines and only assign group->iommufd
> when the compat call succeeds.

Yeah, that is probably a small bug:

-               group->iommufd = iommufd;
                ret = iommufd_vfio_compat_ioas_id(iommufd, &ioas_id);
+               if (ret) {
+                       iommufd_ctx_put(group->iommufd);
+                       goto out_unlock;
+               }
+
+               group->iommufd = iommufd;
                goto out_unlock;


> > @@ -900,7 +940,7 @@ static int vfio_group_ioctl_get_status(struct
> > vfio_group *group,
> >  		return -ENODEV;
> >  	}
> > 
> > -	if (group->container)
> > +	if (group->container || group->iommufd)
> >  		status.flags |= VFIO_GROUP_FLAGS_CONTAINER_SET |
> >  				VFIO_GROUP_FLAGS_VIABLE;
> 
> Copy some explanation from commit msg to explain the subtle difference
> between container and iommufd.

	/*
 	 * With the container FD the iommu_group_claim_dma_owner() is done
	 * during SET_CONTAINER but for IOMMFD this is done during
	 * VFIO_GROUP_GET_DEVICE_FD. Meaning that with iommufd
	 * VFIO_GROUP_FLAGS_VIABLE could be set but GET_DEVICE_FD will fail due
	 * to viability.
	 */

Thanks,
Jason

Re: [PATCH 08/10] vfio-iommufd: Support iommufd for emulated VFIO devices

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 08:37:39AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:51 AM
> > 
> > Emulated VFIO devices are calling vfio_register_emulated_iommu_dev() and
> > consist of all the mdev drivers.
> > 
> > Like the physical drivers, support for iommufd is provided by the driver
> > supplying the correct correct standard ops. Provide ops from the core that
> > duplicate what vfio_register_emulated_iommu_dev() does.
> > 
> > Emulated drivers are where it is more likely to see variation in the
> > iommfd support ops. For instance IDXD will probably need to setup both a
> > iommfd_device context linked to a PASID and an iommufd_access context to
> > support all their mdev operations.
> > 
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> >  drivers/gpu/drm/i915/gvt/kvmgt.c  |   3 +
> >  drivers/s390/cio/vfio_ccw_ops.c   |   3 +
> >  drivers/s390/crypto/vfio_ap_ops.c |   3 +
> >  drivers/vfio/container.c          | 108 ++++++-----------------------
> >  drivers/vfio/iommufd.c            |  57 ++++++++++++++++
> >  drivers/vfio/vfio.h               |  10 ++-
> >  drivers/vfio/vfio_main.c          | 110 +++++++++++++++++++++++++++++-
> >  include/linux/vfio.h              |  14 ++++
> >  8 files changed, 217 insertions(+), 91 deletions(-)
> 
> mtty, mdpy and mbochs?

They don't call rw or pin_pages, so they don't need to do
anything:


	/*
	 * If the driver doesn't provide this op then it means the device does
	 * not do DMA at all. So nothing to do.
	 */
	if (!vdev->ops->bind_iommufd)
		return 0;

> > +int vfio_container_pin_pages(struct vfio_container *container,
> > +			     struct iommu_group *iommu_group, dma_addr_t
> > iova,
> > +			     int npage, int prot, struct page **pages)
> >  {
> > -	struct vfio_container *container;
> > -	struct vfio_group *group = device->group;
> > -	struct vfio_iommu_driver *driver;
> > -	int ret;
> > -
> > -	if (!pages || !npage || !vfio_assert_device_open(device))
> > -		return -EINVAL;
> > +	/* group->container cannot change while a vfio device is open */
> > +	struct vfio_iommu_driver *driver = container->iommu_driver;
> > 
> >  	if (npage > VFIO_PIN_PAGES_MAX_ENTRIES)
> >  		return -E2BIG;
> > 
> >  	/* group->container cannot change while a vfio device is open */
> > -	container = group->container;
> >  	driver = container->iommu_driver;
> 
> duplicated comment and assignment.
> 
> Actually, I'm not sure whether the comment should be put within this
> container helper and other two. There is no group reference in these
> helpers then it sounds like the comment makes more sense to be in the
> caller side?

Yeah, that is better

> > +void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova, int
> > npage)
> > +{
> > +	if (WARN_ON(!vfio_assert_device_open(device)))
> > +		return;
> > +
> > +	if (device->group->container) {
> > +		vfio_container_unpin_pages(device->group->container, iova,
> > +					   npage);
> > +	} else if (device->iommufd_access) {
> 
> be consistent with other two helpers i.e. if-if instead of if-else

Done

> > +		if (WARN_ON(iova > ULONG_MAX))
> > +			return;
> 
> Is there a reason why this is a WARN_ON only in unpin but not in pin?

This is how it has always been. I suppose someone once thought it
would be OK for the driver to do racy stuff during pin - but clearly
that is not the case. Lets fix it while we are here.

> > +int vfio_dma_rw(struct vfio_device *device, dma_addr_t iova, void *data,
> > +		size_t len, bool write)
> > +{
> > +	if (!data || len <= 0 || !vfio_assert_device_open(device))
> > +		return -EINVAL;
> > +
> > +	if (device->group->container)
> > +		return vfio_container_dma_rw(device->group->container,
> > iova,
> > +					     data, len, write);
> > +
> > +	if (device->iommufd_access) {
> > +		unsigned int flags = 0;
> > +
> > +		if (iova > ULONG_MAX)
> > +			return -EINVAL;
> > +
> > +		/* VFIO historically tries to auto-detect a kthread */
> > +		if (!current->mm)
> > +			flags |= IOMMUFD_ACCESS_RW_KTHREAD;
> 
> Can you elaborate why this cannot be put in iommufd as the default
> policy similar to what vfio container does?

Snooping in kernel structs to try to guess the calling execution
context is bad design. The caller should know its own context and it
should declare positively what it is. Someday this should be lifted
out of VFIO as well and into the drivers.

Jason

Re: [PATCH 08/10] vfio-iommufd: Support iommufd for emulated VFIO devices

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 08:37:39AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:51 AM
> > 
> > Emulated VFIO devices are calling vfio_register_emulated_iommu_dev() and
> > consist of all the mdev drivers.
> > 
> > Like the physical drivers, support for iommufd is provided by the driver
> > supplying the correct correct standard ops. Provide ops from the core that
> > duplicate what vfio_register_emulated_iommu_dev() does.
> > 
> > Emulated drivers are where it is more likely to see variation in the
> > iommfd support ops. For instance IDXD will probably need to setup both a
> > iommfd_device context linked to a PASID and an iommufd_access context to
> > support all their mdev operations.
> > 
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> >  drivers/gpu/drm/i915/gvt/kvmgt.c  |   3 +
> >  drivers/s390/cio/vfio_ccw_ops.c   |   3 +
> >  drivers/s390/crypto/vfio_ap_ops.c |   3 +
> >  drivers/vfio/container.c          | 108 ++++++-----------------------
> >  drivers/vfio/iommufd.c            |  57 ++++++++++++++++
> >  drivers/vfio/vfio.h               |  10 ++-
> >  drivers/vfio/vfio_main.c          | 110 +++++++++++++++++++++++++++++-
> >  include/linux/vfio.h              |  14 ++++
> >  8 files changed, 217 insertions(+), 91 deletions(-)
> 
> mtty, mdpy and mbochs?

They don't call rw or pin_pages, so they don't need to do
anything:


	/*
	 * If the driver doesn't provide this op then it means the device does
	 * not do DMA at all. So nothing to do.
	 */
	if (!vdev->ops->bind_iommufd)
		return 0;

> > +int vfio_container_pin_pages(struct vfio_container *container,
> > +			     struct iommu_group *iommu_group, dma_addr_t
> > iova,
> > +			     int npage, int prot, struct page **pages)
> >  {
> > -	struct vfio_container *container;
> > -	struct vfio_group *group = device->group;
> > -	struct vfio_iommu_driver *driver;
> > -	int ret;
> > -
> > -	if (!pages || !npage || !vfio_assert_device_open(device))
> > -		return -EINVAL;
> > +	/* group->container cannot change while a vfio device is open */
> > +	struct vfio_iommu_driver *driver = container->iommu_driver;
> > 
> >  	if (npage > VFIO_PIN_PAGES_MAX_ENTRIES)
> >  		return -E2BIG;
> > 
> >  	/* group->container cannot change while a vfio device is open */
> > -	container = group->container;
> >  	driver = container->iommu_driver;
> 
> duplicated comment and assignment.
> 
> Actually, I'm not sure whether the comment should be put within this
> container helper and other two. There is no group reference in these
> helpers then it sounds like the comment makes more sense to be in the
> caller side?

Yeah, that is better

> > +void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova, int
> > npage)
> > +{
> > +	if (WARN_ON(!vfio_assert_device_open(device)))
> > +		return;
> > +
> > +	if (device->group->container) {
> > +		vfio_container_unpin_pages(device->group->container, iova,
> > +					   npage);
> > +	} else if (device->iommufd_access) {
> 
> be consistent with other two helpers i.e. if-if instead of if-else

Done

> > +		if (WARN_ON(iova > ULONG_MAX))
> > +			return;
> 
> Is there a reason why this is a WARN_ON only in unpin but not in pin?

This is how it has always been. I suppose someone once thought it
would be OK for the driver to do racy stuff during pin - but clearly
that is not the case. Lets fix it while we are here.

> > +int vfio_dma_rw(struct vfio_device *device, dma_addr_t iova, void *data,
> > +		size_t len, bool write)
> > +{
> > +	if (!data || len <= 0 || !vfio_assert_device_open(device))
> > +		return -EINVAL;
> > +
> > +	if (device->group->container)
> > +		return vfio_container_dma_rw(device->group->container,
> > iova,
> > +					     data, len, write);
> > +
> > +	if (device->iommufd_access) {
> > +		unsigned int flags = 0;
> > +
> > +		if (iova > ULONG_MAX)
> > +			return -EINVAL;
> > +
> > +		/* VFIO historically tries to auto-detect a kthread */
> > +		if (!current->mm)
> > +			flags |= IOMMUFD_ACCESS_RW_KTHREAD;
> 
> Can you elaborate why this cannot be put in iommufd as the default
> policy similar to what vfio container does?

Snooping in kernel structs to try to guess the calling execution
context is bad design. The caller should know its own context and it
should declare positively what it is. Someday this should be lifted
out of VFIO as well and into the drivers.

Jason

Re: [Intel-gfx] [PATCH 08/10] vfio-iommufd: Support iommufd for emulated VFIO devices

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 08:37:39AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:51 AM
> > 
> > Emulated VFIO devices are calling vfio_register_emulated_iommu_dev() and
> > consist of all the mdev drivers.
> > 
> > Like the physical drivers, support for iommufd is provided by the driver
> > supplying the correct correct standard ops. Provide ops from the core that
> > duplicate what vfio_register_emulated_iommu_dev() does.
> > 
> > Emulated drivers are where it is more likely to see variation in the
> > iommfd support ops. For instance IDXD will probably need to setup both a
> > iommfd_device context linked to a PASID and an iommufd_access context to
> > support all their mdev operations.
> > 
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> >  drivers/gpu/drm/i915/gvt/kvmgt.c  |   3 +
> >  drivers/s390/cio/vfio_ccw_ops.c   |   3 +
> >  drivers/s390/crypto/vfio_ap_ops.c |   3 +
> >  drivers/vfio/container.c          | 108 ++++++-----------------------
> >  drivers/vfio/iommufd.c            |  57 ++++++++++++++++
> >  drivers/vfio/vfio.h               |  10 ++-
> >  drivers/vfio/vfio_main.c          | 110 +++++++++++++++++++++++++++++-
> >  include/linux/vfio.h              |  14 ++++
> >  8 files changed, 217 insertions(+), 91 deletions(-)
> 
> mtty, mdpy and mbochs?

They don't call rw or pin_pages, so they don't need to do
anything:


	/*
	 * If the driver doesn't provide this op then it means the device does
	 * not do DMA at all. So nothing to do.
	 */
	if (!vdev->ops->bind_iommufd)
		return 0;

> > +int vfio_container_pin_pages(struct vfio_container *container,
> > +			     struct iommu_group *iommu_group, dma_addr_t
> > iova,
> > +			     int npage, int prot, struct page **pages)
> >  {
> > -	struct vfio_container *container;
> > -	struct vfio_group *group = device->group;
> > -	struct vfio_iommu_driver *driver;
> > -	int ret;
> > -
> > -	if (!pages || !npage || !vfio_assert_device_open(device))
> > -		return -EINVAL;
> > +	/* group->container cannot change while a vfio device is open */
> > +	struct vfio_iommu_driver *driver = container->iommu_driver;
> > 
> >  	if (npage > VFIO_PIN_PAGES_MAX_ENTRIES)
> >  		return -E2BIG;
> > 
> >  	/* group->container cannot change while a vfio device is open */
> > -	container = group->container;
> >  	driver = container->iommu_driver;
> 
> duplicated comment and assignment.
> 
> Actually, I'm not sure whether the comment should be put within this
> container helper and other two. There is no group reference in these
> helpers then it sounds like the comment makes more sense to be in the
> caller side?

Yeah, that is better

> > +void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova, int
> > npage)
> > +{
> > +	if (WARN_ON(!vfio_assert_device_open(device)))
> > +		return;
> > +
> > +	if (device->group->container) {
> > +		vfio_container_unpin_pages(device->group->container, iova,
> > +					   npage);
> > +	} else if (device->iommufd_access) {
> 
> be consistent with other two helpers i.e. if-if instead of if-else

Done

> > +		if (WARN_ON(iova > ULONG_MAX))
> > +			return;
> 
> Is there a reason why this is a WARN_ON only in unpin but not in pin?

This is how it has always been. I suppose someone once thought it
would be OK for the driver to do racy stuff during pin - but clearly
that is not the case. Lets fix it while we are here.

> > +int vfio_dma_rw(struct vfio_device *device, dma_addr_t iova, void *data,
> > +		size_t len, bool write)
> > +{
> > +	if (!data || len <= 0 || !vfio_assert_device_open(device))
> > +		return -EINVAL;
> > +
> > +	if (device->group->container)
> > +		return vfio_container_dma_rw(device->group->container,
> > iova,
> > +					     data, len, write);
> > +
> > +	if (device->iommufd_access) {
> > +		unsigned int flags = 0;
> > +
> > +		if (iova > ULONG_MAX)
> > +			return -EINVAL;
> > +
> > +		/* VFIO historically tries to auto-detect a kthread */
> > +		if (!current->mm)
> > +			flags |= IOMMUFD_ACCESS_RW_KTHREAD;
> 
> Can you elaborate why this cannot be put in iommufd as the default
> policy similar to what vfio container does?

Snooping in kernel structs to try to guess the calling execution
context is bad design. The caller should know its own context and it
should declare positively what it is. Someday this should be lifted
out of VFIO as well and into the drivers.

Jason

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Yi Liu]

On 2022/11/1 12:21, Nicolin Chen wrote:
> On Tue, Nov 01, 2022 at 11:04:38AM +0800, Yi Liu wrote:
>> On 2022/11/1 07:24, Jason Gunthorpe wrote:
>>> On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
>>>>> There is something wrong with the test suite that it isn't covering
>>>>> the above, I'm going to look into that today.
>>>>
>>>> sounds to be the cause. I didn't see any significant change in vfio_main.c
>>>> that may fail gvt. So should the iommufd changes. Then we will re-run the
>>>> test after your update.:-)
>>>
>>> I updated the github with all the changes made so far, it is worth
>>> trying again!
>>
>> gvt is still failing with below call trace in host side. vfio_unpin_pages()
>> is still in problem. Any idea on it?
> 
>> [  206.464318] WARNING: CPU: 9 PID: 3362 at
>> drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
> 
> Judging from this WARNING, and since gvt (mdev) needs pin_pages(),
> I assume this might be a fix, though Jason's latest change for the
> iova_alignment seems to be added for CONFIG_IOMMUFD_TEST only.
> 
> ------
> diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
> index 72a289c5f8c9..185075528d5e 100644
> --- a/drivers/vfio/iommufd.c
> +++ b/drivers/vfio/iommufd.c
> @@ -120,6 +120,7 @@ static void vfio_emulated_unmap(void *data, unsigned long iova,
>   }
>   
>   static const struct iommufd_access_ops vfio_user_ops = {
> +	.needs_pin_pages = 1,
>   	.unmap = vfio_emulated_unmap,
>   };
>   
> ------
> 
> Perhaps you can try it first to see if we can test the rest part of
> the routine for now, till Jason acks tomorrow.

fyi. it works so far. :-)

-- 
Regards,
Yi Liu

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Yi Liu]

On 2022/11/1 12:21, Nicolin Chen wrote:
> On Tue, Nov 01, 2022 at 11:04:38AM +0800, Yi Liu wrote:
>> On 2022/11/1 07:24, Jason Gunthorpe wrote:
>>> On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
>>>>> There is something wrong with the test suite that it isn't covering
>>>>> the above, I'm going to look into that today.
>>>>
>>>> sounds to be the cause. I didn't see any significant change in vfio_main.c
>>>> that may fail gvt. So should the iommufd changes. Then we will re-run the
>>>> test after your update.:-)
>>>
>>> I updated the github with all the changes made so far, it is worth
>>> trying again!
>>
>> gvt is still failing with below call trace in host side. vfio_unpin_pages()
>> is still in problem. Any idea on it?
> 
>> [  206.464318] WARNING: CPU: 9 PID: 3362 at
>> drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
> 
> Judging from this WARNING, and since gvt (mdev) needs pin_pages(),
> I assume this might be a fix, though Jason's latest change for the
> iova_alignment seems to be added for CONFIG_IOMMUFD_TEST only.
> 
> ------
> diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
> index 72a289c5f8c9..185075528d5e 100644
> --- a/drivers/vfio/iommufd.c
> +++ b/drivers/vfio/iommufd.c
> @@ -120,6 +120,7 @@ static void vfio_emulated_unmap(void *data, unsigned long iova,
>   }
>   
>   static const struct iommufd_access_ops vfio_user_ops = {
> +	.needs_pin_pages = 1,
>   	.unmap = vfio_emulated_unmap,
>   };
>   
> ------
> 
> Perhaps you can try it first to see if we can test the rest part of
> the routine for now, till Jason acks tomorrow.

fyi. it works so far. :-)

-- 
Regards,
Yi Liu

Re: [Intel-gfx] [PATCH 00/10] Connect VFIO to IOMMUFD

[Yi Liu]

On 2022/11/1 12:21, Nicolin Chen wrote:
> On Tue, Nov 01, 2022 at 11:04:38AM +0800, Yi Liu wrote:
>> On 2022/11/1 07:24, Jason Gunthorpe wrote:
>>> On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
>>>>> There is something wrong with the test suite that it isn't covering
>>>>> the above, I'm going to look into that today.
>>>>
>>>> sounds to be the cause. I didn't see any significant change in vfio_main.c
>>>> that may fail gvt. So should the iommufd changes. Then we will re-run the
>>>> test after your update.:-)
>>>
>>> I updated the github with all the changes made so far, it is worth
>>> trying again!
>>
>> gvt is still failing with below call trace in host side. vfio_unpin_pages()
>> is still in problem. Any idea on it?
> 
>> [  206.464318] WARNING: CPU: 9 PID: 3362 at
>> drivers/iommu/iommufd/device.c:591 iommufd_access_pin_pages+0x337/0x360
> 
> Judging from this WARNING, and since gvt (mdev) needs pin_pages(),
> I assume this might be a fix, though Jason's latest change for the
> iova_alignment seems to be added for CONFIG_IOMMUFD_TEST only.
> 
> ------
> diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
> index 72a289c5f8c9..185075528d5e 100644
> --- a/drivers/vfio/iommufd.c
> +++ b/drivers/vfio/iommufd.c
> @@ -120,6 +120,7 @@ static void vfio_emulated_unmap(void *data, unsigned long iova,
>   }
>   
>   static const struct iommufd_access_ops vfio_user_ops = {
> +	.needs_pin_pages = 1,
>   	.unmap = vfio_emulated_unmap,
>   };
>   
> ------
> 
> Perhaps you can try it first to see if we can test the rest part of
> the routine for now, till Jason acks tomorrow.

fyi. it works so far. :-)

-- 
Regards,
Yi Liu

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Yi Liu]

On 2022/11/1 19:41, Jason Gunthorpe wrote:
> On Tue, Nov 01, 2022 at 11:04:38AM +0800, Yi Liu wrote:
>> On 2022/11/1 07:24, Jason Gunthorpe wrote:
>>> On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
>>>>> There is something wrong with the test suite that it isn't covering
>>>>> the above, I'm going to look into that today.
>>>>
>>>> sounds to be the cause. I didn't see any significant change in vfio_main.c
>>>> that may fail gvt. So should the iommufd changes. Then we will re-run the
>>>> test after your update.:-)
>>>
>>> I updated the github with all the changes made so far, it is worth
>>> trying again!
>>
>> gvt is still failing with below call trace in host side. vfio_unpin_pages()
>> is still in problem. Any idea on it?
> 
> Oh, this is my mistake, I rushed a bit getting updated branches:
> 
> diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
> index 40eb6931ab2321..29e7b1fdd0cd4a 100644
> --- a/drivers/vfio/iommufd.c
> +++ b/drivers/vfio/iommufd.c
> @@ -118,6 +118,7 @@ static void vfio_emulated_unmap(void *data, unsigned long iova,
>   }
>   
>   static const struct iommufd_access_ops vfio_user_ops = {
> +	.needs_pin_pages = 1,
>   	.unmap = vfio_emulated_unmap,
>   };

yes, it is. The call trace goes away. my colleague is running gvt full test 
now.

-- 
Regards,
Yi Liu

Re: [Intel-gfx] [PATCH 00/10] Connect VFIO to IOMMUFD

[Yi Liu]

On 2022/11/1 19:41, Jason Gunthorpe wrote:
> On Tue, Nov 01, 2022 at 11:04:38AM +0800, Yi Liu wrote:
>> On 2022/11/1 07:24, Jason Gunthorpe wrote:
>>> On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
>>>>> There is something wrong with the test suite that it isn't covering
>>>>> the above, I'm going to look into that today.
>>>>
>>>> sounds to be the cause. I didn't see any significant change in vfio_main.c
>>>> that may fail gvt. So should the iommufd changes. Then we will re-run the
>>>> test after your update.:-)
>>>
>>> I updated the github with all the changes made so far, it is worth
>>> trying again!
>>
>> gvt is still failing with below call trace in host side. vfio_unpin_pages()
>> is still in problem. Any idea on it?
> 
> Oh, this is my mistake, I rushed a bit getting updated branches:
> 
> diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
> index 40eb6931ab2321..29e7b1fdd0cd4a 100644
> --- a/drivers/vfio/iommufd.c
> +++ b/drivers/vfio/iommufd.c
> @@ -118,6 +118,7 @@ static void vfio_emulated_unmap(void *data, unsigned long iova,
>   }
>   
>   static const struct iommufd_access_ops vfio_user_ops = {
> +	.needs_pin_pages = 1,
>   	.unmap = vfio_emulated_unmap,
>   };

yes, it is. The call trace goes away. my colleague is running gvt full test 
now.

-- 
Regards,
Yi Liu

Re: [PATCH 00/10] Connect VFIO to IOMMUFD

[Yi Liu]

On 2022/11/1 19:41, Jason Gunthorpe wrote:
> On Tue, Nov 01, 2022 at 11:04:38AM +0800, Yi Liu wrote:
>> On 2022/11/1 07:24, Jason Gunthorpe wrote:
>>> On Mon, Oct 31, 2022 at 08:25:39PM +0800, Yi Liu wrote:
>>>>> There is something wrong with the test suite that it isn't covering
>>>>> the above, I'm going to look into that today.
>>>>
>>>> sounds to be the cause. I didn't see any significant change in vfio_main.c
>>>> that may fail gvt. So should the iommufd changes. Then we will re-run the
>>>> test after your update.:-)
>>>
>>> I updated the github with all the changes made so far, it is worth
>>> trying again!
>>
>> gvt is still failing with below call trace in host side. vfio_unpin_pages()
>> is still in problem. Any idea on it?
> 
> Oh, this is my mistake, I rushed a bit getting updated branches:
> 
> diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c
> index 40eb6931ab2321..29e7b1fdd0cd4a 100644
> --- a/drivers/vfio/iommufd.c
> +++ b/drivers/vfio/iommufd.c
> @@ -118,6 +118,7 @@ static void vfio_emulated_unmap(void *data, unsigned long iova,
>   }
>   
>   static const struct iommufd_access_ops vfio_user_ops = {
> +	.needs_pin_pages = 1,
>   	.unmap = vfio_emulated_unmap,
>   };

yes, it is. The call trace goes away. my colleague is running gvt full test 
now.

-- 
Regards,
Yi Liu

Re: [PATCH 09/10] vfio: Make vfio_container optionally compiled

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 08:41:56AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:51 AM
> > 
> >  if VFIO
> > +config VFIO_CONTAINER
> > +	bool "Support for the VFIO container /dev/vfio/vfio"
> > +	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM ||
> > ARM64)
> > +	default y
> > +	help
> > +	  The VFIO container is the classic interface to VFIO for establishing
> > +	  mappings. If N is selected here then IOMMUFD must be used the
> 
> establishing IOMMU mappings.
> 
> s/used the manage/used to manage/

Done

> > +if VFIO_CONTAINER
> >  config VFIO_IOMMU_TYPE1
> >  	tristate
> > -	default n
> > +	default MMU && (X86 || S390 || ARM || ARM64)
> 
> there are already same conditions for selecting TYPE1 from
> VFIO_CONTAINER. what does duplicating conditions here
> bring to us?

Yah, we can leave this out - this is just the more normal way to
approach this kconfig trick, AFAICT

Thanks,
Jason

Re: [PATCH 09/10] vfio: Make vfio_container optionally compiled

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 08:41:56AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:51 AM
> > 
> >  if VFIO
> > +config VFIO_CONTAINER
> > +	bool "Support for the VFIO container /dev/vfio/vfio"
> > +	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM ||
> > ARM64)
> > +	default y
> > +	help
> > +	  The VFIO container is the classic interface to VFIO for establishing
> > +	  mappings. If N is selected here then IOMMUFD must be used the
> 
> establishing IOMMU mappings.
> 
> s/used the manage/used to manage/

Done

> > +if VFIO_CONTAINER
> >  config VFIO_IOMMU_TYPE1
> >  	tristate
> > -	default n
> > +	default MMU && (X86 || S390 || ARM || ARM64)
> 
> there are already same conditions for selecting TYPE1 from
> VFIO_CONTAINER. what does duplicating conditions here
> bring to us?

Yah, we can leave this out - this is just the more normal way to
approach this kconfig trick, AFAICT

Thanks,
Jason

Re: [Intel-gfx] [PATCH 09/10] vfio: Make vfio_container optionally compiled

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 08:41:56AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:51 AM
> > 
> >  if VFIO
> > +config VFIO_CONTAINER
> > +	bool "Support for the VFIO container /dev/vfio/vfio"
> > +	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM ||
> > ARM64)
> > +	default y
> > +	help
> > +	  The VFIO container is the classic interface to VFIO for establishing
> > +	  mappings. If N is selected here then IOMMUFD must be used the
> 
> establishing IOMMU mappings.
> 
> s/used the manage/used to manage/

Done

> > +if VFIO_CONTAINER
> >  config VFIO_IOMMU_TYPE1
> >  	tristate
> > -	default n
> > +	default MMU && (X86 || S390 || ARM || ARM64)
> 
> there are already same conditions for selecting TYPE1 from
> VFIO_CONTAINER. what does duplicating conditions here
> bring to us?

Yah, we can leave this out - this is just the more normal way to
approach this kconfig trick, AFAICT

Thanks,
Jason

Re: [PATCH 01/10] vfio: Move vfio_device driver open/close code to a function

[Yi Liu]

On 2022/10/26 02:17, Jason Gunthorpe wrote:
> This error unwind is getting complicated. Move all the code into two
> pair'd function. The functions should be called when the open_count == 1
> after incrementing/before decrementing.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>   drivers/vfio/vfio_main.c | 95 ++++++++++++++++++++++------------------
>   1 file changed, 53 insertions(+), 42 deletions(-)

Reviewed-by: Yi Liu <yi.l.liu@intel.com>

> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index 2d168793d4e1ce..d043383fc3ba2b 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -734,6 +734,51 @@ bool vfio_assert_device_open(struct vfio_device *device)
>   	return !WARN_ON_ONCE(!READ_ONCE(device->open_count));
>   }
>   
> +static int vfio_device_first_open(struct vfio_device *device)
> +{
> +	int ret;
> +
> +	lockdep_assert_held(&device->dev_set->lock);
> +
> +	if (!try_module_get(device->dev->driver->owner))
> +		return -ENODEV;
> +
> +	/*
> +	 * Here we pass the KVM pointer with the group under the read lock.  If
> +	 * the device driver will use it, it must obtain a reference and release
> +	 * it during close_device.
> +	 */
> +	mutex_lock(&device->group->group_lock);
> +	device->kvm = device->group->kvm;
> +	if (device->ops->open_device) {
> +		ret = device->ops->open_device(device);
> +		if (ret)
> +			goto err_module_put;
> +	}
> +	vfio_device_container_register(device);
> +	mutex_unlock(&device->group->group_lock);
> +	return 0;
> +
> +err_module_put:
> +	device->kvm = NULL;
> +	mutex_unlock(&device->group->group_lock);
> +	module_put(device->dev->driver->owner);
> +	return ret;
> +}
> +
> +static void vfio_device_last_close(struct vfio_device *device)
> +{
> +	lockdep_assert_held(&device->dev_set->lock);
> +
> +	mutex_lock(&device->group->group_lock);
> +	vfio_device_container_unregister(device);
> +	if (device->ops->close_device)
> +		device->ops->close_device(device);
> +	device->kvm = NULL;
> +	mutex_unlock(&device->group->group_lock);
> +	module_put(device->dev->driver->owner);
> +}
> +
>   static struct file *vfio_device_open(struct vfio_device *device)
>   {
>   	struct file *filep;
> @@ -745,29 +790,12 @@ static struct file *vfio_device_open(struct vfio_device *device)
>   	if (ret)
>   		return ERR_PTR(ret);
>   
> -	if (!try_module_get(device->dev->driver->owner)) {
> -		ret = -ENODEV;
> -		goto err_unassign_container;
> -	}
> -
>   	mutex_lock(&device->dev_set->lock);
>   	device->open_count++;
>   	if (device->open_count == 1) {
> -		/*
> -		 * Here we pass the KVM pointer with the group under the read
> -		 * lock.  If the device driver will use it, it must obtain a
> -		 * reference and release it during close_device.
> -		 */
> -		mutex_lock(&device->group->group_lock);
> -		device->kvm = device->group->kvm;
> -
> -		if (device->ops->open_device) {
> -			ret = device->ops->open_device(device);
> -			if (ret)
> -				goto err_undo_count;
> -		}
> -		vfio_device_container_register(device);
> -		mutex_unlock(&device->group->group_lock);
> +		ret = vfio_device_first_open(device);
> +		if (ret)
> +			goto err_unassign_container;
>   	}
>   	mutex_unlock(&device->dev_set->lock);
>   
> @@ -800,20 +828,11 @@ static struct file *vfio_device_open(struct vfio_device *device)
>   
>   err_close_device:
>   	mutex_lock(&device->dev_set->lock);
> -	mutex_lock(&device->group->group_lock);
> -	if (device->open_count == 1 && device->ops->close_device) {
> -		device->ops->close_device(device);
> -
> -		vfio_device_container_unregister(device);
> -	}
> -err_undo_count:
> -	mutex_unlock(&device->group->group_lock);
> +	if (device->open_count == 1)
> +		vfio_device_last_close(device);
> +err_unassign_container:
>   	device->open_count--;
> -	if (device->open_count == 0 && device->kvm)
> -		device->kvm = NULL;
>   	mutex_unlock(&device->dev_set->lock);
> -	module_put(device->dev->driver->owner);
> -err_unassign_container:
>   	vfio_device_unassign_container(device);
>   	return ERR_PTR(ret);
>   }
> @@ -1016,19 +1035,11 @@ static int vfio_device_fops_release(struct inode *inode, struct file *filep)
>   
>   	mutex_lock(&device->dev_set->lock);
>   	vfio_assert_device_open(device);
> -	mutex_lock(&device->group->group_lock);
> -	if (device->open_count == 1 && device->ops->close_device)
> -		device->ops->close_device(device);
> -
> -	vfio_device_container_unregister(device);
> -	mutex_unlock(&device->group->group_lock);
> +	if (device->open_count == 1)
> +		vfio_device_last_close(device);
>   	device->open_count--;
> -	if (device->open_count == 0)
> -		device->kvm = NULL;
>   	mutex_unlock(&device->dev_set->lock);
>   
> -	module_put(device->dev->driver->owner);
> -
>   	vfio_device_unassign_container(device);
>   
>   	vfio_device_put_registration(device);

-- 
Regards,
Yi Liu

Re: [PATCH 01/10] vfio: Move vfio_device driver open/close code to a function

[Yi Liu]

On 2022/10/26 02:17, Jason Gunthorpe wrote:
> This error unwind is getting complicated. Move all the code into two
> pair'd function. The functions should be called when the open_count == 1
> after incrementing/before decrementing.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>   drivers/vfio/vfio_main.c | 95 ++++++++++++++++++++++------------------
>   1 file changed, 53 insertions(+), 42 deletions(-)

Reviewed-by: Yi Liu <yi.l.liu@intel.com>

> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index 2d168793d4e1ce..d043383fc3ba2b 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -734,6 +734,51 @@ bool vfio_assert_device_open(struct vfio_device *device)
>   	return !WARN_ON_ONCE(!READ_ONCE(device->open_count));
>   }
>   
> +static int vfio_device_first_open(struct vfio_device *device)
> +{
> +	int ret;
> +
> +	lockdep_assert_held(&device->dev_set->lock);
> +
> +	if (!try_module_get(device->dev->driver->owner))
> +		return -ENODEV;
> +
> +	/*
> +	 * Here we pass the KVM pointer with the group under the read lock.  If
> +	 * the device driver will use it, it must obtain a reference and release
> +	 * it during close_device.
> +	 */
> +	mutex_lock(&device->group->group_lock);
> +	device->kvm = device->group->kvm;
> +	if (device->ops->open_device) {
> +		ret = device->ops->open_device(device);
> +		if (ret)
> +			goto err_module_put;
> +	}
> +	vfio_device_container_register(device);
> +	mutex_unlock(&device->group->group_lock);
> +	return 0;
> +
> +err_module_put:
> +	device->kvm = NULL;
> +	mutex_unlock(&device->group->group_lock);
> +	module_put(device->dev->driver->owner);
> +	return ret;
> +}
> +
> +static void vfio_device_last_close(struct vfio_device *device)
> +{
> +	lockdep_assert_held(&device->dev_set->lock);
> +
> +	mutex_lock(&device->group->group_lock);
> +	vfio_device_container_unregister(device);
> +	if (device->ops->close_device)
> +		device->ops->close_device(device);
> +	device->kvm = NULL;
> +	mutex_unlock(&device->group->group_lock);
> +	module_put(device->dev->driver->owner);
> +}
> +
>   static struct file *vfio_device_open(struct vfio_device *device)
>   {
>   	struct file *filep;
> @@ -745,29 +790,12 @@ static struct file *vfio_device_open(struct vfio_device *device)
>   	if (ret)
>   		return ERR_PTR(ret);
>   
> -	if (!try_module_get(device->dev->driver->owner)) {
> -		ret = -ENODEV;
> -		goto err_unassign_container;
> -	}
> -
>   	mutex_lock(&device->dev_set->lock);
>   	device->open_count++;
>   	if (device->open_count == 1) {
> -		/*
> -		 * Here we pass the KVM pointer with the group under the read
> -		 * lock.  If the device driver will use it, it must obtain a
> -		 * reference and release it during close_device.
> -		 */
> -		mutex_lock(&device->group->group_lock);
> -		device->kvm = device->group->kvm;
> -
> -		if (device->ops->open_device) {
> -			ret = device->ops->open_device(device);
> -			if (ret)
> -				goto err_undo_count;
> -		}
> -		vfio_device_container_register(device);
> -		mutex_unlock(&device->group->group_lock);
> +		ret = vfio_device_first_open(device);
> +		if (ret)
> +			goto err_unassign_container;
>   	}
>   	mutex_unlock(&device->dev_set->lock);
>   
> @@ -800,20 +828,11 @@ static struct file *vfio_device_open(struct vfio_device *device)
>   
>   err_close_device:
>   	mutex_lock(&device->dev_set->lock);
> -	mutex_lock(&device->group->group_lock);
> -	if (device->open_count == 1 && device->ops->close_device) {
> -		device->ops->close_device(device);
> -
> -		vfio_device_container_unregister(device);
> -	}
> -err_undo_count:
> -	mutex_unlock(&device->group->group_lock);
> +	if (device->open_count == 1)
> +		vfio_device_last_close(device);
> +err_unassign_container:
>   	device->open_count--;
> -	if (device->open_count == 0 && device->kvm)
> -		device->kvm = NULL;
>   	mutex_unlock(&device->dev_set->lock);
> -	module_put(device->dev->driver->owner);
> -err_unassign_container:
>   	vfio_device_unassign_container(device);
>   	return ERR_PTR(ret);
>   }
> @@ -1016,19 +1035,11 @@ static int vfio_device_fops_release(struct inode *inode, struct file *filep)
>   
>   	mutex_lock(&device->dev_set->lock);
>   	vfio_assert_device_open(device);
> -	mutex_lock(&device->group->group_lock);
> -	if (device->open_count == 1 && device->ops->close_device)
> -		device->ops->close_device(device);
> -
> -	vfio_device_container_unregister(device);
> -	mutex_unlock(&device->group->group_lock);
> +	if (device->open_count == 1)
> +		vfio_device_last_close(device);
>   	device->open_count--;
> -	if (device->open_count == 0)
> -		device->kvm = NULL;
>   	mutex_unlock(&device->dev_set->lock);
>   
> -	module_put(device->dev->driver->owner);
> -
>   	vfio_device_unassign_container(device);
>   
>   	vfio_device_put_registration(device);

-- 
Regards,
Yi Liu

Re: [Intel-gfx] [PATCH 01/10] vfio: Move vfio_device driver open/close code to a function

[Yi Liu]

On 2022/10/26 02:17, Jason Gunthorpe wrote:
> This error unwind is getting complicated. Move all the code into two
> pair'd function. The functions should be called when the open_count == 1
> after incrementing/before decrementing.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>   drivers/vfio/vfio_main.c | 95 ++++++++++++++++++++++------------------
>   1 file changed, 53 insertions(+), 42 deletions(-)

Reviewed-by: Yi Liu <yi.l.liu@intel.com>

> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index 2d168793d4e1ce..d043383fc3ba2b 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -734,6 +734,51 @@ bool vfio_assert_device_open(struct vfio_device *device)
>   	return !WARN_ON_ONCE(!READ_ONCE(device->open_count));
>   }
>   
> +static int vfio_device_first_open(struct vfio_device *device)
> +{
> +	int ret;
> +
> +	lockdep_assert_held(&device->dev_set->lock);
> +
> +	if (!try_module_get(device->dev->driver->owner))
> +		return -ENODEV;
> +
> +	/*
> +	 * Here we pass the KVM pointer with the group under the read lock.  If
> +	 * the device driver will use it, it must obtain a reference and release
> +	 * it during close_device.
> +	 */
> +	mutex_lock(&device->group->group_lock);
> +	device->kvm = device->group->kvm;
> +	if (device->ops->open_device) {
> +		ret = device->ops->open_device(device);
> +		if (ret)
> +			goto err_module_put;
> +	}
> +	vfio_device_container_register(device);
> +	mutex_unlock(&device->group->group_lock);
> +	return 0;
> +
> +err_module_put:
> +	device->kvm = NULL;
> +	mutex_unlock(&device->group->group_lock);
> +	module_put(device->dev->driver->owner);
> +	return ret;
> +}
> +
> +static void vfio_device_last_close(struct vfio_device *device)
> +{
> +	lockdep_assert_held(&device->dev_set->lock);
> +
> +	mutex_lock(&device->group->group_lock);
> +	vfio_device_container_unregister(device);
> +	if (device->ops->close_device)
> +		device->ops->close_device(device);
> +	device->kvm = NULL;
> +	mutex_unlock(&device->group->group_lock);
> +	module_put(device->dev->driver->owner);
> +}
> +
>   static struct file *vfio_device_open(struct vfio_device *device)
>   {
>   	struct file *filep;
> @@ -745,29 +790,12 @@ static struct file *vfio_device_open(struct vfio_device *device)
>   	if (ret)
>   		return ERR_PTR(ret);
>   
> -	if (!try_module_get(device->dev->driver->owner)) {
> -		ret = -ENODEV;
> -		goto err_unassign_container;
> -	}
> -
>   	mutex_lock(&device->dev_set->lock);
>   	device->open_count++;
>   	if (device->open_count == 1) {
> -		/*
> -		 * Here we pass the KVM pointer with the group under the read
> -		 * lock.  If the device driver will use it, it must obtain a
> -		 * reference and release it during close_device.
> -		 */
> -		mutex_lock(&device->group->group_lock);
> -		device->kvm = device->group->kvm;
> -
> -		if (device->ops->open_device) {
> -			ret = device->ops->open_device(device);
> -			if (ret)
> -				goto err_undo_count;
> -		}
> -		vfio_device_container_register(device);
> -		mutex_unlock(&device->group->group_lock);
> +		ret = vfio_device_first_open(device);
> +		if (ret)
> +			goto err_unassign_container;
>   	}
>   	mutex_unlock(&device->dev_set->lock);
>   
> @@ -800,20 +828,11 @@ static struct file *vfio_device_open(struct vfio_device *device)
>   
>   err_close_device:
>   	mutex_lock(&device->dev_set->lock);
> -	mutex_lock(&device->group->group_lock);
> -	if (device->open_count == 1 && device->ops->close_device) {
> -		device->ops->close_device(device);
> -
> -		vfio_device_container_unregister(device);
> -	}
> -err_undo_count:
> -	mutex_unlock(&device->group->group_lock);
> +	if (device->open_count == 1)
> +		vfio_device_last_close(device);
> +err_unassign_container:
>   	device->open_count--;
> -	if (device->open_count == 0 && device->kvm)
> -		device->kvm = NULL;
>   	mutex_unlock(&device->dev_set->lock);
> -	module_put(device->dev->driver->owner);
> -err_unassign_container:
>   	vfio_device_unassign_container(device);
>   	return ERR_PTR(ret);
>   }
> @@ -1016,19 +1035,11 @@ static int vfio_device_fops_release(struct inode *inode, struct file *filep)
>   
>   	mutex_lock(&device->dev_set->lock);
>   	vfio_assert_device_open(device);
> -	mutex_lock(&device->group->group_lock);
> -	if (device->open_count == 1 && device->ops->close_device)
> -		device->ops->close_device(device);
> -
> -	vfio_device_container_unregister(device);
> -	mutex_unlock(&device->group->group_lock);
> +	if (device->open_count == 1)
> +		vfio_device_last_close(device);
>   	device->open_count--;
> -	if (device->open_count == 0)
> -		device->kvm = NULL;
>   	mutex_unlock(&device->dev_set->lock);
>   
> -	module_put(device->dev->driver->owner);
> -
>   	vfio_device_unassign_container(device);
>   
>   	vfio_device_put_registration(device);

-- 
Regards,
Yi Liu

Re: [PATCH 02/10] vfio: Move vfio_device_assign_container() into vfio_device_first_open()

[Yi Liu]

On 2022/10/26 02:17, Jason Gunthorpe wrote:
> The only thing this function does is assert the group has an assigned
> container and incrs refcounts.
> 
> The overall model we have is that once a conatiner_users refcount is

typo.

s/conatiner_users/container_users

> incremented it cannot be de-assigned from the group -
> vfio_group_ioctl_unset_container() will fail and the group FD cannot be
> closed.
> 
> Thus we do not need to check this on evey device FD open, just the

s/evey/every

> first. Reorganize the code so that only the first open and last close
> manages the container.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>   drivers/vfio/container.c |  4 ++--
>   drivers/vfio/vfio_main.c | 18 ++++++++----------
>   2 files changed, 10 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
> index d74164abbf401d..dd79a66ec62cad 100644
> --- a/drivers/vfio/container.c
> +++ b/drivers/vfio/container.c
> @@ -531,11 +531,11 @@ int vfio_device_assign_container(struct vfio_device *device)
>   
>   void vfio_device_unassign_container(struct vfio_device *device)
>   {
> -	mutex_lock(&device->group->group_lock);
> +	lockdep_assert_held_write(&device->group->group_lock);
> +
>   	WARN_ON(device->group->container_users <= 1);
>   	device->group->container_users--;
>   	fput(device->group->opened_file);
> -	mutex_unlock(&device->group->group_lock);
>   }
>   
>   /*
> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index d043383fc3ba2b..204443ba3b3cd9 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -749,16 +749,22 @@ static int vfio_device_first_open(struct vfio_device *device)
>   	 * it during close_device.
>   	 */
>   	mutex_lock(&device->group->group_lock);
> +	ret = vfio_device_assign_container(device);
> +	if (ret)
> +		goto err_module_put;
> +
>   	device->kvm = device->group->kvm;
>   	if (device->ops->open_device) {
>   		ret = device->ops->open_device(device);
>   		if (ret)
> -			goto err_module_put;
> +			goto err_container;
>   	}
>   	vfio_device_container_register(device);
>   	mutex_unlock(&device->group->group_lock);
>   	return 0;
>   
> +err_container:
> +	vfio_device_unassign_container(device);
>   err_module_put:
>   	device->kvm = NULL;
>   	mutex_unlock(&device->group->group_lock);
> @@ -775,6 +781,7 @@ static void vfio_device_last_close(struct vfio_device *device)
>   	if (device->ops->close_device)
>   		device->ops->close_device(device);
>   	device->kvm = NULL;
> +	vfio_device_unassign_container(device);
>   	mutex_unlock(&device->group->group_lock);
>   	module_put(device->dev->driver->owner);
>   }
> @@ -784,12 +791,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
>   	struct file *filep;
>   	int ret;
>   
> -	mutex_lock(&device->group->group_lock);
> -	ret = vfio_device_assign_container(device);
> -	mutex_unlock(&device->group->group_lock);
> -	if (ret)
> -		return ERR_PTR(ret);
> -
>   	mutex_lock(&device->dev_set->lock);
>   	device->open_count++;
>   	if (device->open_count == 1) {
> @@ -833,7 +834,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
>   err_unassign_container:

should the err_unassign_container be renamed to be err_dec_count?

other parts look good to me.

Reviewed-by: Yi Liu <yi.l.liu@intel.com>

>   	device->open_count--;
>   	mutex_unlock(&device->dev_set->lock);
> -	vfio_device_unassign_container(device);
>   	return ERR_PTR(ret);
>   }
>   
> @@ -1040,8 +1040,6 @@ static int vfio_device_fops_release(struct inode *inode, struct file *filep)
>   	device->open_count--;
>   	mutex_unlock(&device->dev_set->lock);
>   
> -	vfio_device_unassign_container(device);
> -
>   	vfio_device_put_registration(device);
>   
>   	return 0;

-- 
Regards,
Yi Liu

Re: [Intel-gfx] [PATCH 02/10] vfio: Move vfio_device_assign_container() into vfio_device_first_open()

[Yi Liu]

On 2022/10/26 02:17, Jason Gunthorpe wrote:
> The only thing this function does is assert the group has an assigned
> container and incrs refcounts.
> 
> The overall model we have is that once a conatiner_users refcount is

typo.

s/conatiner_users/container_users

> incremented it cannot be de-assigned from the group -
> vfio_group_ioctl_unset_container() will fail and the group FD cannot be
> closed.
> 
> Thus we do not need to check this on evey device FD open, just the

s/evey/every

> first. Reorganize the code so that only the first open and last close
> manages the container.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>   drivers/vfio/container.c |  4 ++--
>   drivers/vfio/vfio_main.c | 18 ++++++++----------
>   2 files changed, 10 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
> index d74164abbf401d..dd79a66ec62cad 100644
> --- a/drivers/vfio/container.c
> +++ b/drivers/vfio/container.c
> @@ -531,11 +531,11 @@ int vfio_device_assign_container(struct vfio_device *device)
>   
>   void vfio_device_unassign_container(struct vfio_device *device)
>   {
> -	mutex_lock(&device->group->group_lock);
> +	lockdep_assert_held_write(&device->group->group_lock);
> +
>   	WARN_ON(device->group->container_users <= 1);
>   	device->group->container_users--;
>   	fput(device->group->opened_file);
> -	mutex_unlock(&device->group->group_lock);
>   }
>   
>   /*
> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index d043383fc3ba2b..204443ba3b3cd9 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -749,16 +749,22 @@ static int vfio_device_first_open(struct vfio_device *device)
>   	 * it during close_device.
>   	 */
>   	mutex_lock(&device->group->group_lock);
> +	ret = vfio_device_assign_container(device);
> +	if (ret)
> +		goto err_module_put;
> +
>   	device->kvm = device->group->kvm;
>   	if (device->ops->open_device) {
>   		ret = device->ops->open_device(device);
>   		if (ret)
> -			goto err_module_put;
> +			goto err_container;
>   	}
>   	vfio_device_container_register(device);
>   	mutex_unlock(&device->group->group_lock);
>   	return 0;
>   
> +err_container:
> +	vfio_device_unassign_container(device);
>   err_module_put:
>   	device->kvm = NULL;
>   	mutex_unlock(&device->group->group_lock);
> @@ -775,6 +781,7 @@ static void vfio_device_last_close(struct vfio_device *device)
>   	if (device->ops->close_device)
>   		device->ops->close_device(device);
>   	device->kvm = NULL;
> +	vfio_device_unassign_container(device);
>   	mutex_unlock(&device->group->group_lock);
>   	module_put(device->dev->driver->owner);
>   }
> @@ -784,12 +791,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
>   	struct file *filep;
>   	int ret;
>   
> -	mutex_lock(&device->group->group_lock);
> -	ret = vfio_device_assign_container(device);
> -	mutex_unlock(&device->group->group_lock);
> -	if (ret)
> -		return ERR_PTR(ret);
> -
>   	mutex_lock(&device->dev_set->lock);
>   	device->open_count++;
>   	if (device->open_count == 1) {
> @@ -833,7 +834,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
>   err_unassign_container:

should the err_unassign_container be renamed to be err_dec_count?

other parts look good to me.

Reviewed-by: Yi Liu <yi.l.liu@intel.com>

>   	device->open_count--;
>   	mutex_unlock(&device->dev_set->lock);
> -	vfio_device_unassign_container(device);
>   	return ERR_PTR(ret);
>   }
>   
> @@ -1040,8 +1040,6 @@ static int vfio_device_fops_release(struct inode *inode, struct file *filep)
>   	device->open_count--;
>   	mutex_unlock(&device->dev_set->lock);
>   
> -	vfio_device_unassign_container(device);
> -
>   	vfio_device_put_registration(device);
>   
>   	return 0;

-- 
Regards,
Yi Liu

Re: [PATCH 02/10] vfio: Move vfio_device_assign_container() into vfio_device_first_open()

[Yi Liu]

On 2022/10/26 02:17, Jason Gunthorpe wrote:
> The only thing this function does is assert the group has an assigned
> container and incrs refcounts.
> 
> The overall model we have is that once a conatiner_users refcount is

typo.

s/conatiner_users/container_users

> incremented it cannot be de-assigned from the group -
> vfio_group_ioctl_unset_container() will fail and the group FD cannot be
> closed.
> 
> Thus we do not need to check this on evey device FD open, just the

s/evey/every

> first. Reorganize the code so that only the first open and last close
> manages the container.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>   drivers/vfio/container.c |  4 ++--
>   drivers/vfio/vfio_main.c | 18 ++++++++----------
>   2 files changed, 10 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
> index d74164abbf401d..dd79a66ec62cad 100644
> --- a/drivers/vfio/container.c
> +++ b/drivers/vfio/container.c
> @@ -531,11 +531,11 @@ int vfio_device_assign_container(struct vfio_device *device)
>   
>   void vfio_device_unassign_container(struct vfio_device *device)
>   {
> -	mutex_lock(&device->group->group_lock);
> +	lockdep_assert_held_write(&device->group->group_lock);
> +
>   	WARN_ON(device->group->container_users <= 1);
>   	device->group->container_users--;
>   	fput(device->group->opened_file);
> -	mutex_unlock(&device->group->group_lock);
>   }
>   
>   /*
> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index d043383fc3ba2b..204443ba3b3cd9 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -749,16 +749,22 @@ static int vfio_device_first_open(struct vfio_device *device)
>   	 * it during close_device.
>   	 */
>   	mutex_lock(&device->group->group_lock);
> +	ret = vfio_device_assign_container(device);
> +	if (ret)
> +		goto err_module_put;
> +
>   	device->kvm = device->group->kvm;
>   	if (device->ops->open_device) {
>   		ret = device->ops->open_device(device);
>   		if (ret)
> -			goto err_module_put;
> +			goto err_container;
>   	}
>   	vfio_device_container_register(device);
>   	mutex_unlock(&device->group->group_lock);
>   	return 0;
>   
> +err_container:
> +	vfio_device_unassign_container(device);
>   err_module_put:
>   	device->kvm = NULL;
>   	mutex_unlock(&device->group->group_lock);
> @@ -775,6 +781,7 @@ static void vfio_device_last_close(struct vfio_device *device)
>   	if (device->ops->close_device)
>   		device->ops->close_device(device);
>   	device->kvm = NULL;
> +	vfio_device_unassign_container(device);
>   	mutex_unlock(&device->group->group_lock);
>   	module_put(device->dev->driver->owner);
>   }
> @@ -784,12 +791,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
>   	struct file *filep;
>   	int ret;
>   
> -	mutex_lock(&device->group->group_lock);
> -	ret = vfio_device_assign_container(device);
> -	mutex_unlock(&device->group->group_lock);
> -	if (ret)
> -		return ERR_PTR(ret);
> -
>   	mutex_lock(&device->dev_set->lock);
>   	device->open_count++;
>   	if (device->open_count == 1) {
> @@ -833,7 +834,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
>   err_unassign_container:

should the err_unassign_container be renamed to be err_dec_count?

other parts look good to me.

Reviewed-by: Yi Liu <yi.l.liu@intel.com>

>   	device->open_count--;
>   	mutex_unlock(&device->dev_set->lock);
> -	vfio_device_unassign_container(device);
>   	return ERR_PTR(ret);
>   }
>   
> @@ -1040,8 +1040,6 @@ static int vfio_device_fops_release(struct inode *inode, struct file *filep)
>   	device->open_count--;
>   	mutex_unlock(&device->dev_set->lock);
>   
> -	vfio_device_unassign_container(device);
> -
>   	vfio_device_put_registration(device);
>   
>   	return 0;

-- 
Regards,
Yi Liu

Re: [PATCH 03/10] vfio: Rename vfio_device_assign/unassign_container()

[Yi Liu]

On 2022/10/26 02:17, Jason Gunthorpe wrote:
> These functions don't really assign anything anymore, they just increment
> some refcounts and do a sanity check. Call them
> vfio_group_[un]use_container()
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>   drivers/vfio/container.c | 14 ++++++--------
>   drivers/vfio/vfio.h      |  4 ++--
>   drivers/vfio/vfio_main.c |  6 +++---
>   3 files changed, 11 insertions(+), 13 deletions(-)

Reviewed-by: Yi Liu <yi.l.liu@intel.com>

> diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
> index dd79a66ec62cad..499777930b08fa 100644
> --- a/drivers/vfio/container.c
> +++ b/drivers/vfio/container.c
> @@ -511,10 +511,8 @@ void vfio_group_detach_container(struct vfio_group *group)
>   	vfio_container_put(container);
>   }
>   
> -int vfio_device_assign_container(struct vfio_device *device)
> +int vfio_group_use_container(struct vfio_group *group)
>   {
> -	struct vfio_group *group = device->group;
> -
>   	lockdep_assert_held(&group->group_lock);
>   
>   	if (!group->container || !group->container->iommu_driver ||
> @@ -529,13 +527,13 @@ int vfio_device_assign_container(struct vfio_device *device)
>   	return 0;
>   }
>   
> -void vfio_device_unassign_container(struct vfio_device *device)
> +void vfio_group_unuse_container(struct vfio_group *group)
>   {
> -	lockdep_assert_held_write(&device->group->group_lock);
> +	lockdep_assert_held(&group->group_lock);
>   
> -	WARN_ON(device->group->container_users <= 1);
> -	device->group->container_users--;
> -	fput(device->group->opened_file);
> +	WARN_ON(group->container_users <= 1);
> +	group->container_users--;
> +	fput(group->opened_file);
>   }
>   
>   /*
> diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
> index bcad54bbab08c4..f95f4925b83bbd 100644
> --- a/drivers/vfio/vfio.h
> +++ b/drivers/vfio/vfio.h
> @@ -112,8 +112,8 @@ void vfio_unregister_iommu_driver(const struct vfio_iommu_driver_ops *ops);
>   bool vfio_assert_device_open(struct vfio_device *device);
>   
>   struct vfio_container *vfio_container_from_file(struct file *filep);
> -int vfio_device_assign_container(struct vfio_device *device);
> -void vfio_device_unassign_container(struct vfio_device *device);
> +int vfio_group_use_container(struct vfio_group *group);
> +void vfio_group_unuse_container(struct vfio_group *group);
>   int vfio_container_attach_group(struct vfio_container *container,
>   				struct vfio_group *group);
>   void vfio_group_detach_container(struct vfio_group *group);
> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index 204443ba3b3cd9..8d809ecd982b39 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -749,7 +749,7 @@ static int vfio_device_first_open(struct vfio_device *device)
>   	 * it during close_device.
>   	 */
>   	mutex_lock(&device->group->group_lock);
> -	ret = vfio_device_assign_container(device);
> +	ret = vfio_group_use_container(device->group);
>   	if (ret)
>   		goto err_module_put;
>   
> @@ -764,7 +764,7 @@ static int vfio_device_first_open(struct vfio_device *device)
>   	return 0;
>   
>   err_container:
> -	vfio_device_unassign_container(device);
> +	vfio_group_unuse_container(device->group);
>   err_module_put:
>   	device->kvm = NULL;
>   	mutex_unlock(&device->group->group_lock);
> @@ -781,7 +781,7 @@ static void vfio_device_last_close(struct vfio_device *device)
>   	if (device->ops->close_device)
>   		device->ops->close_device(device);
>   	device->kvm = NULL;
> -	vfio_device_unassign_container(device);
> +	vfio_group_unuse_container(device->group);
>   	mutex_unlock(&device->group->group_lock);
>   	module_put(device->dev->driver->owner);
>   }

-- 
Regards,
Yi Liu

Re: [PATCH 03/10] vfio: Rename vfio_device_assign/unassign_container()

[Yi Liu]

On 2022/10/26 02:17, Jason Gunthorpe wrote:
> These functions don't really assign anything anymore, they just increment
> some refcounts and do a sanity check. Call them
> vfio_group_[un]use_container()
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>   drivers/vfio/container.c | 14 ++++++--------
>   drivers/vfio/vfio.h      |  4 ++--
>   drivers/vfio/vfio_main.c |  6 +++---
>   3 files changed, 11 insertions(+), 13 deletions(-)

Reviewed-by: Yi Liu <yi.l.liu@intel.com>

> diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
> index dd79a66ec62cad..499777930b08fa 100644
> --- a/drivers/vfio/container.c
> +++ b/drivers/vfio/container.c
> @@ -511,10 +511,8 @@ void vfio_group_detach_container(struct vfio_group *group)
>   	vfio_container_put(container);
>   }
>   
> -int vfio_device_assign_container(struct vfio_device *device)
> +int vfio_group_use_container(struct vfio_group *group)
>   {
> -	struct vfio_group *group = device->group;
> -
>   	lockdep_assert_held(&group->group_lock);
>   
>   	if (!group->container || !group->container->iommu_driver ||
> @@ -529,13 +527,13 @@ int vfio_device_assign_container(struct vfio_device *device)
>   	return 0;
>   }
>   
> -void vfio_device_unassign_container(struct vfio_device *device)
> +void vfio_group_unuse_container(struct vfio_group *group)
>   {
> -	lockdep_assert_held_write(&device->group->group_lock);
> +	lockdep_assert_held(&group->group_lock);
>   
> -	WARN_ON(device->group->container_users <= 1);
> -	device->group->container_users--;
> -	fput(device->group->opened_file);
> +	WARN_ON(group->container_users <= 1);
> +	group->container_users--;
> +	fput(group->opened_file);
>   }
>   
>   /*
> diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
> index bcad54bbab08c4..f95f4925b83bbd 100644
> --- a/drivers/vfio/vfio.h
> +++ b/drivers/vfio/vfio.h
> @@ -112,8 +112,8 @@ void vfio_unregister_iommu_driver(const struct vfio_iommu_driver_ops *ops);
>   bool vfio_assert_device_open(struct vfio_device *device);
>   
>   struct vfio_container *vfio_container_from_file(struct file *filep);
> -int vfio_device_assign_container(struct vfio_device *device);
> -void vfio_device_unassign_container(struct vfio_device *device);
> +int vfio_group_use_container(struct vfio_group *group);
> +void vfio_group_unuse_container(struct vfio_group *group);
>   int vfio_container_attach_group(struct vfio_container *container,
>   				struct vfio_group *group);
>   void vfio_group_detach_container(struct vfio_group *group);
> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index 204443ba3b3cd9..8d809ecd982b39 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -749,7 +749,7 @@ static int vfio_device_first_open(struct vfio_device *device)
>   	 * it during close_device.
>   	 */
>   	mutex_lock(&device->group->group_lock);
> -	ret = vfio_device_assign_container(device);
> +	ret = vfio_group_use_container(device->group);
>   	if (ret)
>   		goto err_module_put;
>   
> @@ -764,7 +764,7 @@ static int vfio_device_first_open(struct vfio_device *device)
>   	return 0;
>   
>   err_container:
> -	vfio_device_unassign_container(device);
> +	vfio_group_unuse_container(device->group);
>   err_module_put:
>   	device->kvm = NULL;
>   	mutex_unlock(&device->group->group_lock);
> @@ -781,7 +781,7 @@ static void vfio_device_last_close(struct vfio_device *device)
>   	if (device->ops->close_device)
>   		device->ops->close_device(device);
>   	device->kvm = NULL;
> -	vfio_device_unassign_container(device);
> +	vfio_group_unuse_container(device->group);
>   	mutex_unlock(&device->group->group_lock);
>   	module_put(device->dev->driver->owner);
>   }

-- 
Regards,
Yi Liu

Re: [Intel-gfx] [PATCH 03/10] vfio: Rename vfio_device_assign/unassign_container()

[Yi Liu]

On 2022/10/26 02:17, Jason Gunthorpe wrote:
> These functions don't really assign anything anymore, they just increment
> some refcounts and do a sanity check. Call them
> vfio_group_[un]use_container()
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>   drivers/vfio/container.c | 14 ++++++--------
>   drivers/vfio/vfio.h      |  4 ++--
>   drivers/vfio/vfio_main.c |  6 +++---
>   3 files changed, 11 insertions(+), 13 deletions(-)

Reviewed-by: Yi Liu <yi.l.liu@intel.com>

> diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
> index dd79a66ec62cad..499777930b08fa 100644
> --- a/drivers/vfio/container.c
> +++ b/drivers/vfio/container.c
> @@ -511,10 +511,8 @@ void vfio_group_detach_container(struct vfio_group *group)
>   	vfio_container_put(container);
>   }
>   
> -int vfio_device_assign_container(struct vfio_device *device)
> +int vfio_group_use_container(struct vfio_group *group)
>   {
> -	struct vfio_group *group = device->group;
> -
>   	lockdep_assert_held(&group->group_lock);
>   
>   	if (!group->container || !group->container->iommu_driver ||
> @@ -529,13 +527,13 @@ int vfio_device_assign_container(struct vfio_device *device)
>   	return 0;
>   }
>   
> -void vfio_device_unassign_container(struct vfio_device *device)
> +void vfio_group_unuse_container(struct vfio_group *group)
>   {
> -	lockdep_assert_held_write(&device->group->group_lock);
> +	lockdep_assert_held(&group->group_lock);
>   
> -	WARN_ON(device->group->container_users <= 1);
> -	device->group->container_users--;
> -	fput(device->group->opened_file);
> +	WARN_ON(group->container_users <= 1);
> +	group->container_users--;
> +	fput(group->opened_file);
>   }
>   
>   /*
> diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
> index bcad54bbab08c4..f95f4925b83bbd 100644
> --- a/drivers/vfio/vfio.h
> +++ b/drivers/vfio/vfio.h
> @@ -112,8 +112,8 @@ void vfio_unregister_iommu_driver(const struct vfio_iommu_driver_ops *ops);
>   bool vfio_assert_device_open(struct vfio_device *device);
>   
>   struct vfio_container *vfio_container_from_file(struct file *filep);
> -int vfio_device_assign_container(struct vfio_device *device);
> -void vfio_device_unassign_container(struct vfio_device *device);
> +int vfio_group_use_container(struct vfio_group *group);
> +void vfio_group_unuse_container(struct vfio_group *group);
>   int vfio_container_attach_group(struct vfio_container *container,
>   				struct vfio_group *group);
>   void vfio_group_detach_container(struct vfio_group *group);
> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index 204443ba3b3cd9..8d809ecd982b39 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -749,7 +749,7 @@ static int vfio_device_first_open(struct vfio_device *device)
>   	 * it during close_device.
>   	 */
>   	mutex_lock(&device->group->group_lock);
> -	ret = vfio_device_assign_container(device);
> +	ret = vfio_group_use_container(device->group);
>   	if (ret)
>   		goto err_module_put;
>   
> @@ -764,7 +764,7 @@ static int vfio_device_first_open(struct vfio_device *device)
>   	return 0;
>   
>   err_container:
> -	vfio_device_unassign_container(device);
> +	vfio_group_unuse_container(device->group);
>   err_module_put:
>   	device->kvm = NULL;
>   	mutex_unlock(&device->group->group_lock);
> @@ -781,7 +781,7 @@ static void vfio_device_last_close(struct vfio_device *device)
>   	if (device->ops->close_device)
>   		device->ops->close_device(device);
>   	device->kvm = NULL;
> -	vfio_device_unassign_container(device);
> +	vfio_group_unuse_container(device->group);
>   	mutex_unlock(&device->group->group_lock);
>   	module_put(device->dev->driver->owner);
>   }

-- 
Regards,
Yi Liu

Re: [PATCH 02/10] vfio: Move vfio_device_assign_container() into vfio_device_first_open()

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 10:37:14PM +0800, Yi Liu wrote:
> > @@ -784,12 +791,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
> >   	struct file *filep;
> >   	int ret;
> > -	mutex_lock(&device->group->group_lock);
> > -	ret = vfio_device_assign_container(device);
> > -	mutex_unlock(&device->group->group_lock);
> > -	if (ret)
> > -		return ERR_PTR(ret);
> > -
> >   	mutex_lock(&device->dev_set->lock);
> >   	device->open_count++;
> >   	if (device->open_count == 1) {
> > @@ -833,7 +834,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
> >   err_unassign_container:
> 
> should the err_unassign_container be renamed to be err_dec_count?

Yes, I went with err_unlock

Thanks,
Jason

Re: [PATCH 02/10] vfio: Move vfio_device_assign_container() into vfio_device_first_open()

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 10:37:14PM +0800, Yi Liu wrote:
> > @@ -784,12 +791,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
> >   	struct file *filep;
> >   	int ret;
> > -	mutex_lock(&device->group->group_lock);
> > -	ret = vfio_device_assign_container(device);
> > -	mutex_unlock(&device->group->group_lock);
> > -	if (ret)
> > -		return ERR_PTR(ret);
> > -
> >   	mutex_lock(&device->dev_set->lock);
> >   	device->open_count++;
> >   	if (device->open_count == 1) {
> > @@ -833,7 +834,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
> >   err_unassign_container:
> 
> should the err_unassign_container be renamed to be err_dec_count?

Yes, I went with err_unlock

Thanks,
Jason

Re: [Intel-gfx] [PATCH 02/10] vfio: Move vfio_device_assign_container() into vfio_device_first_open()

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 10:37:14PM +0800, Yi Liu wrote:
> > @@ -784,12 +791,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
> >   	struct file *filep;
> >   	int ret;
> > -	mutex_lock(&device->group->group_lock);
> > -	ret = vfio_device_assign_container(device);
> > -	mutex_unlock(&device->group->group_lock);
> > -	if (ret)
> > -		return ERR_PTR(ret);
> > -
> >   	mutex_lock(&device->dev_set->lock);
> >   	device->open_count++;
> >   	if (device->open_count == 1) {
> > @@ -833,7 +834,6 @@ static struct file *vfio_device_open(struct vfio_device *device)
> >   err_unassign_container:
> 
> should the err_unassign_container be renamed to be err_dec_count?

Yes, I went with err_unlock

Thanks,
Jason

Re: [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Yi Liu]

On 2022/10/26 02:50, Jason Gunthorpe wrote:
> This makes VFIO_GROUP_SET_CONTAINER accept both a vfio container FD and an
> iommufd.
> 
> In iommufd mode an IOAS will exist after the SET_CONTAINER, but it will
> not be attached to any groups.

is there any special reason that we cannot attach the IOAS in the SET
container phase or SET_IOMMU phase?

> 
>  From a VFIO perspective this means that the VFIO_GROUP_GET_STATUS and
> VFIO_GROUP_FLAGS_VIABLE works subtly differently. With the container FD
> the iommu_group_claim_dma_owner() is done during SET_CONTAINER but for
> IOMMFD this is done during VFIO_GROUP_GET_DEVICE_FD. Meaning that

s/IOMMFD/IOMMUFD

> VFIO_GROUP_FLAGS_VIABLE could be set but GET_DEVICE_FD will fail due to
> viability.
> 
> As GET_DEVICE_FD can fail for many reasons already this is not expected to
> be a meaningful difference.
> 
> Reorganize the tests for if the group has an assigned container or iommu
> into a vfio_group_has_iommu() function and consolidate all the duplicated
> WARN_ON's etc related to this.
> 
> Call container functions only if a container is actually present on the
> group.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>   drivers/vfio/Kconfig     |  1 +
>   drivers/vfio/container.c |  7 ++--
>   drivers/vfio/vfio.h      |  2 ++
>   drivers/vfio/vfio_main.c | 76 ++++++++++++++++++++++++++++++++--------
>   4 files changed, 69 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
> index 86c381ceb9a1e9..1118d322eec97d 100644
> --- a/drivers/vfio/Kconfig
> +++ b/drivers/vfio/Kconfig
> @@ -2,6 +2,7 @@
>   menuconfig VFIO
>   	tristate "VFIO Non-Privileged userspace driver framework"
>   	select IOMMU_API
> +	depends on IOMMUFD || !IOMMUFD
>   	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM || ARM64)
>   	select INTERVAL_TREE
>   	help
> diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
> index d97747dfb05d02..8772dad6808539 100644
> --- a/drivers/vfio/container.c
> +++ b/drivers/vfio/container.c
> @@ -516,8 +516,11 @@ int vfio_group_use_container(struct vfio_group *group)
>   {
>   	lockdep_assert_held(&group->group_lock);
>   
> -	if (!group->container || !group->container->iommu_driver ||
> -	    WARN_ON(!group->container_users))
> +	/*
> +	 * The container fd has been assigned with VFIO_GROUP_SET_CONTAINER but
> +	 * VFIO_SET_IOMMU hasn't been done yet.
> +	 */
> +	if (!group->container->iommu_driver)
>   		return -EINVAL;
>   
>   	if (group->type == VFIO_NO_IOMMU && !capable(CAP_SYS_RAWIO))
> diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
> index 247590334e14b0..985e13d52989ca 100644
> --- a/drivers/vfio/vfio.h
> +++ b/drivers/vfio/vfio.h
> @@ -10,6 +10,7 @@
>   #include <linux/cdev.h>
>   #include <linux/module.h>
>   
> +struct iommufd_ctx;
>   struct iommu_group;
>   struct vfio_device;
>   struct vfio_container;
> @@ -60,6 +61,7 @@ struct vfio_group {
>   	struct kvm			*kvm;
>   	struct file			*opened_file;
>   	struct blocking_notifier_head	notifier;
> +	struct iommufd_ctx		*iommufd;
>   };
>   
>   /* events for the backend driver notify callback */
> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index a8d1fbfcc3ddad..cf0ea744de931e 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -35,6 +35,7 @@
>   #include <linux/pm_runtime.h>
>   #include <linux/interval_tree.h>
>   #include <linux/iova_bitmap.h>
> +#include <linux/iommufd.h>
>   #include "vfio.h"
>   
>   #define DRIVER_VERSION	"0.3"
> @@ -665,6 +666,16 @@ EXPORT_SYMBOL_GPL(vfio_unregister_group_dev);
>   /*
>    * VFIO Group fd, /dev/vfio/$GROUP
>    */
> +static bool vfio_group_has_iommu(struct vfio_group *group)
> +{
> +	lockdep_assert_held(&group->group_lock);
> +	if (!group->container)
> +		WARN_ON(group->container_users);
> +	else
> +		WARN_ON(!group->container_users);
> +	return group->container || group->iommufd;
> +}
> +
>   /*
>    * VFIO_GROUP_UNSET_CONTAINER should fail if there are other users or
>    * if there was no container to unset.  Since the ioctl is called on
> @@ -676,15 +687,21 @@ static int vfio_group_ioctl_unset_container(struct vfio_group *group)
>   	int ret = 0;
>   
>   	mutex_lock(&group->group_lock);
> -	if (!group->container) {
> +	if (!vfio_group_has_iommu(group)) {
>   		ret = -EINVAL;
>   		goto out_unlock;
>   	}
> -	if (group->container_users != 1) {
> -		ret = -EBUSY;
> -		goto out_unlock;
> +	if (group->container) {
> +		if (group->container_users != 1) {
> +			ret = -EBUSY;
> +			goto out_unlock;
> +		}
> +		vfio_group_detach_container(group);
> +	}
> +	if (group->iommufd) {
> +		iommufd_ctx_put(group->iommufd);
> +		group->iommufd = NULL;
>   	}
> -	vfio_group_detach_container(group);
>   
>   out_unlock:
>   	mutex_unlock(&group->group_lock);
> @@ -695,6 +712,7 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
>   					  int __user *arg)
>   {
>   	struct vfio_container *container;
> +	struct iommufd_ctx *iommufd;
>   	struct fd f;
>   	int ret;
>   	int fd;
> @@ -707,7 +725,7 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
>   		return -EBADF;
>   
>   	mutex_lock(&group->group_lock);
> -	if (group->container || WARN_ON(group->container_users)) {
> +	if (vfio_group_has_iommu(group)) {
>   		ret = -EINVAL;
>   		goto out_unlock;
>   	}
> @@ -717,12 +735,23 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
>   	}
>   
>   	container = vfio_container_from_file(f.file);
> -	ret = -EINVAL;
>   	if (container) {
>   		ret = vfio_container_attach_group(container, group);
>   		goto out_unlock;
>   	}
>   
> +	iommufd = iommufd_ctx_from_file(f.file);
> +	if (!IS_ERR(iommufd)) {
> +		u32 ioas_id;
> +
> +		group->iommufd = iommufd;
> +		ret = iommufd_vfio_compat_ioas_id(iommufd, &ioas_id);
> +		goto out_unlock;
> +	}
> +
> +	/* The FD passed is not recognized. */
> +	ret = -EBADF;
> +
>   out_unlock:
>   	mutex_unlock(&group->group_lock);
>   	fdput(f);
> @@ -752,9 +781,16 @@ static int vfio_device_first_open(struct vfio_device *device)
>   	 * it during close_device.
>   	 */
>   	mutex_lock(&device->group->group_lock);
> -	ret = vfio_group_use_container(device->group);
> -	if (ret)
> +	if (!vfio_group_has_iommu(device->group)) {
> +		ret = -EINVAL;
>   		goto err_module_put;
> +	}
> +
> +	if (device->group->container) {
> +		ret = vfio_group_use_container(device->group);
> +		if (ret)
> +			goto err_module_put;
> +	}
>   
>   	device->kvm = device->group->kvm;
>   	if (device->ops->open_device) {
> @@ -762,14 +798,16 @@ static int vfio_device_first_open(struct vfio_device *device)
>   		if (ret)
>   			goto err_container;
>   	}
> -	vfio_device_container_register(device);
> +	if (device->group->container)
> +		vfio_device_container_register(device);
>   	mutex_unlock(&device->group->group_lock);
>   	return 0;
>   
>   err_container:
> -	vfio_group_unuse_container(device->group);
> -err_module_put:
> +	if (device->group->container)
> +		vfio_group_unuse_container(device->group);
>   	device->kvm = NULL;
> +err_module_put:
>   	mutex_unlock(&device->group->group_lock);
>   	module_put(device->dev->driver->owner);
>   	return ret;
> @@ -780,11 +818,13 @@ static void vfio_device_last_close(struct vfio_device *device)
>   	lockdep_assert_held(&device->dev_set->lock);
>   
>   	mutex_lock(&device->group->group_lock);
> -	vfio_device_container_unregister(device);
> +	if (device->group->container)
> +		vfio_device_container_unregister(device);
>   	if (device->ops->close_device)
>   		device->ops->close_device(device);
>   	device->kvm = NULL;
> -	vfio_group_unuse_container(device->group);
> +	if (device->group->container)
> +		vfio_group_unuse_container(device->group);
>   	mutex_unlock(&device->group->group_lock);
>   	module_put(device->dev->driver->owner);
>   }
> @@ -900,7 +940,7 @@ static int vfio_group_ioctl_get_status(struct vfio_group *group,
>   		return -ENODEV;
>   	}
>   
> -	if (group->container)
> +	if (group->container || group->iommufd)
>   		status.flags |= VFIO_GROUP_FLAGS_CONTAINER_SET |
>   				VFIO_GROUP_FLAGS_VIABLE;
>   	else if (!iommu_group_dma_owner_claimed(group->iommu_group))
> @@ -983,6 +1023,10 @@ static int vfio_group_fops_release(struct inode *inode, struct file *filep)
>   	WARN_ON(group->notifier.head);
>   	if (group->container)
>   		vfio_group_detach_container(group);
> +	if (group->iommufd) {
> +		iommufd_ctx_put(group->iommufd);
> +		group->iommufd = NULL;
> +	}
>   	group->opened_file = NULL;
>   	mutex_unlock(&group->group_lock);
>   	return 0;
> @@ -1879,6 +1923,8 @@ static void __exit vfio_cleanup(void)
>   module_init(vfio_init);
>   module_exit(vfio_cleanup);
>   
> +MODULE_IMPORT_NS(IOMMUFD);
> +MODULE_IMPORT_NS(IOMMUFD_VFIO);
>   MODULE_VERSION(DRIVER_VERSION);
>   MODULE_LICENSE("GPL v2");
>   MODULE_AUTHOR(DRIVER_AUTHOR);

-- 
Regards,
Yi Liu

Re: [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Yi Liu]

On 2022/10/26 02:50, Jason Gunthorpe wrote:
> This makes VFIO_GROUP_SET_CONTAINER accept both a vfio container FD and an
> iommufd.
> 
> In iommufd mode an IOAS will exist after the SET_CONTAINER, but it will
> not be attached to any groups.

is there any special reason that we cannot attach the IOAS in the SET
container phase or SET_IOMMU phase?

> 
>  From a VFIO perspective this means that the VFIO_GROUP_GET_STATUS and
> VFIO_GROUP_FLAGS_VIABLE works subtly differently. With the container FD
> the iommu_group_claim_dma_owner() is done during SET_CONTAINER but for
> IOMMFD this is done during VFIO_GROUP_GET_DEVICE_FD. Meaning that

s/IOMMFD/IOMMUFD

> VFIO_GROUP_FLAGS_VIABLE could be set but GET_DEVICE_FD will fail due to
> viability.
> 
> As GET_DEVICE_FD can fail for many reasons already this is not expected to
> be a meaningful difference.
> 
> Reorganize the tests for if the group has an assigned container or iommu
> into a vfio_group_has_iommu() function and consolidate all the duplicated
> WARN_ON's etc related to this.
> 
> Call container functions only if a container is actually present on the
> group.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>   drivers/vfio/Kconfig     |  1 +
>   drivers/vfio/container.c |  7 ++--
>   drivers/vfio/vfio.h      |  2 ++
>   drivers/vfio/vfio_main.c | 76 ++++++++++++++++++++++++++++++++--------
>   4 files changed, 69 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
> index 86c381ceb9a1e9..1118d322eec97d 100644
> --- a/drivers/vfio/Kconfig
> +++ b/drivers/vfio/Kconfig
> @@ -2,6 +2,7 @@
>   menuconfig VFIO
>   	tristate "VFIO Non-Privileged userspace driver framework"
>   	select IOMMU_API
> +	depends on IOMMUFD || !IOMMUFD
>   	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM || ARM64)
>   	select INTERVAL_TREE
>   	help
> diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
> index d97747dfb05d02..8772dad6808539 100644
> --- a/drivers/vfio/container.c
> +++ b/drivers/vfio/container.c
> @@ -516,8 +516,11 @@ int vfio_group_use_container(struct vfio_group *group)
>   {
>   	lockdep_assert_held(&group->group_lock);
>   
> -	if (!group->container || !group->container->iommu_driver ||
> -	    WARN_ON(!group->container_users))
> +	/*
> +	 * The container fd has been assigned with VFIO_GROUP_SET_CONTAINER but
> +	 * VFIO_SET_IOMMU hasn't been done yet.
> +	 */
> +	if (!group->container->iommu_driver)
>   		return -EINVAL;
>   
>   	if (group->type == VFIO_NO_IOMMU && !capable(CAP_SYS_RAWIO))
> diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
> index 247590334e14b0..985e13d52989ca 100644
> --- a/drivers/vfio/vfio.h
> +++ b/drivers/vfio/vfio.h
> @@ -10,6 +10,7 @@
>   #include <linux/cdev.h>
>   #include <linux/module.h>
>   
> +struct iommufd_ctx;
>   struct iommu_group;
>   struct vfio_device;
>   struct vfio_container;
> @@ -60,6 +61,7 @@ struct vfio_group {
>   	struct kvm			*kvm;
>   	struct file			*opened_file;
>   	struct blocking_notifier_head	notifier;
> +	struct iommufd_ctx		*iommufd;
>   };
>   
>   /* events for the backend driver notify callback */
> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index a8d1fbfcc3ddad..cf0ea744de931e 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -35,6 +35,7 @@
>   #include <linux/pm_runtime.h>
>   #include <linux/interval_tree.h>
>   #include <linux/iova_bitmap.h>
> +#include <linux/iommufd.h>
>   #include "vfio.h"
>   
>   #define DRIVER_VERSION	"0.3"
> @@ -665,6 +666,16 @@ EXPORT_SYMBOL_GPL(vfio_unregister_group_dev);
>   /*
>    * VFIO Group fd, /dev/vfio/$GROUP
>    */
> +static bool vfio_group_has_iommu(struct vfio_group *group)
> +{
> +	lockdep_assert_held(&group->group_lock);
> +	if (!group->container)
> +		WARN_ON(group->container_users);
> +	else
> +		WARN_ON(!group->container_users);
> +	return group->container || group->iommufd;
> +}
> +
>   /*
>    * VFIO_GROUP_UNSET_CONTAINER should fail if there are other users or
>    * if there was no container to unset.  Since the ioctl is called on
> @@ -676,15 +687,21 @@ static int vfio_group_ioctl_unset_container(struct vfio_group *group)
>   	int ret = 0;
>   
>   	mutex_lock(&group->group_lock);
> -	if (!group->container) {
> +	if (!vfio_group_has_iommu(group)) {
>   		ret = -EINVAL;
>   		goto out_unlock;
>   	}
> -	if (group->container_users != 1) {
> -		ret = -EBUSY;
> -		goto out_unlock;
> +	if (group->container) {
> +		if (group->container_users != 1) {
> +			ret = -EBUSY;
> +			goto out_unlock;
> +		}
> +		vfio_group_detach_container(group);
> +	}
> +	if (group->iommufd) {
> +		iommufd_ctx_put(group->iommufd);
> +		group->iommufd = NULL;
>   	}
> -	vfio_group_detach_container(group);
>   
>   out_unlock:
>   	mutex_unlock(&group->group_lock);
> @@ -695,6 +712,7 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
>   					  int __user *arg)
>   {
>   	struct vfio_container *container;
> +	struct iommufd_ctx *iommufd;
>   	struct fd f;
>   	int ret;
>   	int fd;
> @@ -707,7 +725,7 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
>   		return -EBADF;
>   
>   	mutex_lock(&group->group_lock);
> -	if (group->container || WARN_ON(group->container_users)) {
> +	if (vfio_group_has_iommu(group)) {
>   		ret = -EINVAL;
>   		goto out_unlock;
>   	}
> @@ -717,12 +735,23 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
>   	}
>   
>   	container = vfio_container_from_file(f.file);
> -	ret = -EINVAL;
>   	if (container) {
>   		ret = vfio_container_attach_group(container, group);
>   		goto out_unlock;
>   	}
>   
> +	iommufd = iommufd_ctx_from_file(f.file);
> +	if (!IS_ERR(iommufd)) {
> +		u32 ioas_id;
> +
> +		group->iommufd = iommufd;
> +		ret = iommufd_vfio_compat_ioas_id(iommufd, &ioas_id);
> +		goto out_unlock;
> +	}
> +
> +	/* The FD passed is not recognized. */
> +	ret = -EBADF;
> +
>   out_unlock:
>   	mutex_unlock(&group->group_lock);
>   	fdput(f);
> @@ -752,9 +781,16 @@ static int vfio_device_first_open(struct vfio_device *device)
>   	 * it during close_device.
>   	 */
>   	mutex_lock(&device->group->group_lock);
> -	ret = vfio_group_use_container(device->group);
> -	if (ret)
> +	if (!vfio_group_has_iommu(device->group)) {
> +		ret = -EINVAL;
>   		goto err_module_put;
> +	}
> +
> +	if (device->group->container) {
> +		ret = vfio_group_use_container(device->group);
> +		if (ret)
> +			goto err_module_put;
> +	}
>   
>   	device->kvm = device->group->kvm;
>   	if (device->ops->open_device) {
> @@ -762,14 +798,16 @@ static int vfio_device_first_open(struct vfio_device *device)
>   		if (ret)
>   			goto err_container;
>   	}
> -	vfio_device_container_register(device);
> +	if (device->group->container)
> +		vfio_device_container_register(device);
>   	mutex_unlock(&device->group->group_lock);
>   	return 0;
>   
>   err_container:
> -	vfio_group_unuse_container(device->group);
> -err_module_put:
> +	if (device->group->container)
> +		vfio_group_unuse_container(device->group);
>   	device->kvm = NULL;
> +err_module_put:
>   	mutex_unlock(&device->group->group_lock);
>   	module_put(device->dev->driver->owner);
>   	return ret;
> @@ -780,11 +818,13 @@ static void vfio_device_last_close(struct vfio_device *device)
>   	lockdep_assert_held(&device->dev_set->lock);
>   
>   	mutex_lock(&device->group->group_lock);
> -	vfio_device_container_unregister(device);
> +	if (device->group->container)
> +		vfio_device_container_unregister(device);
>   	if (device->ops->close_device)
>   		device->ops->close_device(device);
>   	device->kvm = NULL;
> -	vfio_group_unuse_container(device->group);
> +	if (device->group->container)
> +		vfio_group_unuse_container(device->group);
>   	mutex_unlock(&device->group->group_lock);
>   	module_put(device->dev->driver->owner);
>   }
> @@ -900,7 +940,7 @@ static int vfio_group_ioctl_get_status(struct vfio_group *group,
>   		return -ENODEV;
>   	}
>   
> -	if (group->container)
> +	if (group->container || group->iommufd)
>   		status.flags |= VFIO_GROUP_FLAGS_CONTAINER_SET |
>   				VFIO_GROUP_FLAGS_VIABLE;
>   	else if (!iommu_group_dma_owner_claimed(group->iommu_group))
> @@ -983,6 +1023,10 @@ static int vfio_group_fops_release(struct inode *inode, struct file *filep)
>   	WARN_ON(group->notifier.head);
>   	if (group->container)
>   		vfio_group_detach_container(group);
> +	if (group->iommufd) {
> +		iommufd_ctx_put(group->iommufd);
> +		group->iommufd = NULL;
> +	}
>   	group->opened_file = NULL;
>   	mutex_unlock(&group->group_lock);
>   	return 0;
> @@ -1879,6 +1923,8 @@ static void __exit vfio_cleanup(void)
>   module_init(vfio_init);
>   module_exit(vfio_cleanup);
>   
> +MODULE_IMPORT_NS(IOMMUFD);
> +MODULE_IMPORT_NS(IOMMUFD_VFIO);
>   MODULE_VERSION(DRIVER_VERSION);
>   MODULE_LICENSE("GPL v2");
>   MODULE_AUTHOR(DRIVER_AUTHOR);

-- 
Regards,
Yi Liu

Re: [Intel-gfx] [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Yi Liu]

On 2022/10/26 02:50, Jason Gunthorpe wrote:
> This makes VFIO_GROUP_SET_CONTAINER accept both a vfio container FD and an
> iommufd.
> 
> In iommufd mode an IOAS will exist after the SET_CONTAINER, but it will
> not be attached to any groups.

is there any special reason that we cannot attach the IOAS in the SET
container phase or SET_IOMMU phase?

> 
>  From a VFIO perspective this means that the VFIO_GROUP_GET_STATUS and
> VFIO_GROUP_FLAGS_VIABLE works subtly differently. With the container FD
> the iommu_group_claim_dma_owner() is done during SET_CONTAINER but for
> IOMMFD this is done during VFIO_GROUP_GET_DEVICE_FD. Meaning that

s/IOMMFD/IOMMUFD

> VFIO_GROUP_FLAGS_VIABLE could be set but GET_DEVICE_FD will fail due to
> viability.
> 
> As GET_DEVICE_FD can fail for many reasons already this is not expected to
> be a meaningful difference.
> 
> Reorganize the tests for if the group has an assigned container or iommu
> into a vfio_group_has_iommu() function and consolidate all the duplicated
> WARN_ON's etc related to this.
> 
> Call container functions only if a container is actually present on the
> group.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>   drivers/vfio/Kconfig     |  1 +
>   drivers/vfio/container.c |  7 ++--
>   drivers/vfio/vfio.h      |  2 ++
>   drivers/vfio/vfio_main.c | 76 ++++++++++++++++++++++++++++++++--------
>   4 files changed, 69 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
> index 86c381ceb9a1e9..1118d322eec97d 100644
> --- a/drivers/vfio/Kconfig
> +++ b/drivers/vfio/Kconfig
> @@ -2,6 +2,7 @@
>   menuconfig VFIO
>   	tristate "VFIO Non-Privileged userspace driver framework"
>   	select IOMMU_API
> +	depends on IOMMUFD || !IOMMUFD
>   	select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM || ARM64)
>   	select INTERVAL_TREE
>   	help
> diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c
> index d97747dfb05d02..8772dad6808539 100644
> --- a/drivers/vfio/container.c
> +++ b/drivers/vfio/container.c
> @@ -516,8 +516,11 @@ int vfio_group_use_container(struct vfio_group *group)
>   {
>   	lockdep_assert_held(&group->group_lock);
>   
> -	if (!group->container || !group->container->iommu_driver ||
> -	    WARN_ON(!group->container_users))
> +	/*
> +	 * The container fd has been assigned with VFIO_GROUP_SET_CONTAINER but
> +	 * VFIO_SET_IOMMU hasn't been done yet.
> +	 */
> +	if (!group->container->iommu_driver)
>   		return -EINVAL;
>   
>   	if (group->type == VFIO_NO_IOMMU && !capable(CAP_SYS_RAWIO))
> diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h
> index 247590334e14b0..985e13d52989ca 100644
> --- a/drivers/vfio/vfio.h
> +++ b/drivers/vfio/vfio.h
> @@ -10,6 +10,7 @@
>   #include <linux/cdev.h>
>   #include <linux/module.h>
>   
> +struct iommufd_ctx;
>   struct iommu_group;
>   struct vfio_device;
>   struct vfio_container;
> @@ -60,6 +61,7 @@ struct vfio_group {
>   	struct kvm			*kvm;
>   	struct file			*opened_file;
>   	struct blocking_notifier_head	notifier;
> +	struct iommufd_ctx		*iommufd;
>   };
>   
>   /* events for the backend driver notify callback */
> diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
> index a8d1fbfcc3ddad..cf0ea744de931e 100644
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -35,6 +35,7 @@
>   #include <linux/pm_runtime.h>
>   #include <linux/interval_tree.h>
>   #include <linux/iova_bitmap.h>
> +#include <linux/iommufd.h>
>   #include "vfio.h"
>   
>   #define DRIVER_VERSION	"0.3"
> @@ -665,6 +666,16 @@ EXPORT_SYMBOL_GPL(vfio_unregister_group_dev);
>   /*
>    * VFIO Group fd, /dev/vfio/$GROUP
>    */
> +static bool vfio_group_has_iommu(struct vfio_group *group)
> +{
> +	lockdep_assert_held(&group->group_lock);
> +	if (!group->container)
> +		WARN_ON(group->container_users);
> +	else
> +		WARN_ON(!group->container_users);
> +	return group->container || group->iommufd;
> +}
> +
>   /*
>    * VFIO_GROUP_UNSET_CONTAINER should fail if there are other users or
>    * if there was no container to unset.  Since the ioctl is called on
> @@ -676,15 +687,21 @@ static int vfio_group_ioctl_unset_container(struct vfio_group *group)
>   	int ret = 0;
>   
>   	mutex_lock(&group->group_lock);
> -	if (!group->container) {
> +	if (!vfio_group_has_iommu(group)) {
>   		ret = -EINVAL;
>   		goto out_unlock;
>   	}
> -	if (group->container_users != 1) {
> -		ret = -EBUSY;
> -		goto out_unlock;
> +	if (group->container) {
> +		if (group->container_users != 1) {
> +			ret = -EBUSY;
> +			goto out_unlock;
> +		}
> +		vfio_group_detach_container(group);
> +	}
> +	if (group->iommufd) {
> +		iommufd_ctx_put(group->iommufd);
> +		group->iommufd = NULL;
>   	}
> -	vfio_group_detach_container(group);
>   
>   out_unlock:
>   	mutex_unlock(&group->group_lock);
> @@ -695,6 +712,7 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
>   					  int __user *arg)
>   {
>   	struct vfio_container *container;
> +	struct iommufd_ctx *iommufd;
>   	struct fd f;
>   	int ret;
>   	int fd;
> @@ -707,7 +725,7 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
>   		return -EBADF;
>   
>   	mutex_lock(&group->group_lock);
> -	if (group->container || WARN_ON(group->container_users)) {
> +	if (vfio_group_has_iommu(group)) {
>   		ret = -EINVAL;
>   		goto out_unlock;
>   	}
> @@ -717,12 +735,23 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
>   	}
>   
>   	container = vfio_container_from_file(f.file);
> -	ret = -EINVAL;
>   	if (container) {
>   		ret = vfio_container_attach_group(container, group);
>   		goto out_unlock;
>   	}
>   
> +	iommufd = iommufd_ctx_from_file(f.file);
> +	if (!IS_ERR(iommufd)) {
> +		u32 ioas_id;
> +
> +		group->iommufd = iommufd;
> +		ret = iommufd_vfio_compat_ioas_id(iommufd, &ioas_id);
> +		goto out_unlock;
> +	}
> +
> +	/* The FD passed is not recognized. */
> +	ret = -EBADF;
> +
>   out_unlock:
>   	mutex_unlock(&group->group_lock);
>   	fdput(f);
> @@ -752,9 +781,16 @@ static int vfio_device_first_open(struct vfio_device *device)
>   	 * it during close_device.
>   	 */
>   	mutex_lock(&device->group->group_lock);
> -	ret = vfio_group_use_container(device->group);
> -	if (ret)
> +	if (!vfio_group_has_iommu(device->group)) {
> +		ret = -EINVAL;
>   		goto err_module_put;
> +	}
> +
> +	if (device->group->container) {
> +		ret = vfio_group_use_container(device->group);
> +		if (ret)
> +			goto err_module_put;
> +	}
>   
>   	device->kvm = device->group->kvm;
>   	if (device->ops->open_device) {
> @@ -762,14 +798,16 @@ static int vfio_device_first_open(struct vfio_device *device)
>   		if (ret)
>   			goto err_container;
>   	}
> -	vfio_device_container_register(device);
> +	if (device->group->container)
> +		vfio_device_container_register(device);
>   	mutex_unlock(&device->group->group_lock);
>   	return 0;
>   
>   err_container:
> -	vfio_group_unuse_container(device->group);
> -err_module_put:
> +	if (device->group->container)
> +		vfio_group_unuse_container(device->group);
>   	device->kvm = NULL;
> +err_module_put:
>   	mutex_unlock(&device->group->group_lock);
>   	module_put(device->dev->driver->owner);
>   	return ret;
> @@ -780,11 +818,13 @@ static void vfio_device_last_close(struct vfio_device *device)
>   	lockdep_assert_held(&device->dev_set->lock);
>   
>   	mutex_lock(&device->group->group_lock);
> -	vfio_device_container_unregister(device);
> +	if (device->group->container)
> +		vfio_device_container_unregister(device);
>   	if (device->ops->close_device)
>   		device->ops->close_device(device);
>   	device->kvm = NULL;
> -	vfio_group_unuse_container(device->group);
> +	if (device->group->container)
> +		vfio_group_unuse_container(device->group);
>   	mutex_unlock(&device->group->group_lock);
>   	module_put(device->dev->driver->owner);
>   }
> @@ -900,7 +940,7 @@ static int vfio_group_ioctl_get_status(struct vfio_group *group,
>   		return -ENODEV;
>   	}
>   
> -	if (group->container)
> +	if (group->container || group->iommufd)
>   		status.flags |= VFIO_GROUP_FLAGS_CONTAINER_SET |
>   				VFIO_GROUP_FLAGS_VIABLE;
>   	else if (!iommu_group_dma_owner_claimed(group->iommu_group))
> @@ -983,6 +1023,10 @@ static int vfio_group_fops_release(struct inode *inode, struct file *filep)
>   	WARN_ON(group->notifier.head);
>   	if (group->container)
>   		vfio_group_detach_container(group);
> +	if (group->iommufd) {
> +		iommufd_ctx_put(group->iommufd);
> +		group->iommufd = NULL;
> +	}
>   	group->opened_file = NULL;
>   	mutex_unlock(&group->group_lock);
>   	return 0;
> @@ -1879,6 +1923,8 @@ static void __exit vfio_cleanup(void)
>   module_init(vfio_init);
>   module_exit(vfio_cleanup);
>   
> +MODULE_IMPORT_NS(IOMMUFD);
> +MODULE_IMPORT_NS(IOMMUFD_VFIO);
>   MODULE_VERSION(DRIVER_VERSION);
>   MODULE_LICENSE("GPL v2");
>   MODULE_AUTHOR(DRIVER_AUTHOR);

-- 
Regards,
Yi Liu

RE: [PATCH 05/10] vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, November 1, 2022 8:26 PM
> And this:
> 
> 	/*
> 	 * If the device does not have
> IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
> 	 * any domain later attached to it will also not support it. If the cap
> 	 * is set then the iommu_domain eventually attached to the
> device/group
> 	 * must must use a domain with enforce_cache_coherency().
> 	 */

duplicated 'must'

RE: [PATCH 05/10] vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, November 1, 2022 8:26 PM
> And this:
> 
> 	/*
> 	 * If the device does not have
> IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
> 	 * any domain later attached to it will also not support it. If the cap
> 	 * is set then the iommu_domain eventually attached to the
> device/group
> 	 * must must use a domain with enforce_cache_coherency().
> 	 */

duplicated 'must'

Re: [Intel-gfx] [PATCH 05/10] vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, November 1, 2022 8:26 PM
> And this:
> 
> 	/*
> 	 * If the device does not have
> IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
> 	 * any domain later attached to it will also not support it. If the cap
> 	 * is set then the iommu_domain eventually attached to the
> device/group
> 	 * must must use a domain with enforce_cache_coherency().
> 	 */

duplicated 'must'

RE: [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, November 1, 2022 7:51 PM
> 
> On Tue, Nov 01, 2022 at 02:19:04AM -0700, Nicolin Chen wrote:
> > On Tue, Nov 01, 2022 at 08:09:52AM +0000, Tian, Kevin wrote:
> >
> > > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > > Sent: Wednesday, October 26, 2022 2:51 AM
> > > >
> > > >  menuconfig VFIO
> > > >       tristate "VFIO Non-Privileged userspace driver framework"
> > > >       select IOMMU_API
> > > > +     depends on IOMMUFD || !IOMMUFD
> > >
> > > Out of curiosity. What is the meaning of this dependency claim?
> >
> > "is it a module or not" -- from https://lwn.net/Articles/683476/
> 
> Yes, it is the kconfig pattern for "This symbol optionally uses the
> other symbol, and if the other symbol is turned on then it has to be
> the right y/m value"
> 
> ie rejects vfio being built-in but iommufd being a module
> 

Thanks. a good learning. 😊

RE: [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, November 1, 2022 7:51 PM
> 
> On Tue, Nov 01, 2022 at 02:19:04AM -0700, Nicolin Chen wrote:
> > On Tue, Nov 01, 2022 at 08:09:52AM +0000, Tian, Kevin wrote:
> >
> > > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > > Sent: Wednesday, October 26, 2022 2:51 AM
> > > >
> > > >  menuconfig VFIO
> > > >       tristate "VFIO Non-Privileged userspace driver framework"
> > > >       select IOMMU_API
> > > > +     depends on IOMMUFD || !IOMMUFD
> > >
> > > Out of curiosity. What is the meaning of this dependency claim?
> >
> > "is it a module or not" -- from https://lwn.net/Articles/683476/
> 
> Yes, it is the kconfig pattern for "This symbol optionally uses the
> other symbol, and if the other symbol is turned on then it has to be
> the right y/m value"
> 
> ie rejects vfio being built-in but iommufd being a module
> 

Thanks. a good learning. 😊

Re: [Intel-gfx] [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, November 1, 2022 7:51 PM
> 
> On Tue, Nov 01, 2022 at 02:19:04AM -0700, Nicolin Chen wrote:
> > On Tue, Nov 01, 2022 at 08:09:52AM +0000, Tian, Kevin wrote:
> >
> > > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > > Sent: Wednesday, October 26, 2022 2:51 AM
> > > >
> > > >  menuconfig VFIO
> > > >       tristate "VFIO Non-Privileged userspace driver framework"
> > > >       select IOMMU_API
> > > > +     depends on IOMMUFD || !IOMMUFD
> > >
> > > Out of curiosity. What is the meaning of this dependency claim?
> >
> > "is it a module or not" -- from https://lwn.net/Articles/683476/
> 
> Yes, it is the kconfig pattern for "This symbol optionally uses the
> other symbol, and if the other symbol is turned on then it has to be
> the right y/m value"
> 
> ie rejects vfio being built-in but iommufd being a module
> 

Thanks. a good learning. 😊

RE: [PATCH 08/10] vfio-iommufd: Support iommufd for emulated VFIO devices

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, November 1, 2022 8:49 PM
> > > ---
> > >  drivers/gpu/drm/i915/gvt/kvmgt.c  |   3 +
> > >  drivers/s390/cio/vfio_ccw_ops.c   |   3 +
> > >  drivers/s390/crypto/vfio_ap_ops.c |   3 +
> > >  drivers/vfio/container.c          | 108 ++++++-----------------------
> > >  drivers/vfio/iommufd.c            |  57 ++++++++++++++++
> > >  drivers/vfio/vfio.h               |  10 ++-
> > >  drivers/vfio/vfio_main.c          | 110 +++++++++++++++++++++++++++++-
> > >  include/linux/vfio.h              |  14 ++++
> > >  8 files changed, 217 insertions(+), 91 deletions(-)
> >
> > mtty, mdpy and mbochs?
> 
> They don't call rw or pin_pages, so they don't need to do
> anything:
> 
> 
> 	/*
> 	 * If the driver doesn't provide this op then it means the device does
> 	 * not do DMA at all. So nothing to do.
> 	 */
> 	if (!vdev->ops->bind_iommufd)
> 		return 0;
> 

OK, I see the point of this check here.

btw It'd be good to document in vfio_device_ops that driver must provide
this op if the device does DMA.

> > > +
> > > +		/* VFIO historically tries to auto-detect a kthread */
> > > +		if (!current->mm)
> > > +			flags |= IOMMUFD_ACCESS_RW_KTHREAD;
> >
> > Can you elaborate why this cannot be put in iommufd as the default
> > policy similar to what vfio container does?
> 
> Snooping in kernel structs to try to guess the calling execution
> context is bad design. The caller should know its own context and it
> should declare positively what it is. Someday this should be lifted
> out of VFIO as well and into the drivers.
> 

with the last sentence it makes more sense. otherwise I didn't see
why putting the guess in vfio makes real difference from doing it
in iommufd as there is no vfio specific state referenced for making
this decision.

RE: [PATCH 08/10] vfio-iommufd: Support iommufd for emulated VFIO devices

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, November 1, 2022 8:49 PM
> > > ---
> > >  drivers/gpu/drm/i915/gvt/kvmgt.c  |   3 +
> > >  drivers/s390/cio/vfio_ccw_ops.c   |   3 +
> > >  drivers/s390/crypto/vfio_ap_ops.c |   3 +
> > >  drivers/vfio/container.c          | 108 ++++++-----------------------
> > >  drivers/vfio/iommufd.c            |  57 ++++++++++++++++
> > >  drivers/vfio/vfio.h               |  10 ++-
> > >  drivers/vfio/vfio_main.c          | 110 +++++++++++++++++++++++++++++-
> > >  include/linux/vfio.h              |  14 ++++
> > >  8 files changed, 217 insertions(+), 91 deletions(-)
> >
> > mtty, mdpy and mbochs?
> 
> They don't call rw or pin_pages, so they don't need to do
> anything:
> 
> 
> 	/*
> 	 * If the driver doesn't provide this op then it means the device does
> 	 * not do DMA at all. So nothing to do.
> 	 */
> 	if (!vdev->ops->bind_iommufd)
> 		return 0;
> 

OK, I see the point of this check here.

btw It'd be good to document in vfio_device_ops that driver must provide
this op if the device does DMA.

> > > +
> > > +		/* VFIO historically tries to auto-detect a kthread */
> > > +		if (!current->mm)
> > > +			flags |= IOMMUFD_ACCESS_RW_KTHREAD;
> >
> > Can you elaborate why this cannot be put in iommufd as the default
> > policy similar to what vfio container does?
> 
> Snooping in kernel structs to try to guess the calling execution
> context is bad design. The caller should know its own context and it
> should declare positively what it is. Someday this should be lifted
> out of VFIO as well and into the drivers.
> 

with the last sentence it makes more sense. otherwise I didn't see
why putting the guess in vfio makes real difference from doing it
in iommufd as there is no vfio specific state referenced for making
this decision.

Re: [Intel-gfx] [PATCH 08/10] vfio-iommufd: Support iommufd for emulated VFIO devices

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, November 1, 2022 8:49 PM
> > > ---
> > >  drivers/gpu/drm/i915/gvt/kvmgt.c  |   3 +
> > >  drivers/s390/cio/vfio_ccw_ops.c   |   3 +
> > >  drivers/s390/crypto/vfio_ap_ops.c |   3 +
> > >  drivers/vfio/container.c          | 108 ++++++-----------------------
> > >  drivers/vfio/iommufd.c            |  57 ++++++++++++++++
> > >  drivers/vfio/vfio.h               |  10 ++-
> > >  drivers/vfio/vfio_main.c          | 110 +++++++++++++++++++++++++++++-
> > >  include/linux/vfio.h              |  14 ++++
> > >  8 files changed, 217 insertions(+), 91 deletions(-)
> >
> > mtty, mdpy and mbochs?
> 
> They don't call rw or pin_pages, so they don't need to do
> anything:
> 
> 
> 	/*
> 	 * If the driver doesn't provide this op then it means the device does
> 	 * not do DMA at all. So nothing to do.
> 	 */
> 	if (!vdev->ops->bind_iommufd)
> 		return 0;
> 

OK, I see the point of this check here.

btw It'd be good to document in vfio_device_ops that driver must provide
this op if the device does DMA.

> > > +
> > > +		/* VFIO historically tries to auto-detect a kthread */
> > > +		if (!current->mm)
> > > +			flags |= IOMMUFD_ACCESS_RW_KTHREAD;
> >
> > Can you elaborate why this cannot be put in iommufd as the default
> > policy similar to what vfio container does?
> 
> Snooping in kernel structs to try to guess the calling execution
> context is bad design. The caller should know its own context and it
> should declare positively what it is. Someday this should be lifted
> out of VFIO as well and into the drivers.
> 

with the last sentence it makes more sense. otherwise I didn't see
why putting the guess in vfio makes real difference from doing it
in iommufd as there is no vfio specific state referenced for making
this decision.

Re: [PATCH 05/10] vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()

[Jason Gunthorpe]

On Thu, Nov 03, 2022 at 04:38:16AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Tuesday, November 1, 2022 8:26 PM
> > And this:
> > 
> > 	/*
> > 	 * If the device does not have
> > IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
> > 	 * any domain later attached to it will also not support it. If the cap
> > 	 * is set then the iommu_domain eventually attached to the
> > device/group
> > 	 * must must use a domain with enforce_cache_coherency().
> > 	 */
> 
> duplicated 'must'

Done

Jason

Re: [PATCH 05/10] vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()

[Jason Gunthorpe]

On Thu, Nov 03, 2022 at 04:38:16AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Tuesday, November 1, 2022 8:26 PM
> > And this:
> > 
> > 	/*
> > 	 * If the device does not have
> > IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
> > 	 * any domain later attached to it will also not support it. If the cap
> > 	 * is set then the iommu_domain eventually attached to the
> > device/group
> > 	 * must must use a domain with enforce_cache_coherency().
> > 	 */
> 
> duplicated 'must'

Done

Jason

Re: [Intel-gfx] [PATCH 05/10] vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()

[Jason Gunthorpe]

On Thu, Nov 03, 2022 at 04:38:16AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Tuesday, November 1, 2022 8:26 PM
> > And this:
> > 
> > 	/*
> > 	 * If the device does not have
> > IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
> > 	 * any domain later attached to it will also not support it. If the cap
> > 	 * is set then the iommu_domain eventually attached to the
> > device/group
> > 	 * must must use a domain with enforce_cache_coherency().
> > 	 */
> 
> duplicated 'must'

Done

Jason

Re: [PATCH 07/10] vfio-iommufd: Support iommufd for physical VFIO devices

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 08:21:20AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:51 AM
> > 
> > +int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx)
> > +{
> > +	u32 ioas_id;
> > +	u32 device_id;
> > +	int ret;
> > +
> > +	lockdep_assert_held(&vdev->dev_set->lock);
> > +
> > +	/*
> > +	 * If the driver doesn't provide this op then it means the device does
> > +	 * not do DMA at all. So nothing to do.
> > +	 */
> > +	if (!vdev->ops->bind_iommufd)
> > +		return 0;
> 
> Nothing to do or return -EOPNOTSUPP?

As in the other email, nothing to do, driver is "bound" but doesn't
actually need iommufd at all.

> > +	ret = vdev->ops->bind_iommufd(vdev, ictx, &device_id);
> > +	if (ret)
> > +		return ret;
> > +
> > +	if (vdev->ops->attach_ioas) {
> 
> __vfio_register_dev() already verifies that all three callbacks must
> co-exist. Then no need to check it again here and later.

Ok

> > +void vfio_iommufd_unbind(struct vfio_device *vdev)
> > +{
> > +	lockdep_assert_held(&vdev->dev_set->lock);
> > +
> > +	if (!vdev->iommufd_device)
> > +		return;
> 
> there is no iommufd_device in the emulated path...

Yes, this if should just be deleted

Thanks,
Jason

Re: [PATCH 07/10] vfio-iommufd: Support iommufd for physical VFIO devices

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 08:21:20AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:51 AM
> > 
> > +int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx)
> > +{
> > +	u32 ioas_id;
> > +	u32 device_id;
> > +	int ret;
> > +
> > +	lockdep_assert_held(&vdev->dev_set->lock);
> > +
> > +	/*
> > +	 * If the driver doesn't provide this op then it means the device does
> > +	 * not do DMA at all. So nothing to do.
> > +	 */
> > +	if (!vdev->ops->bind_iommufd)
> > +		return 0;
> 
> Nothing to do or return -EOPNOTSUPP?

As in the other email, nothing to do, driver is "bound" but doesn't
actually need iommufd at all.

> > +	ret = vdev->ops->bind_iommufd(vdev, ictx, &device_id);
> > +	if (ret)
> > +		return ret;
> > +
> > +	if (vdev->ops->attach_ioas) {
> 
> __vfio_register_dev() already verifies that all three callbacks must
> co-exist. Then no need to check it again here and later.

Ok

> > +void vfio_iommufd_unbind(struct vfio_device *vdev)
> > +{
> > +	lockdep_assert_held(&vdev->dev_set->lock);
> > +
> > +	if (!vdev->iommufd_device)
> > +		return;
> 
> there is no iommufd_device in the emulated path...

Yes, this if should just be deleted

Thanks,
Jason

Re: [Intel-gfx] [PATCH 07/10] vfio-iommufd: Support iommufd for physical VFIO devices

[Jason Gunthorpe]

On Tue, Nov 01, 2022 at 08:21:20AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, October 26, 2022 2:51 AM
> > 
> > +int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx)
> > +{
> > +	u32 ioas_id;
> > +	u32 device_id;
> > +	int ret;
> > +
> > +	lockdep_assert_held(&vdev->dev_set->lock);
> > +
> > +	/*
> > +	 * If the driver doesn't provide this op then it means the device does
> > +	 * not do DMA at all. So nothing to do.
> > +	 */
> > +	if (!vdev->ops->bind_iommufd)
> > +		return 0;
> 
> Nothing to do or return -EOPNOTSUPP?

As in the other email, nothing to do, driver is "bound" but doesn't
actually need iommufd at all.

> > +	ret = vdev->ops->bind_iommufd(vdev, ictx, &device_id);
> > +	if (ret)
> > +		return ret;
> > +
> > +	if (vdev->ops->attach_ioas) {
> 
> __vfio_register_dev() already verifies that all three callbacks must
> co-exist. Then no need to check it again here and later.

Ok

> > +void vfio_iommufd_unbind(struct vfio_device *vdev)
> > +{
> > +	lockdep_assert_held(&vdev->dev_set->lock);
> > +
> > +	if (!vdev->iommufd_device)
> > +		return;
> 
> there is no iommufd_device in the emulated path...

Yes, this if should just be deleted

Thanks,
Jason

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Mon, Oct 31, 2022 at 04:45:26PM -0600, Alex Williamson wrote:

> > It is one idea, it depends how literal you want to be on "module
> > parameters are ABI". IMHO it is a weak form of ABI and the need of
> > this paramter in particular is not that common in modern times, AFAIK.
> > 
> > So perhaps we just also expose it through vfio.ko and expect people to
> > migrate. That would give a window were both options are available.
> 
> That might be best.  Ultimately this is an opt-out of a feature that
> has security implications, so I'd rather error on the side of requiring
> the user to re-assert that opt-out.  It seems the potential good in
> eliminating stale or unnecessary options outweighs any weak claims of
> preserving an ABI for a module that's no longer in service.

Ok, lets do this

--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -55,6 +55,11 @@ static struct vfio {
 bool vfio_allow_unsafe_interrupts;
 EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
 
+module_param_named(allow_unsafe_interrupts,
+                  vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
+MODULE_PARM_DESC(allow_unsafe_interrupts,
+                "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
+
 static DEFINE_XARRAY(vfio_device_set_xa);
 static const struct file_operations vfio_group_fops;

> However, I'd question whether vfio is the right place for that new
> module option.  As proposed, vfio is only passing it through to
> iommufd, where an error related to lack of the hardware feature is
> masked behind an -EPERM by the time it gets back to vfio, making any
> sort of advisory to the user about the module option convoluted.  It
> seems like iommufd should own the option to opt-out universally, not
> just through the vfio use case.  Thanks,

My thinking is this option shouldn't exist at all in other iommufd
users. eg I don't see value in VDPA supporting it.

So, let's wait and see if a need arises first. I'm reluctant to add
options to disable kernel security without really good reasons.

Jason

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Mon, Oct 31, 2022 at 04:45:26PM -0600, Alex Williamson wrote:

> > It is one idea, it depends how literal you want to be on "module
> > parameters are ABI". IMHO it is a weak form of ABI and the need of
> > this paramter in particular is not that common in modern times, AFAIK.
> > 
> > So perhaps we just also expose it through vfio.ko and expect people to
> > migrate. That would give a window were both options are available.
> 
> That might be best.  Ultimately this is an opt-out of a feature that
> has security implications, so I'd rather error on the side of requiring
> the user to re-assert that opt-out.  It seems the potential good in
> eliminating stale or unnecessary options outweighs any weak claims of
> preserving an ABI for a module that's no longer in service.

Ok, lets do this

--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -55,6 +55,11 @@ static struct vfio {
 bool vfio_allow_unsafe_interrupts;
 EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
 
+module_param_named(allow_unsafe_interrupts,
+                  vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
+MODULE_PARM_DESC(allow_unsafe_interrupts,
+                "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
+
 static DEFINE_XARRAY(vfio_device_set_xa);
 static const struct file_operations vfio_group_fops;

> However, I'd question whether vfio is the right place for that new
> module option.  As proposed, vfio is only passing it through to
> iommufd, where an error related to lack of the hardware feature is
> masked behind an -EPERM by the time it gets back to vfio, making any
> sort of advisory to the user about the module option convoluted.  It
> seems like iommufd should own the option to opt-out universally, not
> just through the vfio use case.  Thanks,

My thinking is this option shouldn't exist at all in other iommufd
users. eg I don't see value in VDPA supporting it.

So, let's wait and see if a need arises first. I'm reluctant to add
options to disable kernel security without really good reasons.

Jason

Re: [Intel-gfx] [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Mon, Oct 31, 2022 at 04:45:26PM -0600, Alex Williamson wrote:

> > It is one idea, it depends how literal you want to be on "module
> > parameters are ABI". IMHO it is a weak form of ABI and the need of
> > this paramter in particular is not that common in modern times, AFAIK.
> > 
> > So perhaps we just also expose it through vfio.ko and expect people to
> > migrate. That would give a window were both options are available.
> 
> That might be best.  Ultimately this is an opt-out of a feature that
> has security implications, so I'd rather error on the side of requiring
> the user to re-assert that opt-out.  It seems the potential good in
> eliminating stale or unnecessary options outweighs any weak claims of
> preserving an ABI for a module that's no longer in service.

Ok, lets do this

--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -55,6 +55,11 @@ static struct vfio {
 bool vfio_allow_unsafe_interrupts;
 EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
 
+module_param_named(allow_unsafe_interrupts,
+                  vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
+MODULE_PARM_DESC(allow_unsafe_interrupts,
+                "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
+
 static DEFINE_XARRAY(vfio_device_set_xa);
 static const struct file_operations vfio_group_fops;

> However, I'd question whether vfio is the right place for that new
> module option.  As proposed, vfio is only passing it through to
> iommufd, where an error related to lack of the hardware feature is
> masked behind an -EPERM by the time it gets back to vfio, making any
> sort of advisory to the user about the module option convoluted.  It
> seems like iommufd should own the option to opt-out universally, not
> just through the vfio use case.  Thanks,

My thinking is this option shouldn't exist at all in other iommufd
users. eg I don't see value in VDPA supporting it.

So, let's wait and see if a need arises first. I'm reluctant to add
options to disable kernel security without really good reasons.

Jason

Re: [PATCH 10/10] iommufd: Allow iommufd to supply /dev/vfio/vfio

[Jason Gunthorpe]

On Mon, Oct 31, 2022 at 04:53:11PM -0600, Alex Williamson wrote:
> On Fri, 28 Oct 2022 15:44:36 -0300
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > On Wed, Oct 26, 2022 at 03:31:33PM -0600, Alex Williamson wrote:
> > > On Tue, 25 Oct 2022 15:50:45 -0300
> > > Jason Gunthorpe <jgg@nvidia.com> wrote:
> > >   
> > > > If the VFIO container is compiled out, give a kconfig option for iommufd
> > > > to provide the miscdev node with the same name and permissions as vfio
> > > > uses.
> > > > 
> > > > The compatibility node supports the same ioctls as VFIO and automatically
> > > > enables the VFIO compatible pinned page accounting mode.  
> > > 
> > > I think I'd like to see some sort of breadcrumb when /dev/vfio/vfio is
> > > provided by something other than the vfio container code.  If we intend
> > > to include this before P2P is resolved, that breadcrumb   
> > 
> > I don't belive I can get P2P done soon enough. I plan to do it after
> > this is merged. Right now these two series are taking all my time.
> > 
> > > (dmesg I'm guessing) might also list any known limitations of the
> > > compatibility to save time with debugging.  Thanks,  
> > 
> > Yes, that makes sense.
> > 
> > Do you want a dmesg at module load time, on every open, or a sysfs
> > something? What seems like it would make it into a bug report?
> 
> I think dmesg at module load time should probably be ok, every open
> seems like harassment and sysfs would require updated support in
> various bug reporting tools.  Users are often terrible about reporting
> full dmesg in bugs, but they do often filter it for "IOMMU" or "VFIO",
> so keep that in mind when crafting the log message.  Thanks,

This seems like the right approach, the message comes out once when it
might be most useful:

@@ -176,8 +176,11 @@ static int iommufd_fops_open(struct inode *inode, struct file *filp)
         * For compatibility with VFIO when /dev/vfio/vfio is opened we default
         * to the same rlimit accounting as vfio uses.
         */
-       if (filp->private_data == &vfio_misc_dev)
+       if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER) &&
+           filp->private_data == &vfio_misc_dev) {
                ictx->account_mode = IOPT_PAGES_ACCOUNT_MM;
+               pr_info_once("IOMMUFD is providing /dev/vfio/vfio, not VFIO.\n");
+       }
 
        xa_init_flags(&ictx->objects, XA_FLAGS_ALLOC1 | XA_FLAGS_ACCOUNT);
        ictx->file = filp;

Also this is needed:

@@ -446,6 +449,7 @@ module_exit(iommufd_exit);
 
 #if IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER)
 MODULE_ALIAS_MISCDEV(VFIO_MINOR);
+MODULE_ALIAS("devname:vfio/vfio");
 #endif
 MODULE_DESCRIPTION("I/O Address Space Management for passthrough devices");
 MODULE_LICENSE("GPL");

Thanks,
Jason

Re: [PATCH 10/10] iommufd: Allow iommufd to supply /dev/vfio/vfio

[Jason Gunthorpe]

On Mon, Oct 31, 2022 at 04:53:11PM -0600, Alex Williamson wrote:
> On Fri, 28 Oct 2022 15:44:36 -0300
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > On Wed, Oct 26, 2022 at 03:31:33PM -0600, Alex Williamson wrote:
> > > On Tue, 25 Oct 2022 15:50:45 -0300
> > > Jason Gunthorpe <jgg@nvidia.com> wrote:
> > >   
> > > > If the VFIO container is compiled out, give a kconfig option for iommufd
> > > > to provide the miscdev node with the same name and permissions as vfio
> > > > uses.
> > > > 
> > > > The compatibility node supports the same ioctls as VFIO and automatically
> > > > enables the VFIO compatible pinned page accounting mode.  
> > > 
> > > I think I'd like to see some sort of breadcrumb when /dev/vfio/vfio is
> > > provided by something other than the vfio container code.  If we intend
> > > to include this before P2P is resolved, that breadcrumb   
> > 
> > I don't belive I can get P2P done soon enough. I plan to do it after
> > this is merged. Right now these two series are taking all my time.
> > 
> > > (dmesg I'm guessing) might also list any known limitations of the
> > > compatibility to save time with debugging.  Thanks,  
> > 
> > Yes, that makes sense.
> > 
> > Do you want a dmesg at module load time, on every open, or a sysfs
> > something? What seems like it would make it into a bug report?
> 
> I think dmesg at module load time should probably be ok, every open
> seems like harassment and sysfs would require updated support in
> various bug reporting tools.  Users are often terrible about reporting
> full dmesg in bugs, but they do often filter it for "IOMMU" or "VFIO",
> so keep that in mind when crafting the log message.  Thanks,

This seems like the right approach, the message comes out once when it
might be most useful:

@@ -176,8 +176,11 @@ static int iommufd_fops_open(struct inode *inode, struct file *filp)
         * For compatibility with VFIO when /dev/vfio/vfio is opened we default
         * to the same rlimit accounting as vfio uses.
         */
-       if (filp->private_data == &vfio_misc_dev)
+       if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER) &&
+           filp->private_data == &vfio_misc_dev) {
                ictx->account_mode = IOPT_PAGES_ACCOUNT_MM;
+               pr_info_once("IOMMUFD is providing /dev/vfio/vfio, not VFIO.\n");
+       }
 
        xa_init_flags(&ictx->objects, XA_FLAGS_ALLOC1 | XA_FLAGS_ACCOUNT);
        ictx->file = filp;

Also this is needed:

@@ -446,6 +449,7 @@ module_exit(iommufd_exit);
 
 #if IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER)
 MODULE_ALIAS_MISCDEV(VFIO_MINOR);
+MODULE_ALIAS("devname:vfio/vfio");
 #endif
 MODULE_DESCRIPTION("I/O Address Space Management for passthrough devices");
 MODULE_LICENSE("GPL");

Thanks,
Jason

Re: [Intel-gfx] [PATCH 10/10] iommufd: Allow iommufd to supply /dev/vfio/vfio

[Jason Gunthorpe]

On Mon, Oct 31, 2022 at 04:53:11PM -0600, Alex Williamson wrote:
> On Fri, 28 Oct 2022 15:44:36 -0300
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > On Wed, Oct 26, 2022 at 03:31:33PM -0600, Alex Williamson wrote:
> > > On Tue, 25 Oct 2022 15:50:45 -0300
> > > Jason Gunthorpe <jgg@nvidia.com> wrote:
> > >   
> > > > If the VFIO container is compiled out, give a kconfig option for iommufd
> > > > to provide the miscdev node with the same name and permissions as vfio
> > > > uses.
> > > > 
> > > > The compatibility node supports the same ioctls as VFIO and automatically
> > > > enables the VFIO compatible pinned page accounting mode.  
> > > 
> > > I think I'd like to see some sort of breadcrumb when /dev/vfio/vfio is
> > > provided by something other than the vfio container code.  If we intend
> > > to include this before P2P is resolved, that breadcrumb   
> > 
> > I don't belive I can get P2P done soon enough. I plan to do it after
> > this is merged. Right now these two series are taking all my time.
> > 
> > > (dmesg I'm guessing) might also list any known limitations of the
> > > compatibility to save time with debugging.  Thanks,  
> > 
> > Yes, that makes sense.
> > 
> > Do you want a dmesg at module load time, on every open, or a sysfs
> > something? What seems like it would make it into a bug report?
> 
> I think dmesg at module load time should probably be ok, every open
> seems like harassment and sysfs would require updated support in
> various bug reporting tools.  Users are often terrible about reporting
> full dmesg in bugs, but they do often filter it for "IOMMU" or "VFIO",
> so keep that in mind when crafting the log message.  Thanks,

This seems like the right approach, the message comes out once when it
might be most useful:

@@ -176,8 +176,11 @@ static int iommufd_fops_open(struct inode *inode, struct file *filp)
         * For compatibility with VFIO when /dev/vfio/vfio is opened we default
         * to the same rlimit accounting as vfio uses.
         */
-       if (filp->private_data == &vfio_misc_dev)
+       if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER) &&
+           filp->private_data == &vfio_misc_dev) {
                ictx->account_mode = IOPT_PAGES_ACCOUNT_MM;
+               pr_info_once("IOMMUFD is providing /dev/vfio/vfio, not VFIO.\n");
+       }
 
        xa_init_flags(&ictx->objects, XA_FLAGS_ALLOC1 | XA_FLAGS_ACCOUNT);
        ictx->file = filp;

Also this is needed:

@@ -446,6 +449,7 @@ module_exit(iommufd_exit);
 
 #if IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER)
 MODULE_ALIAS_MISCDEV(VFIO_MINOR);
+MODULE_ALIAS("devname:vfio/vfio");
 #endif
 MODULE_DESCRIPTION("I/O Address Space Management for passthrough devices");
 MODULE_LICENSE("GPL");

Thanks,
Jason

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Mon, 7 Nov 2022 09:19:43 -0400
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Mon, Oct 31, 2022 at 04:45:26PM -0600, Alex Williamson wrote:
> 
> > > It is one idea, it depends how literal you want to be on "module
> > > parameters are ABI". IMHO it is a weak form of ABI and the need of
> > > this paramter in particular is not that common in modern times, AFAIK.
> > > 
> > > So perhaps we just also expose it through vfio.ko and expect people to
> > > migrate. That would give a window were both options are available.  
> > 
> > That might be best.  Ultimately this is an opt-out of a feature that
> > has security implications, so I'd rather error on the side of requiring
> > the user to re-assert that opt-out.  It seems the potential good in
> > eliminating stale or unnecessary options outweighs any weak claims of
> > preserving an ABI for a module that's no longer in service.  
> 
> Ok, lets do this
> 
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -55,6 +55,11 @@ static struct vfio {
>  bool vfio_allow_unsafe_interrupts;
>  EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
>  
> +module_param_named(allow_unsafe_interrupts,
> +                  vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
> +MODULE_PARM_DESC(allow_unsafe_interrupts,
> +                "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
> +
>  static DEFINE_XARRAY(vfio_device_set_xa);
>  static const struct file_operations vfio_group_fops;
> 
> > However, I'd question whether vfio is the right place for that new
> > module option.  As proposed, vfio is only passing it through to
> > iommufd, where an error related to lack of the hardware feature is
> > masked behind an -EPERM by the time it gets back to vfio, making any
> > sort of advisory to the user about the module option convoluted.  It
> > seems like iommufd should own the option to opt-out universally, not
> > just through the vfio use case.  Thanks,  
> 
> My thinking is this option shouldn't exist at all in other iommufd
> users. eg I don't see value in VDPA supporting it.

I disagree, the IOMMU interface is responsible for isolating the
device, this option doesn't make any sense to live in vfio-main, which
is the reason it was always a type1 option.  If vdpa doesn't allow full
device access such that it can guarantee that a device cannot generate
a DMA that can spoof MSI, then it sounds like the flag we pass when
attaching a device to iommfd should to reflect this difference in usage.
The driver either requires full isolation, default, or can indicate a
form of restricted DMA programming that prevents interrupt spoofing.
The policy whether to permit unsafe configurations should exist in one
place, iommufd.  Thanks,

Alex

Re: [Intel-gfx] [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Mon, 7 Nov 2022 09:19:43 -0400
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Mon, Oct 31, 2022 at 04:45:26PM -0600, Alex Williamson wrote:
> 
> > > It is one idea, it depends how literal you want to be on "module
> > > parameters are ABI". IMHO it is a weak form of ABI and the need of
> > > this paramter in particular is not that common in modern times, AFAIK.
> > > 
> > > So perhaps we just also expose it through vfio.ko and expect people to
> > > migrate. That would give a window were both options are available.  
> > 
> > That might be best.  Ultimately this is an opt-out of a feature that
> > has security implications, so I'd rather error on the side of requiring
> > the user to re-assert that opt-out.  It seems the potential good in
> > eliminating stale or unnecessary options outweighs any weak claims of
> > preserving an ABI for a module that's no longer in service.  
> 
> Ok, lets do this
> 
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -55,6 +55,11 @@ static struct vfio {
>  bool vfio_allow_unsafe_interrupts;
>  EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
>  
> +module_param_named(allow_unsafe_interrupts,
> +                  vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
> +MODULE_PARM_DESC(allow_unsafe_interrupts,
> +                "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
> +
>  static DEFINE_XARRAY(vfio_device_set_xa);
>  static const struct file_operations vfio_group_fops;
> 
> > However, I'd question whether vfio is the right place for that new
> > module option.  As proposed, vfio is only passing it through to
> > iommufd, where an error related to lack of the hardware feature is
> > masked behind an -EPERM by the time it gets back to vfio, making any
> > sort of advisory to the user about the module option convoluted.  It
> > seems like iommufd should own the option to opt-out universally, not
> > just through the vfio use case.  Thanks,  
> 
> My thinking is this option shouldn't exist at all in other iommufd
> users. eg I don't see value in VDPA supporting it.

I disagree, the IOMMU interface is responsible for isolating the
device, this option doesn't make any sense to live in vfio-main, which
is the reason it was always a type1 option.  If vdpa doesn't allow full
device access such that it can guarantee that a device cannot generate
a DMA that can spoof MSI, then it sounds like the flag we pass when
attaching a device to iommfd should to reflect this difference in usage.
The driver either requires full isolation, default, or can indicate a
form of restricted DMA programming that prevents interrupt spoofing.
The policy whether to permit unsafe configurations should exist in one
place, iommufd.  Thanks,

Alex

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Mon, 7 Nov 2022 09:19:43 -0400
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Mon, Oct 31, 2022 at 04:45:26PM -0600, Alex Williamson wrote:
> 
> > > It is one idea, it depends how literal you want to be on "module
> > > parameters are ABI". IMHO it is a weak form of ABI and the need of
> > > this paramter in particular is not that common in modern times, AFAIK.
> > > 
> > > So perhaps we just also expose it through vfio.ko and expect people to
> > > migrate. That would give a window were both options are available.  
> > 
> > That might be best.  Ultimately this is an opt-out of a feature that
> > has security implications, so I'd rather error on the side of requiring
> > the user to re-assert that opt-out.  It seems the potential good in
> > eliminating stale or unnecessary options outweighs any weak claims of
> > preserving an ABI for a module that's no longer in service.  
> 
> Ok, lets do this
> 
> --- a/drivers/vfio/vfio_main.c
> +++ b/drivers/vfio/vfio_main.c
> @@ -55,6 +55,11 @@ static struct vfio {
>  bool vfio_allow_unsafe_interrupts;
>  EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
>  
> +module_param_named(allow_unsafe_interrupts,
> +                  vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
> +MODULE_PARM_DESC(allow_unsafe_interrupts,
> +                "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
> +
>  static DEFINE_XARRAY(vfio_device_set_xa);
>  static const struct file_operations vfio_group_fops;
> 
> > However, I'd question whether vfio is the right place for that new
> > module option.  As proposed, vfio is only passing it through to
> > iommufd, where an error related to lack of the hardware feature is
> > masked behind an -EPERM by the time it gets back to vfio, making any
> > sort of advisory to the user about the module option convoluted.  It
> > seems like iommufd should own the option to opt-out universally, not
> > just through the vfio use case.  Thanks,  
> 
> My thinking is this option shouldn't exist at all in other iommufd
> users. eg I don't see value in VDPA supporting it.

I disagree, the IOMMU interface is responsible for isolating the
device, this option doesn't make any sense to live in vfio-main, which
is the reason it was always a type1 option.  If vdpa doesn't allow full
device access such that it can guarantee that a device cannot generate
a DMA that can spoof MSI, then it sounds like the flag we pass when
attaching a device to iommfd should to reflect this difference in usage.
The driver either requires full isolation, default, or can indicate a
form of restricted DMA programming that prevents interrupt spoofing.
The policy whether to permit unsafe configurations should exist in one
place, iommufd.  Thanks,

Alex

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Mon, Nov 07, 2022 at 08:18:53AM -0700, Alex Williamson wrote:
> On Mon, 7 Nov 2022 09:19:43 -0400
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > On Mon, Oct 31, 2022 at 04:45:26PM -0600, Alex Williamson wrote:
> > 
> > > > It is one idea, it depends how literal you want to be on "module
> > > > parameters are ABI". IMHO it is a weak form of ABI and the need of
> > > > this paramter in particular is not that common in modern times, AFAIK.
> > > > 
> > > > So perhaps we just also expose it through vfio.ko and expect people to
> > > > migrate. That would give a window were both options are available.  
> > > 
> > > That might be best.  Ultimately this is an opt-out of a feature that
> > > has security implications, so I'd rather error on the side of requiring
> > > the user to re-assert that opt-out.  It seems the potential good in
> > > eliminating stale or unnecessary options outweighs any weak claims of
> > > preserving an ABI for a module that's no longer in service.  
> > 
> > Ok, lets do this
> > 
> > --- a/drivers/vfio/vfio_main.c
> > +++ b/drivers/vfio/vfio_main.c
> > @@ -55,6 +55,11 @@ static struct vfio {
> >  bool vfio_allow_unsafe_interrupts;
> >  EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
> >  
> > +module_param_named(allow_unsafe_interrupts,
> > +                  vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
> > +MODULE_PARM_DESC(allow_unsafe_interrupts,
> > +                "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
> > +
> >  static DEFINE_XARRAY(vfio_device_set_xa);
> >  static const struct file_operations vfio_group_fops;
> > 
> > > However, I'd question whether vfio is the right place for that new
> > > module option.  As proposed, vfio is only passing it through to
> > > iommufd, where an error related to lack of the hardware feature is
> > > masked behind an -EPERM by the time it gets back to vfio, making any
> > > sort of advisory to the user about the module option convoluted.  It
> > > seems like iommufd should own the option to opt-out universally, not
> > > just through the vfio use case.  Thanks,  
> > 
> > My thinking is this option shouldn't exist at all in other iommufd
> > users. eg I don't see value in VDPA supporting it.
> 
> I disagree, the IOMMU interface is responsible for isolating the
> device, this option doesn't make any sense to live in vfio-main, which
> is the reason it was always a type1 option.  

You just agreed to this above?

> If vdpa doesn't allow full device access such that it can guarantee
> that a device cannot generate a DMA that can spoof MSI, then it
> sounds like the flag we pass when attaching a device to iommfd
> should to reflect this difference in usage.

VDPA allows arbitary DMA just like VFIO. At most VDPA limits the MMIO
touches.

> The driver either requires full isolation, default, or can indicate a
> form of restricted DMA programming that prevents interrupt spoofing.
> The policy whether to permit unsafe configurations should exist in one
> place, iommufd.

iommufd doesn't know the level of unsafely the external driver is
creating, and IMHO we don't actually want to enable this more
widely. So I don't want to see a global kernel wide flag at this point
until we get reason to make more than just VFIO insecure.

Jason

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Mon, Nov 07, 2022 at 08:18:53AM -0700, Alex Williamson wrote:
> On Mon, 7 Nov 2022 09:19:43 -0400
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > On Mon, Oct 31, 2022 at 04:45:26PM -0600, Alex Williamson wrote:
> > 
> > > > It is one idea, it depends how literal you want to be on "module
> > > > parameters are ABI". IMHO it is a weak form of ABI and the need of
> > > > this paramter in particular is not that common in modern times, AFAIK.
> > > > 
> > > > So perhaps we just also expose it through vfio.ko and expect people to
> > > > migrate. That would give a window were both options are available.  
> > > 
> > > That might be best.  Ultimately this is an opt-out of a feature that
> > > has security implications, so I'd rather error on the side of requiring
> > > the user to re-assert that opt-out.  It seems the potential good in
> > > eliminating stale or unnecessary options outweighs any weak claims of
> > > preserving an ABI for a module that's no longer in service.  
> > 
> > Ok, lets do this
> > 
> > --- a/drivers/vfio/vfio_main.c
> > +++ b/drivers/vfio/vfio_main.c
> > @@ -55,6 +55,11 @@ static struct vfio {
> >  bool vfio_allow_unsafe_interrupts;
> >  EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
> >  
> > +module_param_named(allow_unsafe_interrupts,
> > +                  vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
> > +MODULE_PARM_DESC(allow_unsafe_interrupts,
> > +                "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
> > +
> >  static DEFINE_XARRAY(vfio_device_set_xa);
> >  static const struct file_operations vfio_group_fops;
> > 
> > > However, I'd question whether vfio is the right place for that new
> > > module option.  As proposed, vfio is only passing it through to
> > > iommufd, where an error related to lack of the hardware feature is
> > > masked behind an -EPERM by the time it gets back to vfio, making any
> > > sort of advisory to the user about the module option convoluted.  It
> > > seems like iommufd should own the option to opt-out universally, not
> > > just through the vfio use case.  Thanks,  
> > 
> > My thinking is this option shouldn't exist at all in other iommufd
> > users. eg I don't see value in VDPA supporting it.
> 
> I disagree, the IOMMU interface is responsible for isolating the
> device, this option doesn't make any sense to live in vfio-main, which
> is the reason it was always a type1 option.  

You just agreed to this above?

> If vdpa doesn't allow full device access such that it can guarantee
> that a device cannot generate a DMA that can spoof MSI, then it
> sounds like the flag we pass when attaching a device to iommfd
> should to reflect this difference in usage.

VDPA allows arbitary DMA just like VFIO. At most VDPA limits the MMIO
touches.

> The driver either requires full isolation, default, or can indicate a
> form of restricted DMA programming that prevents interrupt spoofing.
> The policy whether to permit unsafe configurations should exist in one
> place, iommufd.

iommufd doesn't know the level of unsafely the external driver is
creating, and IMHO we don't actually want to enable this more
widely. So I don't want to see a global kernel wide flag at this point
until we get reason to make more than just VFIO insecure.

Jason

Re: [Intel-gfx] [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Mon, Nov 07, 2022 at 08:18:53AM -0700, Alex Williamson wrote:
> On Mon, 7 Nov 2022 09:19:43 -0400
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > On Mon, Oct 31, 2022 at 04:45:26PM -0600, Alex Williamson wrote:
> > 
> > > > It is one idea, it depends how literal you want to be on "module
> > > > parameters are ABI". IMHO it is a weak form of ABI and the need of
> > > > this paramter in particular is not that common in modern times, AFAIK.
> > > > 
> > > > So perhaps we just also expose it through vfio.ko and expect people to
> > > > migrate. That would give a window were both options are available.  
> > > 
> > > That might be best.  Ultimately this is an opt-out of a feature that
> > > has security implications, so I'd rather error on the side of requiring
> > > the user to re-assert that opt-out.  It seems the potential good in
> > > eliminating stale or unnecessary options outweighs any weak claims of
> > > preserving an ABI for a module that's no longer in service.  
> > 
> > Ok, lets do this
> > 
> > --- a/drivers/vfio/vfio_main.c
> > +++ b/drivers/vfio/vfio_main.c
> > @@ -55,6 +55,11 @@ static struct vfio {
> >  bool vfio_allow_unsafe_interrupts;
> >  EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
> >  
> > +module_param_named(allow_unsafe_interrupts,
> > +                  vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
> > +MODULE_PARM_DESC(allow_unsafe_interrupts,
> > +                "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
> > +
> >  static DEFINE_XARRAY(vfio_device_set_xa);
> >  static const struct file_operations vfio_group_fops;
> > 
> > > However, I'd question whether vfio is the right place for that new
> > > module option.  As proposed, vfio is only passing it through to
> > > iommufd, where an error related to lack of the hardware feature is
> > > masked behind an -EPERM by the time it gets back to vfio, making any
> > > sort of advisory to the user about the module option convoluted.  It
> > > seems like iommufd should own the option to opt-out universally, not
> > > just through the vfio use case.  Thanks,  
> > 
> > My thinking is this option shouldn't exist at all in other iommufd
> > users. eg I don't see value in VDPA supporting it.
> 
> I disagree, the IOMMU interface is responsible for isolating the
> device, this option doesn't make any sense to live in vfio-main, which
> is the reason it was always a type1 option.  

You just agreed to this above?

> If vdpa doesn't allow full device access such that it can guarantee
> that a device cannot generate a DMA that can spoof MSI, then it
> sounds like the flag we pass when attaching a device to iommfd
> should to reflect this difference in usage.

VDPA allows arbitary DMA just like VFIO. At most VDPA limits the MMIO
touches.

> The driver either requires full isolation, default, or can indicate a
> form of restricted DMA programming that prevents interrupt spoofing.
> The policy whether to permit unsafe configurations should exist in one
> place, iommufd.

iommufd doesn't know the level of unsafely the external driver is
creating, and IMHO we don't actually want to enable this more
widely. So I don't want to see a global kernel wide flag at this point
until we get reason to make more than just VFIO insecure.

Jason

[Intel-gfx] ✗ Fi.CI.BUILD: failure for Connect VFIO to IOMMUFD (rev2)

[Patchwork]

== Series Details ==

Series: Connect VFIO to IOMMUFD (rev2)
URL   : https://patchwork.freedesktop.org/series/110133/
State : failure

== Summary ==

Error: patch https://patchwork.freedesktop.org/api/1.0/series/110133/revisions/2/mbox/ not applied
Applying: vfio: Move vfio_device driver open/close code to a function
Applying: vfio: Move vfio_device_assign_container() into vfio_device_first_open()
Applying: vfio: Rename vfio_device_assign/unassign_container()
Applying: vfio: Move storage of allow_unsafe_interrupts to vfio_main.c
error: sha1 information is lacking or useless (drivers/vfio/vfio_main.c).
error: could not build fake ancestor
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0004 vfio: Move storage of allow_unsafe_interrupts to vfio_main.c
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Mon, 7 Nov 2022 11:32:40 -0400
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Mon, Nov 07, 2022 at 08:18:53AM -0700, Alex Williamson wrote:
> > On Mon, 7 Nov 2022 09:19:43 -0400
> > Jason Gunthorpe <jgg@nvidia.com> wrote:
> >   
> > > On Mon, Oct 31, 2022 at 04:45:26PM -0600, Alex Williamson wrote:
> > >   
> > > > > It is one idea, it depends how literal you want to be on "module
> > > > > parameters are ABI". IMHO it is a weak form of ABI and the need of
> > > > > this paramter in particular is not that common in modern times, AFAIK.
> > > > > 
> > > > > So perhaps we just also expose it through vfio.ko and expect people to
> > > > > migrate. That would give a window were both options are available.    
> > > > 
> > > > That might be best.  Ultimately this is an opt-out of a feature that
> > > > has security implications, so I'd rather error on the side of requiring
> > > > the user to re-assert that opt-out.  It seems the potential good in
> > > > eliminating stale or unnecessary options outweighs any weak claims of
> > > > preserving an ABI for a module that's no longer in service.    
> > > 
> > > Ok, lets do this
> > > 
> > > --- a/drivers/vfio/vfio_main.c
> > > +++ b/drivers/vfio/vfio_main.c
> > > @@ -55,6 +55,11 @@ static struct vfio {
> > >  bool vfio_allow_unsafe_interrupts;
> > >  EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
> > >  
> > > +module_param_named(allow_unsafe_interrupts,
> > > +                  vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
> > > +MODULE_PARM_DESC(allow_unsafe_interrupts,
> > > +                "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
> > > +
> > >  static DEFINE_XARRAY(vfio_device_set_xa);
> > >  static const struct file_operations vfio_group_fops;
> > >   
> > > > However, I'd question whether vfio is the right place for that new
> > > > module option.  As proposed, vfio is only passing it through to
> > > > iommufd, where an error related to lack of the hardware feature is
> > > > masked behind an -EPERM by the time it gets back to vfio, making any
> > > > sort of advisory to the user about the module option convoluted.  It
> > > > seems like iommufd should own the option to opt-out universally, not
> > > > just through the vfio use case.  Thanks,    
> > > 
> > > My thinking is this option shouldn't exist at all in other iommufd
> > > users. eg I don't see value in VDPA supporting it.  
> > 
> > I disagree, the IOMMU interface is responsible for isolating the
> > device, this option doesn't make any sense to live in vfio-main, which
> > is the reason it was always a type1 option.    
> 
> You just agreed to this above?

After further consideration... I don't think the option on vfio-main
makes sense, basically for the same reason that the original option
existed on the IOMMU backend rather than vfio-core.  The option
describes a means to relax a specific aspect of IOMMU isolation, which
makes more sense to expose via the IOMMU provider, imo.  For example,
vfio-main cannot generate an equivalent error message as provided in
type1 today, it's too far removed from the IOMMU feature support.

> > If vdpa doesn't allow full device access such that it can guarantee
> > that a device cannot generate a DMA that can spoof MSI, then it
> > sounds like the flag we pass when attaching a device to iommfd
> > should to reflect this difference in usage.  
> 
> VDPA allows arbitary DMA just like VFIO. At most VDPA limits the MMIO
> touches.

So why exactly isn't this an issue for VDPA?  Are we just burying our
head in the sand that such platforms exists and can still be useful
given the appropriate risk vs reward trade-off?

> > The driver either requires full isolation, default, or can indicate a
> > form of restricted DMA programming that prevents interrupt spoofing.
> > The policy whether to permit unsafe configurations should exist in one
> > place, iommufd.  
> 
> iommufd doesn't know the level of unsafely the external driver is
> creating,

Thus the proposed flag.  But maybe we don't need it if VDPA has no
inherent protection against MSI spoofing itself.

> and IMHO we don't actually want to enable this more
> widely. So I don't want to see a global kernel wide flag at this point
> until we get reason to make more than just VFIO insecure.

But this brings into question the entire existence of the opt-in.  Do
we agree that there are valid use cases for such an option?

Unlike things like ACS overrides, lack of interrupt isolation really
requires a malicious actor.  We're not going to inadvertently overlap
DMA to interrupt addresses like we might to a non-isolated MMIO ranges.
Therefore an admin can make a reasonable determination relative to the
extent to which the userspace is trusted.  This is not unlike opt-outs
to CPU vulnerability mitigation imo, there are use cases where the
performance or functionality is more important than the isolation.
Hand waving this away as a vfio-unique insecurity is a bad precedent
for iommufd.  Thanks,

Alex

Re: [Intel-gfx] [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Mon, 7 Nov 2022 11:32:40 -0400
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Mon, Nov 07, 2022 at 08:18:53AM -0700, Alex Williamson wrote:
> > On Mon, 7 Nov 2022 09:19:43 -0400
> > Jason Gunthorpe <jgg@nvidia.com> wrote:
> >   
> > > On Mon, Oct 31, 2022 at 04:45:26PM -0600, Alex Williamson wrote:
> > >   
> > > > > It is one idea, it depends how literal you want to be on "module
> > > > > parameters are ABI". IMHO it is a weak form of ABI and the need of
> > > > > this paramter in particular is not that common in modern times, AFAIK.
> > > > > 
> > > > > So perhaps we just also expose it through vfio.ko and expect people to
> > > > > migrate. That would give a window were both options are available.    
> > > > 
> > > > That might be best.  Ultimately this is an opt-out of a feature that
> > > > has security implications, so I'd rather error on the side of requiring
> > > > the user to re-assert that opt-out.  It seems the potential good in
> > > > eliminating stale or unnecessary options outweighs any weak claims of
> > > > preserving an ABI for a module that's no longer in service.    
> > > 
> > > Ok, lets do this
> > > 
> > > --- a/drivers/vfio/vfio_main.c
> > > +++ b/drivers/vfio/vfio_main.c
> > > @@ -55,6 +55,11 @@ static struct vfio {
> > >  bool vfio_allow_unsafe_interrupts;
> > >  EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
> > >  
> > > +module_param_named(allow_unsafe_interrupts,
> > > +                  vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
> > > +MODULE_PARM_DESC(allow_unsafe_interrupts,
> > > +                "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
> > > +
> > >  static DEFINE_XARRAY(vfio_device_set_xa);
> > >  static const struct file_operations vfio_group_fops;
> > >   
> > > > However, I'd question whether vfio is the right place for that new
> > > > module option.  As proposed, vfio is only passing it through to
> > > > iommufd, where an error related to lack of the hardware feature is
> > > > masked behind an -EPERM by the time it gets back to vfio, making any
> > > > sort of advisory to the user about the module option convoluted.  It
> > > > seems like iommufd should own the option to opt-out universally, not
> > > > just through the vfio use case.  Thanks,    
> > > 
> > > My thinking is this option shouldn't exist at all in other iommufd
> > > users. eg I don't see value in VDPA supporting it.  
> > 
> > I disagree, the IOMMU interface is responsible for isolating the
> > device, this option doesn't make any sense to live in vfio-main, which
> > is the reason it was always a type1 option.    
> 
> You just agreed to this above?

After further consideration... I don't think the option on vfio-main
makes sense, basically for the same reason that the original option
existed on the IOMMU backend rather than vfio-core.  The option
describes a means to relax a specific aspect of IOMMU isolation, which
makes more sense to expose via the IOMMU provider, imo.  For example,
vfio-main cannot generate an equivalent error message as provided in
type1 today, it's too far removed from the IOMMU feature support.

> > If vdpa doesn't allow full device access such that it can guarantee
> > that a device cannot generate a DMA that can spoof MSI, then it
> > sounds like the flag we pass when attaching a device to iommfd
> > should to reflect this difference in usage.  
> 
> VDPA allows arbitary DMA just like VFIO. At most VDPA limits the MMIO
> touches.

So why exactly isn't this an issue for VDPA?  Are we just burying our
head in the sand that such platforms exists and can still be useful
given the appropriate risk vs reward trade-off?

> > The driver either requires full isolation, default, or can indicate a
> > form of restricted DMA programming that prevents interrupt spoofing.
> > The policy whether to permit unsafe configurations should exist in one
> > place, iommufd.  
> 
> iommufd doesn't know the level of unsafely the external driver is
> creating,

Thus the proposed flag.  But maybe we don't need it if VDPA has no
inherent protection against MSI spoofing itself.

> and IMHO we don't actually want to enable this more
> widely. So I don't want to see a global kernel wide flag at this point
> until we get reason to make more than just VFIO insecure.

But this brings into question the entire existence of the opt-in.  Do
we agree that there are valid use cases for such an option?

Unlike things like ACS overrides, lack of interrupt isolation really
requires a malicious actor.  We're not going to inadvertently overlap
DMA to interrupt addresses like we might to a non-isolated MMIO ranges.
Therefore an admin can make a reasonable determination relative to the
extent to which the userspace is trusted.  This is not unlike opt-outs
to CPU vulnerability mitigation imo, there are use cases where the
performance or functionality is more important than the isolation.
Hand waving this away as a vfio-unique insecurity is a bad precedent
for iommufd.  Thanks,

Alex

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Mon, 7 Nov 2022 11:32:40 -0400
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Mon, Nov 07, 2022 at 08:18:53AM -0700, Alex Williamson wrote:
> > On Mon, 7 Nov 2022 09:19:43 -0400
> > Jason Gunthorpe <jgg@nvidia.com> wrote:
> >   
> > > On Mon, Oct 31, 2022 at 04:45:26PM -0600, Alex Williamson wrote:
> > >   
> > > > > It is one idea, it depends how literal you want to be on "module
> > > > > parameters are ABI". IMHO it is a weak form of ABI and the need of
> > > > > this paramter in particular is not that common in modern times, AFAIK.
> > > > > 
> > > > > So perhaps we just also expose it through vfio.ko and expect people to
> > > > > migrate. That would give a window were both options are available.    
> > > > 
> > > > That might be best.  Ultimately this is an opt-out of a feature that
> > > > has security implications, so I'd rather error on the side of requiring
> > > > the user to re-assert that opt-out.  It seems the potential good in
> > > > eliminating stale or unnecessary options outweighs any weak claims of
> > > > preserving an ABI for a module that's no longer in service.    
> > > 
> > > Ok, lets do this
> > > 
> > > --- a/drivers/vfio/vfio_main.c
> > > +++ b/drivers/vfio/vfio_main.c
> > > @@ -55,6 +55,11 @@ static struct vfio {
> > >  bool vfio_allow_unsafe_interrupts;
> > >  EXPORT_SYMBOL_GPL(vfio_allow_unsafe_interrupts);
> > >  
> > > +module_param_named(allow_unsafe_interrupts,
> > > +                  vfio_allow_unsafe_interrupts, bool, S_IRUGO | S_IWUSR);
> > > +MODULE_PARM_DESC(allow_unsafe_interrupts,
> > > +                "Enable VFIO IOMMU support for on platforms without interrupt remapping support.");
> > > +
> > >  static DEFINE_XARRAY(vfio_device_set_xa);
> > >  static const struct file_operations vfio_group_fops;
> > >   
> > > > However, I'd question whether vfio is the right place for that new
> > > > module option.  As proposed, vfio is only passing it through to
> > > > iommufd, where an error related to lack of the hardware feature is
> > > > masked behind an -EPERM by the time it gets back to vfio, making any
> > > > sort of advisory to the user about the module option convoluted.  It
> > > > seems like iommufd should own the option to opt-out universally, not
> > > > just through the vfio use case.  Thanks,    
> > > 
> > > My thinking is this option shouldn't exist at all in other iommufd
> > > users. eg I don't see value in VDPA supporting it.  
> > 
> > I disagree, the IOMMU interface is responsible for isolating the
> > device, this option doesn't make any sense to live in vfio-main, which
> > is the reason it was always a type1 option.    
> 
> You just agreed to this above?

After further consideration... I don't think the option on vfio-main
makes sense, basically for the same reason that the original option
existed on the IOMMU backend rather than vfio-core.  The option
describes a means to relax a specific aspect of IOMMU isolation, which
makes more sense to expose via the IOMMU provider, imo.  For example,
vfio-main cannot generate an equivalent error message as provided in
type1 today, it's too far removed from the IOMMU feature support.

> > If vdpa doesn't allow full device access such that it can guarantee
> > that a device cannot generate a DMA that can spoof MSI, then it
> > sounds like the flag we pass when attaching a device to iommfd
> > should to reflect this difference in usage.  
> 
> VDPA allows arbitary DMA just like VFIO. At most VDPA limits the MMIO
> touches.

So why exactly isn't this an issue for VDPA?  Are we just burying our
head in the sand that such platforms exists and can still be useful
given the appropriate risk vs reward trade-off?

> > The driver either requires full isolation, default, or can indicate a
> > form of restricted DMA programming that prevents interrupt spoofing.
> > The policy whether to permit unsafe configurations should exist in one
> > place, iommufd.  
> 
> iommufd doesn't know the level of unsafely the external driver is
> creating,

Thus the proposed flag.  But maybe we don't need it if VDPA has no
inherent protection against MSI spoofing itself.

> and IMHO we don't actually want to enable this more
> widely. So I don't want to see a global kernel wide flag at this point
> until we get reason to make more than just VFIO insecure.

But this brings into question the entire existence of the opt-in.  Do
we agree that there are valid use cases for such an option?

Unlike things like ACS overrides, lack of interrupt isolation really
requires a malicious actor.  We're not going to inadvertently overlap
DMA to interrupt addresses like we might to a non-isolated MMIO ranges.
Therefore an admin can make a reasonable determination relative to the
extent to which the userspace is trusted.  This is not unlike opt-outs
to CPU vulnerability mitigation imo, there are use cases where the
performance or functionality is more important than the isolation.
Hand waving this away as a vfio-unique insecurity is a bad precedent
for iommufd.  Thanks,

Alex

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Mon, Nov 07, 2022 at 11:05:08AM -0700, Alex Williamson wrote:

> After further consideration... I don't think the option on vfio-main
> makes sense, basically for the same reason that the original option
> existed on the IOMMU backend rather than vfio-core.  The option
> describes a means to relax a specific aspect of IOMMU isolation, which
> makes more sense to expose via the IOMMU provider, imo.  For example,
> vfio-main cannot generate an equivalent error message as provided in
> type1 today, it's too far removed from the IOMMU feature support.

vfio-main can do it, we just have to be strict that the EPERM code is
always going to be this case.
 
> > > If vdpa doesn't allow full device access such that it can guarantee
> > > that a device cannot generate a DMA that can spoof MSI, then it
> > > sounds like the flag we pass when attaching a device to iommfd
> > > should to reflect this difference in usage.  
> > 
> > VDPA allows arbitary DMA just like VFIO. At most VDPA limits the MMIO
> > touches.
>
> So why exactly isn't this an issue for VDPA?  Are we just burying our
> head in the sand that such platforms exists and can still be useful
> given the appropriate risk vs reward trade-off?

Simply that nobody has asked for it, and might never ask for it. This
is all support for old platforms, and there just doesn't seem to be a
"real" use case for very new (and actually rare) NIC hardware stuck
into ancient platforms with this security problem.

So I'd rather leave this in the past than carry forward a security
exception as some ongoing 1st class thing.

> > and IMHO we don't actually want to enable this more
> > widely. So I don't want to see a global kernel wide flag at this point
> > until we get reason to make more than just VFIO insecure.
> 
> But this brings into question the entire existence of the opt-in.  Do
> we agree that there are valid use cases for such an option?

I think it is something VFIO has historically allowed and I think we
can continue to allow it, but I don't think we should encourage its
use or encourage it to propogate to wider areas given that the
legitimate use cases are focused on fairly old hardware at this point.

So, I'd rather wait for someone to ask for it, and explain why they
need to use a combination of stuff where we need to have a true global
option.

> Unlike things like ACS overrides, lack of interrupt isolation really
> requires a malicious actor.  We're not going to inadvertently overlap
> DMA to interrupt addresses like we might to a non-isolated MMIO ranges.
> Therefore an admin can make a reasonable determination relative to the
> extent to which the userspace is trusted.  This is not unlike opt-outs
> to CPU vulnerability mitigation imo, there are use cases where the
> performance or functionality is more important than the isolation.
> Hand waving this away as a vfio-unique insecurity is a bad precedent
> for iommufd.

I agree with this, which is why I think it should come from the actual
user facing subsystem not be a system wide flag. The "is userspace
trusted" for VFIO may be quite different than from VDPA or whatever
else comes next.

I'd be much more comfortable with this as a system wide iommufd flag
if we also tied it to do some demonstration of privilege - eg a
requirement to open iommufd with CAP_SYS_RAWIO for instance.

That is the usual protocol for these kinds of insecurities..

I think right now we can leave this as-is and we can wait for some
more information to decide how best to proceed.

Thanks,
Jason

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Mon, Nov 07, 2022 at 11:05:08AM -0700, Alex Williamson wrote:

> After further consideration... I don't think the option on vfio-main
> makes sense, basically for the same reason that the original option
> existed on the IOMMU backend rather than vfio-core.  The option
> describes a means to relax a specific aspect of IOMMU isolation, which
> makes more sense to expose via the IOMMU provider, imo.  For example,
> vfio-main cannot generate an equivalent error message as provided in
> type1 today, it's too far removed from the IOMMU feature support.

vfio-main can do it, we just have to be strict that the EPERM code is
always going to be this case.
 
> > > If vdpa doesn't allow full device access such that it can guarantee
> > > that a device cannot generate a DMA that can spoof MSI, then it
> > > sounds like the flag we pass when attaching a device to iommfd
> > > should to reflect this difference in usage.  
> > 
> > VDPA allows arbitary DMA just like VFIO. At most VDPA limits the MMIO
> > touches.
>
> So why exactly isn't this an issue for VDPA?  Are we just burying our
> head in the sand that such platforms exists and can still be useful
> given the appropriate risk vs reward trade-off?

Simply that nobody has asked for it, and might never ask for it. This
is all support for old platforms, and there just doesn't seem to be a
"real" use case for very new (and actually rare) NIC hardware stuck
into ancient platforms with this security problem.

So I'd rather leave this in the past than carry forward a security
exception as some ongoing 1st class thing.

> > and IMHO we don't actually want to enable this more
> > widely. So I don't want to see a global kernel wide flag at this point
> > until we get reason to make more than just VFIO insecure.
> 
> But this brings into question the entire existence of the opt-in.  Do
> we agree that there are valid use cases for such an option?

I think it is something VFIO has historically allowed and I think we
can continue to allow it, but I don't think we should encourage its
use or encourage it to propogate to wider areas given that the
legitimate use cases are focused on fairly old hardware at this point.

So, I'd rather wait for someone to ask for it, and explain why they
need to use a combination of stuff where we need to have a true global
option.

> Unlike things like ACS overrides, lack of interrupt isolation really
> requires a malicious actor.  We're not going to inadvertently overlap
> DMA to interrupt addresses like we might to a non-isolated MMIO ranges.
> Therefore an admin can make a reasonable determination relative to the
> extent to which the userspace is trusted.  This is not unlike opt-outs
> to CPU vulnerability mitigation imo, there are use cases where the
> performance or functionality is more important than the isolation.
> Hand waving this away as a vfio-unique insecurity is a bad precedent
> for iommufd.

I agree with this, which is why I think it should come from the actual
user facing subsystem not be a system wide flag. The "is userspace
trusted" for VFIO may be quite different than from VDPA or whatever
else comes next.

I'd be much more comfortable with this as a system wide iommufd flag
if we also tied it to do some demonstration of privilege - eg a
requirement to open iommufd with CAP_SYS_RAWIO for instance.

That is the usual protocol for these kinds of insecurities..

I think right now we can leave this as-is and we can wait for some
more information to decide how best to proceed.

Thanks,
Jason

Re: [Intel-gfx] [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Mon, Nov 07, 2022 at 11:05:08AM -0700, Alex Williamson wrote:

> After further consideration... I don't think the option on vfio-main
> makes sense, basically for the same reason that the original option
> existed on the IOMMU backend rather than vfio-core.  The option
> describes a means to relax a specific aspect of IOMMU isolation, which
> makes more sense to expose via the IOMMU provider, imo.  For example,
> vfio-main cannot generate an equivalent error message as provided in
> type1 today, it's too far removed from the IOMMU feature support.

vfio-main can do it, we just have to be strict that the EPERM code is
always going to be this case.
 
> > > If vdpa doesn't allow full device access such that it can guarantee
> > > that a device cannot generate a DMA that can spoof MSI, then it
> > > sounds like the flag we pass when attaching a device to iommfd
> > > should to reflect this difference in usage.  
> > 
> > VDPA allows arbitary DMA just like VFIO. At most VDPA limits the MMIO
> > touches.
>
> So why exactly isn't this an issue for VDPA?  Are we just burying our
> head in the sand that such platforms exists and can still be useful
> given the appropriate risk vs reward trade-off?

Simply that nobody has asked for it, and might never ask for it. This
is all support for old platforms, and there just doesn't seem to be a
"real" use case for very new (and actually rare) NIC hardware stuck
into ancient platforms with this security problem.

So I'd rather leave this in the past than carry forward a security
exception as some ongoing 1st class thing.

> > and IMHO we don't actually want to enable this more
> > widely. So I don't want to see a global kernel wide flag at this point
> > until we get reason to make more than just VFIO insecure.
> 
> But this brings into question the entire existence of the opt-in.  Do
> we agree that there are valid use cases for such an option?

I think it is something VFIO has historically allowed and I think we
can continue to allow it, but I don't think we should encourage its
use or encourage it to propogate to wider areas given that the
legitimate use cases are focused on fairly old hardware at this point.

So, I'd rather wait for someone to ask for it, and explain why they
need to use a combination of stuff where we need to have a true global
option.

> Unlike things like ACS overrides, lack of interrupt isolation really
> requires a malicious actor.  We're not going to inadvertently overlap
> DMA to interrupt addresses like we might to a non-isolated MMIO ranges.
> Therefore an admin can make a reasonable determination relative to the
> extent to which the userspace is trusted.  This is not unlike opt-outs
> to CPU vulnerability mitigation imo, there are use cases where the
> performance or functionality is more important than the isolation.
> Hand waving this away as a vfio-unique insecurity is a bad precedent
> for iommufd.

I agree with this, which is why I think it should come from the actual
user facing subsystem not be a system wide flag. The "is userspace
trusted" for VFIO may be quite different than from VDPA or whatever
else comes next.

I'd be much more comfortable with this as a system wide iommufd flag
if we also tied it to do some demonstration of privilege - eg a
requirement to open iommufd with CAP_SYS_RAWIO for instance.

That is the usual protocol for these kinds of insecurities..

I think right now we can leave this as-is and we can wait for some
more information to decide how best to proceed.

Thanks,
Jason

Re: [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Jason Gunthorpe]

On Wed, Nov 02, 2022 at 03:28:20PM +0800, Yi Liu wrote:
> On 2022/10/26 02:50, Jason Gunthorpe wrote:
> > This makes VFIO_GROUP_SET_CONTAINER accept both a vfio container FD and an
> > iommufd.
> > 
> > In iommufd mode an IOAS will exist after the SET_CONTAINER, but it will
> > not be attached to any groups.
> 
> is there any special reason that we cannot attach the IOAS in the SET
> container phase or SET_IOMMU phase?

It is because iommufd has been deliberately made to work only on
struct device * not iommu_groups, and when we go to do the
SET_CONTAINER we have no idea what the device will be.

So defering the operation is the cleanest approach.

> >  From a VFIO perspective this means that the VFIO_GROUP_GET_STATUS and
> > VFIO_GROUP_FLAGS_VIABLE works subtly differently. With the container FD
> > the iommu_group_claim_dma_owner() is done during SET_CONTAINER but for
> > IOMMFD this is done during VFIO_GROUP_GET_DEVICE_FD. Meaning that
> 
> s/IOMMFD/IOMMUFD

Done

Jason

Re: [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Jason Gunthorpe]

On Wed, Nov 02, 2022 at 03:28:20PM +0800, Yi Liu wrote:
> On 2022/10/26 02:50, Jason Gunthorpe wrote:
> > This makes VFIO_GROUP_SET_CONTAINER accept both a vfio container FD and an
> > iommufd.
> > 
> > In iommufd mode an IOAS will exist after the SET_CONTAINER, but it will
> > not be attached to any groups.
> 
> is there any special reason that we cannot attach the IOAS in the SET
> container phase or SET_IOMMU phase?

It is because iommufd has been deliberately made to work only on
struct device * not iommu_groups, and when we go to do the
SET_CONTAINER we have no idea what the device will be.

So defering the operation is the cleanest approach.

> >  From a VFIO perspective this means that the VFIO_GROUP_GET_STATUS and
> > VFIO_GROUP_FLAGS_VIABLE works subtly differently. With the container FD
> > the iommu_group_claim_dma_owner() is done during SET_CONTAINER but for
> > IOMMFD this is done during VFIO_GROUP_GET_DEVICE_FD. Meaning that
> 
> s/IOMMFD/IOMMUFD

Done

Jason

Re: [Intel-gfx] [PATCH 06/10] vfio-iommufd: Allow iommufd to be used in place of a container fd

[Jason Gunthorpe]

On Wed, Nov 02, 2022 at 03:28:20PM +0800, Yi Liu wrote:
> On 2022/10/26 02:50, Jason Gunthorpe wrote:
> > This makes VFIO_GROUP_SET_CONTAINER accept both a vfio container FD and an
> > iommufd.
> > 
> > In iommufd mode an IOAS will exist after the SET_CONTAINER, but it will
> > not be attached to any groups.
> 
> is there any special reason that we cannot attach the IOAS in the SET
> container phase or SET_IOMMU phase?

It is because iommufd has been deliberately made to work only on
struct device * not iommu_groups, and when we go to do the
SET_CONTAINER we have no idea what the device will be.

So defering the operation is the cleanest approach.

> >  From a VFIO perspective this means that the VFIO_GROUP_GET_STATUS and
> > VFIO_GROUP_FLAGS_VIABLE works subtly differently. With the container FD
> > the iommu_group_claim_dma_owner() is done during SET_CONTAINER but for
> > IOMMFD this is done during VFIO_GROUP_GET_DEVICE_FD. Meaning that
> 
> s/IOMMFD/IOMMUFD

Done

Jason

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Mon, 7 Nov 2022 14:45:59 -0400
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Mon, Nov 07, 2022 at 11:05:08AM -0700, Alex Williamson wrote:
> 
> > After further consideration... I don't think the option on vfio-main
> > makes sense, basically for the same reason that the original option
> > existed on the IOMMU backend rather than vfio-core.  The option
> > describes a means to relax a specific aspect of IOMMU isolation, which
> > makes more sense to expose via the IOMMU provider, imo.  For example,
> > vfio-main cannot generate an equivalent error message as provided in
> > type1 today, it's too far removed from the IOMMU feature support.  
> 
> vfio-main can do it, we just have to be strict that the EPERM code is
> always going to be this case.

This doesn't seem very practical.

> > > > If vdpa doesn't allow full device access such that it can guarantee
> > > > that a device cannot generate a DMA that can spoof MSI, then it
> > > > sounds like the flag we pass when attaching a device to iommfd
> > > > should to reflect this difference in usage.    
> > > 
> > > VDPA allows arbitary DMA just like VFIO. At most VDPA limits the MMIO
> > > touches.  
> >
> > So why exactly isn't this an issue for VDPA?  Are we just burying our
> > head in the sand that such platforms exists and can still be useful
> > given the appropriate risk vs reward trade-off?  
> 
> Simply that nobody has asked for it, and might never ask for it. This
> is all support for old platforms, and there just doesn't seem to be a
> "real" use case for very new (and actually rare) NIC hardware stuck
> into ancient platforms with this security problem.

vIOMMU support for interrupt remapping is relatively new, the nesting
case is important as well.

> So I'd rather leave this in the past than carry forward a security
> exception as some ongoing 1st class thing.
> 
> > > and IMHO we don't actually want to enable this more
> > > widely. So I don't want to see a global kernel wide flag at this point
> > > until we get reason to make more than just VFIO insecure.  
> > 
> > But this brings into question the entire existence of the opt-in.  Do
> > we agree that there are valid use cases for such an option?  
> 
> I think it is something VFIO has historically allowed and I think we
> can continue to allow it, but I don't think we should encourage its
> use or encourage it to propogate to wider areas given that the
> legitimate use cases are focused on fairly old hardware at this point.
> 
> So, I'd rather wait for someone to ask for it, and explain why they
> need to use a combination of stuff where we need to have a true global
> option.
> 
> > Unlike things like ACS overrides, lack of interrupt isolation really
> > requires a malicious actor.  We're not going to inadvertently overlap
> > DMA to interrupt addresses like we might to a non-isolated MMIO ranges.
> > Therefore an admin can make a reasonable determination relative to the
> > extent to which the userspace is trusted.  This is not unlike opt-outs
> > to CPU vulnerability mitigation imo, there are use cases where the
> > performance or functionality is more important than the isolation.
> > Hand waving this away as a vfio-unique insecurity is a bad precedent
> > for iommufd.  
> 
> I agree with this, which is why I think it should come from the actual
> user facing subsystem not be a system wide flag. The "is userspace
> trusted" for VFIO may be quite different than from VDPA or whatever
> else comes next.
> 
> I'd be much more comfortable with this as a system wide iommufd flag
> if we also tied it to do some demonstration of privilege - eg a
> requirement to open iommufd with CAP_SYS_RAWIO for instance.

Which is not compatible to existing use cases, which is also why we
can't invent some way to allow some applications to run without CPU
mitigations, while requiring it for others as a baseline.

> That is the usual protocol for these kinds of insecurities..

Hmm, is it?

> I think right now we can leave this as-is and we can wait for some
> more information to decide how best to proceed.

It's certainly not acceptable in the latest proposal, iommufd consumes
an option set by another module and when that module goes away, so does
any claim of compatibility.  The code becomes dead and the feature not
present.  The option doesn't belong on the vfio module.  Do we need a
vfio-iommufd module to host it?  Thanks,

Alex

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Mon, 7 Nov 2022 14:45:59 -0400
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Mon, Nov 07, 2022 at 11:05:08AM -0700, Alex Williamson wrote:
> 
> > After further consideration... I don't think the option on vfio-main
> > makes sense, basically for the same reason that the original option
> > existed on the IOMMU backend rather than vfio-core.  The option
> > describes a means to relax a specific aspect of IOMMU isolation, which
> > makes more sense to expose via the IOMMU provider, imo.  For example,
> > vfio-main cannot generate an equivalent error message as provided in
> > type1 today, it's too far removed from the IOMMU feature support.  
> 
> vfio-main can do it, we just have to be strict that the EPERM code is
> always going to be this case.

This doesn't seem very practical.

> > > > If vdpa doesn't allow full device access such that it can guarantee
> > > > that a device cannot generate a DMA that can spoof MSI, then it
> > > > sounds like the flag we pass when attaching a device to iommfd
> > > > should to reflect this difference in usage.    
> > > 
> > > VDPA allows arbitary DMA just like VFIO. At most VDPA limits the MMIO
> > > touches.  
> >
> > So why exactly isn't this an issue for VDPA?  Are we just burying our
> > head in the sand that such platforms exists and can still be useful
> > given the appropriate risk vs reward trade-off?  
> 
> Simply that nobody has asked for it, and might never ask for it. This
> is all support for old platforms, and there just doesn't seem to be a
> "real" use case for very new (and actually rare) NIC hardware stuck
> into ancient platforms with this security problem.

vIOMMU support for interrupt remapping is relatively new, the nesting
case is important as well.

> So I'd rather leave this in the past than carry forward a security
> exception as some ongoing 1st class thing.
> 
> > > and IMHO we don't actually want to enable this more
> > > widely. So I don't want to see a global kernel wide flag at this point
> > > until we get reason to make more than just VFIO insecure.  
> > 
> > But this brings into question the entire existence of the opt-in.  Do
> > we agree that there are valid use cases for such an option?  
> 
> I think it is something VFIO has historically allowed and I think we
> can continue to allow it, but I don't think we should encourage its
> use or encourage it to propogate to wider areas given that the
> legitimate use cases are focused on fairly old hardware at this point.
> 
> So, I'd rather wait for someone to ask for it, and explain why they
> need to use a combination of stuff where we need to have a true global
> option.
> 
> > Unlike things like ACS overrides, lack of interrupt isolation really
> > requires a malicious actor.  We're not going to inadvertently overlap
> > DMA to interrupt addresses like we might to a non-isolated MMIO ranges.
> > Therefore an admin can make a reasonable determination relative to the
> > extent to which the userspace is trusted.  This is not unlike opt-outs
> > to CPU vulnerability mitigation imo, there are use cases where the
> > performance or functionality is more important than the isolation.
> > Hand waving this away as a vfio-unique insecurity is a bad precedent
> > for iommufd.  
> 
> I agree with this, which is why I think it should come from the actual
> user facing subsystem not be a system wide flag. The "is userspace
> trusted" for VFIO may be quite different than from VDPA or whatever
> else comes next.
> 
> I'd be much more comfortable with this as a system wide iommufd flag
> if we also tied it to do some demonstration of privilege - eg a
> requirement to open iommufd with CAP_SYS_RAWIO for instance.

Which is not compatible to existing use cases, which is also why we
can't invent some way to allow some applications to run without CPU
mitigations, while requiring it for others as a baseline.

> That is the usual protocol for these kinds of insecurities..

Hmm, is it?

> I think right now we can leave this as-is and we can wait for some
> more information to decide how best to proceed.

It's certainly not acceptable in the latest proposal, iommufd consumes
an option set by another module and when that module goes away, so does
any claim of compatibility.  The code becomes dead and the feature not
present.  The option doesn't belong on the vfio module.  Do we need a
vfio-iommufd module to host it?  Thanks,

Alex

Re: [Intel-gfx] [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Mon, 7 Nov 2022 14:45:59 -0400
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Mon, Nov 07, 2022 at 11:05:08AM -0700, Alex Williamson wrote:
> 
> > After further consideration... I don't think the option on vfio-main
> > makes sense, basically for the same reason that the original option
> > existed on the IOMMU backend rather than vfio-core.  The option
> > describes a means to relax a specific aspect of IOMMU isolation, which
> > makes more sense to expose via the IOMMU provider, imo.  For example,
> > vfio-main cannot generate an equivalent error message as provided in
> > type1 today, it's too far removed from the IOMMU feature support.  
> 
> vfio-main can do it, we just have to be strict that the EPERM code is
> always going to be this case.

This doesn't seem very practical.

> > > > If vdpa doesn't allow full device access such that it can guarantee
> > > > that a device cannot generate a DMA that can spoof MSI, then it
> > > > sounds like the flag we pass when attaching a device to iommfd
> > > > should to reflect this difference in usage.    
> > > 
> > > VDPA allows arbitary DMA just like VFIO. At most VDPA limits the MMIO
> > > touches.  
> >
> > So why exactly isn't this an issue for VDPA?  Are we just burying our
> > head in the sand that such platforms exists and can still be useful
> > given the appropriate risk vs reward trade-off?  
> 
> Simply that nobody has asked for it, and might never ask for it. This
> is all support for old platforms, and there just doesn't seem to be a
> "real" use case for very new (and actually rare) NIC hardware stuck
> into ancient platforms with this security problem.

vIOMMU support for interrupt remapping is relatively new, the nesting
case is important as well.

> So I'd rather leave this in the past than carry forward a security
> exception as some ongoing 1st class thing.
> 
> > > and IMHO we don't actually want to enable this more
> > > widely. So I don't want to see a global kernel wide flag at this point
> > > until we get reason to make more than just VFIO insecure.  
> > 
> > But this brings into question the entire existence of the opt-in.  Do
> > we agree that there are valid use cases for such an option?  
> 
> I think it is something VFIO has historically allowed and I think we
> can continue to allow it, but I don't think we should encourage its
> use or encourage it to propogate to wider areas given that the
> legitimate use cases are focused on fairly old hardware at this point.
> 
> So, I'd rather wait for someone to ask for it, and explain why they
> need to use a combination of stuff where we need to have a true global
> option.
> 
> > Unlike things like ACS overrides, lack of interrupt isolation really
> > requires a malicious actor.  We're not going to inadvertently overlap
> > DMA to interrupt addresses like we might to a non-isolated MMIO ranges.
> > Therefore an admin can make a reasonable determination relative to the
> > extent to which the userspace is trusted.  This is not unlike opt-outs
> > to CPU vulnerability mitigation imo, there are use cases where the
> > performance or functionality is more important than the isolation.
> > Hand waving this away as a vfio-unique insecurity is a bad precedent
> > for iommufd.  
> 
> I agree with this, which is why I think it should come from the actual
> user facing subsystem not be a system wide flag. The "is userspace
> trusted" for VFIO may be quite different than from VDPA or whatever
> else comes next.
> 
> I'd be much more comfortable with this as a system wide iommufd flag
> if we also tied it to do some demonstration of privilege - eg a
> requirement to open iommufd with CAP_SYS_RAWIO for instance.

Which is not compatible to existing use cases, which is also why we
can't invent some way to allow some applications to run without CPU
mitigations, while requiring it for others as a baseline.

> That is the usual protocol for these kinds of insecurities..

Hmm, is it?

> I think right now we can leave this as-is and we can wait for some
> more information to decide how best to proceed.

It's certainly not acceptable in the latest proposal, iommufd consumes
an option set by another module and when that module goes away, so does
any claim of compatibility.  The code becomes dead and the feature not
present.  The option doesn't belong on the vfio module.  Do we need a
vfio-iommufd module to host it?  Thanks,

Alex

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Tue, Nov 08, 2022 at 03:55:20PM -0700, Alex Williamson wrote:

> > > So why exactly isn't this an issue for VDPA?  Are we just burying our
> > > head in the sand that such platforms exists and can still be useful
> > > given the appropriate risk vs reward trade-off?  
> > 
> > Simply that nobody has asked for it, and might never ask for it. This
> > is all support for old platforms, and there just doesn't seem to be a
> > "real" use case for very new (and actually rare) NIC hardware stuck
> > into ancient platforms with this security problem.
> 
> vIOMMU support for interrupt remapping is relatively new, the nesting
> case is important as well.

This is where we got hit. In the end we fixed the qemu..

> > I'd be much more comfortable with this as a system wide iommufd flag
> > if we also tied it to do some demonstration of privilege - eg a
> > requirement to open iommufd with CAP_SYS_RAWIO for instance.
> 
> Which is not compatible to existing use cases, which is also why we
> can't invent some way to allow some applications to run without CPU
> mitigations, while requiring it for others as a baseline.

Isn't it? Didn't we learn that libvirt runs as root and will open and
pass the iommufd as root?

> > That is the usual protocol for these kinds of insecurities..
> 
> Hmm, is it?

I think so. At least you should have something to shut down an
insecure feature in kernel lockdown modes. CAP_SYS_RAWIO is a simple
way to do it.

> > I think right now we can leave this as-is and we can wait for some
> > more information to decide how best to proceed.
> 
> It's certainly not acceptable in the latest proposal, iommufd consumes
> an option set by another module and when that module goes away, so does
> any claim of compatibility.  The code becomes dead and the feature not
> present.  The option doesn't belong on the vfio module.  Do we need a
> vfio-iommufd module to host it?  Thanks,

I don't know, as I said in the other email, these little things need
work and discussion to resolve. We need to recheck the security stuff
against the 2022 kernel where things have changed. We don't need to do
it all right now.

People who want allow_unsafe_interrupts to work will simply not set
VFIO_CONTAINER=n at this time. Same with P2P, vfio-no-iommu and any
other gaps we haven't discovered.

vfio-iommufd seems like overkill, I think your first suggestion to put
in vfio.ko was more practical.

My only doubt is if we should make it system wide for everything - and
I'm just a bit uncomfortable with that from a security POV. But maybe
I don't quite know exactly what the risks are.

Jason

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Tue, Nov 08, 2022 at 03:55:20PM -0700, Alex Williamson wrote:

> > > So why exactly isn't this an issue for VDPA?  Are we just burying our
> > > head in the sand that such platforms exists and can still be useful
> > > given the appropriate risk vs reward trade-off?  
> > 
> > Simply that nobody has asked for it, and might never ask for it. This
> > is all support for old platforms, and there just doesn't seem to be a
> > "real" use case for very new (and actually rare) NIC hardware stuck
> > into ancient platforms with this security problem.
> 
> vIOMMU support for interrupt remapping is relatively new, the nesting
> case is important as well.

This is where we got hit. In the end we fixed the qemu..

> > I'd be much more comfortable with this as a system wide iommufd flag
> > if we also tied it to do some demonstration of privilege - eg a
> > requirement to open iommufd with CAP_SYS_RAWIO for instance.
> 
> Which is not compatible to existing use cases, which is also why we
> can't invent some way to allow some applications to run without CPU
> mitigations, while requiring it for others as a baseline.

Isn't it? Didn't we learn that libvirt runs as root and will open and
pass the iommufd as root?

> > That is the usual protocol for these kinds of insecurities..
> 
> Hmm, is it?

I think so. At least you should have something to shut down an
insecure feature in kernel lockdown modes. CAP_SYS_RAWIO is a simple
way to do it.

> > I think right now we can leave this as-is and we can wait for some
> > more information to decide how best to proceed.
> 
> It's certainly not acceptable in the latest proposal, iommufd consumes
> an option set by another module and when that module goes away, so does
> any claim of compatibility.  The code becomes dead and the feature not
> present.  The option doesn't belong on the vfio module.  Do we need a
> vfio-iommufd module to host it?  Thanks,

I don't know, as I said in the other email, these little things need
work and discussion to resolve. We need to recheck the security stuff
against the 2022 kernel where things have changed. We don't need to do
it all right now.

People who want allow_unsafe_interrupts to work will simply not set
VFIO_CONTAINER=n at this time. Same with P2P, vfio-no-iommu and any
other gaps we haven't discovered.

vfio-iommufd seems like overkill, I think your first suggestion to put
in vfio.ko was more practical.

My only doubt is if we should make it system wide for everything - and
I'm just a bit uncomfortable with that from a security POV. But maybe
I don't quite know exactly what the risks are.

Jason

Re: [Intel-gfx] [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Tue, Nov 08, 2022 at 03:55:20PM -0700, Alex Williamson wrote:

> > > So why exactly isn't this an issue for VDPA?  Are we just burying our
> > > head in the sand that such platforms exists and can still be useful
> > > given the appropriate risk vs reward trade-off?  
> > 
> > Simply that nobody has asked for it, and might never ask for it. This
> > is all support for old platforms, and there just doesn't seem to be a
> > "real" use case for very new (and actually rare) NIC hardware stuck
> > into ancient platforms with this security problem.
> 
> vIOMMU support for interrupt remapping is relatively new, the nesting
> case is important as well.

This is where we got hit. In the end we fixed the qemu..

> > I'd be much more comfortable with this as a system wide iommufd flag
> > if we also tied it to do some demonstration of privilege - eg a
> > requirement to open iommufd with CAP_SYS_RAWIO for instance.
> 
> Which is not compatible to existing use cases, which is also why we
> can't invent some way to allow some applications to run without CPU
> mitigations, while requiring it for others as a baseline.

Isn't it? Didn't we learn that libvirt runs as root and will open and
pass the iommufd as root?

> > That is the usual protocol for these kinds of insecurities..
> 
> Hmm, is it?

I think so. At least you should have something to shut down an
insecure feature in kernel lockdown modes. CAP_SYS_RAWIO is a simple
way to do it.

> > I think right now we can leave this as-is and we can wait for some
> > more information to decide how best to proceed.
> 
> It's certainly not acceptable in the latest proposal, iommufd consumes
> an option set by another module and when that module goes away, so does
> any claim of compatibility.  The code becomes dead and the feature not
> present.  The option doesn't belong on the vfio module.  Do we need a
> vfio-iommufd module to host it?  Thanks,

I don't know, as I said in the other email, these little things need
work and discussion to resolve. We need to recheck the security stuff
against the 2022 kernel where things have changed. We don't need to do
it all right now.

People who want allow_unsafe_interrupts to work will simply not set
VFIO_CONTAINER=n at this time. Same with P2P, vfio-no-iommu and any
other gaps we haven't discovered.

vfio-iommufd seems like overkill, I think your first suggestion to put
in vfio.ko was more practical.

My only doubt is if we should make it system wide for everything - and
I'm just a bit uncomfortable with that from a security POV. But maybe
I don't quite know exactly what the risks are.

Jason

RE: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, November 9, 2022 9:05 AM
> 
> On Tue, Nov 08, 2022 at 03:55:20PM -0700, Alex Williamson wrote:
> 
> > > > So why exactly isn't this an issue for VDPA?  Are we just burying our
> > > > head in the sand that such platforms exists and can still be useful
> > > > given the appropriate risk vs reward trade-off?
> > >
> > > Simply that nobody has asked for it, and might never ask for it. This
> > > is all support for old platforms, and there just doesn't seem to be a
> > > "real" use case for very new (and actually rare) NIC hardware stuck
> > > into ancient platforms with this security problem.
> >
> > vIOMMU support for interrupt remapping is relatively new, the nesting
> > case is important as well.
> 
> This is where we got hit. In the end we fixed the qemu..

but the point is that old qemu could have a much longer lifespan than
old platforms then when running newer kernel which supports vdpa
on old qemu the same tradeoff consideration is then not vfio specific...

> > It's certainly not acceptable in the latest proposal, iommufd consumes
> > an option set by another module and when that module goes away, so
> does
> > any claim of compatibility.  The code becomes dead and the feature not
> > present.  The option doesn't belong on the vfio module.  Do we need a
> > vfio-iommufd module to host it?  Thanks,
> 
> I don't know, as I said in the other email, these little things need
> work and discussion to resolve. We need to recheck the security stuff
> against the 2022 kernel where things have changed. We don't need to do
> it all right now.
> 
> People who want allow_unsafe_interrupts to work will simply not set
> VFIO_CONTAINER=n at this time. Same with P2P, vfio-no-iommu and any
> other gaps we haven't discovered.
> 

If all agree that VFIO_CONTAINER=n is a process to evolve, does it make
more sense to remove this patch from this series i.e. let it buried in
VFIO_CONTAINER=y for now? Then resolve it in a follow up patch if
no consensus can be made quickly at this point.

RE: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, November 9, 2022 9:05 AM
> 
> On Tue, Nov 08, 2022 at 03:55:20PM -0700, Alex Williamson wrote:
> 
> > > > So why exactly isn't this an issue for VDPA?  Are we just burying our
> > > > head in the sand that such platforms exists and can still be useful
> > > > given the appropriate risk vs reward trade-off?
> > >
> > > Simply that nobody has asked for it, and might never ask for it. This
> > > is all support for old platforms, and there just doesn't seem to be a
> > > "real" use case for very new (and actually rare) NIC hardware stuck
> > > into ancient platforms with this security problem.
> >
> > vIOMMU support for interrupt remapping is relatively new, the nesting
> > case is important as well.
> 
> This is where we got hit. In the end we fixed the qemu..

but the point is that old qemu could have a much longer lifespan than
old platforms then when running newer kernel which supports vdpa
on old qemu the same tradeoff consideration is then not vfio specific...

> > It's certainly not acceptable in the latest proposal, iommufd consumes
> > an option set by another module and when that module goes away, so
> does
> > any claim of compatibility.  The code becomes dead and the feature not
> > present.  The option doesn't belong on the vfio module.  Do we need a
> > vfio-iommufd module to host it?  Thanks,
> 
> I don't know, as I said in the other email, these little things need
> work and discussion to resolve. We need to recheck the security stuff
> against the 2022 kernel where things have changed. We don't need to do
> it all right now.
> 
> People who want allow_unsafe_interrupts to work will simply not set
> VFIO_CONTAINER=n at this time. Same with P2P, vfio-no-iommu and any
> other gaps we haven't discovered.
> 

If all agree that VFIO_CONTAINER=n is a process to evolve, does it make
more sense to remove this patch from this series i.e. let it buried in
VFIO_CONTAINER=y for now? Then resolve it in a follow up patch if
no consensus can be made quickly at this point.

Re: [Intel-gfx] [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, November 9, 2022 9:05 AM
> 
> On Tue, Nov 08, 2022 at 03:55:20PM -0700, Alex Williamson wrote:
> 
> > > > So why exactly isn't this an issue for VDPA?  Are we just burying our
> > > > head in the sand that such platforms exists and can still be useful
> > > > given the appropriate risk vs reward trade-off?
> > >
> > > Simply that nobody has asked for it, and might never ask for it. This
> > > is all support for old platforms, and there just doesn't seem to be a
> > > "real" use case for very new (and actually rare) NIC hardware stuck
> > > into ancient platforms with this security problem.
> >
> > vIOMMU support for interrupt remapping is relatively new, the nesting
> > case is important as well.
> 
> This is where we got hit. In the end we fixed the qemu..

but the point is that old qemu could have a much longer lifespan than
old platforms then when running newer kernel which supports vdpa
on old qemu the same tradeoff consideration is then not vfio specific...

> > It's certainly not acceptable in the latest proposal, iommufd consumes
> > an option set by another module and when that module goes away, so
> does
> > any claim of compatibility.  The code becomes dead and the feature not
> > present.  The option doesn't belong on the vfio module.  Do we need a
> > vfio-iommufd module to host it?  Thanks,
> 
> I don't know, as I said in the other email, these little things need
> work and discussion to resolve. We need to recheck the security stuff
> against the 2022 kernel where things have changed. We don't need to do
> it all right now.
> 
> People who want allow_unsafe_interrupts to work will simply not set
> VFIO_CONTAINER=n at this time. Same with P2P, vfio-no-iommu and any
> other gaps we haven't discovered.
> 

If all agree that VFIO_CONTAINER=n is a process to evolve, does it make
more sense to remove this patch from this series i.e. let it buried in
VFIO_CONTAINER=y for now? Then resolve it in a follow up patch if
no consensus can be made quickly at this point.

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Wed, Nov 09, 2022 at 03:21:29AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, November 9, 2022 9:05 AM
> > 
> > On Tue, Nov 08, 2022 at 03:55:20PM -0700, Alex Williamson wrote:
> > 
> > > > > So why exactly isn't this an issue for VDPA?  Are we just burying our
> > > > > head in the sand that such platforms exists and can still be useful
> > > > > given the appropriate risk vs reward trade-off?
> > > >
> > > > Simply that nobody has asked for it, and might never ask for it. This
> > > > is all support for old platforms, and there just doesn't seem to be a
> > > > "real" use case for very new (and actually rare) NIC hardware stuck
> > > > into ancient platforms with this security problem.
> > >
> > > vIOMMU support for interrupt remapping is relatively new, the nesting
> > > case is important as well.
> > 
> > This is where we got hit. In the end we fixed the qemu..
> 
> but the point is that old qemu could have a much longer lifespan than
> old platforms then when running newer kernel which supports vdpa
> on old qemu the same tradeoff consideration is then not vfio
> specific...

I think we are reaching into incredible hypotheticals here. We don't
know of any real uses cases where a very new VDPA capable device would
be assinged into a VM using an old qemu and the entire system is
expected to work. What we are seeing is people using this stuff are
making highly engineered systems and will meet the constraints.

Today VDPA doesn't support allow_unsafe_interrupts, and I don't see a
compelling reason to change that.

The threshold for introducing a kernel security hole should be much
higher than "someone could possibly do this".

> If all agree that VFIO_CONTAINER=n is a process to evolve, does it make
> more sense to remove this patch from this series i.e. let it buried in
> VFIO_CONTAINER=y for now? Then resolve it in a follow up patch if
> no consensus can be made quickly at this point.

This is worse, it would make iommufd completely unusable in situations
where we need allow_unsafe_interrupts. If we belive that is important
we should keep this patch so existing systems on kernels with
VFIO_CONTAINER=y continue to work after libvirt/qemu are upgraded to
iommufd.

Jason

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Wed, Nov 09, 2022 at 03:21:29AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, November 9, 2022 9:05 AM
> > 
> > On Tue, Nov 08, 2022 at 03:55:20PM -0700, Alex Williamson wrote:
> > 
> > > > > So why exactly isn't this an issue for VDPA?  Are we just burying our
> > > > > head in the sand that such platforms exists and can still be useful
> > > > > given the appropriate risk vs reward trade-off?
> > > >
> > > > Simply that nobody has asked for it, and might never ask for it. This
> > > > is all support for old platforms, and there just doesn't seem to be a
> > > > "real" use case for very new (and actually rare) NIC hardware stuck
> > > > into ancient platforms with this security problem.
> > >
> > > vIOMMU support for interrupt remapping is relatively new, the nesting
> > > case is important as well.
> > 
> > This is where we got hit. In the end we fixed the qemu..
> 
> but the point is that old qemu could have a much longer lifespan than
> old platforms then when running newer kernel which supports vdpa
> on old qemu the same tradeoff consideration is then not vfio
> specific...

I think we are reaching into incredible hypotheticals here. We don't
know of any real uses cases where a very new VDPA capable device would
be assinged into a VM using an old qemu and the entire system is
expected to work. What we are seeing is people using this stuff are
making highly engineered systems and will meet the constraints.

Today VDPA doesn't support allow_unsafe_interrupts, and I don't see a
compelling reason to change that.

The threshold for introducing a kernel security hole should be much
higher than "someone could possibly do this".

> If all agree that VFIO_CONTAINER=n is a process to evolve, does it make
> more sense to remove this patch from this series i.e. let it buried in
> VFIO_CONTAINER=y for now? Then resolve it in a follow up patch if
> no consensus can be made quickly at this point.

This is worse, it would make iommufd completely unusable in situations
where we need allow_unsafe_interrupts. If we belive that is important
we should keep this patch so existing systems on kernels with
VFIO_CONTAINER=y continue to work after libvirt/qemu are upgraded to
iommufd.

Jason

Re: [Intel-gfx] [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Wed, Nov 09, 2022 at 03:21:29AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, November 9, 2022 9:05 AM
> > 
> > On Tue, Nov 08, 2022 at 03:55:20PM -0700, Alex Williamson wrote:
> > 
> > > > > So why exactly isn't this an issue for VDPA?  Are we just burying our
> > > > > head in the sand that such platforms exists and can still be useful
> > > > > given the appropriate risk vs reward trade-off?
> > > >
> > > > Simply that nobody has asked for it, and might never ask for it. This
> > > > is all support for old platforms, and there just doesn't seem to be a
> > > > "real" use case for very new (and actually rare) NIC hardware stuck
> > > > into ancient platforms with this security problem.
> > >
> > > vIOMMU support for interrupt remapping is relatively new, the nesting
> > > case is important as well.
> > 
> > This is where we got hit. In the end we fixed the qemu..
> 
> but the point is that old qemu could have a much longer lifespan than
> old platforms then when running newer kernel which supports vdpa
> on old qemu the same tradeoff consideration is then not vfio
> specific...

I think we are reaching into incredible hypotheticals here. We don't
know of any real uses cases where a very new VDPA capable device would
be assinged into a VM using an old qemu and the entire system is
expected to work. What we are seeing is people using this stuff are
making highly engineered systems and will meet the constraints.

Today VDPA doesn't support allow_unsafe_interrupts, and I don't see a
compelling reason to change that.

The threshold for introducing a kernel security hole should be much
higher than "someone could possibly do this".

> If all agree that VFIO_CONTAINER=n is a process to evolve, does it make
> more sense to remove this patch from this series i.e. let it buried in
> VFIO_CONTAINER=y for now? Then resolve it in a follow up patch if
> no consensus can be made quickly at this point.

This is worse, it would make iommufd completely unusable in situations
where we need allow_unsafe_interrupts. If we belive that is important
we should keep this patch so existing systems on kernels with
VFIO_CONTAINER=y continue to work after libvirt/qemu are upgraded to
iommufd.

Jason

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Tue, 8 Nov 2022 21:05:21 -0400
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Tue, Nov 08, 2022 at 03:55:20PM -0700, Alex Williamson wrote:
> 
> > > > So why exactly isn't this an issue for VDPA?  Are we just burying our
> > > > head in the sand that such platforms exists and can still be useful
> > > > given the appropriate risk vs reward trade-off?    
> > > 
> > > Simply that nobody has asked for it, and might never ask for it. This
> > > is all support for old platforms, and there just doesn't seem to be a
> > > "real" use case for very new (and actually rare) NIC hardware stuck
> > > into ancient platforms with this security problem.  
> > 
> > vIOMMU support for interrupt remapping is relatively new, the nesting
> > case is important as well.  
> 
> This is where we got hit. In the end we fixed the qemu..
> 
> > > I'd be much more comfortable with this as a system wide iommufd flag
> > > if we also tied it to do some demonstration of privilege - eg a
> > > requirement to open iommufd with CAP_SYS_RAWIO for instance.  
> > 
> > Which is not compatible to existing use cases, which is also why we
> > can't invent some way to allow some applications to run without CPU
> > mitigations, while requiring it for others as a baseline.  
> 
> Isn't it? Didn't we learn that libvirt runs as root and will open and
> pass the iommufd as root?

We're jumping ahead to native iommufd support here, what happens when
VFIO_CONTAINER=n and it's QEMU opening the fds, with only file access
privileges?

> > > That is the usual protocol for these kinds of insecurities..  
> > 
> > Hmm, is it?  
> 
> I think so. At least you should have something to shut down an
> insecure feature in kernel lockdown modes. CAP_SYS_RAWIO is a simple
> way to do it.

How are CPU vulnerabilities handled in lockdown mode, do apps require
certain capabilities to run fast vs safe, or do we simply disallow
unsafe globally in lockdown?  I think we have a lot more leniency to
ignore/disallow flags that enable global insecurities when any sort of
lockdown is imposed.

> > > I think right now we can leave this as-is and we can wait for some
> > > more information to decide how best to proceed.  
> > 
> > It's certainly not acceptable in the latest proposal, iommufd consumes
> > an option set by another module and when that module goes away, so does
> > any claim of compatibility.  The code becomes dead and the feature not
> > present.  The option doesn't belong on the vfio module.  Do we need a
> > vfio-iommufd module to host it?  Thanks,  
> 
> I don't know, as I said in the other email, these little things need
> work and discussion to resolve. We need to recheck the security stuff
> against the 2022 kernel where things have changed. We don't need to do
> it all right now.
> 
> People who want allow_unsafe_interrupts to work will simply not set
> VFIO_CONTAINER=n at this time. Same with P2P, vfio-no-iommu and any
> other gaps we haven't discovered.
> 
> vfio-iommufd seems like overkill, I think your first suggestion to put
> in vfio.ko was more practical.

Convenient perhaps, but architecturally the wrong place for it.

> My only doubt is if we should make it system wide for everything - and
> I'm just a bit uncomfortable with that from a security POV. But maybe
> I don't quite know exactly what the risks are.

There's a paper about these sorts of attacks here[1].  As I noted
earlier, a non-malicious DMA targeting an address that would trigger an
interrupt is extremely unlikely, and the resulting vulnerability is
largely more of a denial of service, IIRC.  It would certainly be
strongly dis-recommended in any scenario where the userspace drivers
are untrusted, such as a cloud hosting provider, but there are
certainly other scenarios where either the guest or userspace drivers
are also under the control of the hosting provider and this is not such
a concern.  Thanks,

Alex

[1]https://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Tue, 8 Nov 2022 21:05:21 -0400
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Tue, Nov 08, 2022 at 03:55:20PM -0700, Alex Williamson wrote:
> 
> > > > So why exactly isn't this an issue for VDPA?  Are we just burying our
> > > > head in the sand that such platforms exists and can still be useful
> > > > given the appropriate risk vs reward trade-off?    
> > > 
> > > Simply that nobody has asked for it, and might never ask for it. This
> > > is all support for old platforms, and there just doesn't seem to be a
> > > "real" use case for very new (and actually rare) NIC hardware stuck
> > > into ancient platforms with this security problem.  
> > 
> > vIOMMU support for interrupt remapping is relatively new, the nesting
> > case is important as well.  
> 
> This is where we got hit. In the end we fixed the qemu..
> 
> > > I'd be much more comfortable with this as a system wide iommufd flag
> > > if we also tied it to do some demonstration of privilege - eg a
> > > requirement to open iommufd with CAP_SYS_RAWIO for instance.  
> > 
> > Which is not compatible to existing use cases, which is also why we
> > can't invent some way to allow some applications to run without CPU
> > mitigations, while requiring it for others as a baseline.  
> 
> Isn't it? Didn't we learn that libvirt runs as root and will open and
> pass the iommufd as root?

We're jumping ahead to native iommufd support here, what happens when
VFIO_CONTAINER=n and it's QEMU opening the fds, with only file access
privileges?

> > > That is the usual protocol for these kinds of insecurities..  
> > 
> > Hmm, is it?  
> 
> I think so. At least you should have something to shut down an
> insecure feature in kernel lockdown modes. CAP_SYS_RAWIO is a simple
> way to do it.

How are CPU vulnerabilities handled in lockdown mode, do apps require
certain capabilities to run fast vs safe, or do we simply disallow
unsafe globally in lockdown?  I think we have a lot more leniency to
ignore/disallow flags that enable global insecurities when any sort of
lockdown is imposed.

> > > I think right now we can leave this as-is and we can wait for some
> > > more information to decide how best to proceed.  
> > 
> > It's certainly not acceptable in the latest proposal, iommufd consumes
> > an option set by another module and when that module goes away, so does
> > any claim of compatibility.  The code becomes dead and the feature not
> > present.  The option doesn't belong on the vfio module.  Do we need a
> > vfio-iommufd module to host it?  Thanks,  
> 
> I don't know, as I said in the other email, these little things need
> work and discussion to resolve. We need to recheck the security stuff
> against the 2022 kernel where things have changed. We don't need to do
> it all right now.
> 
> People who want allow_unsafe_interrupts to work will simply not set
> VFIO_CONTAINER=n at this time. Same with P2P, vfio-no-iommu and any
> other gaps we haven't discovered.
> 
> vfio-iommufd seems like overkill, I think your first suggestion to put
> in vfio.ko was more practical.

Convenient perhaps, but architecturally the wrong place for it.

> My only doubt is if we should make it system wide for everything - and
> I'm just a bit uncomfortable with that from a security POV. But maybe
> I don't quite know exactly what the risks are.

There's a paper about these sorts of attacks here[1].  As I noted
earlier, a non-malicious DMA targeting an address that would trigger an
interrupt is extremely unlikely, and the resulting vulnerability is
largely more of a denial of service, IIRC.  It would certainly be
strongly dis-recommended in any scenario where the userspace drivers
are untrusted, such as a cloud hosting provider, but there are
certainly other scenarios where either the guest or userspace drivers
are also under the control of the hosting provider and this is not such
a concern.  Thanks,

Alex

[1]https://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf

Re: [Intel-gfx] [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Alex Williamson]

On Tue, 8 Nov 2022 21:05:21 -0400
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Tue, Nov 08, 2022 at 03:55:20PM -0700, Alex Williamson wrote:
> 
> > > > So why exactly isn't this an issue for VDPA?  Are we just burying our
> > > > head in the sand that such platforms exists and can still be useful
> > > > given the appropriate risk vs reward trade-off?    
> > > 
> > > Simply that nobody has asked for it, and might never ask for it. This
> > > is all support for old platforms, and there just doesn't seem to be a
> > > "real" use case for very new (and actually rare) NIC hardware stuck
> > > into ancient platforms with this security problem.  
> > 
> > vIOMMU support for interrupt remapping is relatively new, the nesting
> > case is important as well.  
> 
> This is where we got hit. In the end we fixed the qemu..
> 
> > > I'd be much more comfortable with this as a system wide iommufd flag
> > > if we also tied it to do some demonstration of privilege - eg a
> > > requirement to open iommufd with CAP_SYS_RAWIO for instance.  
> > 
> > Which is not compatible to existing use cases, which is also why we
> > can't invent some way to allow some applications to run without CPU
> > mitigations, while requiring it for others as a baseline.  
> 
> Isn't it? Didn't we learn that libvirt runs as root and will open and
> pass the iommufd as root?

We're jumping ahead to native iommufd support here, what happens when
VFIO_CONTAINER=n and it's QEMU opening the fds, with only file access
privileges?

> > > That is the usual protocol for these kinds of insecurities..  
> > 
> > Hmm, is it?  
> 
> I think so. At least you should have something to shut down an
> insecure feature in kernel lockdown modes. CAP_SYS_RAWIO is a simple
> way to do it.

How are CPU vulnerabilities handled in lockdown mode, do apps require
certain capabilities to run fast vs safe, or do we simply disallow
unsafe globally in lockdown?  I think we have a lot more leniency to
ignore/disallow flags that enable global insecurities when any sort of
lockdown is imposed.

> > > I think right now we can leave this as-is and we can wait for some
> > > more information to decide how best to proceed.  
> > 
> > It's certainly not acceptable in the latest proposal, iommufd consumes
> > an option set by another module and when that module goes away, so does
> > any claim of compatibility.  The code becomes dead and the feature not
> > present.  The option doesn't belong on the vfio module.  Do we need a
> > vfio-iommufd module to host it?  Thanks,  
> 
> I don't know, as I said in the other email, these little things need
> work and discussion to resolve. We need to recheck the security stuff
> against the 2022 kernel where things have changed. We don't need to do
> it all right now.
> 
> People who want allow_unsafe_interrupts to work will simply not set
> VFIO_CONTAINER=n at this time. Same with P2P, vfio-no-iommu and any
> other gaps we haven't discovered.
> 
> vfio-iommufd seems like overkill, I think your first suggestion to put
> in vfio.ko was more practical.

Convenient perhaps, but architecturally the wrong place for it.

> My only doubt is if we should make it system wide for everything - and
> I'm just a bit uncomfortable with that from a security POV. But maybe
> I don't quite know exactly what the risks are.

There's a paper about these sorts of attacks here[1].  As I noted
earlier, a non-malicious DMA targeting an address that would trigger an
interrupt is extremely unlikely, and the resulting vulnerability is
largely more of a denial of service, IIRC.  It would certainly be
strongly dis-recommended in any scenario where the userspace drivers
are untrusted, such as a cloud hosting provider, but there are
certainly other scenarios where either the guest or userspace drivers
are also under the control of the hosting provider and this is not such
a concern.  Thanks,

Alex

[1]https://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf

RE: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, November 9, 2022 9:11 PM
> 
> > If all agree that VFIO_CONTAINER=n is a process to evolve, does it make
> > more sense to remove this patch from this series i.e. let it buried in
> > VFIO_CONTAINER=y for now? Then resolve it in a follow up patch if
> > no consensus can be made quickly at this point.
> 
> This is worse, it would make iommufd completely unusable in situations
> where we need allow_unsafe_interrupts. If we belive that is important
> we should keep this patch so existing systems on kernels with
> VFIO_CONTAINER=y continue to work after libvirt/qemu are upgraded to
> iommufd.
> 

You are right. I kept a wrong thought that v2 has moved the option into
vfio_main which is what I commented to hold before consensus was made.

btw is it a good tradeoff by making vfio-compat as a module to carry this
option? anyway it's not necessarily to be in iommufd core when VFIO=n.

RE: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, November 9, 2022 9:11 PM
> 
> > If all agree that VFIO_CONTAINER=n is a process to evolve, does it make
> > more sense to remove this patch from this series i.e. let it buried in
> > VFIO_CONTAINER=y for now? Then resolve it in a follow up patch if
> > no consensus can be made quickly at this point.
> 
> This is worse, it would make iommufd completely unusable in situations
> where we need allow_unsafe_interrupts. If we belive that is important
> we should keep this patch so existing systems on kernels with
> VFIO_CONTAINER=y continue to work after libvirt/qemu are upgraded to
> iommufd.
> 

You are right. I kept a wrong thought that v2 has moved the option into
vfio_main which is what I commented to hold before consensus was made.

btw is it a good tradeoff by making vfio-compat as a module to carry this
option? anyway it's not necessarily to be in iommufd core when VFIO=n.

Re: [Intel-gfx] [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Wednesday, November 9, 2022 9:11 PM
> 
> > If all agree that VFIO_CONTAINER=n is a process to evolve, does it make
> > more sense to remove this patch from this series i.e. let it buried in
> > VFIO_CONTAINER=y for now? Then resolve it in a follow up patch if
> > no consensus can be made quickly at this point.
> 
> This is worse, it would make iommufd completely unusable in situations
> where we need allow_unsafe_interrupts. If we belive that is important
> we should keep this patch so existing systems on kernels with
> VFIO_CONTAINER=y continue to work after libvirt/qemu are upgraded to
> iommufd.
> 

You are right. I kept a wrong thought that v2 has moved the option into
vfio_main which is what I commented to hold before consensus was made.

btw is it a good tradeoff by making vfio-compat as a module to carry this
option? anyway it's not necessarily to be in iommufd core when VFIO=n.

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Wed, Nov 09, 2022 at 11:28:22AM -0700, Alex Williamson wrote:
> > > > I'd be much more comfortable with this as a system wide iommufd flag
> > > > if we also tied it to do some demonstration of privilege - eg a
> > > > requirement to open iommufd with CAP_SYS_RAWIO for instance.  
> > > 
> > > Which is not compatible to existing use cases, which is also why we
> > > can't invent some way to allow some applications to run without CPU
> > > mitigations, while requiring it for others as a baseline.  
> > 
> > Isn't it? Didn't we learn that libvirt runs as root and will open and
> > pass the iommufd as root?
> 
> We're jumping ahead to native iommufd support here, what happens when
> VFIO_CONTAINER=n and it's QEMU opening the fds, with only file access
> privileges?

Yes, but I am thinking aloud about how to best to do this in native
iommufd modes.

> > I think so. At least you should have something to shut down an
> > insecure feature in kernel lockdown modes. CAP_SYS_RAWIO is a simple
> > way to do it.
> 
> How are CPU vulnerabilities handled in lockdown mode, do apps require
> certain capabilities to run fast vs safe, or do we simply disallow
> unsafe globally in lockdown?  I think we have a lot more leniency to
> ignore/disallow flags that enable global insecurities when any sort of
> lockdown is imposed.

The CPU things are all information leaks from the kernel to
userspace. lockdown is about preserving kernel operating integrity -
eg preventing modification of hijacking of the running kernel.

So, like you say below, this is kind of in between, it is not
information leakage, and it is is hopefully not an integrity issue.

Being more of a DOS maybe it is fine under the lockdown scenarios. At
least I am happier to hear that.

> > vfio-iommufd seems like overkill, I think your first suggestion to put
> > in vfio.ko was more practical.
> 
> Convenient perhaps, but architecturally the wrong place for it.

Ah, that is pretty subjective. If the architecture is that the iommufd
user subsystem opts-in to this insecurity then it is an OK place

If it is that iommufd sets it globaly for the whole system it is the
wrong place.

We could also talk about a per-vfio_device sysfs to control this? Then
we can make the sysfs only appear for vfio_devices using the
iommu_domain part of iommufd/vfio. That has a nice sort of compat
shape as we can make the existing module option default the sysfs to
insecure

Jason

Re: [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Wed, Nov 09, 2022 at 11:28:22AM -0700, Alex Williamson wrote:
> > > > I'd be much more comfortable with this as a system wide iommufd flag
> > > > if we also tied it to do some demonstration of privilege - eg a
> > > > requirement to open iommufd with CAP_SYS_RAWIO for instance.  
> > > 
> > > Which is not compatible to existing use cases, which is also why we
> > > can't invent some way to allow some applications to run without CPU
> > > mitigations, while requiring it for others as a baseline.  
> > 
> > Isn't it? Didn't we learn that libvirt runs as root and will open and
> > pass the iommufd as root?
> 
> We're jumping ahead to native iommufd support here, what happens when
> VFIO_CONTAINER=n and it's QEMU opening the fds, with only file access
> privileges?

Yes, but I am thinking aloud about how to best to do this in native
iommufd modes.

> > I think so. At least you should have something to shut down an
> > insecure feature in kernel lockdown modes. CAP_SYS_RAWIO is a simple
> > way to do it.
> 
> How are CPU vulnerabilities handled in lockdown mode, do apps require
> certain capabilities to run fast vs safe, or do we simply disallow
> unsafe globally in lockdown?  I think we have a lot more leniency to
> ignore/disallow flags that enable global insecurities when any sort of
> lockdown is imposed.

The CPU things are all information leaks from the kernel to
userspace. lockdown is about preserving kernel operating integrity -
eg preventing modification of hijacking of the running kernel.

So, like you say below, this is kind of in between, it is not
information leakage, and it is is hopefully not an integrity issue.

Being more of a DOS maybe it is fine under the lockdown scenarios. At
least I am happier to hear that.

> > vfio-iommufd seems like overkill, I think your first suggestion to put
> > in vfio.ko was more practical.
> 
> Convenient perhaps, but architecturally the wrong place for it.

Ah, that is pretty subjective. If the architecture is that the iommufd
user subsystem opts-in to this insecurity then it is an OK place

If it is that iommufd sets it globaly for the whole system it is the
wrong place.

We could also talk about a per-vfio_device sysfs to control this? Then
we can make the sysfs only appear for vfio_devices using the
iommu_domain part of iommufd/vfio. That has a nice sort of compat
shape as we can make the existing module option default the sysfs to
insecure

Jason

Re: [Intel-gfx] [PATCH 04/10] vfio: Move storage of allow_unsafe_interrupts to vfio_main.c

[Jason Gunthorpe]

On Wed, Nov 09, 2022 at 11:28:22AM -0700, Alex Williamson wrote:
> > > > I'd be much more comfortable with this as a system wide iommufd flag
> > > > if we also tied it to do some demonstration of privilege - eg a
> > > > requirement to open iommufd with CAP_SYS_RAWIO for instance.  
> > > 
> > > Which is not compatible to existing use cases, which is also why we
> > > can't invent some way to allow some applications to run without CPU
> > > mitigations, while requiring it for others as a baseline.  
> > 
> > Isn't it? Didn't we learn that libvirt runs as root and will open and
> > pass the iommufd as root?
> 
> We're jumping ahead to native iommufd support here, what happens when
> VFIO_CONTAINER=n and it's QEMU opening the fds, with only file access
> privileges?

Yes, but I am thinking aloud about how to best to do this in native
iommufd modes.

> > I think so. At least you should have something to shut down an
> > insecure feature in kernel lockdown modes. CAP_SYS_RAWIO is a simple
> > way to do it.
> 
> How are CPU vulnerabilities handled in lockdown mode, do apps require
> certain capabilities to run fast vs safe, or do we simply disallow
> unsafe globally in lockdown?  I think we have a lot more leniency to
> ignore/disallow flags that enable global insecurities when any sort of
> lockdown is imposed.

The CPU things are all information leaks from the kernel to
userspace. lockdown is about preserving kernel operating integrity -
eg preventing modification of hijacking of the running kernel.

So, like you say below, this is kind of in between, it is not
information leakage, and it is is hopefully not an integrity issue.

Being more of a DOS maybe it is fine under the lockdown scenarios. At
least I am happier to hear that.

> > vfio-iommufd seems like overkill, I think your first suggestion to put
> > in vfio.ko was more practical.
> 
> Convenient perhaps, but architecturally the wrong place for it.

Ah, that is pretty subjective. If the architecture is that the iommufd
user subsystem opts-in to this insecurity then it is an OK place

If it is that iommufd sets it globaly for the whole system it is the
wrong place.

We could also talk about a per-vfio_device sysfs to control this? Then
we can make the sysfs only appear for vfio_devices using the
iommu_domain part of iommufd/vfio. That has a nice sort of compat
shape as we can make the existing module option default the sysfs to
insecure

Jason