[RFC][PATCH v2 00/13] iommu/arm-smmu-v3: Add NVIDIA implementation

[RFC][PATCH v2 00/13] iommu/arm-smmu-v3: Add NVIDIA implementation

[Nicolin Chen]

The SMMUv3 devices implemented in the Grace SoC support NVIDIA's custom
CMDQ-Virtualization (CMDQV) hardware. Like the new ECMDQ feature first
introduced in the ARM SMMUv3.3 specification, CMDQV adds multiple VCMDQ
interfaces to supplement the single architected SMMU_CMDQ in an effort
to reduce contention.

This series of patches add CMDQV support with its preparational changes:

* PATCH-1 to PATCH-8 are related to shared VMID feature: they are used
  first to improve TLB utilization, second to bind a shared VMID with a
  VCMDQ interface for hardware configuring requirement.

* PATCH-9 and PATCH-10 are to accommodate the NVIDIA implementation with
  the existing arm-smmu-v3 driver.

* PATCH-11 borrows the "implementation infrastructure" from the arm-smmu
  driver so later change can build upon it.

* PATCH-12 adds an initial NVIDIA implementation related to host feature,
  and also adds implementation specific ->device_reset() and ->get_cmdq()
  callback functions.

* PATCH-13 adds virtualization features using VFIO mdev interface, which
  allows user space hypervisor to map and get access to one of the VCMDQ
  interfaces of CMDQV module.

( Thinking that reviewers can get a better view of this implementation,
  I am attaching QEMU changes here for reference purpose:
      https://github.com/nicolinc/qemu/commits/dev/cmdqv_v6.0.0-rc2
  The branch has all preparational changes, while I'm still integrating
  device model and ARM-VIRT changes, and will push them these two days,
  although they might not be in a good shape of being sent to review yet )

Above all, I marked RFC for this series, as I feel that we may come up
some better solution. So please kindly share your reviews and insights.

Thank you!

Changelog
v1->v2:
 * Added mdev interface support for hypervisor and VMs.
 * Added preparational changes for mdev interface implementation.
 * PATCH-12 Changed ->issue_cmdlist() to ->get_cmdq() for a better
   integration with recently merged ECMDQ-related changes.

Nate Watterson (3):
  iommu/arm-smmu-v3: Add implementation infrastructure
  iommu/arm-smmu-v3: Add support for NVIDIA CMDQ-Virtualization hw
  iommu/nvidia-smmu-v3: Add mdev interface support

Nicolin Chen (10):
  iommu: Add set_nesting_vmid/get_nesting_vmid functions
  vfio: add VFIO_IOMMU_GET_VMID and VFIO_IOMMU_SET_VMID
  vfio: Document VMID control for IOMMU Virtualization
  vfio: add set_vmid and get_vmid for vfio_iommu_type1
  vfio/type1: Implement set_vmid and get_vmid
  vfio/type1: Set/get VMID to/from iommu driver
  iommu/arm-smmu-v3: Add shared VMID support for NESTING
  iommu/arm-smmu-v3: Add VMID alloc/free helpers
  iommu/arm-smmu-v3: Pass dev pointer to arm_smmu_detach_dev
  iommu/arm-smmu-v3: Pass cmdq pointer in arm_smmu_cmdq_issue_cmdlist()

 Documentation/driver-api/vfio.rst             |   34 +
 MAINTAINERS                                   |    2 +
 drivers/iommu/arm/arm-smmu-v3/Makefile        |    2 +-
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c  |   15 +
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |  121 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |   18 +
 .../iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c    | 1249 +++++++++++++++++
 drivers/iommu/iommu.c                         |   20 +
 drivers/vfio/vfio.c                           |   25 +
 drivers/vfio/vfio_iommu_type1.c               |   37 +
 include/linux/iommu.h                         |    5 +
 include/linux/vfio.h                          |    2 +
 include/uapi/linux/vfio.h                     |   26 +
 13 files changed, 1537 insertions(+), 19 deletions(-)
 create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c
 create mode 100644 drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c

-- 
2.17.1

[RFC][PATCH v2 01/13] iommu: Add set_nesting_vmid/get_nesting_vmid functions

[Nicolin Chen]

VMID stands for Virtual Machine Identifier, being used to tag
TLB entries to indicate which VM they belong to. This is used
by some IOMMU like SMMUv3 for virtualization case, in nesting
mode.

So this patch adds a pair of new iommu_ops callback functions
with a pair of exported set/get functions to allow VFIO core
to get access of the VMID value in an IOMMU driver.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/iommu.c | 20 ++++++++++++++++++++
 include/linux/iommu.h |  5 +++++
 2 files changed, 25 insertions(+)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index 3303d707bab4..051f2df36dc0 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -2774,6 +2774,26 @@ int iommu_enable_nesting(struct iommu_domain *domain)
 }
 EXPORT_SYMBOL_GPL(iommu_enable_nesting);
 
+int iommu_set_nesting_vmid(struct iommu_domain *domain, u32 vmid)
+{
+	if (domain->type != IOMMU_DOMAIN_UNMANAGED)
+		return -EINVAL;
+	if (!domain->ops->set_nesting_vmid)
+		return -EINVAL;
+	return domain->ops->set_nesting_vmid(domain, vmid);
+}
+EXPORT_SYMBOL_GPL(iommu_set_nesting_vmid);
+
+int iommu_get_nesting_vmid(struct iommu_domain *domain, u32 *vmid)
+{
+	if (domain->type != IOMMU_DOMAIN_UNMANAGED)
+		return -EINVAL;
+	if (!domain->ops->get_nesting_vmid)
+		return -EINVAL;
+	return domain->ops->get_nesting_vmid(domain, vmid);
+}
+EXPORT_SYMBOL_GPL(iommu_get_nesting_vmid);
+
 int iommu_set_pgtable_quirks(struct iommu_domain *domain,
 		unsigned long quirk)
 {
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index d2f3435e7d17..bda6b3450909 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -163,6 +163,7 @@ enum iommu_dev_features {
 };
 
 #define IOMMU_PASID_INVALID	(-1U)
+#define IOMMU_VMID_INVALID	(-1U)
 
 #ifdef CONFIG_IOMMU_API
 
@@ -269,6 +270,8 @@ struct iommu_ops {
 	void (*probe_finalize)(struct device *dev);
 	struct iommu_group *(*device_group)(struct device *dev);
 	int (*enable_nesting)(struct iommu_domain *domain);
+	int (*set_nesting_vmid)(struct iommu_domain *domain, u32 vmid);
+	int (*get_nesting_vmid)(struct iommu_domain *domain, u32 *vmid);
 	int (*set_pgtable_quirks)(struct iommu_domain *domain,
 				  unsigned long quirks);
 
@@ -500,6 +503,8 @@ extern int iommu_group_id(struct iommu_group *group);
 extern struct iommu_domain *iommu_group_default_domain(struct iommu_group *);
 
 int iommu_enable_nesting(struct iommu_domain *domain);
+int iommu_set_nesting_vmid(struct iommu_domain *domain, u32 vmid);
+int iommu_get_nesting_vmid(struct iommu_domain *domain, u32 *vmid);
 int iommu_set_pgtable_quirks(struct iommu_domain *domain,
 		unsigned long quirks);
 
-- 
2.17.1

[RFC][PATCH v2 02/13] vfio: add VFIO_IOMMU_GET_VMID and VFIO_IOMMU_SET_VMID

[Nicolin Chen]

This patch adds a pair of new ioctl commands to communicate with
user space (virtual machine hypervisor) to get and set VMID that
indicates a Virtual Machine Identifier, being used by some IOMMU
to tag TLB entries -- similar to CPU MMU, using this VMID number
allows IOMMU to invalidate at the same time TLBs of the same VM.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/vfio/vfio.c       | 13 +++++++++++++
 include/uapi/linux/vfio.h | 26 ++++++++++++++++++++++++++
 2 files changed, 39 insertions(+)

diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
index 3c034fe14ccb..c17b25c127a2 100644
--- a/drivers/vfio/vfio.c
+++ b/drivers/vfio/vfio.c
@@ -59,6 +59,7 @@ struct vfio_container {
 	struct rw_semaphore		group_lock;
 	struct vfio_iommu_driver	*iommu_driver;
 	void				*iommu_data;
+	u32				vmid;
 	bool				noiommu;
 };
 
@@ -1190,6 +1191,16 @@ static long vfio_fops_unl_ioctl(struct file *filep,
 	case VFIO_SET_IOMMU:
 		ret = vfio_ioctl_set_iommu(container, arg);
 		break;
+	case VFIO_IOMMU_GET_VMID:
+		ret = copy_to_user((void __user *)arg, &container->vmid,
+				   sizeof(u32)) ? -EFAULT : 0;
+		break;
+	case VFIO_IOMMU_SET_VMID:
+		if ((u32)arg == VFIO_IOMMU_VMID_INVALID)
+			return -EINVAL;
+		container->vmid = (u32)arg;
+		ret = 0;
+		break;
 	default:
 		driver = container->iommu_driver;
 		data = container->iommu_data;
@@ -1213,6 +1224,8 @@ static int vfio_fops_open(struct inode *inode, struct file *filep)
 	init_rwsem(&container->group_lock);
 	kref_init(&container->kref);
 
+	container->vmid = VFIO_IOMMU_VMID_INVALID;
+
 	filep->private_data = container;
 
 	return 0;
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index ef33ea002b0b..58c5fa6aaca6 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -1216,6 +1216,32 @@ struct vfio_iommu_type1_dirty_bitmap_get {
 
 #define VFIO_IOMMU_DIRTY_PAGES             _IO(VFIO_TYPE, VFIO_BASE + 17)
 
+/**
+ * VFIO_IOMMU_GET_VMID - _IOWR(VFIO_TYPE, VFIO_BASE + 22, __u32 *vmid)
+ * VFIO_IOMMU_SET_VMID - _IOWR(VFIO_TYPE, VFIO_BASE + 23, __u32 vmid)
+ *
+ * IOCTLs are used for VMID alignment between Kernel and User Space hypervisor.
+ * In a virtualization use case, a guest owns the first stage translation, and
+ * the hypervisor owns the second stage translation. VMID is an Virtual Machine
+ * Identifier that is to tag TLB entries of a VM. If a VM has multiple physical
+ * devices being assigned to it, while these devices are under different IOMMU
+ * domains, the VMIDs in the second stage configurations of these IOMMU domains
+ * could be aligned to a unified VMID value. This could be achieved by using
+ * these two IOCTLs.
+ *
+ * Caller should get VMID upon its initial value when the first physical device
+ * is assigned to the VM.
+ *
+ * Caller then should set VMID to share the same VMID value with other physical
+ * devices being assigned to the same VM.
+ *
+ */
+#define VFIO_IOMMU_VMID_INVALID		(-1U)
+
+#define VFIO_IOMMU_GET_VMID		_IO(VFIO_TYPE, VFIO_BASE + 22)
+
+#define VFIO_IOMMU_SET_VMID		_IO(VFIO_TYPE, VFIO_BASE + 23)
+
 /* -------- Additional API for SPAPR TCE (Server POWERPC) IOMMU -------- */
 
 /*
-- 
2.17.1

[RFC][PATCH v2 03/13] vfio: Document VMID control for IOMMU Virtualization

[Nicolin Chen]

The VFIO API was enhanced to support VMID control with two
new iotcls to set and get VMID between the kernel and the
virtual machine hypervisor. So updating the document.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 Documentation/driver-api/vfio.rst | 34 +++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/Documentation/driver-api/vfio.rst b/Documentation/driver-api/vfio.rst
index c663b6f97825..a76a17065cdd 100644
--- a/Documentation/driver-api/vfio.rst
+++ b/Documentation/driver-api/vfio.rst
@@ -239,6 +239,40 @@ group and can access them as follows::
 	/* Gratuitous device reset and go... */
 	ioctl(device, VFIO_DEVICE_RESET);
 
+IOMMU Virtual Machine Identifier (VMID)
+------------------------
+In case of virtualization, each VM is assigned a Virtual Machine Identifier
+(VMID). This VMID is used to tag translation lookaside buffer (TLB) entries,
+to identify which VM each entry belongs to. This tagging allows translations
+for multiple different VMs to be present in the TLBs at the same time.
+
+The IOMMU Kernel driver is responsible for allocating a VMID. However, only
+a hypervisor knows what physical devices get assigned to the same VM. Thus,
+when the first physical device gets assigned to the VM, once the hypervisor
+finishes its IOCTL call of VFIO_SET_IOMMU, it should call the following:
+
+struct vm {
+	int iommu_type;
+	uint32_t vmid;	/* initial value is VFIO_IOMMU_VMID_INVALID */
+} vm0;
+
+	/* ... */
+	ioctl(container->fd, VFIO_SET_IOMMU, vm0->iommu_type);
+	/* ... */
+	if (vm0->vmid == VFIO_IOMMU_VMID_INVALID)
+		ioctl(container->fd, VFIO_IOMMU_GET_VMID, &vm0->vmid);
+
+This VMID would be the shared value, across the entire VM, between all the
+physical devices that are assigned to it. So, when other physical devices
+get assigned to the VM, before the hypervisor runs into the IOCTL call of
+VFIO_IOMMU_SET_VMID, it should call the following:
+
+	/* ... */
+	ioctl(container->fd, VFIO_SET_IOMMU, vm0->iommu_type);
+	/* ... */
+	if (vm0->vmid != VFIO_IOMMU_VMID_INVALID)
+		ioctl(container->fd, VFIO_IOMMU_SET_VMID, vmid);
+
 VFIO User API
 -------------------------------------------------------------------------------
 
-- 
2.17.1

[RFC][PATCH v2 04/13] vfio: add set_vmid and get_vmid for vfio_iommu_type1

[Nicolin Chen]

A VMID is generated in an IOMMU driver, being called from this
->attach_group() callback. So call ->get_vmid() right after it
creates a new VMID, and call ->set_vmid() before it, to let it
reuse the same VMID.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/vfio/vfio.c  | 12 ++++++++++++
 include/linux/vfio.h |  2 ++
 2 files changed, 14 insertions(+)

diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
index c17b25c127a2..8b7442deca93 100644
--- a/drivers/vfio/vfio.c
+++ b/drivers/vfio/vfio.c
@@ -1080,9 +1080,21 @@ static int __vfio_container_attach_groups(struct vfio_container *container,
 	int ret = -ENODEV;
 
 	list_for_each_entry(group, &container->group_list, container_next) {
+		if (driver->ops->set_vmid && container->vmid != VFIO_IOMMU_VMID_INVALID) {
+			ret = driver->ops->set_vmid(data, container->vmid);
+			if (ret)
+				goto unwind;
+		}
+
 		ret = driver->ops->attach_group(data, group->iommu_group);
 		if (ret)
 			goto unwind;
+
+		if (driver->ops->get_vmid && container->vmid == VFIO_IOMMU_VMID_INVALID) {
+			ret = driver->ops->get_vmid(data, &container->vmid);
+			if (ret)
+				goto unwind;
+		}
 	}
 
 	return ret;
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index b53a9557884a..b43e7cbef4ab 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -126,6 +126,8 @@ struct vfio_iommu_driver_ops {
 						   struct iommu_group *group);
 	void		(*notify)(void *iommu_data,
 				  enum vfio_iommu_notify_type event);
+	int		(*set_vmid)(void *iommu_data, u32 vmid);
+	int		(*get_vmid)(void *iommu_data, u32 *vmid);
 };
 
 extern int vfio_register_iommu_driver(const struct vfio_iommu_driver_ops *ops);
-- 
2.17.1

[RFC][PATCH v2 05/13] vfio/type1: Implement set_vmid and get_vmid

[Nicolin Chen]

Now we have a pair of ->set_vmid() and ->get_vmid() function
pointers. This patch implements them, to exchange VMID value
between vfio container and vfio_iommu_type1.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/vfio/vfio_iommu_type1.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 0e9217687f5c..bb5d949bc1af 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -74,6 +74,7 @@ struct vfio_iommu {
 	uint64_t		pgsize_bitmap;
 	uint64_t		num_non_pinned_groups;
 	wait_queue_head_t	vaddr_wait;
+	uint32_t		vmid;
 	bool			v2;
 	bool			nesting;
 	bool			dirty_page_tracking;
@@ -2674,6 +2675,7 @@ static void *vfio_iommu_type1_open(unsigned long arg)
 	iommu->dma_list = RB_ROOT;
 	iommu->dma_avail = dma_entry_limit;
 	iommu->container_open = true;
+	iommu->vmid = VFIO_IOMMU_VMID_INVALID;
 	mutex_init(&iommu->lock);
 	BLOCKING_INIT_NOTIFIER_HEAD(&iommu->notifier);
 	init_waitqueue_head(&iommu->vaddr_wait);
@@ -3255,6 +3257,27 @@ static void vfio_iommu_type1_notify(void *iommu_data,
 	wake_up_all(&iommu->vaddr_wait);
 }
 
+static int vfio_iommu_type1_get_vmid(void *iommu_data, u32 *vmid)
+{
+	struct vfio_iommu *iommu = iommu_data;
+
+	*vmid = iommu->vmid;
+
+	return 0;
+}
+
+static int vfio_iommu_type1_set_vmid(void *iommu_data, u32 vmid)
+{
+	struct vfio_iommu *iommu = iommu_data;
+
+	if (vmid == VFIO_IOMMU_VMID_INVALID)
+		return -EINVAL;
+
+	iommu->vmid = vmid;
+
+	return 0;
+}
+
 static const struct vfio_iommu_driver_ops vfio_iommu_driver_ops_type1 = {
 	.name			= "vfio-iommu-type1",
 	.owner			= THIS_MODULE,
@@ -3270,6 +3293,8 @@ static const struct vfio_iommu_driver_ops vfio_iommu_driver_ops_type1 = {
 	.dma_rw			= vfio_iommu_type1_dma_rw,
 	.group_iommu_domain	= vfio_iommu_type1_group_iommu_domain,
 	.notify			= vfio_iommu_type1_notify,
+	.set_vmid		= vfio_iommu_type1_set_vmid,
+	.get_vmid		= vfio_iommu_type1_get_vmid,
 };
 
 static int __init vfio_iommu_type1_init(void)
-- 
2.17.1

[RFC][PATCH v2 06/13] vfio/type1: Set/get VMID to/from iommu driver

[Nicolin Chen]

This patch adds a pair of callbacks of iommu_set_nesting_vmid() and
iommu_get_nesting_vmid() to exchange VMID with the IOMMU core (then
an IOMMU driver).

As a VMID is generated in an IOMMU driver, which is called from the
vfio_iommu_attach_group() function call, add iommu_get_nesting_vmid
right after it creates a VMID and add iommu_set_nesting_vmid before
it to let IOMMU driver reuse it.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/vfio/vfio_iommu_type1.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index bb5d949bc1af..9e72d74dedcd 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -2322,12 +2322,24 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
 		ret = iommu_enable_nesting(domain->domain);
 		if (ret)
 			goto out_domain;
+
+		if (iommu->vmid != VFIO_IOMMU_VMID_INVALID) {
+			ret = iommu_set_nesting_vmid(domain->domain, iommu->vmid);
+			if (ret)
+				goto out_domain;
+		}
 	}
 
 	ret = vfio_iommu_attach_group(domain, group);
 	if (ret)
 		goto out_domain;
 
+	if (iommu->nesting && iommu->vmid == VFIO_IOMMU_VMID_INVALID) {
+		ret = iommu_get_nesting_vmid(domain->domain, &iommu->vmid);
+		if (ret)
+			goto out_domain;
+	}
+
 	/* Get aperture info */
 	geo = &domain->domain->geometry;
 	if (vfio_iommu_aper_conflict(iommu, geo->aperture_start,
-- 
2.17.1

[RFC][PATCH v2 07/13] iommu/arm-smmu-v3: Add shared VMID support for NESTING

[Nicolin Chen]

A VMID can be shared among iommu domains being attached to the same
Virtual Machine in order to improve utilization of TLB cache.

This patch implements ->set_nesting_vmid() and ->get_nesting_vmid()
to set/get s2_cfg->vmid for nesting cases, and then changes to reuse
the VMID.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 65 +++++++++++++++++++--
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h |  1 +
 2 files changed, 60 insertions(+), 6 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index a388e318f86e..c0ae117711fa 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2051,7 +2051,7 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
 		mutex_unlock(&arm_smmu_asid_lock);
 	} else {
 		struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
-		if (cfg->vmid)
+		if (cfg->vmid && !atomic_dec_return(&smmu->vmid_refcnts[cfg->vmid]))
 			arm_smmu_bitmap_free(smmu->vmid_map, cfg->vmid);
 	}
 
@@ -2121,17 +2121,28 @@ static int arm_smmu_domain_finalise_s2(struct arm_smmu_domain *smmu_domain,
 				       struct arm_smmu_master *master,
 				       struct io_pgtable_cfg *pgtbl_cfg)
 {
-	int vmid;
 	struct arm_smmu_device *smmu = smmu_domain->smmu;
 	struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
 	typeof(&pgtbl_cfg->arm_lpae_s2_cfg.vtcr) vtcr;
 
-	vmid = arm_smmu_bitmap_alloc(smmu->vmid_map, smmu->vmid_bits);
-	if (vmid < 0)
-		return vmid;
+	/*
+	 * For a nested case where there are multiple passthrough devices to a
+	 * VM, they share a commond VMID, allocated when the first passthrough
+	 * device is attached to the VM. So the cfg->vmid might be already set
+	 * in arm_smmu_set_nesting_vmid(), reported from the hypervisor. In this
+	 * case, simply reuse the shared VMID and increase its refcount.
+	 */
+	if (!cfg->vmid) {
+		int vmid = arm_smmu_bitmap_alloc(smmu->vmid_map, smmu->vmid_bits);
+
+		if (vmid < 0)
+			return vmid;
+		cfg->vmid = (u16)vmid;
+	}
+
+	atomic_inc(&smmu->vmid_refcnts[cfg->vmid]);
 
 	vtcr = &pgtbl_cfg->arm_lpae_s2_cfg.vtcr;
-	cfg->vmid	= (u16)vmid;
 	cfg->vttbr	= pgtbl_cfg->arm_lpae_s2_cfg.vttbr;
 	cfg->vtcr	= FIELD_PREP(STRTAB_STE_2_VTCR_S2T0SZ, vtcr->tsz) |
 			  FIELD_PREP(STRTAB_STE_2_VTCR_S2SL0, vtcr->sl) |
@@ -2731,6 +2742,44 @@ static int arm_smmu_enable_nesting(struct iommu_domain *domain)
 	return ret;
 }
 
+static int arm_smmu_set_nesting_vmid(struct iommu_domain *domain, u32 vmid)
+{
+	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+	struct arm_smmu_s2_cfg *s2_cfg = &smmu_domain->s2_cfg;
+	int ret = 0;
+
+	if (vmid == IOMMU_VMID_INVALID)
+		return -EINVAL;
+
+	mutex_lock(&smmu_domain->init_mutex);
+	if (smmu_domain->smmu || smmu_domain->stage != ARM_SMMU_DOMAIN_NESTED)
+		ret = -EPERM;
+	else
+		s2_cfg->vmid = vmid;
+	mutex_unlock(&smmu_domain->init_mutex);
+
+	return ret;
+}
+
+static int arm_smmu_get_nesting_vmid(struct iommu_domain *domain, u32 *vmid)
+{
+	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+	struct arm_smmu_s2_cfg *s2_cfg = &smmu_domain->s2_cfg;
+	int ret = 0;
+
+	if (!vmid)
+		return -EINVAL;
+
+	mutex_lock(&smmu_domain->init_mutex);
+	if (smmu_domain->smmu || smmu_domain->stage != ARM_SMMU_DOMAIN_NESTED)
+		ret = -EPERM;
+	else
+		*vmid = s2_cfg->vmid;
+	mutex_unlock(&smmu_domain->init_mutex);
+
+	return ret;
+}
+
 static int arm_smmu_of_xlate(struct device *dev, struct of_phandle_args *args)
 {
 	return iommu_fwspec_add_ids(dev, args->args, 1);
@@ -2845,6 +2894,8 @@ static struct iommu_ops arm_smmu_ops = {
 	.release_device		= arm_smmu_release_device,
 	.device_group		= arm_smmu_device_group,
 	.enable_nesting		= arm_smmu_enable_nesting,
+	.set_nesting_vmid	= arm_smmu_set_nesting_vmid,
+	.get_nesting_vmid	= arm_smmu_get_nesting_vmid,
 	.of_xlate		= arm_smmu_of_xlate,
 	.get_resv_regions	= arm_smmu_get_resv_regions,
 	.put_resv_regions	= generic_iommu_put_resv_regions,
@@ -3530,6 +3581,8 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
 	/* ASID/VMID sizes */
 	smmu->asid_bits = reg & IDR0_ASID16 ? 16 : 8;
 	smmu->vmid_bits = reg & IDR0_VMID16 ? 16 : 8;
+	smmu->vmid_refcnts = devm_kcalloc(smmu->dev, 1 << smmu->vmid_bits,
+					  sizeof(*smmu->vmid_refcnts), GFP_KERNEL);
 
 	/* IDR1 */
 	reg = readl_relaxed(smmu->base + ARM_SMMU_IDR1);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 4cb136f07914..ea2c61d52df8 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -664,6 +664,7 @@ struct arm_smmu_device {
 #define ARM_SMMU_MAX_VMIDS		(1 << 16)
 	unsigned int			vmid_bits;
 	DECLARE_BITMAP(vmid_map, ARM_SMMU_MAX_VMIDS);
+	atomic_t			*vmid_refcnts;
 
 	unsigned int			ssid_bits;
 	unsigned int			sid_bits;
-- 
2.17.1

[RFC][PATCH v2 08/13] iommu/arm-smmu-v3: Add VMID alloc/free helpers

[Nicolin Chen]

NVIDIA implementation needs to link its Virtual Interface to a
VMID, before a device gets attached to the corresponding iommu
domain. One way to ensure that is to allocate a VMID from impl
side and to pass it down to virtual machine hypervisor so that
later it can set it back to passthrough devices' iommu domains
calling newly added arm_smmu_set/get_nesting_vmid() functions.

This patch adds a pair of helpers to allocate and free VMID in
the bitmap.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 10 ++++++++++
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h |  3 +++
 2 files changed, 13 insertions(+)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index c0ae117711fa..497d55ec659b 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2032,6 +2032,16 @@ static void arm_smmu_bitmap_free(unsigned long *map, int idx)
 	clear_bit(idx, map);
 }
 
+int arm_smmu_vmid_alloc(struct arm_smmu_device *smmu)
+{
+	return arm_smmu_bitmap_alloc(smmu->vmid_map, smmu->vmid_bits);
+}
+
+void arm_smmu_vmid_free(struct arm_smmu_device *smmu, u16 vmid)
+{
+	arm_smmu_bitmap_free(smmu->vmid_map, vmid);
+}
+
 static void arm_smmu_domain_free(struct iommu_domain *domain)
 {
 	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index ea2c61d52df8..20463d17fd9f 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -749,6 +749,9 @@ bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd);
 int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
 			    unsigned long iova, size_t size);
 
+int arm_smmu_vmid_alloc(struct arm_smmu_device *smmu);
+void arm_smmu_vmid_free(struct arm_smmu_device *smmu, u16 vmid);
+
 #ifdef CONFIG_ARM_SMMU_V3_SVA
 bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
 bool arm_smmu_master_sva_supported(struct arm_smmu_master *master);
-- 
2.17.1

[RFC][PATCH v2 09/13] iommu/arm-smmu-v3: Pass dev pointer to arm_smmu_detach_dev

[Nicolin Chen]

We are adding NVIDIA implementation that will need a ->detach_dev()
callback along with the dev pointer to grab client information.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 497d55ec659b..6878a83582b9 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2377,7 +2377,7 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master)
 	pci_disable_pasid(pdev);
 }
 
-static void arm_smmu_detach_dev(struct arm_smmu_master *master)
+static void arm_smmu_detach_dev(struct arm_smmu_master *master, struct device *dev)
 {
 	unsigned long flags;
 	struct arm_smmu_domain *smmu_domain = master->domain;
@@ -2421,7 +2421,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
 		return -EBUSY;
 	}
 
-	arm_smmu_detach_dev(master);
+	arm_smmu_detach_dev(master, dev);
 
 	mutex_lock(&smmu_domain->init_mutex);
 
@@ -2713,7 +2713,7 @@ static void arm_smmu_release_device(struct device *dev)
 	master = dev_iommu_priv_get(dev);
 	if (WARN_ON(arm_smmu_master_sva_enabled(master)))
 		iopf_queue_remove_device(master->smmu->evtq.iopf, dev);
-	arm_smmu_detach_dev(master);
+	arm_smmu_detach_dev(master, dev);
 	arm_smmu_disable_pasid(master);
 	arm_smmu_remove_master(master);
 	kfree(master);
-- 
2.17.1

[RFC][PATCH v2 10/13] iommu/arm-smmu-v3: Pass cmdq pointer in arm_smmu_cmdq_issue_cmdlist()

[Nicolin Chen]

The driver currently calls arm_smmu_get_cmdq() helper internally in
different places, though they are all actually called from the same
source -- arm_smmu_cmdq_issue_cmdlist() function.

This patch changes this to pass the cmdq pointer to these functions
instead of calling arm_smmu_get_cmdq() every time.

This also helps NVIDIA implementation, which maintains its own cmdq
pointers and needs to redirect the cmdq pointer from arm_smmu->cmdq
pointer to its own, upon scanning the illegal commands by checking
the opcode of the cmdlist.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 6878a83582b9..216f3442aac4 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -584,11 +584,11 @@ static void arm_smmu_cmdq_poll_valid_map(struct arm_smmu_cmdq *cmdq,
 
 /* Wait for the command queue to become non-full */
 static int arm_smmu_cmdq_poll_until_not_full(struct arm_smmu_device *smmu,
+					     struct arm_smmu_cmdq *cmdq,
 					     struct arm_smmu_ll_queue *llq)
 {
 	unsigned long flags;
 	struct arm_smmu_queue_poll qp;
-	struct arm_smmu_cmdq *cmdq = arm_smmu_get_cmdq(smmu);
 	int ret = 0;
 
 	/*
@@ -619,11 +619,11 @@ static int arm_smmu_cmdq_poll_until_not_full(struct arm_smmu_device *smmu,
  * Must be called with the cmdq lock held in some capacity.
  */
 static int __arm_smmu_cmdq_poll_until_msi(struct arm_smmu_device *smmu,
+					  struct arm_smmu_cmdq *cmdq,
 					  struct arm_smmu_ll_queue *llq)
 {
 	int ret = 0;
 	struct arm_smmu_queue_poll qp;
-	struct arm_smmu_cmdq *cmdq = arm_smmu_get_cmdq(smmu);
 	u32 *cmd = (u32 *)(Q_ENT(&cmdq->q, llq->prod));
 
 	queue_poll_init(smmu, &qp);
@@ -643,10 +643,10 @@ static int __arm_smmu_cmdq_poll_until_msi(struct arm_smmu_device *smmu,
  * Must be called with the cmdq lock held in some capacity.
  */
 static int __arm_smmu_cmdq_poll_until_consumed(struct arm_smmu_device *smmu,
+					       struct arm_smmu_cmdq *cmdq,
 					       struct arm_smmu_ll_queue *llq)
 {
 	struct arm_smmu_queue_poll qp;
-	struct arm_smmu_cmdq *cmdq = arm_smmu_get_cmdq(smmu);
 	u32 prod = llq->prod;
 	int ret = 0;
 
@@ -693,12 +693,13 @@ static int __arm_smmu_cmdq_poll_until_consumed(struct arm_smmu_device *smmu,
 }
 
 static int arm_smmu_cmdq_poll_until_sync(struct arm_smmu_device *smmu,
+					 struct arm_smmu_cmdq *cmdq,
 					 struct arm_smmu_ll_queue *llq)
 {
 	if (smmu->options & ARM_SMMU_OPT_MSIPOLL)
-		return __arm_smmu_cmdq_poll_until_msi(smmu, llq);
+		return __arm_smmu_cmdq_poll_until_msi(smmu, cmdq, llq);
 
-	return __arm_smmu_cmdq_poll_until_consumed(smmu, llq);
+	return __arm_smmu_cmdq_poll_until_consumed(smmu, cmdq, llq);
 }
 
 static void arm_smmu_cmdq_write_entries(struct arm_smmu_cmdq *cmdq, u64 *cmds,
@@ -755,7 +756,7 @@ static int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
 
 		while (!queue_has_space(&llq, n + sync)) {
 			local_irq_restore(flags);
-			if (arm_smmu_cmdq_poll_until_not_full(smmu, &llq))
+			if (arm_smmu_cmdq_poll_until_not_full(smmu, cmdq, &llq))
 				dev_err_ratelimited(smmu->dev, "CMDQ timeout\n");
 			local_irq_save(flags);
 		}
@@ -831,7 +832,7 @@ static int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
 	/* 5. If we are inserting a CMD_SYNC, we must wait for it to complete */
 	if (sync) {
 		llq.prod = queue_inc_prod_n(&llq, n);
-		ret = arm_smmu_cmdq_poll_until_sync(smmu, &llq);
+		ret = arm_smmu_cmdq_poll_until_sync(smmu, cmdq, &llq);
 		if (ret) {
 			dev_err_ratelimited(smmu->dev,
 					    "CMD_SYNC timeout at 0x%08x [hwprod 0x%08x, hwcons 0x%08x]\n",
-- 
2.17.1

[RFC][PATCH v2 11/13] iommu/arm-smmu-v3: Add implementation infrastructure

[Nicolin Chen]

From: Nate Watterson <nwatterson@nvidia.com>

Follow arm-smmu driver's infrastructure for handling implementation
specific details outside the flow of architectural code.

Signed-off-by: Nate Watterson <nwatterson@nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/Makefile           | 2 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c | 8 ++++++++
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c      | 4 ++++
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h      | 4 ++++
 4 files changed, 17 insertions(+), 1 deletion(-)
 create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c

diff --git a/drivers/iommu/arm/arm-smmu-v3/Makefile b/drivers/iommu/arm/arm-smmu-v3/Makefile
index 54feb1ecccad..1f5838d3351b 100644
--- a/drivers/iommu/arm/arm-smmu-v3/Makefile
+++ b/drivers/iommu/arm/arm-smmu-v3/Makefile
@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ARM_SMMU_V3) += arm_smmu_v3.o
-arm_smmu_v3-objs-y += arm-smmu-v3.o
+arm_smmu_v3-objs-y += arm-smmu-v3.o arm-smmu-v3-impl.o
 arm_smmu_v3-objs-$(CONFIG_ARM_SMMU_V3_SVA) += arm-smmu-v3-sva.o
 arm_smmu_v3-objs := $(arm_smmu_v3-objs-y)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c
new file mode 100644
index 000000000000..6947d28067a8
--- /dev/null
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c
@@ -0,0 +1,8 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include "arm-smmu-v3.h"
+
+struct arm_smmu_device *arm_smmu_v3_impl_init(struct arm_smmu_device *smmu)
+{
+	return smmu;
+}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 216f3442aac4..510e1493fd5a 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -3844,6 +3844,10 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
 			return ret;
 	}
 
+	smmu = arm_smmu_v3_impl_init(smmu);
+	if (IS_ERR(smmu))
+		return PTR_ERR(smmu);
+
 	/* Set bypass mode according to firmware probing result */
 	bypass = !!ret;
 
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 20463d17fd9f..c65c39336916 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -810,4 +810,8 @@ static inline u32 arm_smmu_sva_get_pasid(struct iommu_sva *handle)
 
 static inline void arm_smmu_sva_notifier_synchronize(void) {}
 #endif /* CONFIG_ARM_SMMU_V3_SVA */
+
+/* Implementation details */
+struct arm_smmu_device *arm_smmu_v3_impl_init(struct arm_smmu_device *smmu);
+
 #endif /* _ARM_SMMU_V3_H */
-- 
2.17.1

[RFC][PATCH v2 12/13] iommu/arm-smmu-v3: Add support for NVIDIA CMDQ-Virtualization hw

[Nicolin Chen]

From: Nate Watterson <nwatterson@nvidia.com>

NVIDIA's Grace SoC has a CMDQ-Virtualization (CMDQV) hardware,
which adds multiple VCMDQ interfaces (VINTFs) to supplement the
architected SMMU_CMDQ in an effort to reduce contention.

To make use of these supplemental CMDQs in arm-smmu-v3 driver,
this patch borrows the "implemenatation infrastructure" design
from the arm-smmu driver, and then adds implementation specific
supports for ->device_reset() and ->get_cmdq() functions. Since
nvidia's ->get_cmdq() implemenatation needs to check the first
command of the cmdlist to determine whether to redirect to its
own vcmdq, this patch also adds augments to arm_smmu_get_cmdq()
function.

For the CMDQV driver itself, this patch only adds the essential
parts for the host kernel, in terms of virtualization use cases.
VINTF0 is being reserved for host kernel use, so is initialized
with the driver also.

Note that, for the current plan, the CMDQV driver only supports
ACPI configuration.

Signed-off-by: Nate Watterson <nwatterson@nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 MAINTAINERS                                   |   2 +
 drivers/iommu/arm/arm-smmu-v3/Makefile        |   2 +-
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c  |   7 +
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |  15 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |   8 +
 .../iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c    | 432 ++++++++++++++++++
 6 files changed, 463 insertions(+), 3 deletions(-)
 create mode 100644 drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c

diff --git a/MAINTAINERS b/MAINTAINERS
index f800abca74b0..7a2f21279d35 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -18428,8 +18428,10 @@ F:	drivers/i2c/busses/i2c-tegra.c
 TEGRA IOMMU DRIVERS
 M:	Thierry Reding <thierry.reding@gmail.com>
 R:	Krishna Reddy <vdumpa@nvidia.com>
+R:	Nicolin Chen <nicoleotsuka@gmail.com>
 L:	linux-tegra@vger.kernel.org
 S:	Supported
+F:	drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c
 F:	drivers/iommu/arm/arm-smmu/arm-smmu-nvidia.c
 F:	drivers/iommu/tegra*
 
diff --git a/drivers/iommu/arm/arm-smmu-v3/Makefile b/drivers/iommu/arm/arm-smmu-v3/Makefile
index 1f5838d3351b..0aa84c0a50ea 100644
--- a/drivers/iommu/arm/arm-smmu-v3/Makefile
+++ b/drivers/iommu/arm/arm-smmu-v3/Makefile
@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_ARM_SMMU_V3) += arm_smmu_v3.o
-arm_smmu_v3-objs-y += arm-smmu-v3.o arm-smmu-v3-impl.o
+arm_smmu_v3-objs-y += arm-smmu-v3.o arm-smmu-v3-impl.o nvidia-smmu-v3.o
 arm_smmu_v3-objs-$(CONFIG_ARM_SMMU_V3_SVA) += arm-smmu-v3-sva.o
 arm_smmu_v3-objs := $(arm_smmu_v3-objs-y)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c
index 6947d28067a8..37d062e40eb5 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c
@@ -4,5 +4,12 @@
 
 struct arm_smmu_device *arm_smmu_v3_impl_init(struct arm_smmu_device *smmu)
 {
+	/*
+	 * Nvidia implementation supports ACPI only, so calling its init()
+	 * unconditionally to walk through ACPI tables to probe the device.
+	 * It will keep the smmu pointer intact, if it fails.
+	 */
+	smmu = nvidia_smmu_v3_impl_init(smmu);
+
 	return smmu;
 }
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 510e1493fd5a..1b9459592f76 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -335,8 +335,11 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)
 	return 0;
 }
 
-static struct arm_smmu_cmdq *arm_smmu_get_cmdq(struct arm_smmu_device *smmu)
+static struct arm_smmu_cmdq *arm_smmu_get_cmdq(struct arm_smmu_device *smmu, u64 *cmds, int n)
 {
+	if (smmu->impl && smmu->impl->get_cmdq)
+		return smmu->impl->get_cmdq(smmu, cmds, n);
+
 	return &smmu->cmdq;
 }
 
@@ -742,7 +745,7 @@ static int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
 	u32 prod;
 	unsigned long flags;
 	bool owner;
-	struct arm_smmu_cmdq *cmdq = arm_smmu_get_cmdq(smmu);
+	struct arm_smmu_cmdq *cmdq = arm_smmu_get_cmdq(smmu, cmds, n);
 	struct arm_smmu_ll_queue llq, head;
 	int ret = 0;
 
@@ -3487,6 +3490,14 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass)
 		return ret;
 	}
 
+	if (smmu->impl && smmu->impl->device_reset) {
+		ret = smmu->impl->device_reset(smmu);
+		if (ret) {
+			dev_err(smmu->dev, "failed at implementation specific device_reset\n");
+			return ret;
+		}
+	}
+
 	return 0;
 }
 
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index c65c39336916..bb903a7fa662 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -647,6 +647,8 @@ struct arm_smmu_device {
 #define ARM_SMMU_OPT_MSIPOLL		(1 << 2)
 	u32				options;
 
+	const struct arm_smmu_impl	*impl;
+
 	struct arm_smmu_cmdq		cmdq;
 	struct arm_smmu_evtq		evtq;
 	struct arm_smmu_priq		priq;
@@ -812,6 +814,12 @@ static inline void arm_smmu_sva_notifier_synchronize(void) {}
 #endif /* CONFIG_ARM_SMMU_V3_SVA */
 
 /* Implementation details */
+struct arm_smmu_impl {
+	int (*device_reset)(struct arm_smmu_device *smmu);
+	struct arm_smmu_cmdq *(*get_cmdq)(struct arm_smmu_device *smmu, u64 *cmds, int n);
+};
+
 struct arm_smmu_device *arm_smmu_v3_impl_init(struct arm_smmu_device *smmu);
+struct arm_smmu_device *nvidia_smmu_v3_impl_init(struct arm_smmu_device *smmu);
 
 #endif /* _ARM_SMMU_V3_H */
diff --git a/drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c
new file mode 100644
index 000000000000..0c92fe433c6e
--- /dev/null
+++ b/drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c
@@ -0,0 +1,432 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#define dev_fmt(fmt) "nvidia_smmu_cmdqv: " fmt
+
+#include <linux/acpi.h>
+#include <linux/dma-mapping.h>
+#include <linux/interrupt.h>
+#include <linux/iommu.h>
+#include <linux/iopoll.h>
+#include <linux/platform_device.h>
+
+#include <acpi/acpixf.h>
+
+#include "arm-smmu-v3.h"
+
+#define NVIDIA_SMMU_CMDQV_HID		"NVDA0600"
+
+/* CMDQV register page base and size defines */
+#define NVIDIA_CMDQV_CONFIG_BASE	(0)
+#define NVIDIA_CMDQV_CONFIG_SIZE	(SZ_64K)
+#define NVIDIA_VCMDQ_BASE		(0 + SZ_64K)
+#define NVIDIA_VCMDQ_SIZE		(SZ_64K * 2) /* PAGE0 and PAGE1 */
+
+/* CMDQV global config regs */
+#define NVIDIA_CMDQV_CONFIG		0x0000
+#define  CMDQV_EN			BIT(0)
+
+#define NVIDIA_CMDQV_PARAM		0x0004
+#define  CMDQV_NUM_VINTF_LOG2		GENMASK(11, 8)
+#define  CMDQV_NUM_VCMDQ_LOG2		GENMASK(7, 4)
+
+#define NVIDIA_CMDQV_STATUS		0x0008
+#define  CMDQV_STATUS			GENMASK(2, 1)
+#define  CMDQV_ENABLED			BIT(0)
+
+#define NVIDIA_CMDQV_VINTF_ERR_MAP	0x000C
+#define NVIDIA_CMDQV_VINTF_INT_MASK	0x0014
+#define NVIDIA_CMDQV_VCMDQ_ERR_MAP	0x001C
+
+#define NVIDIA_CMDQV_CMDQ_ALLOC(q)	(0x0200 + 0x4*(q))
+#define  CMDQV_CMDQ_ALLOC_VINTF		GENMASK(20, 15)
+#define  CMDQV_CMDQ_ALLOC_LVCMDQ	GENMASK(7, 1)
+#define  CMDQV_CMDQ_ALLOCATED		BIT(0)
+
+/* VINTF config regs */
+#define NVIDIA_CMDQV_VINTF(v)		(0x1000 + 0x100*(v))
+
+#define NVIDIA_VINTF_CONFIG		0x0000
+#define  VINTF_HYP_OWN			BIT(17)
+#define  VINTF_VMID			GENMASK(16, 1)
+#define  VINTF_EN			BIT(0)
+
+#define NVIDIA_VINTF_STATUS		0x0004
+#define  VINTF_STATUS			GENMASK(3, 1)
+#define  VINTF_ENABLED			BIT(0)
+
+/* VCMDQ config regs */
+/* -- PAGE0 -- */
+#define NVIDIA_CMDQV_VCMDQ(q)		(NVIDIA_VCMDQ_BASE + 0x80*(q))
+
+#define NVIDIA_VCMDQ_CONS		0x00000
+#define  VCMDQ_CONS_ERR			GENMASK(30, 24)
+
+#define NVIDIA_VCMDQ_PROD		0x00004
+
+#define NVIDIA_VCMDQ_CONFIG		0x00008
+#define  VCMDQ_EN			BIT(0)
+
+#define NVIDIA_VCMDQ_STATUS		0x0000C
+#define  VCMDQ_ENABLED			BIT(0)
+
+#define NVIDIA_VCMDQ_GERROR		0x00010
+#define NVIDIA_VCMDQ_GERRORN		0x00014
+
+/* -- PAGE1 -- */
+#define NVIDIA_VCMDQ_BASE_L(q)		(NVIDIA_CMDQV_VCMDQ(q) + SZ_64K)
+#define  VCMDQ_ADDR			GENMASK(63, 5)
+#define  VCMDQ_LOG2SIZE			GENMASK(4, 0)
+
+struct nvidia_smmu_vintf {
+	u16			idx;
+	u32			cfg;
+	u32			status;
+
+	void __iomem		*base;
+	struct arm_smmu_cmdq	*vcmdqs;
+};
+
+struct nvidia_smmu {
+	struct arm_smmu_device	smmu;
+
+	struct device		*cmdqv_dev;
+	void __iomem		*cmdqv_base;
+	int			cmdqv_irq;
+
+	/* CMDQV Hardware Params */
+	u16			num_total_vintfs;
+	u16			num_total_vcmdqs;
+	u16			num_vcmdqs_per_vintf;
+
+	/* CMDQV_VINTF(0) reserved for host kernel use */
+	struct nvidia_smmu_vintf vintf0;
+};
+
+static irqreturn_t nvidia_smmu_cmdqv_isr(int irq, void *devid)
+{
+	struct nvidia_smmu *nsmmu = (struct nvidia_smmu *)devid;
+	struct nvidia_smmu_vintf *vintf0 = &nsmmu->vintf0;
+	u32 vintf_err_map[2];
+	u32 vcmdq_err_map[4];
+
+	vintf_err_map[0] = readl_relaxed(nsmmu->cmdqv_base + NVIDIA_CMDQV_VINTF_ERR_MAP);
+	vintf_err_map[1] = readl_relaxed(nsmmu->cmdqv_base + NVIDIA_CMDQV_VINTF_ERR_MAP + 0x4);
+
+	vcmdq_err_map[0] = readl_relaxed(nsmmu->cmdqv_base + NVIDIA_CMDQV_VCMDQ_ERR_MAP);
+	vcmdq_err_map[1] = readl_relaxed(nsmmu->cmdqv_base + NVIDIA_CMDQV_VCMDQ_ERR_MAP + 0x4);
+	vcmdq_err_map[2] = readl_relaxed(nsmmu->cmdqv_base + NVIDIA_CMDQV_VCMDQ_ERR_MAP + 0x8);
+	vcmdq_err_map[3] = readl_relaxed(nsmmu->cmdqv_base + NVIDIA_CMDQV_VCMDQ_ERR_MAP + 0xC);
+
+	dev_warn(nsmmu->cmdqv_dev,
+		 "unexpected cmdqv error reported: vintf_map %08X %08X, vcmdq_map %08X %08X %08X %08X\n",
+		 vintf_err_map[0], vintf_err_map[1], vcmdq_err_map[0], vcmdq_err_map[1],
+		 vcmdq_err_map[2], vcmdq_err_map[3]);
+
+	/* If the error was reported by vintf0, avoid using any of its VCMDQs */
+	if (vintf_err_map[vintf0->idx / 32] & (1 << (vintf0->idx % 32))) {
+		vintf0->status = readl_relaxed(vintf0->base + NVIDIA_VINTF_STATUS);
+
+		dev_warn(nsmmu->cmdqv_dev, "error (0x%lX) reported by host vintf0 - disabling its vcmdqs\n",
+			 FIELD_GET(VINTF_STATUS, vintf0->status));
+	} else if (vintf_err_map[0] || vintf_err_map[1]) {
+		dev_err(nsmmu->cmdqv_dev, "cmdqv error interrupt triggered by unassigned vintf!\n");
+	}
+
+	return IRQ_HANDLED;
+}
+
+/* Adapt struct arm_smmu_cmdq init sequences from arm-smmu-v3.c for VCMDQs */
+static int nvidia_smmu_init_one_arm_smmu_cmdq(struct nvidia_smmu *nsmmu,
+					      struct arm_smmu_cmdq *cmdq,
+					      void __iomem *vcmdq_base,
+					      u16 qidx)
+{
+	struct arm_smmu_queue *q = &cmdq->q;
+	size_t qsz;
+
+	/* struct arm_smmu_cmdq config normally done in arm_smmu_device_hw_probe() */
+	q->llq.max_n_shift = ilog2(SZ_64K >> CMDQ_ENT_SZ_SHIFT);
+
+	/* struct arm_smmu_cmdq config normally done in arm_smmu_init_one_queue() */
+	qsz = (1 << q->llq.max_n_shift) << CMDQ_ENT_SZ_SHIFT;
+	q->base = dmam_alloc_coherent(nsmmu->cmdqv_dev, qsz, &q->base_dma, GFP_KERNEL);
+	if (!q->base) {
+		dev_err(nsmmu->cmdqv_dev, "failed to allocate 0x%zX bytes for VCMDQ%u\n",
+			qsz, qidx);
+		return -ENOMEM;
+	}
+	dev_dbg(nsmmu->cmdqv_dev, "allocated %u entries for VCMDQ%u @ 0x%llX [%pad] ++ %zX",
+		1 << q->llq.max_n_shift, qidx, (u64)q->base, &q->base_dma, qsz);
+
+	q->prod_reg = vcmdq_base + NVIDIA_VCMDQ_PROD;
+	q->cons_reg = vcmdq_base + NVIDIA_VCMDQ_CONS;
+	q->ent_dwords = CMDQ_ENT_DWORDS;
+
+	q->q_base  = q->base_dma & VCMDQ_ADDR;
+	q->q_base |= FIELD_PREP(VCMDQ_LOG2SIZE, q->llq.max_n_shift);
+
+	q->llq.prod = q->llq.cons = 0;
+
+	/* struct arm_smmu_cmdq config normally done in arm_smmu_cmdq_init() */
+	atomic_set(&cmdq->owner_prod, 0);
+	atomic_set(&cmdq->lock, 0);
+
+	cmdq->valid_map = (atomic_long_t *)bitmap_zalloc(1 << q->llq.max_n_shift, GFP_KERNEL);
+	if (!cmdq->valid_map) {
+		dev_err(nsmmu->cmdqv_dev, "failed to allocate valid_map for VCMDQ%u\n", qidx);
+		return -ENOMEM;
+	}
+
+	return 0;
+}
+
+static int nvidia_smmu_cmdqv_init(struct nvidia_smmu *nsmmu)
+{
+	struct nvidia_smmu_vintf *vintf0 = &nsmmu->vintf0;
+	u32 regval;
+	u16 qidx;
+	int ret;
+
+	/* Setup vintf0 for host kernel */
+	vintf0->idx = 0;
+	vintf0->base = nsmmu->cmdqv_base + NVIDIA_CMDQV_VINTF(0);
+
+	regval = FIELD_PREP(VINTF_HYP_OWN, nsmmu->num_total_vintfs > 1);
+	writel_relaxed(regval, vintf0->base + NVIDIA_VINTF_CONFIG);
+
+	regval |= FIELD_PREP(VINTF_EN, 1);
+	writel_relaxed(regval, vintf0->base + NVIDIA_VINTF_CONFIG);
+
+	vintf0->cfg = regval;
+
+	ret = readl_relaxed_poll_timeout(vintf0->base + NVIDIA_VINTF_STATUS,
+					 regval, regval == VINTF_ENABLED,
+					 1, ARM_SMMU_POLL_TIMEOUT_US);
+	vintf0->status = regval;
+	if (ret) {
+		dev_err(nsmmu->cmdqv_dev, "failed to enable VINTF%u: STATUS = 0x%08X\n",
+			vintf0->idx, regval);
+		return ret;
+	}
+
+	/* Allocate vcmdqs to vintf0 */
+	for (qidx = 0; qidx < nsmmu->num_vcmdqs_per_vintf; qidx++) {
+		regval  = FIELD_PREP(CMDQV_CMDQ_ALLOC_VINTF, vintf0->idx);
+		regval |= FIELD_PREP(CMDQV_CMDQ_ALLOC_LVCMDQ, qidx);
+		regval |= CMDQV_CMDQ_ALLOCATED;
+		writel_relaxed(regval, nsmmu->cmdqv_base + NVIDIA_CMDQV_CMDQ_ALLOC(qidx));
+	}
+
+	/* Build an arm_smmu_cmdq for each vcmdq allocated to vintf0 */
+	vintf0->vcmdqs = devm_kcalloc(nsmmu->cmdqv_dev, nsmmu->num_vcmdqs_per_vintf,
+				      sizeof(*vintf0->vcmdqs), GFP_KERNEL);
+	if (!vintf0->vcmdqs)
+		return -ENOMEM;
+
+	for (qidx = 0; qidx < nsmmu->num_vcmdqs_per_vintf; qidx++) {
+		void __iomem *vcmdq_base = nsmmu->cmdqv_base + NVIDIA_CMDQV_VCMDQ(qidx);
+		struct arm_smmu_cmdq *cmdq = &vintf0->vcmdqs[qidx];
+
+		/* Setup struct arm_smmu_cmdq data members */
+		nvidia_smmu_init_one_arm_smmu_cmdq(nsmmu, cmdq, vcmdq_base, qidx);
+
+		/* Configure and enable the vcmdq */
+		writel_relaxed(0, vcmdq_base + NVIDIA_VCMDQ_PROD);
+		writel_relaxed(0, vcmdq_base + NVIDIA_VCMDQ_CONS);
+
+		writeq_relaxed(cmdq->q.q_base, nsmmu->cmdqv_base + NVIDIA_VCMDQ_BASE_L(qidx));
+
+		writel_relaxed(VCMDQ_EN, vcmdq_base + NVIDIA_VCMDQ_CONFIG);
+		ret = readl_poll_timeout(vcmdq_base + NVIDIA_VCMDQ_STATUS,
+					 regval, regval == VCMDQ_ENABLED,
+					 1, ARM_SMMU_POLL_TIMEOUT_US);
+		if (ret) {
+			u32 gerror = readl_relaxed(vcmdq_base + NVIDIA_VCMDQ_GERROR);
+			u32 gerrorn = readl_relaxed(vcmdq_base + NVIDIA_VCMDQ_GERRORN);
+			u32 cons = readl_relaxed(vcmdq_base + NVIDIA_VCMDQ_CONS);
+
+			dev_err(nsmmu->cmdqv_dev,
+				"failed to enable VCMDQ%u: GERROR=0x%X, GERRORN=0x%X, CONS=0x%X\n",
+				qidx, gerror, gerrorn, cons);
+			return ret;
+		}
+
+		dev_info(nsmmu->cmdqv_dev, "VCMDQ%u allocated to VINTF%u as logical-VCMDQ%u\n",
+			 qidx, vintf0->idx, qidx);
+	}
+
+	return 0;
+}
+
+static int nvidia_smmu_probe(struct nvidia_smmu *nsmmu)
+{
+	struct platform_device *cmdqv_pdev = to_platform_device(nsmmu->cmdqv_dev);
+	struct resource *res;
+	u32 regval;
+
+	/* Base address */
+	res = platform_get_resource(cmdqv_pdev, IORESOURCE_MEM, 0);
+	if (!res)
+		return -ENXIO;
+
+	nsmmu->cmdqv_base = devm_ioremap_resource(nsmmu->cmdqv_dev, res);
+	if (IS_ERR(nsmmu->cmdqv_base))
+		return PTR_ERR(nsmmu->cmdqv_base);
+
+	/* Interrupt */
+	nsmmu->cmdqv_irq = platform_get_irq(cmdqv_pdev, 0);
+	if (nsmmu->cmdqv_irq < 0) {
+		dev_warn(nsmmu->cmdqv_dev, "no cmdqv interrupt - errors will not be reported\n");
+		nsmmu->cmdqv_irq = 0;
+	}
+
+	/* Probe the h/w */
+	regval = readl_relaxed(nsmmu->cmdqv_base + NVIDIA_CMDQV_CONFIG);
+	if (!FIELD_GET(CMDQV_EN, regval)) {
+		dev_err(nsmmu->cmdqv_dev, "CMDQV h/w is disabled: CMDQV_CONFIG=0x%08X\n", regval);
+		return -ENODEV;
+	}
+
+	regval = readl_relaxed(nsmmu->cmdqv_base + NVIDIA_CMDQV_STATUS);
+	if (!FIELD_GET(CMDQV_ENABLED, regval) || FIELD_GET(CMDQV_STATUS, regval)) {
+		dev_err(nsmmu->cmdqv_dev, "CMDQV h/w not ready: CMDQV_STATUS=0x%08X\n", regval);
+		return -ENODEV;
+	}
+
+	regval = readl_relaxed(nsmmu->cmdqv_base + NVIDIA_CMDQV_PARAM);
+	nsmmu->num_total_vintfs = 1 << FIELD_GET(CMDQV_NUM_VINTF_LOG2, regval);
+	nsmmu->num_total_vcmdqs = 1 << FIELD_GET(CMDQV_NUM_VCMDQ_LOG2, regval);
+	nsmmu->num_vcmdqs_per_vintf = nsmmu->num_total_vcmdqs / nsmmu->num_total_vintfs;
+
+	return 0;
+}
+
+static struct arm_smmu_cmdq *nvidia_smmu_get_cmdq(struct arm_smmu_device *smmu, u64 *cmds, int n)
+{
+	struct nvidia_smmu *nsmmu = (struct nvidia_smmu *)smmu;
+	struct nvidia_smmu_vintf *vintf0 = &nsmmu->vintf0;
+	u16 qidx;
+
+	/* Make sure vintf0 is enabled and healthy */
+	if (vintf0->status != VINTF_ENABLED)
+		return &smmu->cmdq;
+
+	/* Check for illegal CMDs */
+	if (!FIELD_GET(VINTF_HYP_OWN, vintf0->cfg)) {
+		u64 opcode = (n) ? FIELD_GET(CMDQ_0_OP, cmds[0]) : CMDQ_OP_CMD_SYNC;
+
+		/* List all non-illegal CMDs for cmdq overriding */
+		switch (opcode) {
+		case CMDQ_OP_TLBI_NH_ASID:
+		case CMDQ_OP_TLBI_NH_VA:
+		case CMDQ_OP_TLBI_S12_VMALL:
+		case CMDQ_OP_TLBI_S2_IPA:
+		case CMDQ_OP_ATC_INV:
+			break;
+		default:
+			/* Skip overriding for illegal CMDs */
+			return &smmu->cmdq;
+		}
+	}
+
+	/*
+	 * Select a vcmdq to use. Here we use a temporal solution to
+	 * balance out traffic on cmdq issuing: each cmdq has its own
+	 * lock, if all cpus issue cmdlist using the same cmdq, only
+	 * one CPU at a time can enter the process, while the others
+	 * will be spinning at the same lock.
+	 */
+	qidx = smp_processor_id() % nsmmu->num_vcmdqs_per_vintf;
+	return &vintf0->vcmdqs[qidx];
+}
+
+static int nvidia_smmu_device_reset(struct arm_smmu_device *smmu)
+{
+	struct nvidia_smmu *nsmmu = (struct nvidia_smmu *)smmu;
+	int ret;
+
+	ret = nvidia_smmu_cmdqv_init(nsmmu);
+	if (ret)
+		return ret;
+
+	if (nsmmu->cmdqv_irq) {
+		ret = devm_request_irq(nsmmu->cmdqv_dev, nsmmu->cmdqv_irq, nvidia_smmu_cmdqv_isr,
+				       IRQF_SHARED, "nvidia-smmu-cmdqv", nsmmu);
+		if (ret) {
+			dev_err(nsmmu->cmdqv_dev, "failed to claim irq (%d): %d\n",
+				nsmmu->cmdqv_irq, ret);
+			return ret;
+		}
+	}
+
+	/* Disable FEAT_MSI and OPT_MSIPOLL since VCMDQs only support CMD_SYNC w/CS_NONE */
+	smmu->features &= ~ARM_SMMU_FEAT_MSI;
+	smmu->options &= ~ARM_SMMU_OPT_MSIPOLL;
+
+	return 0;
+}
+
+const struct arm_smmu_impl nvidia_smmu_impl = {
+	.device_reset = nvidia_smmu_device_reset,
+	.get_cmdq = nvidia_smmu_get_cmdq,
+};
+
+#ifdef CONFIG_ACPI
+struct nvidia_smmu *nvidia_smmu_create(struct arm_smmu_device *smmu)
+{
+	struct nvidia_smmu *nsmmu = NULL;
+	struct acpi_iort_node *node;
+	struct acpi_device *adev;
+	struct device *cmdqv_dev;
+	const char *match_uid;
+
+	if (acpi_disabled)
+		return NULL;
+
+	/* Look for a device in the DSDT whose _UID matches the SMMU's iort_node identifier */
+	node = *(struct acpi_iort_node **)dev_get_platdata(smmu->dev);
+	match_uid = kasprintf(GFP_KERNEL, "%u", node->identifier);
+	adev = acpi_dev_get_first_match_dev(NVIDIA_SMMU_CMDQV_HID, match_uid, -1);
+	kfree(match_uid);
+
+	if (!adev)
+		return NULL;
+
+	cmdqv_dev = bus_find_device_by_acpi_dev(&platform_bus_type, adev);
+	if (!cmdqv_dev)
+		return NULL;
+
+	dev_info(smmu->dev, "found companion CMDQV device, %s", dev_name(cmdqv_dev));
+
+	nsmmu = devm_krealloc(smmu->dev, smmu, sizeof(*nsmmu), GFP_KERNEL);
+	if (!nsmmu)
+		return ERR_PTR(-ENOMEM);
+
+	nsmmu->cmdqv_dev = cmdqv_dev;
+
+	return nsmmu;
+}
+#else
+struct nvidia_smmu *nvidia_smmu_create(struct arm_smmu_device *smmu)
+{
+	return NULL;
+}
+#endif
+
+struct arm_smmu_device *nvidia_smmu_v3_impl_init(struct arm_smmu_device *smmu)
+{
+	struct nvidia_smmu *nsmmu;
+	int ret;
+
+	nsmmu = nvidia_smmu_create(smmu);
+	if (!nsmmu)
+		return smmu;
+
+	ret = nvidia_smmu_probe(nsmmu);
+	if (ret)
+		return ERR_PTR(ret);
+
+	nsmmu->smmu.impl = &nvidia_smmu_impl;
+
+	return &nsmmu->smmu;
+}
-- 
2.17.1

[RFC][PATCH v2 13/13] iommu/nvidia-smmu-v3: Add mdev interface support

[Nicolin Chen]

From: Nate Watterson <nwatterson@nvidia.com>

This patch adds initial mdev interface support for NVIDIA SMMU CMDQV
driver.

The NVIDIA SMMU CMDQV module has multiple virtual interfaces (VINTFs),
designed to be exposed to virtual machines running on the user space,
while each VINTF can allocate dedicated VCMDQs for TLB invalidations.

The hypervisor can import, to a VM, one of these interfaces via VFIO
mdev interface, to get access to VINTF registers in the host kernel.

Each VINTF has two pages of MMIO regions: PAGE0 and PAGE1. PAGE0 has
performance sensitive registers such as CONS_INDX and PROD_INDX that
should be programmed by the guest directly, so the driver has a mmap
implementation via the mdev interface to let user space get acces to
PAGE0 directly. PAGE1 then has two base address configuring registers
where the addresses should be translated from guest PAs to host PAs,
so they are handled via mdev read/write() to trap for replacements.

As previous patch mentioned, VINTF0 is reserved for the host kernel
(or hypervisor) use, a VINTFx (x > 0) should be allocated to a guest
VM. And from the guest perspective, the VINTFx (host) is seen as the
VINTF0 of the guest. Beside the two MMIO regions of VINTF0, the guest
VM also has the global configuration MMIO region as the host kernel
does, and this global region is also handled via mdev read/write()
to limit the guest to access the bits of its own.

Additionally, there were a couple of issues for this implementation:
1) Setting into VINTF CONFIG register the same VMID as SMMU's s2_cfg.
2) Before enabling the VINTF, programing up-to-16 sets of SID_REPLACE
   and SID_MATCH registers that stores physical stream IDs of host's
   and corresponding virtual stream IDs of guest's respectively.

And in this patch, we add a pair of ->attach_dev and ->detach_dev and
implement them in the following ways:
1) For each VINTF, pre-allocating a VMID on the bitmap of arm_smmu_v3
   driver to create a link between VINTF index and VMID, so either of
   them can be quickly looked up using the counterpart later.
2) Programming PHY_SID into SID_REPLACE (corresponding register), yet
   writing iommu_group_id (a fake VIRT_SID) into SID_MATCH, as it is
   the only shared information of a passthrough device between a host
   kernel and a hypervisor. So the hypervisor is responsible to match
   the iommu_group_id and then to replace it with a virtual SID.
3) Note that, by doing (1) the VMID is now created along with a VINTF
   in the nvidia_smmu_cmdqv_mdev_create() function, which is executed
   before a hypervisor or VM starts, comparing to previous situation:
   we added a few patches to let arm-smmu-v3 driver allocate a shared
   VMID in arm_smmu_attach_dev() function, when the first passthrough
   device is added to the VM. This means that, in the new situation,
   the shared VMID needs to be passed to the hypervisor, before any
   passthrough device gets attached. So, we reuse VFIO_IOMMU_GET_VMID
   command via the mdev ioctl interface to pass the VMID to the CMDQV
   device model, then to the SMMUv3 device model, so that hypervisor
   can set the same VMID to all IOMMU domains of passthrough devices
   using the previous pathway via VFIO core back to SMMUv3 driver.

Signed-off-by: Nate Watterson <nwatterson@nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |   6 +
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |   2 +
 .../iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c    | 817 ++++++++++++++++++
 3 files changed, 825 insertions(+)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 1b9459592f76..fc543181ddde 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2389,6 +2389,9 @@ static void arm_smmu_detach_dev(struct arm_smmu_master *master, struct device *d
 	if (!smmu_domain)
 		return;
 
+	if (master->smmu->impl && master->smmu->impl->detach_dev)
+		master->smmu->impl->detach_dev(smmu_domain, dev);
+
 	arm_smmu_disable_ats(master);
 
 	spin_lock_irqsave(&smmu_domain->devices_lock, flags);
@@ -2471,6 +2474,9 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
 
 	arm_smmu_enable_ats(master);
 
+	if (smmu->impl && smmu->impl->attach_dev)
+		ret = smmu->impl->attach_dev(smmu_domain, dev);
+
 out_unlock:
 	mutex_unlock(&smmu_domain->init_mutex);
 	return ret;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index bb903a7fa662..a872c0d2f23c 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -817,6 +817,8 @@ static inline void arm_smmu_sva_notifier_synchronize(void) {}
 struct arm_smmu_impl {
 	int (*device_reset)(struct arm_smmu_device *smmu);
 	struct arm_smmu_cmdq *(*get_cmdq)(struct arm_smmu_device *smmu, u64 *cmds, int n);
+	int (*attach_dev)(struct arm_smmu_domain *smmu_domain, struct device *dev);
+	void (*detach_dev)(struct arm_smmu_domain *smmu_domain, struct device *dev);
 };
 
 struct arm_smmu_device *arm_smmu_v3_impl_init(struct arm_smmu_device *smmu);
diff --git a/drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c
index 0c92fe433c6e..265681ba96bc 100644
--- a/drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c
@@ -7,7 +7,10 @@
 #include <linux/interrupt.h>
 #include <linux/iommu.h>
 #include <linux/iopoll.h>
+#include <linux/kvm_host.h>
+#include <linux/mdev.h>
 #include <linux/platform_device.h>
+#include <linux/vfio.h>
 
 #include <acpi/acpixf.h>
 
@@ -20,14 +23,17 @@
 #define NVIDIA_CMDQV_CONFIG_SIZE	(SZ_64K)
 #define NVIDIA_VCMDQ_BASE		(0 + SZ_64K)
 #define NVIDIA_VCMDQ_SIZE		(SZ_64K * 2) /* PAGE0 and PAGE1 */
+#define NVIDIA_VINTF_VCMDQ_BASE		(NVIDIA_VCMDQ_BASE + NVIDIA_VCMDQ_SIZE)
 
 /* CMDQV global config regs */
 #define NVIDIA_CMDQV_CONFIG		0x0000
 #define  CMDQV_EN			BIT(0)
 
 #define NVIDIA_CMDQV_PARAM		0x0004
+#define  CMDQV_NUM_SID_PER_VM_LOG2	GENMASK(15, 12)
 #define  CMDQV_NUM_VINTF_LOG2		GENMASK(11, 8)
 #define  CMDQV_NUM_VCMDQ_LOG2		GENMASK(7, 4)
+#define  CMDQV_VER			GENMASK(3, 0)
 
 #define NVIDIA_CMDQV_STATUS		0x0008
 #define  CMDQV_STATUS			GENMASK(2, 1)
@@ -45,6 +51,12 @@
 /* VINTF config regs */
 #define NVIDIA_CMDQV_VINTF(v)		(0x1000 + 0x100*(v))
 
+#define NVIDIA_VINTFi_CONFIG(i)		(NVIDIA_CMDQV_VINTF(i) + NVIDIA_VINTF_CONFIG)
+#define NVIDIA_VINTFi_STATUS(i)		(NVIDIA_CMDQV_VINTF(i) + NVIDIA_VINTF_STATUS)
+#define NVIDIA_VINTFi_SID_MATCH(i, s)	(NVIDIA_CMDQV_VINTF(i) + NVIDIA_VINTF_SID_MATCH(s))
+#define NVIDIA_VINTFi_SID_REPLACE(i, s)	(NVIDIA_CMDQV_VINTF(i) + NVIDIA_VINTF_SID_REPLACE(s))
+#define NVIDIA_VINTFi_CMDQ_ERR_MAP(i)	(NVIDIA_CMDQV_VINTF(i) + NVIDIA_VINTF_CMDQ_ERR_MAP)
+
 #define NVIDIA_VINTF_CONFIG		0x0000
 #define  VINTF_HYP_OWN			BIT(17)
 #define  VINTF_VMID			GENMASK(16, 1)
@@ -54,6 +66,11 @@
 #define  VINTF_STATUS			GENMASK(3, 1)
 #define  VINTF_ENABLED			BIT(0)
 
+#define NVIDIA_VINTF_SID_MATCH(s)	(0x0040 + 0x4*(s))
+#define NVIDIA_VINTF_SID_REPLACE(s)	(0x0080 + 0x4*(s))
+
+#define NVIDIA_VINTF_CMDQ_ERR_MAP	0x00C0
+
 /* VCMDQ config regs */
 /* -- PAGE0 -- */
 #define NVIDIA_CMDQV_VCMDQ(q)		(NVIDIA_VCMDQ_BASE + 0x80*(q))
@@ -77,13 +94,30 @@
 #define  VCMDQ_ADDR			GENMASK(63, 5)
 #define  VCMDQ_LOG2SIZE			GENMASK(4, 0)
 
+#define NVIDIA_VCMDQ0_BASE_L		0x00000	/* offset to NVIDIA_VCMDQ_BASE_L(0) */
+#define NVIDIA_VCMDQ0_BASE_H		0x00004	/* offset to NVIDIA_VCMDQ_BASE_L(0) */
+#define NVIDIA_VCMDQ0_CONS_INDX_BASE_L	0x00008	/* offset to NVIDIA_VCMDQ_BASE_L(0) */
+#define NVIDIA_VCMDQ0_CONS_INDX_BASE_H	0x0000C	/* offset to NVIDIA_VCMDQ_BASE_L(0) */
+
+/* VINTF logical-VCMDQ regs */
+#define NVIDIA_VINTFi_VCMDQ_BASE(i)	(NVIDIA_VINTF_VCMDQ_BASE + NVIDIA_VCMDQ_SIZE*(i))
+#define NVIDIA_VINTFi_VCMDQ(i, q)	(NVIDIA_VINTFi_VCMDQ_BASE(i) + 0x80*(q))
+
 struct nvidia_smmu_vintf {
 	u16			idx;
+	u16			vmid;
 	u32			cfg;
 	u32			status;
 
 	void __iomem		*base;
+	void __iomem		*vcmdq_base;
 	struct arm_smmu_cmdq	*vcmdqs;
+
+#define NVIDIA_SMMU_VINTF_MAX_SIDS 16
+	DECLARE_BITMAP(sid_map, NVIDIA_SMMU_VINTF_MAX_SIDS);
+	u32			sid_replace[NVIDIA_SMMU_VINTF_MAX_SIDS];
+
+	spinlock_t		lock;
 };
 
 struct nvidia_smmu {
@@ -91,6 +125,8 @@ struct nvidia_smmu {
 
 	struct device		*cmdqv_dev;
 	void __iomem		*cmdqv_base;
+	resource_size_t		ioaddr;
+	resource_size_t		ioaddr_size;
 	int			cmdqv_irq;
 
 	/* CMDQV Hardware Params */
@@ -98,10 +134,38 @@ struct nvidia_smmu {
 	u16			num_total_vcmdqs;
 	u16			num_vcmdqs_per_vintf;
 
+#define NVIDIA_SMMU_MAX_VINTFS	(1 << 6)
+	DECLARE_BITMAP(vintf_map, NVIDIA_SMMU_MAX_VINTFS);
+
 	/* CMDQV_VINTF(0) reserved for host kernel use */
 	struct nvidia_smmu_vintf vintf0;
+
+	struct nvidia_smmu_vintf **vmid_mappings;
+
+#ifdef CONFIG_VFIO_MDEV_DEVICE
+	/* CMDQV_VINTFs exposed to userspace via mdev */
+	struct nvidia_cmdqv_mdev **vintf_mdev;
+	/* Cache for two 64-bit VCMDQ base addresses */
+	struct nvidia_cmdqv_vcmdq_regcache {
+		u64		base_addr;
+		u64		cons_addr;
+	} *vcmdq_regcache;
+	struct mutex		mdev_lock;
+	struct mutex		vmid_lock;
+#endif
 };
 
+#ifdef CONFIG_VFIO_MDEV_DEVICE
+struct nvidia_cmdqv_mdev {
+	struct nvidia_smmu	*nsmmu;
+	struct mdev_device	*mdev;
+	struct nvidia_smmu_vintf *vintf;
+
+	struct notifier_block	group_notifier;
+	struct kvm		*kvm;
+};
+#endif
+
 static irqreturn_t nvidia_smmu_cmdqv_isr(int irq, void *devid)
 {
 	struct nvidia_smmu *nsmmu = (struct nvidia_smmu *)devid;
@@ -135,6 +199,61 @@ static irqreturn_t nvidia_smmu_cmdqv_isr(int irq, void *devid)
 	return IRQ_HANDLED;
 }
 
+#ifdef CONFIG_VFIO_MDEV_DEVICE
+struct mdev_parent_ops nvidia_smmu_cmdqv_mdev_ops;
+
+int nvidia_smmu_cmdqv_mdev_init(struct nvidia_smmu *nsmmu)
+{
+	struct nvidia_cmdqv_mdev *cmdqv_mdev;
+	int ret;
+
+	/* Skip mdev init unless there are available VINTFs */
+	if (nsmmu->num_total_vintfs <= 1)
+		return 0;
+
+	nsmmu->vintf_mdev = devm_kcalloc(nsmmu->cmdqv_dev, nsmmu->num_total_vintfs,
+					 sizeof(*nsmmu->vintf_mdev), GFP_KERNEL);
+	if (!nsmmu->vintf_mdev)
+		return -ENOMEM;
+
+	nsmmu->vcmdq_regcache = devm_kcalloc(nsmmu->cmdqv_dev, nsmmu->num_total_vcmdqs,
+					     sizeof(*nsmmu->vcmdq_regcache), GFP_KERNEL);
+	if (!nsmmu->vcmdq_regcache)
+		return -ENOMEM;
+
+	nsmmu->vmid_mappings = devm_kcalloc(nsmmu->cmdqv_dev, 1 << nsmmu->smmu.vmid_bits,
+					    sizeof(*nsmmu->vmid_mappings), GFP_KERNEL);
+	if (!nsmmu->vmid_mappings)
+		return -ENOMEM;
+
+	mutex_init(&nsmmu->mdev_lock);
+	mutex_init(&nsmmu->vmid_lock);
+
+	/* Add a dummy mdev instance to represent vintf0 */
+	cmdqv_mdev = devm_kzalloc(nsmmu->cmdqv_dev, sizeof(*cmdqv_mdev), GFP_KERNEL);
+	if (!cmdqv_mdev)
+		return -ENOMEM;
+
+	cmdqv_mdev->nsmmu = nsmmu;
+	nsmmu->vintf_mdev[0] = cmdqv_mdev;
+
+	ret = mdev_register_device(nsmmu->cmdqv_dev, &nvidia_smmu_cmdqv_mdev_ops);
+	if (ret) {
+		dev_err(nsmmu->cmdqv_dev, "failed to register mdev device: %d\n", ret);
+		return ret;
+	}
+
+	platform_set_drvdata(to_platform_device(nsmmu->cmdqv_dev), nsmmu);
+
+	return ret;
+}
+#else
+int nvidia_smmu_cmdqv_mdev_init(struct nvidia_smmu *nsmmu)
+{
+	return 0;
+}
+#endif
+
 /* Adapt struct arm_smmu_cmdq init sequences from arm-smmu-v3.c for VCMDQs */
 static int nvidia_smmu_init_one_arm_smmu_cmdq(struct nvidia_smmu *nsmmu,
 					      struct arm_smmu_cmdq *cmdq,
@@ -255,6 +374,16 @@ static int nvidia_smmu_cmdqv_init(struct nvidia_smmu *nsmmu)
 			 qidx, vintf0->idx, qidx);
 	}
 
+	/* Log this vintf0 in vintf_map */
+	set_bit(0, nsmmu->vintf_map);
+
+	spin_lock_init(&vintf0->lock);
+
+#ifdef CONFIG_VFIO_MDEV_DEVICE
+	if (nsmmu->vintf_mdev && nsmmu->vintf_mdev[0])
+		nsmmu->vintf_mdev[0]->vintf = vintf0;
+#endif
+
 	return 0;
 }
 
@@ -269,6 +398,9 @@ static int nvidia_smmu_probe(struct nvidia_smmu *nsmmu)
 	if (!res)
 		return -ENXIO;
 
+	nsmmu->ioaddr = res->start;
+	nsmmu->ioaddr_size = resource_size(res);
+
 	nsmmu->cmdqv_base = devm_ioremap_resource(nsmmu->cmdqv_dev, res);
 	if (IS_ERR(nsmmu->cmdqv_base))
 		return PTR_ERR(nsmmu->cmdqv_base);
@@ -366,9 +498,131 @@ static int nvidia_smmu_device_reset(struct arm_smmu_device *smmu)
 	return 0;
 }
 
+static int nvidia_smmu_bitmap_alloc(unsigned long *map, int size)
+{
+	int idx;
+
+	do {
+		idx = find_first_zero_bit(map, size);
+		if (idx == size)
+			return -ENOSPC;
+	} while (test_and_set_bit(idx, map));
+
+	return idx;
+}
+
+static void nvidia_smmu_bitmap_free(unsigned long *map, int idx)
+{
+	clear_bit(idx, map);
+}
+
+static int nvidia_smmu_attach_dev(struct arm_smmu_domain *smmu_domain, struct device *dev)
+{
+	struct nvidia_smmu *nsmmu = (struct nvidia_smmu *)smmu_domain->smmu;
+	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+	struct nvidia_smmu_vintf *vintf = &nsmmu->vintf0;
+	int i, slot;
+
+#ifdef CONFIG_VFIO_MDEV_DEVICE
+	/* Repoint vintf to the corresponding one for Nested Translation mode */
+	if (smmu_domain->stage == ARM_SMMU_DOMAIN_NESTED) {
+		u16 vmid = smmu_domain->s2_cfg.vmid;
+
+		mutex_lock(&nsmmu->vmid_lock);
+		vintf = nsmmu->vmid_mappings[vmid];
+		mutex_unlock(&nsmmu->vmid_lock);
+		if (!vintf) {
+			dev_err(nsmmu->cmdqv_dev, "failed to find vintf\n");
+			return -EINVAL;
+		}
+	}
+#endif
+
+	for (i = 0; i < fwspec->num_ids; i++) {
+		unsigned int sid = fwspec->ids[i];
+		unsigned long flags;
+
+		/* Find an empty slot of SID_MATCH and SID_REPLACE */
+		slot = nvidia_smmu_bitmap_alloc(vintf->sid_map, NVIDIA_SMMU_VINTF_MAX_SIDS);
+		if (slot < 0)
+			return -EBUSY;
+
+		/* Write PHY_SID to SID_REPLACE and cache it for quick lookup */
+		writel_relaxed(sid, vintf->base + NVIDIA_VINTF_SID_REPLACE(slot));
+
+		spin_lock_irqsave(&vintf->lock, flags);
+		vintf->sid_replace[slot] = sid;
+		spin_unlock_irqrestore(&vintf->lock, flags);
+
+		if (smmu_domain->stage == ARM_SMMU_DOMAIN_NESTED) {
+			struct iommu_group *group = iommu_group_get(dev);
+
+			/*
+			 * Mark SID_MATCH with iommu_group_id, without setting ENABLE bit
+			 * This allows hypervisor to look up one SID_MATCH register that
+			 * matches with the same iommu_group_id, and to eventually update
+			 * VIRT_SID in SID_MATCH.
+			 */
+			writel_relaxed(iommu_group_id(group) << 1,
+				       vintf->base + NVIDIA_VINTF_SID_MATCH(slot));
+		}
+	}
+
+	return 0;
+}
+
+static void nvidia_smmu_detach_dev(struct arm_smmu_domain *smmu_domain, struct device *dev)
+{
+	struct nvidia_smmu *nsmmu = (struct nvidia_smmu *)smmu_domain->smmu;
+	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+	struct nvidia_smmu_vintf *vintf = &nsmmu->vintf0;
+	int i, slot;
+
+#ifdef CONFIG_VFIO_MDEV_DEVICE
+	/* Replace vintf0 with the corresponding one for Nested Translation mode */
+	if (smmu_domain->stage == ARM_SMMU_DOMAIN_NESTED) {
+		u16 vmid =  smmu_domain->s2_cfg.vmid;
+
+		mutex_lock(&nsmmu->vmid_lock);
+		vintf = nsmmu->vmid_mappings[vmid];
+		mutex_unlock(&nsmmu->vmid_lock);
+		if (!vintf) {
+			dev_err(nsmmu->cmdqv_dev, "failed to find vintf\n");
+			return;
+		}
+	}
+#endif
+
+	for (i = 0; i < fwspec->num_ids; i++) {
+		unsigned int sid = fwspec->ids[i];
+		unsigned long flags;
+
+		spin_lock_irqsave(&vintf->lock, flags);
+
+		/* Find a SID_REPLACE register matching sid */
+		for (slot = 0; slot < ARRAY_SIZE(vintf->sid_replace); slot++)
+			if (sid == vintf->sid_replace[slot])
+				break;
+
+		spin_unlock_irqrestore(&vintf->lock, flags);
+
+		if (slot == ARRAY_SIZE(vintf->sid_replace)) {
+			dev_dbg(nsmmu->cmdqv_dev, "failed to find vintf\n");
+			return;
+		}
+
+		writel_relaxed(0, vintf->base + NVIDIA_VINTF_SID_REPLACE(slot));
+		writel_relaxed(0, vintf->base + NVIDIA_VINTF_SID_MATCH(slot));
+
+		nvidia_smmu_bitmap_free(vintf->sid_map, slot);
+	}
+}
+
 const struct arm_smmu_impl nvidia_smmu_impl = {
 	.device_reset = nvidia_smmu_device_reset,
 	.get_cmdq = nvidia_smmu_get_cmdq,
+	.attach_dev = nvidia_smmu_attach_dev,
+	.detach_dev = nvidia_smmu_detach_dev,
 };
 
 #ifdef CONFIG_ACPI
@@ -426,7 +680,570 @@ struct arm_smmu_device *nvidia_smmu_v3_impl_init(struct arm_smmu_device *smmu)
 	if (ret)
 		return ERR_PTR(ret);
 
+	ret = nvidia_smmu_cmdqv_mdev_init(nsmmu);
+	if (ret)
+		return ERR_PTR(ret);
+
 	nsmmu->smmu.impl = &nvidia_smmu_impl;
 
 	return &nsmmu->smmu;
 }
+
+#ifdef CONFIG_VFIO_MDEV_DEVICE
+#define mdev_name(m) dev_name(mdev_dev(m))
+
+int nvidia_smmu_cmdqv_mdev_create(struct mdev_device *mdev)
+{
+	struct device *parent_dev = mdev_parent_dev(mdev);
+	struct nvidia_smmu *nsmmu = platform_get_drvdata(to_platform_device(parent_dev));
+	struct nvidia_cmdqv_mdev *cmdqv_mdev;
+	struct nvidia_smmu_vintf *vintf;
+	int vmid, idx, ret;
+	u32 regval;
+
+	cmdqv_mdev = kzalloc(sizeof(*cmdqv_mdev), GFP_KERNEL);
+	if (!cmdqv_mdev)
+		return -ENOMEM;
+
+	cmdqv_mdev->vintf = kzalloc(sizeof(*cmdqv_mdev->vintf), GFP_KERNEL);
+	if (!cmdqv_mdev->vintf) {
+		ret = -ENOMEM;
+		goto free_mdev;
+	}
+
+	cmdqv_mdev->mdev = mdev;
+	cmdqv_mdev->nsmmu = nsmmu;
+	vintf = cmdqv_mdev->vintf;
+
+	mutex_lock(&nsmmu->mdev_lock);
+	idx = nvidia_smmu_bitmap_alloc(nsmmu->vintf_map, nsmmu->num_total_vintfs);
+	if (idx < 0) {
+		dev_err(nsmmu->cmdqv_dev, "failed to allocate vintfs\n");
+		mutex_unlock(&nsmmu->mdev_lock);
+		ret = -EBUSY;
+		goto free_vintf;
+	}
+	nsmmu->vintf_mdev[idx] = cmdqv_mdev;
+	mutex_unlock(&nsmmu->mdev_lock);
+
+	mutex_lock(&nsmmu->vmid_lock);
+	vmid = arm_smmu_vmid_alloc(&nsmmu->smmu);
+	if (vmid < 0) {
+		dev_err(nsmmu->cmdqv_dev, "failed to allocate vmid\n");
+		mutex_unlock(&nsmmu->vmid_lock);
+		ret = -EBUSY;
+		goto free_vintf_map;
+	}
+
+	/* Create mapping between vmid and vintf */
+	nsmmu->vmid_mappings[vmid] = vintf;
+	mutex_unlock(&nsmmu->vmid_lock);
+
+	vintf->idx = idx;
+	vintf->vmid = vmid;
+	vintf->base = nsmmu->cmdqv_base + NVIDIA_CMDQV_VINTF(idx);
+
+	spin_lock_init(&vintf->lock);
+	mdev_set_drvdata(mdev, cmdqv_mdev);
+
+	writel_relaxed(0, vintf->base + NVIDIA_VINTF_CONFIG);
+
+	/* Point to NVIDIA_VINTFi_VCMDQ_BASE */
+	vintf->vcmdq_base = nsmmu->cmdqv_base + NVIDIA_VINTFi_VCMDQ_BASE(vintf->idx);
+
+	/* Alloc VCMDQs (2n, 2n+1, 2n+2, ...) to VINTF(idx) as logical-VCMDQ (0, 1, 2, ...) */
+	for (idx = 0; idx < nsmmu->num_vcmdqs_per_vintf; idx++) {
+		u16 vcmdq_idx = nsmmu->num_vcmdqs_per_vintf * vintf->idx + idx;
+
+		regval = FIELD_PREP(CMDQV_CMDQ_ALLOC_VINTF, vintf->idx);
+		regval |= FIELD_PREP(CMDQV_CMDQ_ALLOC_LVCMDQ, idx);
+		regval |= CMDQV_CMDQ_ALLOCATED;
+		writel_relaxed(regval, nsmmu->cmdqv_base + NVIDIA_CMDQV_CMDQ_ALLOC(vcmdq_idx));
+
+		dev_info(nsmmu->cmdqv_dev, "allocated VCMDQ%u to VINTF%u as logical-VCMDQ%u\n",
+			 vcmdq_idx, vintf->idx, idx);
+	}
+
+	dev_dbg(nsmmu->cmdqv_dev, "allocated VINTF%u to mdev_device (%s) binding to vmid (%d)\n",
+		vintf->idx, dev_name(mdev_dev(mdev)), vintf->vmid);
+
+	return 0;
+
+free_vintf_map:
+	nvidia_smmu_bitmap_free(nsmmu->vintf_map, idx);
+free_vintf:
+	kfree(cmdqv_mdev->vintf);
+free_mdev:
+	kfree(cmdqv_mdev);
+
+	return ret;
+}
+
+int nvidia_smmu_cmdqv_mdev_remove(struct mdev_device *mdev)
+{
+	struct nvidia_cmdqv_mdev *cmdqv_mdev = mdev_get_drvdata(mdev);
+	struct nvidia_smmu_vintf *vintf = cmdqv_mdev->vintf;
+	struct nvidia_smmu *nsmmu = cmdqv_mdev->nsmmu;
+	u16 idx;
+
+	/* Deallocate VCMDQs of the VINTF(idx) */
+	for (idx = 0; idx < nsmmu->num_vcmdqs_per_vintf; idx++) {
+		u16 vcmdq_idx = nsmmu->num_vcmdqs_per_vintf * vintf->idx + idx;
+
+		writel_relaxed(0, nsmmu->cmdqv_base + NVIDIA_CMDQV_CMDQ_ALLOC(vcmdq_idx));
+
+		dev_info(nsmmu->cmdqv_dev, "deallocated VCMDQ%u to VINTF%u\n",
+			 vcmdq_idx, vintf->idx);
+	}
+
+	/* Disable and cleanup VINTF configurations */
+	writel_relaxed(0, vintf->base + NVIDIA_VINTF_CONFIG);
+
+	mutex_lock(&nsmmu->mdev_lock);
+	nvidia_smmu_bitmap_free(nsmmu->vintf_map, vintf->idx);
+	nsmmu->vintf_mdev[vintf->idx] = NULL;
+	mutex_unlock(&nsmmu->mdev_lock);
+
+	mutex_lock(&nsmmu->vmid_lock);
+	arm_smmu_vmid_free(&nsmmu->smmu, vintf->vmid);
+	nsmmu->vmid_mappings[vintf->vmid] = NULL;
+	mutex_unlock(&nsmmu->vmid_lock);
+
+	mdev_set_drvdata(mdev, NULL);
+	kfree(cmdqv_mdev->vintf);
+	kfree(cmdqv_mdev);
+
+	return 0;
+}
+
+static int nvidia_smmu_cmdqv_mdev_group_notifier(struct notifier_block *nb,
+						 unsigned long action, void *data)
+{
+	struct nvidia_cmdqv_mdev *cmdqv_mdev =
+		container_of(nb, struct nvidia_cmdqv_mdev, group_notifier);
+
+	if (action == VFIO_GROUP_NOTIFY_SET_KVM)
+		cmdqv_mdev->kvm = data;
+
+	return NOTIFY_OK;
+}
+
+int nvidia_smmu_cmdqv_mdev_open(struct mdev_device *mdev)
+{
+	struct nvidia_cmdqv_mdev *cmdqv_mdev = mdev_get_drvdata(mdev);
+	unsigned long events = VFIO_GROUP_NOTIFY_SET_KVM;
+	struct device *dev = mdev_dev(mdev);
+	int ret;
+
+	cmdqv_mdev->group_notifier.notifier_call = nvidia_smmu_cmdqv_mdev_group_notifier;
+
+	ret = vfio_register_notifier(dev, VFIO_GROUP_NOTIFY, &events, &cmdqv_mdev->group_notifier);
+	if (ret)
+		dev_err(mdev_dev(mdev), "failed to register group notifier: %d\n", ret);
+
+	return ret;
+}
+
+void nvidia_smmu_cmdqv_mdev_release(struct mdev_device *mdev)
+{
+	struct nvidia_cmdqv_mdev *cmdqv_mdev = mdev_get_drvdata(mdev);
+	struct device *dev = mdev_dev(mdev);
+
+	vfio_unregister_notifier(dev, VFIO_GROUP_NOTIFY, &cmdqv_mdev->group_notifier);
+}
+
+ssize_t nvidia_smmu_cmdqv_mdev_read(struct mdev_device *mdev, char __user *buf,
+				    size_t count, loff_t *ppos)
+{
+	struct nvidia_cmdqv_mdev *cmdqv_mdev = mdev_get_drvdata(mdev);
+	struct nvidia_smmu_vintf *vintf = cmdqv_mdev->vintf;
+	struct nvidia_smmu *nsmmu = cmdqv_mdev->nsmmu;
+	struct device *dev = mdev_dev(mdev);
+	loff_t reg_offset = *ppos, reg;
+	u64 regval = 0;
+	u16 idx, slot;
+
+	/* Only support aligned 32/64-bit accesses */
+	if (!count || (count % 4) || count > 8 || (reg_offset % count))
+		return -EINVAL;
+
+	switch (reg_offset) {
+	case NVIDIA_CMDQV_CONFIG:
+		regval = readl_relaxed(nsmmu->cmdqv_base + NVIDIA_CMDQV_CONFIG);
+		break;
+	case NVIDIA_CMDQV_STATUS:
+		regval = readl_relaxed(nsmmu->cmdqv_base + NVIDIA_CMDQV_STATUS);
+		break;
+	case NVIDIA_CMDQV_PARAM:
+		/*
+		 * Guest shall import only one of the VINTFs using mdev interface,
+		 * so limit the numbers of VINTF and VCMDQs in the PARAM register.
+		 */
+		regval = readl_relaxed(nsmmu->cmdqv_base + NVIDIA_CMDQV_PARAM);
+		regval &= ~(CMDQV_NUM_VINTF_LOG2 | CMDQV_NUM_VCMDQ_LOG2);
+		regval |= FIELD_PREP(CMDQV_NUM_VINTF_LOG2, 0);
+		regval |= FIELD_PREP(CMDQV_NUM_VCMDQ_LOG2, ilog2(nsmmu->num_vcmdqs_per_vintf));
+		break;
+	case NVIDIA_CMDQV_VINTF_ERR_MAP:
+		/* Translate the value to bit 0 as guest can only see vintf0 */
+		regval = readl_relaxed(vintf->base + NVIDIA_VINTF_STATUS);
+		regval = !!FIELD_GET(VINTF_STATUS, regval);
+		break;
+	case NVIDIA_CMDQV_VINTF_INT_MASK:
+		/* Translate the value to bit 0 as guest can only see vintf0 */
+		regval = readq_relaxed(nsmmu->cmdqv_base + NVIDIA_CMDQV_VINTF_INT_MASK);
+		regval = !!(regval & BIT(vintf->idx));
+		break;
+	case NVIDIA_CMDQV_VCMDQ_ERR_MAP:
+		regval = readq_relaxed(vintf->base + NVIDIA_VINTF_CMDQ_ERR_MAP);
+		break;
+	case NVIDIA_CMDQV_CMDQ_ALLOC(0) ... NVIDIA_CMDQV_CMDQ_ALLOC(128):
+		if (idx >= nsmmu->num_vcmdqs_per_vintf) {
+			/* Guest only has limited number of VMCDQs for one VINTF */
+			regval = 0;
+		} else {
+			/* We have allocated VCMDQs, so just report it constantly */
+			idx = (reg_offset - NVIDIA_CMDQV_CMDQ_ALLOC(0)) / 4;
+			regval = FIELD_PREP(CMDQV_CMDQ_ALLOC_LVCMDQ, idx) | CMDQV_CMDQ_ALLOCATED;
+		}
+		break;
+	case NVIDIA_VINTFi_CONFIG(0):
+		regval = readl_relaxed(vintf->base + NVIDIA_VINTF_CONFIG);
+		/* Guest should not see the VMID field */
+		regval &= ~(VINTF_VMID);
+		break;
+	case NVIDIA_VINTFi_STATUS(0):
+		regval = readl_relaxed(vintf->base + NVIDIA_VINTF_STATUS);
+		break;
+	case NVIDIA_VINTFi_SID_MATCH(0, 0) ... NVIDIA_VINTFi_SID_MATCH(0, 15):
+		slot = (reg_offset - NVIDIA_VINTFi_SID_MATCH(0, 0)) / 0x4;
+		regval = readl_relaxed(vintf->base + NVIDIA_VINTF_SID_MATCH(slot));
+		break;
+	case NVIDIA_VINTFi_SID_REPLACE(0, 0) ... NVIDIA_VINTFi_SID_REPLACE(0, 15):
+		/* Guest should not see the PHY_SID but know whether it is set or not */
+		slot = (reg_offset - NVIDIA_VINTFi_SID_REPLACE(0, 0)) / 0x4;
+		regval = !!readl_relaxed(vintf->base + NVIDIA_VINTF_SID_REPLACE(slot));
+		break;
+	case NVIDIA_VINTFi_CMDQ_ERR_MAP(0):
+		regval = readl_relaxed(vintf->base + NVIDIA_VINTF_CMDQ_ERR_MAP);
+		break;
+	case NVIDIA_CMDQV_VCMDQ(0) ... NVIDIA_CMDQV_VCMDQ(128):
+		/* We allow fallback reading of VCMDQ PAGE0 upon a warning */
+		dev_warn(dev, "read access at 0x%llx should go through mmap instead!", reg_offset);
+
+		/* Adjust reg_offset since we're reading base on VINTF logical-VCMDQ space */
+		regval = readl_relaxed(vintf->vcmdq_base + reg_offset - NVIDIA_CMDQV_VCMDQ(0));
+		break;
+	case NVIDIA_VCMDQ_BASE_L(0) ... NVIDIA_VCMDQ_BASE_L(128):
+		/* Decipher idx and reg of VCMDQ */
+		idx = (reg_offset - NVIDIA_VCMDQ_BASE_L(0)) / 0x80;
+		reg = reg_offset - NVIDIA_VCMDQ_BASE_L(idx);
+
+		switch (reg) {
+		case NVIDIA_VCMDQ0_BASE_L:
+			regval = nsmmu->vcmdq_regcache[idx].base_addr;
+			if (count == 4)
+				regval = lower_32_bits(regval);
+			break;
+		case NVIDIA_VCMDQ0_BASE_H:
+			regval = upper_32_bits(nsmmu->vcmdq_regcache[idx].base_addr);
+			break;
+		case NVIDIA_VCMDQ0_CONS_INDX_BASE_L:
+			regval = nsmmu->vcmdq_regcache[idx].cons_addr;
+			if (count == 4)
+				regval = lower_32_bits(regval);
+			break;
+		case NVIDIA_VCMDQ0_CONS_INDX_BASE_H:
+			regval = upper_32_bits(nsmmu->vcmdq_regcache[idx].cons_addr);
+			break;
+		default:
+			dev_err(dev, "unknown base address read access at 0x%llX\n", reg_offset);
+			break;
+		}
+		break;
+	default:
+		dev_err(dev, "unhandled read access at 0x%llX\n", reg_offset);
+		return -EINVAL;
+	}
+
+	if (copy_to_user(buf, &regval, count))
+		return -EFAULT;
+	*ppos += count;
+
+	return count;
+}
+
+static u64 nvidia_smmu_cmdqv_mdev_gpa_to_pa(struct nvidia_cmdqv_mdev *cmdqv_mdev, u64 gpa)
+{
+	u64 gfn, hfn, hva, hpa, pg_offset;
+	struct page *pg;
+	long num_pages;
+
+	gfn = gpa_to_gfn(gpa);
+	pg_offset = gpa ^ gfn_to_gpa(gfn);
+
+	hva = gfn_to_hva(cmdqv_mdev->kvm, gfn);
+	if (kvm_is_error_hva(hva))
+		return 0;
+
+	num_pages = get_user_pages(hva, 1, FOLL_GET | FOLL_WRITE, &pg, NULL);
+	if (num_pages < 1)
+		return 0;
+
+	hfn = page_to_pfn(pg);
+	hpa = pfn_to_hpa(hfn);
+
+	return hpa | pg_offset;
+}
+
+ssize_t nvidia_smmu_cmdqv_mdev_write(struct mdev_device *mdev, const char __user *buf,
+				     size_t count, loff_t *ppos)
+{
+	struct nvidia_cmdqv_mdev *cmdqv_mdev = mdev_get_drvdata(mdev);
+	struct nvidia_smmu_vintf *vintf = cmdqv_mdev->vintf;
+	struct nvidia_smmu *nsmmu = cmdqv_mdev->nsmmu;
+	struct device *dev = mdev_dev(mdev);
+	loff_t reg_offset = *ppos, reg;
+	u64 mask = U32_MAX;
+	u64 regval = 0x0;
+	u16 idx, slot;
+
+	/* Only support aligned 32/64-bit accesses */
+	if (!count || (count % 4) || count > 8 || (reg_offset % count))
+		return -EINVAL;
+
+	/* Get the value to be written to the register at reg_offset */
+	if (copy_from_user(&regval, buf, count))
+		return -EFAULT;
+
+	switch (reg_offset) {
+	case NVIDIA_VINTFi_CONFIG(0):
+		regval &= ~(VINTF_VMID);
+		regval |= FIELD_PREP(VINTF_VMID, vintf->vmid);
+		writel_relaxed(regval, vintf->base + NVIDIA_VINTF_CONFIG);
+		break;
+	case NVIDIA_CMDQV_CMDQ_ALLOC(0) ... NVIDIA_CMDQV_CMDQ_ALLOC(128):
+		/* Ignore since VCMDQs were already allocated to the VINTF */
+		break;
+	case NVIDIA_VINTFi_SID_MATCH(0, 0) ... NVIDIA_VINTFi_SID_MATCH(0, 15):
+		slot = (reg_offset - NVIDIA_VINTFi_SID_MATCH(0, 0)) / 0x4;
+		writel_relaxed(regval, vintf->base + NVIDIA_VINTF_SID_MATCH(slot));
+		break;
+	case NVIDIA_VINTFi_SID_REPLACE(0, 0) ... NVIDIA_VINTFi_SID_REPLACE(0, 15):
+		/* Guest should not alter the value */
+		break;
+	case NVIDIA_CMDQV_VCMDQ(0) ... NVIDIA_CMDQV_VCMDQ(128):
+		/* We allow fallback writing at VCMDQ PAGE0 upon a warning */
+		dev_warn(dev, "write access at 0x%llx should go through mmap instead!", reg_offset);
+
+		/* Adjust reg_offset since we're reading base on VINTF logical-VCMDQ space */
+		writel_relaxed(regval, vintf->vcmdq_base + reg_offset - NVIDIA_CMDQV_VCMDQ(0));
+		break;
+	case NVIDIA_VCMDQ_BASE_L(0) ... NVIDIA_VCMDQ_BASE_L(128):
+		/* Decipher idx and reg of VCMDQ */
+		idx = (reg_offset - NVIDIA_VCMDQ_BASE_L(0)) / 0x80;
+		reg = reg_offset - NVIDIA_VCMDQ_BASE_L(idx);
+
+		switch (reg) {
+		case NVIDIA_VCMDQ0_BASE_L:
+			if (count == 8)
+				mask = U64_MAX;
+			regval &= mask;
+			nsmmu->vcmdq_regcache[idx].base_addr &= ~mask;
+			nsmmu->vcmdq_regcache[idx].base_addr |= regval;
+			regval = nsmmu->vcmdq_regcache[idx].base_addr;
+			break;
+		case NVIDIA_VCMDQ0_BASE_H:
+			nsmmu->vcmdq_regcache[idx].base_addr &= U32_MAX;
+			nsmmu->vcmdq_regcache[idx].base_addr |= regval << 32;
+			regval = nsmmu->vcmdq_regcache[idx].base_addr;
+			break;
+		case NVIDIA_VCMDQ0_CONS_INDX_BASE_L:
+			if (count == 8)
+				mask = U64_MAX;
+			regval &= mask;
+			nsmmu->vcmdq_regcache[idx].cons_addr &= ~mask;
+			nsmmu->vcmdq_regcache[idx].cons_addr |= regval;
+			regval = nsmmu->vcmdq_regcache[idx].cons_addr;
+			break;
+		case NVIDIA_VCMDQ0_CONS_INDX_BASE_H:
+			nsmmu->vcmdq_regcache[idx].cons_addr &= U32_MAX;
+			nsmmu->vcmdq_regcache[idx].cons_addr |= regval << 32;
+			regval = nsmmu->vcmdq_regcache[idx].cons_addr;
+			break;
+		default:
+			dev_err(dev, "unknown base address write access at 0x%llX\n", reg_offset);
+			return -EFAULT;
+		}
+
+		/* Translate guest PA to host PA before writing to the address register */
+		regval = nvidia_smmu_cmdqv_mdev_gpa_to_pa(cmdqv_mdev, regval);
+
+		/* Do not fail mdev write as higher/lower addresses can be written separately */
+		if (!regval)
+			dev_dbg(dev, "failed to convert guest address for VCMDQ%d\n", idx);
+
+		/* Adjust reg_offset since we're accessing it via the VINTF CMDQ aperture */
+		reg_offset -= NVIDIA_CMDQV_VCMDQ(0);
+		if (count == 8)
+			writeq_relaxed(regval, vintf->vcmdq_base + reg_offset);
+		else
+			writel_relaxed(regval, vintf->vcmdq_base + reg_offset);
+		break;
+
+	default:
+		dev_err(dev, "unhandled write access at 0x%llX\n", reg_offset);
+		return -EINVAL;
+	}
+
+	*ppos += count;
+	return count;
+}
+
+long nvidia_smmu_cmdqv_mdev_ioctl(struct mdev_device *mdev, unsigned int cmd, unsigned long arg)
+{
+	struct nvidia_cmdqv_mdev *cmdqv_mdev = mdev_get_drvdata(mdev);
+	struct nvidia_smmu_vintf *vintf = cmdqv_mdev->vintf;
+	struct device *dev = mdev_dev(mdev);
+	struct vfio_device_info device_info;
+	struct vfio_region_info region_info;
+	unsigned long minsz;
+
+	switch (cmd) {
+	case VFIO_DEVICE_GET_INFO:
+		minsz = offsetofend(struct vfio_device_info, num_irqs);
+
+		if (copy_from_user(&device_info, (void __user *)arg, minsz))
+			return -EFAULT;
+
+		if (device_info.argsz < minsz)
+			return -EINVAL;
+
+		device_info.flags = 0;
+		device_info.num_irqs = 0;
+		/* MMIO Regions: [0] - CMDQV_CONFIG, [1] - VCMDQ_PAGE0, [2] - VCMDQ_PAGE1 */
+		device_info.num_regions = 3;
+
+		return copy_to_user((void __user *)arg, &device_info, minsz) ? -EFAULT : 0;
+	case VFIO_DEVICE_GET_REGION_INFO:
+		minsz = offsetofend(struct vfio_region_info, offset);
+
+		if (copy_from_user(&region_info, (void __user *)arg, minsz))
+			return -EFAULT;
+
+		if (region_info.argsz < minsz)
+			return -EINVAL;
+
+		if (region_info.index >= 3)
+			return -EINVAL;
+
+		/* MMIO Regions: [0] - CMDQV_CONFIG, [1] - VCMDQ_PAGE0, [2] - VCMDQ_PAGE1 */
+		region_info.size = SZ_64K;
+		region_info.offset = region_info.index * SZ_64K;
+		region_info.flags = VFIO_REGION_INFO_FLAG_READ | VFIO_REGION_INFO_FLAG_WRITE;
+		/* In case of VCMDQ_PAGE0, add FLAG_MMAP */
+		if (region_info.index == 1)
+			region_info.flags |= VFIO_REGION_INFO_FLAG_MMAP;
+
+		return copy_to_user((void __user *)arg, &region_info, minsz) ? -EFAULT : 0;
+	case VFIO_IOMMU_GET_VMID:
+		return copy_to_user((void __user *)arg, &vintf->vmid, sizeof(u16)) ? -EFAULT : 0;
+	default:
+		dev_err(dev, "unhandled ioctl cmd 0x%X\n", cmd);
+		return -ENOTTY;
+	}
+
+	return 0;
+}
+
+int nvidia_smmu_cmdqv_mdev_mmap(struct mdev_device *mdev, struct vm_area_struct *vma)
+{
+	struct nvidia_cmdqv_mdev *cmdqv_mdev = mdev_get_drvdata(mdev);
+	struct nvidia_smmu_vintf *vintf = cmdqv_mdev->vintf;
+	struct nvidia_smmu *nsmmu = cmdqv_mdev->nsmmu;
+	struct device *dev = mdev_dev(mdev);
+	unsigned int region_idx;
+	unsigned long size;
+
+	/* Make sure that only VCMDQ_PAGE0 MMIO region can be mmapped */
+	region_idx = (vma->vm_pgoff << PAGE_SHIFT) / SZ_64K;
+	if (region_idx != 0x1) {
+		dev_err(dev, "mmap unsupported for region_idx %d", region_idx);
+		return -EINVAL;
+	}
+
+	size = vma->vm_end - vma->vm_start;
+	if (size > SZ_64K)
+		return -EINVAL;
+
+	/* Fixup the VMA */
+	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+
+	/* Map PAGE0 of VINTF[idx] */
+	vma->vm_pgoff = nsmmu->ioaddr + NVIDIA_VINTFi_VCMDQ_BASE(vintf->idx);
+	vma->vm_pgoff >>= PAGE_SHIFT;
+
+	return remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff, size, vma->vm_page_prot);
+}
+
+static ssize_t name_show(struct mdev_type *mtype,
+			 struct mdev_type_attribute *attr, char *buf)
+{
+	return sprintf(buf, "%s\n", "NVIDIA_SMMU_CMDQV_VINTF - (2 VCMDQs/VINTF)");
+}
+static MDEV_TYPE_ATTR_RO(name);
+
+static ssize_t available_instances_show(struct mdev_type *mtype,
+					struct mdev_type_attribute *attr, char *buf)
+{
+	struct device *parent_dev = mtype_get_parent_dev(mtype);
+	struct nvidia_smmu *nsmmu = platform_get_drvdata(to_platform_device(parent_dev));
+	u16 idx, cnt = 0;
+
+	mutex_lock(&nsmmu->mdev_lock);
+	for (idx = 0; idx < nsmmu->num_total_vintfs; idx++)
+		cnt += !nsmmu->vintf_mdev[idx];
+	mutex_unlock(&nsmmu->mdev_lock);
+
+	return sprintf(buf, "%d\n", cnt);
+}
+static MDEV_TYPE_ATTR_RO(available_instances);
+
+static ssize_t device_api_show(struct mdev_type *mtype,
+			       struct mdev_type_attribute *attr, char *buf)
+{
+	return sprintf(buf, "%s\n", VFIO_DEVICE_API_PLATFORM_STRING);
+}
+static MDEV_TYPE_ATTR_RO(device_api);
+
+static struct attribute *mdev_types_attrs[] = {
+	&mdev_type_attr_name.attr,
+	&mdev_type_attr_device_api.attr,
+	&mdev_type_attr_available_instances.attr,
+	NULL,
+};
+
+static struct attribute_group mdev_type_group1 = {
+	.name  = "nvidia_cmdqv_vintf",
+	.attrs = mdev_types_attrs,
+};
+
+static struct attribute_group *mdev_type_groups[] = {
+	&mdev_type_group1,
+	NULL,
+};
+
+struct mdev_parent_ops nvidia_smmu_cmdqv_mdev_ops = {
+	.owner = THIS_MODULE,
+	.supported_type_groups = mdev_type_groups,
+	.create = nvidia_smmu_cmdqv_mdev_create,
+	.remove = nvidia_smmu_cmdqv_mdev_remove,
+	.open = nvidia_smmu_cmdqv_mdev_open,
+	.release = nvidia_smmu_cmdqv_mdev_release,
+	.read = nvidia_smmu_cmdqv_mdev_read,
+	.write = nvidia_smmu_cmdqv_mdev_write,
+	.ioctl = nvidia_smmu_cmdqv_mdev_ioctl,
+	.mmap = nvidia_smmu_cmdqv_mdev_mmap,
+};
+
+#endif /* CONFIG_VFIO_MDEV_DEVICE */
-- 
2.17.1

Re: [RFC][PATCH v2 00/13] iommu/arm-smmu-v3: Add NVIDIA implementation

[Alex Williamson]

On Mon, 30 Aug 2021 19:59:10 -0700
Nicolin Chen <nicolinc@nvidia.com> wrote:

> The SMMUv3 devices implemented in the Grace SoC support NVIDIA's custom
> CMDQ-Virtualization (CMDQV) hardware. Like the new ECMDQ feature first
> introduced in the ARM SMMUv3.3 specification, CMDQV adds multiple VCMDQ
> interfaces to supplement the single architected SMMU_CMDQ in an effort
> to reduce contention.
> 
> This series of patches add CMDQV support with its preparational changes:
> 
> * PATCH-1 to PATCH-8 are related to shared VMID feature: they are used
>   first to improve TLB utilization, second to bind a shared VMID with a
>   VCMDQ interface for hardware configuring requirement.

The vfio changes would need to be implemented in alignment with the
/dev/iommu proposals[1].  AIUI, the VMID is essentially binding
multiple containers together for TLB invalidation, which I expect in
the proposal below is largely already taken care of in that a single
iommu-fd can support multiple I/O address spaces and it's largely
expected that a hypervisor would use a single iommu-fd so this explicit
connection by userspace across containers wouldn't be necessary.

We're expecting to talk more about the /dev/iommu approach at Plumbers
in few weeks.  Thanks,

Alex

[1]https://lore.kernel.org/kvm/BN9PR11MB5433B1E4AE5B0480369F97178C189@BN9PR11MB5433.namprd11.prod.outlook.com/

 
> * PATCH-9 and PATCH-10 are to accommodate the NVIDIA implementation with
>   the existing arm-smmu-v3 driver.
> 
> * PATCH-11 borrows the "implementation infrastructure" from the arm-smmu
>   driver so later change can build upon it.
> 
> * PATCH-12 adds an initial NVIDIA implementation related to host feature,
>   and also adds implementation specific ->device_reset() and ->get_cmdq()
>   callback functions.
> 
> * PATCH-13 adds virtualization features using VFIO mdev interface, which
>   allows user space hypervisor to map and get access to one of the VCMDQ
>   interfaces of CMDQV module.
> 
> ( Thinking that reviewers can get a better view of this implementation,
>   I am attaching QEMU changes here for reference purpose:
>       https://github.com/nicolinc/qemu/commits/dev/cmdqv_v6.0.0-rc2
>   The branch has all preparational changes, while I'm still integrating
>   device model and ARM-VIRT changes, and will push them these two days,
>   although they might not be in a good shape of being sent to review yet )
> 
> Above all, I marked RFC for this series, as I feel that we may come up
> some better solution. So please kindly share your reviews and insights.
> 
> Thank you!
> 
> Changelog
> v1->v2:
>  * Added mdev interface support for hypervisor and VMs.
>  * Added preparational changes for mdev interface implementation.
>  * PATCH-12 Changed ->issue_cmdlist() to ->get_cmdq() for a better
>    integration with recently merged ECMDQ-related changes.
> 
> Nate Watterson (3):
>   iommu/arm-smmu-v3: Add implementation infrastructure
>   iommu/arm-smmu-v3: Add support for NVIDIA CMDQ-Virtualization hw
>   iommu/nvidia-smmu-v3: Add mdev interface support
> 
> Nicolin Chen (10):
>   iommu: Add set_nesting_vmid/get_nesting_vmid functions
>   vfio: add VFIO_IOMMU_GET_VMID and VFIO_IOMMU_SET_VMID
>   vfio: Document VMID control for IOMMU Virtualization
>   vfio: add set_vmid and get_vmid for vfio_iommu_type1
>   vfio/type1: Implement set_vmid and get_vmid
>   vfio/type1: Set/get VMID to/from iommu driver
>   iommu/arm-smmu-v3: Add shared VMID support for NESTING
>   iommu/arm-smmu-v3: Add VMID alloc/free helpers
>   iommu/arm-smmu-v3: Pass dev pointer to arm_smmu_detach_dev
>   iommu/arm-smmu-v3: Pass cmdq pointer in arm_smmu_cmdq_issue_cmdlist()
> 
>  Documentation/driver-api/vfio.rst             |   34 +
>  MAINTAINERS                                   |    2 +
>  drivers/iommu/arm/arm-smmu-v3/Makefile        |    2 +-
>  .../iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c  |   15 +
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |  121 +-
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |   18 +
>  .../iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c    | 1249 +++++++++++++++++
>  drivers/iommu/iommu.c                         |   20 +
>  drivers/vfio/vfio.c                           |   25 +
>  drivers/vfio/vfio_iommu_type1.c               |   37 +
>  include/linux/iommu.h                         |    5 +
>  include/linux/vfio.h                          |    2 +
>  include/uapi/linux/vfio.h                     |   26 +
>  13 files changed, 1537 insertions(+), 19 deletions(-)
>  create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c
>  create mode 100644 drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c
> 

RE: [RFC][PATCH v2 00/13] iommu/arm-smmu-v3: Add NVIDIA implementation

[Tian, Kevin]

> From: Alex Williamson
> Sent: Wednesday, September 1, 2021 12:16 AM
> 
> On Mon, 30 Aug 2021 19:59:10 -0700
> Nicolin Chen <nicolinc@nvidia.com> wrote:
> 
> > The SMMUv3 devices implemented in the Grace SoC support NVIDIA's
> custom
> > CMDQ-Virtualization (CMDQV) hardware. Like the new ECMDQ feature first
> > introduced in the ARM SMMUv3.3 specification, CMDQV adds multiple
> VCMDQ
> > interfaces to supplement the single architected SMMU_CMDQ in an effort
> > to reduce contention.
> >
> > This series of patches add CMDQV support with its preparational changes:
> >
> > * PATCH-1 to PATCH-8 are related to shared VMID feature: they are used
> >   first to improve TLB utilization, second to bind a shared VMID with a
> >   VCMDQ interface for hardware configuring requirement.
> 
> The vfio changes would need to be implemented in alignment with the
> /dev/iommu proposals[1].  AIUI, the VMID is essentially binding
> multiple containers together for TLB invalidation, which I expect in
> the proposal below is largely already taken care of in that a single
> iommu-fd can support multiple I/O address spaces and it's largely
> expected that a hypervisor would use a single iommu-fd so this explicit
> connection by userspace across containers wouldn't be necessary.

Agree. VMID is equivalent to DID (domain id) in other vendor iommus.
with /dev/iommu multiple I/O address spaces can share the same VMID
via nesting. No need of exposing VMID to userspace to build the 
connection.

Thanks,
Kevin

> 
> We're expecting to talk more about the /dev/iommu approach at Plumbers
> in few weeks.  Thanks,
> 
> Alex
> 
> [1]https://lore.kernel.org/kvm/BN9PR11MB5433B1E4AE5B0480369F97178C1
> 89@BN9PR11MB5433.namprd11.prod.outlook.com/
> 
> 
> > * PATCH-9 and PATCH-10 are to accommodate the NVIDIA implementation
> with
> >   the existing arm-smmu-v3 driver.
> >
> > * PATCH-11 borrows the "implementation infrastructure" from the arm-
> smmu
> >   driver so later change can build upon it.
> >
> > * PATCH-12 adds an initial NVIDIA implementation related to host feature,
> >   and also adds implementation specific ->device_reset() and ->get_cmdq()
> >   callback functions.
> >
> > * PATCH-13 adds virtualization features using VFIO mdev interface, which
> >   allows user space hypervisor to map and get access to one of the VCMDQ
> >   interfaces of CMDQV module.
> >
> > ( Thinking that reviewers can get a better view of this implementation,
> >   I am attaching QEMU changes here for reference purpose:
> >       https://github.com/nicolinc/qemu/commits/dev/cmdqv_v6.0.0-rc2
> >   The branch has all preparational changes, while I'm still integrating
> >   device model and ARM-VIRT changes, and will push them these two days,
> >   although they might not be in a good shape of being sent to review yet )
> >
> > Above all, I marked RFC for this series, as I feel that we may come up
> > some better solution. So please kindly share your reviews and insights.
> >
> > Thank you!
> >
> > Changelog
> > v1->v2:
> >  * Added mdev interface support for hypervisor and VMs.
> >  * Added preparational changes for mdev interface implementation.
> >  * PATCH-12 Changed ->issue_cmdlist() to ->get_cmdq() for a better
> >    integration with recently merged ECMDQ-related changes.
> >
> > Nate Watterson (3):
> >   iommu/arm-smmu-v3: Add implementation infrastructure
> >   iommu/arm-smmu-v3: Add support for NVIDIA CMDQ-Virtualization hw
> >   iommu/nvidia-smmu-v3: Add mdev interface support
> >
> > Nicolin Chen (10):
> >   iommu: Add set_nesting_vmid/get_nesting_vmid functions
> >   vfio: add VFIO_IOMMU_GET_VMID and VFIO_IOMMU_SET_VMID
> >   vfio: Document VMID control for IOMMU Virtualization
> >   vfio: add set_vmid and get_vmid for vfio_iommu_type1
> >   vfio/type1: Implement set_vmid and get_vmid
> >   vfio/type1: Set/get VMID to/from iommu driver
> >   iommu/arm-smmu-v3: Add shared VMID support for NESTING
> >   iommu/arm-smmu-v3: Add VMID alloc/free helpers
> >   iommu/arm-smmu-v3: Pass dev pointer to arm_smmu_detach_dev
> >   iommu/arm-smmu-v3: Pass cmdq pointer in
> arm_smmu_cmdq_issue_cmdlist()
> >
> >  Documentation/driver-api/vfio.rst             |   34 +
> >  MAINTAINERS                                   |    2 +
> >  drivers/iommu/arm/arm-smmu-v3/Makefile        |    2 +-
> >  .../iommu/arm/arm-smmu-v3/arm-smmu-v3-impl.c  |   15 +
> >  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |  121 +-
> >  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |   18 +
> >  .../iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c    | 1249
> +++++++++++++++++
> >  drivers/iommu/iommu.c                         |   20 +
> >  drivers/vfio/vfio.c                           |   25 +
> >  drivers/vfio/vfio_iommu_type1.c               |   37 +
> >  include/linux/iommu.h                         |    5 +
> >  include/linux/vfio.h                          |    2 +
> >  include/uapi/linux/vfio.h                     |   26 +
> >  13 files changed, 1537 insertions(+), 19 deletions(-)
> >  create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-
> impl.c
> >  create mode 100644 drivers/iommu/arm/arm-smmu-v3/nvidia-smmu-v3.c
> >
> 
> _______________________________________________
> iommu mailing list
> iommu@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC][PATCH v2 00/13] iommu/arm-smmu-v3: Add NVIDIA implementation

[Jason Gunthorpe]

On Wed, Sep 01, 2021 at 06:55:55AM +0000, Tian, Kevin wrote:
> > From: Alex Williamson
> > Sent: Wednesday, September 1, 2021 12:16 AM
> > 
> > On Mon, 30 Aug 2021 19:59:10 -0700
> > Nicolin Chen <nicolinc@nvidia.com> wrote:
> > 
> > > The SMMUv3 devices implemented in the Grace SoC support NVIDIA's
> > custom
> > > CMDQ-Virtualization (CMDQV) hardware. Like the new ECMDQ feature first
> > > introduced in the ARM SMMUv3.3 specification, CMDQV adds multiple
> > VCMDQ
> > > interfaces to supplement the single architected SMMU_CMDQ in an effort
> > > to reduce contention.
> > >
> > > This series of patches add CMDQV support with its preparational changes:
> > >
> > > * PATCH-1 to PATCH-8 are related to shared VMID feature: they are used
> > >   first to improve TLB utilization, second to bind a shared VMID with a
> > >   VCMDQ interface for hardware configuring requirement.
> > 
> > The vfio changes would need to be implemented in alignment with the
> > /dev/iommu proposals[1].  AIUI, the VMID is essentially binding
> > multiple containers together for TLB invalidation, which I expect in
> > the proposal below is largely already taken care of in that a single
> > iommu-fd can support multiple I/O address spaces and it's largely
> > expected that a hypervisor would use a single iommu-fd so this explicit
> > connection by userspace across containers wouldn't be necessary.
> 
> Agree. VMID is equivalent to DID (domain id) in other vendor iommus.
> with /dev/iommu multiple I/O address spaces can share the same VMID
> via nesting. No need of exposing VMID to userspace to build the 
> connection.

Indeed, this looks like a flavour of the accelerated invalidation
stuff we've talked about already.

I would see it probably exposed as some HW specific IOCTL on the iommu
fd to get access to the accelerated invalidation for IOASID's in the
FD.

Indeed, this seems like a further example of why /dev/iommu is looking
like a good idea as this RFC is very complicated to do something
fairly simple.

Where are thing on the /dev/iommu work these days?

Thanks,
Jason

RE: [RFC][PATCH v2 00/13] iommu/arm-smmu-v3: Add NVIDIA implementation

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Thursday, September 2, 2021 10:45 PM
> 
> On Wed, Sep 01, 2021 at 06:55:55AM +0000, Tian, Kevin wrote:
> > > From: Alex Williamson
> > > Sent: Wednesday, September 1, 2021 12:16 AM
> > >
> > > On Mon, 30 Aug 2021 19:59:10 -0700
> > > Nicolin Chen <nicolinc@nvidia.com> wrote:
> > >
> > > > The SMMUv3 devices implemented in the Grace SoC support NVIDIA's
> > > custom
> > > > CMDQ-Virtualization (CMDQV) hardware. Like the new ECMDQ feature
> first
> > > > introduced in the ARM SMMUv3.3 specification, CMDQV adds multiple
> > > VCMDQ
> > > > interfaces to supplement the single architected SMMU_CMDQ in an
> effort
> > > > to reduce contention.
> > > >
> > > > This series of patches add CMDQV support with its preparational
> changes:
> > > >
> > > > * PATCH-1 to PATCH-8 are related to shared VMID feature: they are
> used
> > > >   first to improve TLB utilization, second to bind a shared VMID with a
> > > >   VCMDQ interface for hardware configuring requirement.
> > >
> > > The vfio changes would need to be implemented in alignment with the
> > > /dev/iommu proposals[1].  AIUI, the VMID is essentially binding
> > > multiple containers together for TLB invalidation, which I expect in
> > > the proposal below is largely already taken care of in that a single
> > > iommu-fd can support multiple I/O address spaces and it's largely
> > > expected that a hypervisor would use a single iommu-fd so this explicit
> > > connection by userspace across containers wouldn't be necessary.
> >
> > Agree. VMID is equivalent to DID (domain id) in other vendor iommus.
> > with /dev/iommu multiple I/O address spaces can share the same VMID
> > via nesting. No need of exposing VMID to userspace to build the
> > connection.
> 
> Indeed, this looks like a flavour of the accelerated invalidation
> stuff we've talked about already.
> 
> I would see it probably exposed as some HW specific IOCTL on the iommu
> fd to get access to the accelerated invalidation for IOASID's in the
> FD.
> 
> Indeed, this seems like a further example of why /dev/iommu is looking
> like a good idea as this RFC is very complicated to do something
> fairly simple.
> 
> Where are thing on the /dev/iommu work these days?
> 

We are actively working on the basic skeleton. Our original plan is to send
out the 1st draft before LPC, with support of vfio type1 semantics and
and pci dev only (single-device group). But later we realized that adding
multi-devices group support is also necessary even in the 1st draft to avoid
some dirty hacks and build the complete picture. This will add some time
though. If things go well, we'll still try hit the original plan. If not, it will be
soon after LPC.

Thanks
Kevin

Re: [RFC][PATCH v2 00/13] iommu/arm-smmu-v3: Add NVIDIA implementation

[Nicolin Chen]

Hi Kevin,

On Thu, Sep 02, 2021 at 10:27:06PM +0000, Tian, Kevin wrote:

> > Indeed, this looks like a flavour of the accelerated invalidation
> > stuff we've talked about already.
> >
> > I would see it probably exposed as some HW specific IOCTL on the iommu
> > fd to get access to the accelerated invalidation for IOASID's in the
> > FD.
> >
> > Indeed, this seems like a further example of why /dev/iommu is looking
> > like a good idea as this RFC is very complicated to do something
> > fairly simple.
> >
> > Where are thing on the /dev/iommu work these days?
> 
> We are actively working on the basic skeleton. Our original plan is to send
> out the 1st draft before LPC, with support of vfio type1 semantics and
> and pci dev only (single-device group). But later we realized that adding
> multi-devices group support is also necessary even in the 1st draft to avoid
> some dirty hacks and build the complete picture. This will add some time
> though. If things go well, we'll still try hit the original plan. If not, it will be
> soon after LPC.

As I also want to take a look at your work for this implementation,
would you please CC me when you send out your patches? Thank you!