RE: [PATCH v2 3/8] iommu/vt-d: Implement device_pasid domain attach ops

["Tian, Kevin" <kevin.tian@intel.com>]

> From: Jacob Pan <jacob.jun.pan@linux.intel.com>
> Sent: Tuesday, March 15, 2022 1:07 PM
> 
> On VT-d platforms with scalable mode enabled, devices issue DMA requests
> with PASID need to attach to the correct IOMMU domains.
> The attach operation involves the following:
> - programming the PASID into device's PASID table
> - tracking device domain and the PASID relationship
> - managing IOTLB and device TLB invalidations
> 
> This patch extends DMAR domain and device domain info with xarrays to
> track per domain and per device PASIDs.  It provides the flexibility to
> be used beyond DMA API PASID support.
> 
> Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
> Signed-off-by: Jacob Pan <jacob.jun.pan@linux.intel.com>
> ---
>  drivers/iommu/intel/iommu.c | 194
> +++++++++++++++++++++++++++++++++++-
>  include/linux/intel-iommu.h |  12 ++-
>  2 files changed, 202 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
> index 881f8361eca2..9267194eaed3 100644
> --- a/drivers/iommu/intel/iommu.c
> +++ b/drivers/iommu/intel/iommu.c
> @@ -1622,20 +1622,48 @@ static void __iommu_flush_dev_iotlb(struct
> device_domain_info *info,
>  			   qdep, addr, mask);
>  }
> 
> +static void __iommu_flush_dev_piotlb(struct device_domain_info *info,

piotlb is confusing, better be:

	__iommu_flush_dev_iotlb_pasid()

> +					u64 address,
> +				     ioasid_t pasid, unsigned int mask)
> +{
> +	u16 sid, qdep;
> +
> +	if (!info || !info->ats_enabled)
> +		return;
> +
> +	sid = info->bus << 8 | info->devfn;
> +	qdep = info->ats_qdep;
> +	qi_flush_dev_iotlb_pasid(info->iommu, sid, info->pfsid,
> +				 pasid, qdep, address, mask);
> +}
> +
>  static void iommu_flush_dev_iotlb(struct dmar_domain *domain,
>  				  u64 addr, unsigned mask)
>  {
>  	unsigned long flags;
>  	struct device_domain_info *info;
>  	struct subdev_domain_info *sinfo;
> +	unsigned long pasid;
> +	struct pasid_info *pinfo;
> 
>  	if (!domain->has_iotlb_device)
>  		return;
> 
>  	spin_lock_irqsave(&device_domain_lock, flags);
> -	list_for_each_entry(info, &domain->devices, link)
> -		__iommu_flush_dev_iotlb(info, addr, mask);
> -
> +	list_for_each_entry(info, &domain->devices, link) {
> +		/*
> +		 * We cannot use PASID based devTLB invalidation on
> RID2PASID
> +		 * Device does not understand RID2PASID/0. This is different

Lack of a conjunction word between 'RID2PASID' and 'Device'.

and what is RID2PASID/0? It would be clearer to point out that RID2PASID
is visible only within the iommu to mark out requests without PASID, 
thus this PASID value should never be sent to the device side.

> +		 * than IOTLB invalidation where RID2PASID is also used for
> +		 * tagging.

Then it would be obvious because IOTLB is iommu internal agent thus takes 
use of RID2PASID for tagging.

> +		 */
> +		xa_for_each(&info->pasids, pasid, pinfo) {
> +			if (!pasid)

this should be compared to PASID_RID2PASID (though it's zero today).

> +				__iommu_flush_dev_iotlb(info, addr, mask);
> +			else
> +				__iommu_flush_dev_piotlb(info, addr, pasid,
> mask);
> +		}
> +	}
>  	list_for_each_entry(sinfo, &domain->subdevices, link_domain) {
>  		info = get_domain_info(sinfo->pdev);
>  		__iommu_flush_dev_iotlb(info, addr, mask);

Thanks
Kevin