Re: [PATCH v4 3/6] iommu/vt-d: Implement domain ops for attach_dev_pasid

From: Jacob Pan <jacob.jun.pan@linux.intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: iommu@lists.linux-foundation.org,
	LKML <linux-kernel@vger.kernel.org>,
	dmaengine@vger.kernel.org, Joerg Roedel <joro@8bytes.org>,
	David Woodhouse <dwmw2@infradead.org>,
	Jean-Philippe Brucker <jean-philippe@linaro.com>,
	Lu Baolu <baolu.lu@linux.intel.com>,
	Christoph Hellwig <hch@infradead.org>,
	vkoul@kernel.org, robin.murphy@arm.com, will@kernel.org,
	Yi Liu <yi.l.liu@intel.com>, Dave Jiang <dave.jiang@intel.com>,
	"Tian, Kevin" <kevin.tian@intel.com>,
	Raj Ashok <ashok.raj@intel.com>,
	Eric Auger <eric.auger@redhat.com>,
	jacob.jun.pan@linux.intel.com
Subject: Re: [PATCH v4 3/6] iommu/vt-d: Implement domain ops for attach_dev_pasid
Date: Tue, 24 May 2022 13:45:26 -0700	[thread overview]
Message-ID: <20220524134526.409519ac@jacob-builder> (raw)
In-Reply-To: <20220524180241.GY1343366@nvidia.com>

Hi Jason,

On Tue, 24 May 2022 15:02:41 -0300, Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Tue, May 24, 2022 at 09:12:35AM -0700, Jacob Pan wrote:
> > Hi Jason,
> > 
> > On Tue, 24 May 2022 10:51:35 -0300, Jason Gunthorpe <jgg@nvidia.com>
> > wrote: 
> > > On Wed, May 18, 2022 at 11:21:17AM -0700, Jacob Pan wrote:  
> > > > On VT-d platforms with scalable mode enabled, devices issue DMA
> > > > requests with PASID need to attach PASIDs to given IOMMU domains.
> > > > The attach operation involves the following:
> > > > - Programming the PASID into the device's PASID table
> > > > - Tracking device domain and the PASID relationship
> > > > - Managing IOTLB and device TLB invalidations
> > > > 
> > > > This patch add attach_dev_pasid functions to the default domain ops
> > > > which is used by DMA and identity domain types. It could be
> > > > extended to support other domain types whenever necessary.
> > > > 
> > > > Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
> > > > Signed-off-by: Jacob Pan <jacob.jun.pan@linux.intel.com>
> > > >  drivers/iommu/intel/iommu.c | 72
> > > > +++++++++++++++++++++++++++++++++++-- 1 file changed, 70
> > > > insertions(+), 2 deletions(-)
> > > > 
> > > > diff --git a/drivers/iommu/intel/iommu.c
> > > > b/drivers/iommu/intel/iommu.c index 1c2c92b657c7..75615c105fdf
> > > > 100644 +++ b/drivers/iommu/intel/iommu.c
> > > > @@ -1556,12 +1556,18 @@ static void __iommu_flush_dev_iotlb(struct
> > > > device_domain_info *info, u64 addr, unsigned int mask)
> > > >  {
> > > >  	u16 sid, qdep;
> > > > +	ioasid_t pasid;
> > > >  
> > > >  	if (!info || !info->ats_enabled)
> > > >  		return;
> > > >  
> > > >  	sid = info->bus << 8 | info->devfn;
> > > >  	qdep = info->ats_qdep;
> > > > +	pasid = iommu_get_pasid_from_domain(info->dev,
> > > > &info->domain->domain);    
> > > 
> > > No, a simgple domain can be attached to multiple pasids, all need to
> > > be flushed.
> > >   
> > Here is device TLB flush, why would I want to flush PASIDs other than my
> > own device attached?  
> 
> Again, a domain can be attached to multiple PASID's *on the same
> device*
> 
> The idea that there is only one PASID per domain per device is not
> right.
> 
Got you, I was under the impression that there is no use case yet for
multiple PASIDs per device-domain based on our early discussion.
https://lore.kernel.org/lkml/20220315142216.GV11336@nvidia.com/

Perhaps I misunderstood. I will make the API more future proof and search
through the pasid_array xa for *all* domain-device matches. Like you
suggested earlier, may need to retrieve the xa in the first place and use
xas_for_each to get a faster search.

Thanks,

Jacob