Re: [PATCH v3 01/18] iommu: Add device dma ownership set/release interfaces

From: Lu Baolu <baolu.lu@linux.intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>, Joerg Roedel <joro@8bytes.org>
Cc: baolu.lu@linux.intel.com,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Alex Williamson <alex.williamson@redhat.com>,
	Bjorn Helgaas <bhelgaas@google.com>,
	Christoph Hellwig <hch@infradead.org>,
	Kevin Tian <kevin.tian@intel.com>,
	Ashok Raj <ashok.raj@intel.com>, Will Deacon <will@kernel.org>,
	Robin Murphy <robin.murphy@arm.com>,
	Dan Williams <dan.j.williams@intel.com>,
	rafael@kernel.org, Diana Craciun <diana.craciun@oss.nxp.com>,
	Cornelia Huck <cohuck@redhat.com>,
	Eric Auger <eric.auger@redhat.com>, Liu Yi L <yi.l.liu@intel.com>,
	Jacob jun Pan <jacob.jun.pan@intel.com>,
	Chaitanya Kulkarni <kch@nvidia.com>,
	Stuart Yoder <stuyoder@gmail.com>,
	Laurentiu Tudor <laurentiu.tudor@nxp.com>,
	Thierry Reding <thierry.reding@gmail.com>,
	David Airlie <airlied@linux.ie>, Daniel Vetter <daniel@ffwll.ch>,
	Jonathan Hunter <jonathanh@nvidia.com>,
	Li Yang <leoyang.li@nxp.com>, Dmitry Osipenko <digetx@gmail.com>,
	iommu@lists.linux-foundation.org, linux-pci@vger.kernel.org,
	kvm@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3 01/18] iommu: Add device dma ownership set/release interfaces
Date: Tue, 7 Dec 2021 09:52:37 +0800	[thread overview]
Message-ID: <fa30d398-746c-c7d0-830f-40e3aaee16d6@linux.intel.com> (raw)
In-Reply-To: <20211206150144.GC4670@nvidia.com>

On 12/6/21 11:01 PM, Jason Gunthorpe wrote:
> On Mon, Dec 06, 2021 at 02:35:55PM +0100, Joerg Roedel wrote:
>> On Mon, Dec 06, 2021 at 09:58:46AM +0800, Lu Baolu wrote:
>>> >From the perspective of who is initiating the device to do DMA, device
>>> DMA could be divided into the following types:
>>>
>>>          DMA_OWNER_DMA_API: Device DMAs are initiated by a kernel driver
>>> 			through the kernel DMA API.
>>>          DMA_OWNER_PRIVATE_DOMAIN: Device DMAs are initiated by a kernel
>>> 			driver with its own PRIVATE domain.
>>> 	DMA_OWNER_PRIVATE_DOMAIN_USER: Device DMAs are initiated by
>>> 			userspace.
>>
>> I have looked at the other iommu patches in this series, but I still
>> don't quite get what the difference in the code flow is between
>> DMA_OWNER_PRIVATE_DOMAIN and DMA_OWNER_PRIVATE_DOMAIN_USER. What are the
>> differences in the iommu core behavior based on this setting?
> 
> USER causes the IOMMU code to spend extra work to never assign the
> default domain. Lu, it would be good to update the comment with this
> detail
> 
> Once in USER mode the domain is always a /dev/null domain or a domain
> controlled by userspace. Never a domain pointing at kernel memory.

Yes. The __iommu_detach_group() re-attaches the default domain
automatically. This is not allowed once in USER mode.

I will update the comments whit this detail.

> 
>>>   struct group_device {
>>> @@ -621,6 +624,7 @@ struct iommu_group *iommu_group_alloc(void)
>>>   	INIT_LIST_HEAD(&group->devices);
>>>   	INIT_LIST_HEAD(&group->entry);
>>>   	BLOCKING_INIT_NOTIFIER_HEAD(&group->notifier);
>>> +	group->dma_owner = DMA_OWNER_NONE;
>>
>>
>> DMA_OWNER_NONE is also questionable. All devices are always in one
>> domain, and the default domain is always the one used for DMA-API, so
>> why isn't the initial value DMA_OWNER_DMA_API?
> 
> 'NONE' means the group is in the default domain but no driver is bound
> and thus DMA isn't being used. Seeing NONE is the only condition when
> it is OK to change the domain.
> 
> This could be reworked to instead rely on the refcount == 0 as the
> signal to know it is OK to change the domain and then we never have
> NONE at all. Lu?

NONE is just a parking state. It's okay to rely on the "refcount == 0"
for state transition as far as I see. I will work towards this.

Best regards,
baolu