From: Baolu Lu <baolu.lu@linux.intel.com>
To: Jean-Philippe Brucker <jean-philippe@linaro.org>
Cc: Joerg Roedel <joro@8bytes.org>, Jason Gunthorpe <jgg@nvidia.com>,
Christoph Hellwig <hch@infradead.org>,
Kevin Tian <kevin.tian@intel.com>,
Ashok Raj <ashok.raj@intel.com>, Will Deacon <will@kernel.org>,
Robin Murphy <robin.murphy@arm.com>,
Jean-Philippe Brucker <jean-philippe@linaro.com>,
Dave Jiang <dave.jiang@intel.com>, Vinod Koul <vkoul@kernel.org>,
Eric Auger <eric.auger@redhat.com>, Liu Yi L <yi.l.liu@intel.com>,
Jacob jun Pan <jacob.jun.pan@intel.com>,
iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v5 10/12] iommu: Prepare IOMMU domain for IOPF
Date: Fri, 6 May 2022 13:40:01 +0800 [thread overview]
Message-ID: <5bbf6ccf-2a49-7611-b8af-143252decc1f@linux.intel.com> (raw)
In-Reply-To: <YnPTXwjpHnnGp4j2@myrica>
On 2022/5/5 21:38, Jean-Philippe Brucker wrote:
> Hi Baolu,
>
> On Thu, May 05, 2022 at 04:31:38PM +0800, Baolu Lu wrote:
>> On 2022/5/4 02:20, Jean-Philippe Brucker wrote:
>>>> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
>>>> index 7cae631c1baa..33449523afbe 100644
>>>> --- a/drivers/iommu/iommu.c
>>>> +++ b/drivers/iommu/iommu.c
>>>> @@ -3174,3 +3174,24 @@ void iommu_detach_device_pasid(struct iommu_domain *domain,
>>>> iommu_group_put(group);
>>>> }
>>>> +
>>>> +struct iommu_domain *iommu_get_domain_for_dev_pasid(struct device *dev,
>>>> + ioasid_t pasid)
>>>> +{
>>>> + struct iommu_domain *domain;
>>>> + struct iommu_group *group;
>>>> +
>>>> + if (!pasid_valid(pasid))
>>>> + return NULL;
>>>> +
>>>> + group = iommu_group_get(dev);
>>>> + if (!group)
>>>> + return NULL;
>>>> +
>>>> + mutex_lock(&group->mutex);
>>> Unfortunately this still causes the deadlock when unbind() flushes the
>>> IOPF queue while holding the group mutex.
>>
>> Sorry, I didn't get your point here.
>>
>> Do you mean unbind() could hold group mutex before calling this helper?
>> The group mutex is only available in iommu.c. The unbind() has no means
>> to hold this lock. Or, I missed anything?
>
> I wasn't clear, it's iommu_detach_device_pasid() that holds the
> group->mutex:
>
> iommu_sva_unbind_device() |
> iommu_detach_device_pasid() |
> mutex_lock(&group->mutex) |
> domain->ops->detach_dev_pasid() | iopf_handle_group()
> iopf_queue_flush_dev() | iommu_get_domain_for_dev_pasid()
> ... wait for IOPF work | mutex_lock(&group->mutex)
> | ... deadlock
Ah! Yes. Thank you for the clarification.
>
> Thanks,
> Jean
>
>>
>> Best regards,
>> baolu
>>
>>>
>>> If we make this function private to IOPF, then we can get rid of this
>>> mutex_lock(). It's OK because:
>>>
>>> * xarray protects its internal state with RCU, so we can call
>>> xa_load() outside the lock.
>>>
>>> * The domain obtained from xa_load is finalized. Its content is valid
>>> because xarray stores the domain using rcu_assign_pointer(), which has a
>>> release memory barrier, which pairs with data dependencies in IOPF
>>> (domain->sva_ioas etc).
>>>
>>> We'll need to be careful about this when allowing other users to install
>>> a fault handler. Should be fine as long as the handler and data are
>>> installed before the domain is added to pasid_array.
>>>
>>> * We know the domain is valid the whole time IOPF is using it, because
>>> unbind() waits for pending faults.
>>>
>>> We just need a comment explaining the last point, something like:
>>>
>>> /*
>>> * Safe to fetch outside the group mutex because:
>>> * - xarray protects its internal state with RCU
>>> * - the domain obtained is either NULL or fully formed
>>> * - the IOPF work is the only caller and is flushed before the
>>> * domain is freed.
>>> */
Agreed. The mutex is needed only when domain could possibly be freed
before unbind(). In that case, we need this mutex and get a reference
from the domain. As we have dropped the domain user reference, this lock
is unnecessary.
>>>
>>> Thanks,
>>> Jean
>>>
>>>> + domain = xa_load(&group->pasid_array, pasid);
>>>> + mutex_unlock(&group->mutex);
>>>> + iommu_group_put(group);
>>>> +
>>>> + return domain;
>>>> +}
>>
Best regards,
baolu
next prev parent reply other threads:[~2022-05-06 5:40 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-05-02 1:48 [PATCH v5 00/12] iommu: SVA and IOPF refactoring Lu Baolu
2022-05-02 1:48 ` [PATCH v5 01/12] dmaengine: idxd: Separate user and kernel pasid enabling Lu Baolu
2022-05-02 1:48 ` [PATCH v5 02/12] iommu: Add pasid_bits field in struct dev_iommu Lu Baolu
2022-05-03 18:02 ` Jean-Philippe Brucker
2022-05-05 6:25 ` Baolu Lu
2022-05-02 1:48 ` [PATCH v5 03/12] iommu: Add attach/detach_dev_pasid domain ops Lu Baolu
2022-05-03 18:07 ` Jean-Philippe Brucker
2022-05-05 6:28 ` Baolu Lu
2022-05-02 1:48 ` [PATCH v5 04/12] iommu/sva: Basic data structures for SVA Lu Baolu
2022-05-03 18:09 ` Jean-Philippe Brucker
2022-05-05 6:42 ` Baolu Lu
2022-05-07 8:32 ` Baolu Lu
2022-05-07 12:39 ` Baolu Lu
2022-05-02 1:48 ` [PATCH v5 05/12] iommu/vt-d: Remove SVM_FLAG_SUPERVISOR_MODE support Lu Baolu
2022-05-02 1:48 ` [PATCH v5 06/12] iommu/vt-d: Add SVA domain support Lu Baolu
2022-05-02 1:48 ` [PATCH v5 07/12] arm-smmu-v3/sva: " Lu Baolu
2022-05-03 18:12 ` Jean-Philippe Brucker
2022-05-05 7:09 ` Baolu Lu
2022-05-02 1:48 ` [PATCH v5 08/12] iommu/sva: Use attach/detach_pasid_dev in SVA interfaces Lu Baolu
2022-05-02 1:48 ` [PATCH v5 09/12] iommu: Remove SVA related callbacks from iommu ops Lu Baolu
2022-05-03 18:14 ` Jean-Philippe Brucker
2022-05-02 1:48 ` [PATCH v5 10/12] iommu: Prepare IOMMU domain for IOPF Lu Baolu
2022-05-03 18:20 ` Jean-Philippe Brucker
2022-05-05 8:31 ` Baolu Lu
2022-05-05 13:38 ` Jean-Philippe Brucker
2022-05-06 5:40 ` Baolu Lu [this message]
2022-05-02 1:48 ` [PATCH v5 11/12] iommu: Per-domain I/O page fault handling Lu Baolu
2022-05-03 18:27 ` Jean-Philippe Brucker
2022-05-02 1:48 ` [PATCH v5 12/12] iommu: Rename iommu-sva-lib.{c,h} Lu Baolu
2022-05-03 18:28 ` Jean-Philippe Brucker
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5bbf6ccf-2a49-7611-b8af-143252decc1f@linux.intel.com \
--to=baolu.lu@linux.intel.com \
--cc=ashok.raj@intel.com \
--cc=dave.jiang@intel.com \
--cc=eric.auger@redhat.com \
--cc=hch@infradead.org \
--cc=iommu@lists.linux-foundation.org \
--cc=jacob.jun.pan@intel.com \
--cc=jean-philippe@linaro.com \
--cc=jean-philippe@linaro.org \
--cc=jgg@nvidia.com \
--cc=joro@8bytes.org \
--cc=kevin.tian@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=robin.murphy@arm.com \
--cc=vkoul@kernel.org \
--cc=will@kernel.org \
--cc=yi.l.liu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).