From: Mostafa Saleh <smostafa@google.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: iommu@lists.linux.dev, Joerg Roedel <joro@8bytes.org>,
linux-arm-kernel@lists.infradead.org,
Robin Murphy <robin.murphy@arm.com>,
Will Deacon <will@kernel.org>, Moritz Fischer <mdf@kernel.org>,
Moritz Fischer <moritzf@google.com>,
Michael Shavit <mshavit@google.com>,
Nicolin Chen <nicolinc@nvidia.com>,
patches@lists.linux.dev,
Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
Subject: Re: [PATCH v4 06/16] iommu/arm-smmu-v3: Hold arm_smmu_asid_lock during all of attach_dev
Date: Tue, 13 Feb 2024 13:30:12 +0000 [thread overview]
Message-ID: <Zctu5HOJOPaTNBt-@google.com> (raw)
In-Reply-To: <20240201132443.GP1455070@nvidia.com>
On Thu, Feb 01, 2024 at 09:24:43AM -0400, Jason Gunthorpe wrote:
> On Thu, Feb 01, 2024 at 12:15:53PM +0000, Mostafa Saleh wrote:
> > Hi Jason,
> >
> > On Thu, Jan 25, 2024 at 07:57:16PM -0400, Jason Gunthorpe wrote:
> > > The BTM support wants to be able to change the ASID of any smmu_domain.
> > > When it goes to do this it holds the arm_smmu_asid_lock and iterates over
> > > the target domain's devices list.
> > >
> > > During attach of a S1 domain we must ensure that the devices list and
> > > CD are in sync, otherwise we could miss CD updates or a parallel CD update
> > > could push an out of date CD.
> > >
> > > This is pretty complicated, and almost works today because
> > > arm_smmu_detach_dev() removes the master from the linked list before
> > > working on the CD entries, preventing parallel update of the CD.
> > >
> > > However, it does have an issue where the CD can remain programed while the
> > > domain appears to be unattached. arm_smmu_share_asid() will then not clear
> > > any CD entriess and install its own CD entry with the same ASID
> > > concurrently. This creates a small race window where the IOMMU can see two
> > > ASIDs pointing to different translations.
> >
> > I don’t see the race condition.
> >
> > The current flow is as follows,
> > For SVA, if the asid was used by domain_x, it will do:
> >
> > lock(arm_smmu_asid_lock)
> > Alloc new asid and set cd->asid.
> > lock(domain_x->devices_lock)
> > Write new CD with the new asid
> > unlock(domain_x->devices_lock)
> > unlock(arm_smmu_asid_lock)
> >
> > For attach_dev (domain_y), if the device was attached to domain_z
> > //Detach old domain
> > lock(domain_z->devices_lock)
> > Remove master from old domain
> > unlock(domain_z->devices_lock)
>
> At this moment all locks are dropped and the RID's CD entry continues
> to use the ASID.
>
> The racing BTM flow now runs and will do your above:
>
> arm_smmu_mmu_notifier_get()
> arm_smmu_alloc_shared_cd()
> arm_smmu_share_asid():
> arm_smmu_update_ctx_desc_devices() <<- Does nothing due to list_del above
> arm_smmu_tlb_inv_asid() <<-- Woops, we are invalidating an ASID that is still in a CD!
> arm_smmu_write_ctx_desc() <<-- Install a new translation on a PASID's CD
>
> Now the HW can observe two installed CDs using the same ASID but they
> point to different translations. This is illegal.
>
> > Clear CD
>
> Now we remove the RID CD, but it is too late, the PASID CD is already
> installed.
>
> ASID/VMID lifecycle must be strictly contained to ensure the cache
> remains coherent:
>
> 1. All programmed STE/CDs using the ASID/VMID must always point to the
> same translation
>
> 2. All references to a ASID/VMID must be removed from their STE/CDs
> before the ASID is flushed
>
> 3. The ASID/VMID must be flushed before it is assigned to a STE/CD
> with a new translation.
>
> We solve this by requiring that the arm_smmu_asid_lock must be held
> such that the smmu_domains->devices list AND the actual content of the
> CD tables are always observed to be consistent.
>
> Jason
I see, thanks a lot for the detailed explanation.
Maybe this can be added to the change log, so it’s documented somewhere.
Also, I guess this is mainly theoretical, as it requires the detached device to
issue DMA while being detached?
Thanks,
Mostafa
next prev parent reply other threads:[~2024-02-13 13:30 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-25 23:57 [PATCH v4 00/16] Update SMMUv3 to the modern iommu API (part 1/3) Jason Gunthorpe
2024-01-25 23:57 ` [PATCH v4 01/16] iommu/arm-smmu-v3: Make STE programming independent of the callers Jason Gunthorpe
2024-01-26 4:03 ` Michael Shavit
2024-01-29 19:53 ` Moritz Fischer
2024-01-30 22:42 ` Mostafa Saleh
2024-01-30 23:56 ` Jason Gunthorpe
2024-01-31 14:34 ` Mostafa Saleh
2024-01-31 14:40 ` Jason Gunthorpe
2024-01-25 23:57 ` [PATCH v4 02/16] iommu/arm-smmu-v3: Consolidate the STE generation for abort/bypass Jason Gunthorpe
2024-01-31 14:40 ` Mostafa Saleh
2024-01-31 14:47 ` Jason Gunthorpe
2024-02-01 11:32 ` Mostafa Saleh
2024-02-01 13:02 ` Jason Gunthorpe
2024-01-25 23:57 ` [PATCH v4 03/16] iommu/arm-smmu-v3: Move arm_smmu_rmr_install_bypass_ste() Jason Gunthorpe
2024-01-29 15:07 ` Shameerali Kolothum Thodi
2024-01-29 15:43 ` Jason Gunthorpe
2024-01-25 23:57 ` [PATCH v4 04/16] iommu/arm-smmu-v3: Move the STE generation for S1 and S2 domains into functions Jason Gunthorpe
2024-01-31 14:50 ` Mostafa Saleh
2024-01-31 15:05 ` Jason Gunthorpe
2024-01-25 23:57 ` [PATCH v4 05/16] iommu/arm-smmu-v3: Build the whole STE in arm_smmu_make_s2_domain_ste() Jason Gunthorpe
2024-02-01 11:34 ` Mostafa Saleh
2024-01-25 23:57 ` [PATCH v4 06/16] iommu/arm-smmu-v3: Hold arm_smmu_asid_lock during all of attach_dev Jason Gunthorpe
2024-02-01 12:15 ` Mostafa Saleh
2024-02-01 13:24 ` Jason Gunthorpe
2024-02-13 13:30 ` Mostafa Saleh [this message]
2024-01-25 23:57 ` [PATCH v4 07/16] iommu/arm-smmu-v3: Compute the STE only once for each master Jason Gunthorpe
2024-02-01 12:18 ` Mostafa Saleh
2024-01-25 23:57 ` [PATCH v4 08/16] iommu/arm-smmu-v3: Do not change the STE twice during arm_smmu_attach_dev() Jason Gunthorpe
2024-01-25 23:57 ` [PATCH v4 09/16] iommu/arm-smmu-v3: Put writing the context descriptor in the right order Jason Gunthorpe
2024-01-25 23:57 ` [PATCH v4 10/16] iommu/arm-smmu-v3: Pass smmu_domain to arm_enable/disable_ats() Jason Gunthorpe
2024-01-25 23:57 ` [PATCH v4 11/16] iommu/arm-smmu-v3: Remove arm_smmu_master->domain Jason Gunthorpe
2024-01-25 23:57 ` [PATCH v4 12/16] iommu/arm-smmu-v3: Add a global static IDENTITY domain Jason Gunthorpe
2024-01-29 18:11 ` Shameerali Kolothum Thodi
2024-01-29 18:37 ` Jason Gunthorpe
2024-01-30 8:35 ` Shameerali Kolothum Thodi
2024-01-25 23:57 ` [PATCH v4 13/16] iommu/arm-smmu-v3: Add a global static BLOCKED domain Jason Gunthorpe
2024-01-25 23:57 ` [PATCH v4 14/16] iommu/arm-smmu-v3: Use the identity/blocked domain during release Jason Gunthorpe
2024-01-25 23:57 ` [PATCH v4 15/16] iommu/arm-smmu-v3: Pass arm_smmu_domain and arm_smmu_device to finalize Jason Gunthorpe
2024-01-25 23:57 ` [PATCH v4 16/16] iommu/arm-smmu-v3: Convert to domain_alloc_paging() Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Zctu5HOJOPaTNBt-@google.com \
--to=smostafa@google.com \
--cc=iommu@lists.linux.dev \
--cc=jgg@nvidia.com \
--cc=joro@8bytes.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=mdf@kernel.org \
--cc=moritzf@google.com \
--cc=mshavit@google.com \
--cc=nicolinc@nvidia.com \
--cc=patches@lists.linux.dev \
--cc=robin.murphy@arm.com \
--cc=shameerali.kolothum.thodi@huawei.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).