* [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
@ 2024-03-04 23:43 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Continuing the work of part 1 this focuses on the CD, PASID and SVA
components:
- attach_dev failure does not change the HW configuration.
- Full PASID API support including:
- S1/SVA domains attached to PASIDs
- IDENTITY/BLOCKED/S1 attached to RID
- Change of the RID domain while PASIDs are attached
- Streamlined SVA support using the core infrastructure
- Hitless, whenever possible, change between two domains
Making the CD programming work like the new STE programming allows
untangling some of the confusing SVA flows. From there the focus is on
building out the core infrastructure for dealing with PASID and CD
entries, then keeping track of unique SSID's for ATS invalidation.
The ATS ordering is generalized so that the PASID flow can use it and put
into a form where it is fully hitless, whenever possible. Care is taken to
ensure that ATC flushes are present after any change in translation.
Finally we simply kill the entire outdated SVA mmu_notifier implementation
in one shot and switch it over to the newly created generic PASID & CD
code. This avoids the messy and confusing approach of trying to
incrementally untangle this in place. The new code is small and simple
enough this is much better than trying to figure out smaller steps.
Once SVA is resting on the right CD code it is straightforward to make the
PASID interface functionally complete.
It achieves the same goals as the several series from Michael and the S1DSS
series from Nicolin that were trying to improve portions of the API.
This is on github:
https://github.com/jgunthorpe/linux/commits/smmuv3_newapi
v5:
- Rebase on v6.8-rc7 & Will's tree
- Accomdate the SVA rc patch removing the master list iteration
- Move the kfree(to_smmu_domain(domain)) hunk to the right patch
- Move S1DSS get_used hunk to "Allow IDENTITY/BLOCKED to be set while
PASID is used"
v4: https://lore.kernel.org/r/0-v4-e7091cdd9e8d+43b1-smmuv3_newapi_p2_jgg@nvidia.com
- Rebase on v6.8-rc1, adjust to use mm_get_enqcmd_pasid() and eventually
remove all references from ARM. Move the new ARM_SMMU_FEAT_STALL_FORCE
stuff to arm_smmu_make_sva_cd()
- Adjust to use the new shared STE/CD writer logic. Disable some of the
sanity checks for the interior of the series
- Return ERR_PTR from domain_alloc functions
- Move the ATS disablement flow into arm_smmu_attach_prepare()/commit()
which lets all the STE update flows use the same sequence. This is
needed for nesting in part 3
- Put ssid in attach_state
- Replace to_smmu_domain_safe() with to_smmu_domain_devices()
v3: https://lore.kernel.org/r/0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com
- Rebase on the latest part 1
- update comments and commit messages
- Fix error exit in arm_smmu_set_pasid()
- Fix inverted logic for btm_invalidation
- Add missing ATC invalidation on mm release
- Add a big comment explaining that BTM is not enabled and what is
missing to enable it.
v2: https://lore.kernel.org/r/0-v2-16665a652079+5947-smmuv3_newapi_p2_jgg@nvidia.com
- Rebased on iommmufd + Joerg's tree
- Use sid_smmu_domain consistently to refer to the domain attached to the
device (eg the PCIe RID)
- Rework how arm_smmu_attach_*() and callers flow to be more careful
about ordering around ATC invalidation. The ATC must be invalidated
after it is impossible to establish stale entires.
- ATS disable is now entirely part of arm_smmu_attach_dev_ste(), which is
the only STE type that ever disables ATS.
- Remove the 'existing_master_domain' optimization, the code is
functionally fine without it.
- Whitespace, spelling, and checkpatch related items
- Fixed wrong value stored in the xa for the BTM flows
- Use pasid more consistently instead of id
v1: https://lore.kernel.org/r/0-v1-afbb86647bbd+5-smmuv3_newapi_p2_jgg@nvidia.com
Jason Gunthorpe (27):
iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong
PASID
iommu/arm-smmu-v3: Do not ATC invalidate the entire domain
iommu/arm-smmu-v3: Add a type for the CD entry
iommu/arm-smmu-v3: Add an ops indirection to the STE code
iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
iommu/arm-smmu-v3: Consolidate clearing a CD table entry
iommu/arm-smmu-v3: Move the CD generation for S1 domains into a
function
iommu/arm-smmu-v3: Move allocation of the cdtable into
arm_smmu_get_cd_ptr()
iommu/arm-smmu-v3: Allocate the CD table entry in advance
iommu/arm-smmu-v3: Move the CD generation for SVA into a function
iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd()
iommu/arm-smmu-v3: Start building a generic PASID layer
iommu/arm-smmu-v3: Make smmu_domain->devices into an allocated list
iommu/arm-smmu-v3: Make changing domains be hitless for ATS
iommu/arm-smmu-v3: Add ssid to struct arm_smmu_master_domain
iommu/arm-smmu-v3: Keep track of valid CD entries in the cd_table
iommu/arm-smmu-v3: Thread SSID through the arm_smmu_attach_*()
interface
iommu/arm-smmu-v3: Make SVA allocate a normal arm_smmu_domain
iommu/arm-smmu-v3: Keep track of arm_smmu_master_domain for SVA
iommu: Add ops->domain_alloc_sva()
iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain
iommu/arm-smmu-v3: Consolidate freeing the ASID/VMID
iommu/arm-smmu-v3: Move the arm_smmu_asid_xa to per-smmu like vmid
iommu/arm-smmu-v3: Bring back SVA BTM support
iommu/arm-smmu-v3: Allow IDENTITY/BLOCKED to be set while PASID is
used
iommu/arm-smmu-v3: Allow a PASID to be set when RID is
IDENTITY/BLOCKED
iommu/arm-smmu-v3: Allow setting a S1 domain to a PASID
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 639 +++++-----
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 1036 +++++++++++------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 79 +-
drivers/iommu/iommu-sva.c | 4 +-
drivers/iommu/iommu.c | 12 +-
include/linux/iommu.h | 3 +
6 files changed, 1024 insertions(+), 749 deletions(-)
base-commit: 98b23ebb0c84657a135957d727eedebd1280cbbf
--
2.43.2
^ permalink raw reply [flat|nested] 232+ messages in thread
* [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
@ 2024-03-04 23:43 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Continuing the work of part 1 this focuses on the CD, PASID and SVA
components:
- attach_dev failure does not change the HW configuration.
- Full PASID API support including:
- S1/SVA domains attached to PASIDs
- IDENTITY/BLOCKED/S1 attached to RID
- Change of the RID domain while PASIDs are attached
- Streamlined SVA support using the core infrastructure
- Hitless, whenever possible, change between two domains
Making the CD programming work like the new STE programming allows
untangling some of the confusing SVA flows. From there the focus is on
building out the core infrastructure for dealing with PASID and CD
entries, then keeping track of unique SSID's for ATS invalidation.
The ATS ordering is generalized so that the PASID flow can use it and put
into a form where it is fully hitless, whenever possible. Care is taken to
ensure that ATC flushes are present after any change in translation.
Finally we simply kill the entire outdated SVA mmu_notifier implementation
in one shot and switch it over to the newly created generic PASID & CD
code. This avoids the messy and confusing approach of trying to
incrementally untangle this in place. The new code is small and simple
enough this is much better than trying to figure out smaller steps.
Once SVA is resting on the right CD code it is straightforward to make the
PASID interface functionally complete.
It achieves the same goals as the several series from Michael and the S1DSS
series from Nicolin that were trying to improve portions of the API.
This is on github:
https://github.com/jgunthorpe/linux/commits/smmuv3_newapi
v5:
- Rebase on v6.8-rc7 & Will's tree
- Accomdate the SVA rc patch removing the master list iteration
- Move the kfree(to_smmu_domain(domain)) hunk to the right patch
- Move S1DSS get_used hunk to "Allow IDENTITY/BLOCKED to be set while
PASID is used"
v4: https://lore.kernel.org/r/0-v4-e7091cdd9e8d+43b1-smmuv3_newapi_p2_jgg@nvidia.com
- Rebase on v6.8-rc1, adjust to use mm_get_enqcmd_pasid() and eventually
remove all references from ARM. Move the new ARM_SMMU_FEAT_STALL_FORCE
stuff to arm_smmu_make_sva_cd()
- Adjust to use the new shared STE/CD writer logic. Disable some of the
sanity checks for the interior of the series
- Return ERR_PTR from domain_alloc functions
- Move the ATS disablement flow into arm_smmu_attach_prepare()/commit()
which lets all the STE update flows use the same sequence. This is
needed for nesting in part 3
- Put ssid in attach_state
- Replace to_smmu_domain_safe() with to_smmu_domain_devices()
v3: https://lore.kernel.org/r/0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com
- Rebase on the latest part 1
- update comments and commit messages
- Fix error exit in arm_smmu_set_pasid()
- Fix inverted logic for btm_invalidation
- Add missing ATC invalidation on mm release
- Add a big comment explaining that BTM is not enabled and what is
missing to enable it.
v2: https://lore.kernel.org/r/0-v2-16665a652079+5947-smmuv3_newapi_p2_jgg@nvidia.com
- Rebased on iommmufd + Joerg's tree
- Use sid_smmu_domain consistently to refer to the domain attached to the
device (eg the PCIe RID)
- Rework how arm_smmu_attach_*() and callers flow to be more careful
about ordering around ATC invalidation. The ATC must be invalidated
after it is impossible to establish stale entires.
- ATS disable is now entirely part of arm_smmu_attach_dev_ste(), which is
the only STE type that ever disables ATS.
- Remove the 'existing_master_domain' optimization, the code is
functionally fine without it.
- Whitespace, spelling, and checkpatch related items
- Fixed wrong value stored in the xa for the BTM flows
- Use pasid more consistently instead of id
v1: https://lore.kernel.org/r/0-v1-afbb86647bbd+5-smmuv3_newapi_p2_jgg@nvidia.com
Jason Gunthorpe (27):
iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong
PASID
iommu/arm-smmu-v3: Do not ATC invalidate the entire domain
iommu/arm-smmu-v3: Add a type for the CD entry
iommu/arm-smmu-v3: Add an ops indirection to the STE code
iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
iommu/arm-smmu-v3: Consolidate clearing a CD table entry
iommu/arm-smmu-v3: Move the CD generation for S1 domains into a
function
iommu/arm-smmu-v3: Move allocation of the cdtable into
arm_smmu_get_cd_ptr()
iommu/arm-smmu-v3: Allocate the CD table entry in advance
iommu/arm-smmu-v3: Move the CD generation for SVA into a function
iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd()
iommu/arm-smmu-v3: Start building a generic PASID layer
iommu/arm-smmu-v3: Make smmu_domain->devices into an allocated list
iommu/arm-smmu-v3: Make changing domains be hitless for ATS
iommu/arm-smmu-v3: Add ssid to struct arm_smmu_master_domain
iommu/arm-smmu-v3: Keep track of valid CD entries in the cd_table
iommu/arm-smmu-v3: Thread SSID through the arm_smmu_attach_*()
interface
iommu/arm-smmu-v3: Make SVA allocate a normal arm_smmu_domain
iommu/arm-smmu-v3: Keep track of arm_smmu_master_domain for SVA
iommu: Add ops->domain_alloc_sva()
iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain
iommu/arm-smmu-v3: Consolidate freeing the ASID/VMID
iommu/arm-smmu-v3: Move the arm_smmu_asid_xa to per-smmu like vmid
iommu/arm-smmu-v3: Bring back SVA BTM support
iommu/arm-smmu-v3: Allow IDENTITY/BLOCKED to be set while PASID is
used
iommu/arm-smmu-v3: Allow a PASID to be set when RID is
IDENTITY/BLOCKED
iommu/arm-smmu-v3: Allow setting a S1 domain to a PASID
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 639 +++++-----
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 1036 +++++++++++------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 79 +-
drivers/iommu/iommu-sva.c | 4 +-
drivers/iommu/iommu.c | 12 +-
include/linux/iommu.h | 3 +
6 files changed, 1024 insertions(+), 749 deletions(-)
base-commit: 98b23ebb0c84657a135957d727eedebd1280cbbf
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* [PATCH v5 01/27] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:43 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
The SVA code is wired to assume that the SVA is programmed onto the
mm->pasid. The current core code always does this, so it is fine.
Add a check for clarity.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 2610e82c0ecd0d..347c2fdd865c1a 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -581,6 +581,9 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
int ret = 0;
struct mm_struct *mm = domain->mm;
+ if (mm_get_enqcmd_pasid(mm) != id)
+ return -EINVAL;
+
mutex_lock(&sva_lock);
ret = __arm_smmu_sva_bind(dev, id, mm);
mutex_unlock(&sva_lock);
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 01/27] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID
@ 2024-03-04 23:43 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
The SVA code is wired to assume that the SVA is programmed onto the
mm->pasid. The current core code always does this, so it is fine.
Add a check for clarity.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 2610e82c0ecd0d..347c2fdd865c1a 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -581,6 +581,9 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
int ret = 0;
struct mm_struct *mm = domain->mm;
+ if (mm_get_enqcmd_pasid(mm) != id)
+ return -EINVAL;
+
mutex_lock(&sva_lock);
ret = __arm_smmu_sva_bind(dev, id, mm);
mutex_unlock(&sva_lock);
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 02/27] iommu/arm-smmu-v3: Do not ATC invalidate the entire domain
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:43 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
At this point we know which master we are going to change the PCI config
on, this is the only device we need to invalidate. Switch
arm_smmu_atc_inv_domain() for arm_smmu_atc_inv_master().
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index d1bc151a5dff8c..9e9233331c4636 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2447,7 +2447,10 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master,
pdev = to_pci_dev(master->dev);
atomic_inc(&smmu_domain->nr_ats_masters);
- arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, 0, 0);
+ /*
+ * ATC invalidation of PASID 0 causes the entire ATC to be flushed.
+ */
+ arm_smmu_atc_inv_master(master);
if (pci_enable_ats(pdev, stu))
dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu);
}
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 02/27] iommu/arm-smmu-v3: Do not ATC invalidate the entire domain
@ 2024-03-04 23:43 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
At this point we know which master we are going to change the PCI config
on, this is the only device we need to invalidate. Switch
arm_smmu_atc_inv_domain() for arm_smmu_atc_inv_master().
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index d1bc151a5dff8c..9e9233331c4636 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2447,7 +2447,10 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master,
pdev = to_pci_dev(master->dev);
atomic_inc(&smmu_domain->nr_ats_masters);
- arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, 0, 0);
+ /*
+ * ATC invalidation of PASID 0 causes the entire ATC to be flushed.
+ */
+ arm_smmu_atc_inv_master(master);
if (pci_enable_ats(pdev, stu))
dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu);
}
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 03/27] iommu/arm-smmu-v3: Add a type for the CD entry
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:43 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Instead of passing a naked __le16 * around to represent a CD table entry
wrap it in a "struct arm_smmu_cd" with an array of the correct size. This
makes it much clearer which functions will comprise the "CD API".
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 +++++++++++---------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 ++++++-
2 files changed, 17 insertions(+), 10 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 9e9233331c4636..c60b067c1f553e 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1219,7 +1219,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
WRITE_ONCE(*dst, cpu_to_le64(val));
}
-static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
+static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
+ u32 ssid)
{
__le64 *l1ptr;
unsigned int idx;
@@ -1228,7 +1229,8 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
- return cd_table->cdtab + ssid * CTXDESC_CD_DWORDS;
+ return (struct arm_smmu_cd *)(cd_table->cdtab +
+ ssid * CTXDESC_CD_DWORDS);
idx = ssid >> CTXDESC_SPLIT;
l1_desc = &cd_table->l1_desc[idx];
@@ -1242,7 +1244,7 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
arm_smmu_sync_cd(master, ssid, false);
}
idx = ssid & (CTXDESC_L2_ENTRIES - 1);
- return l1_desc->l2ptr + idx * CTXDESC_CD_DWORDS;
+ return &l1_desc->l2ptr[idx];
}
int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
@@ -1261,7 +1263,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
*/
u64 val;
bool cd_live;
- __le64 *cdptr;
+ struct arm_smmu_cd *cdptr;
struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
struct arm_smmu_device *smmu = master->smmu;
@@ -1272,7 +1274,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
if (!cdptr)
return -ENOMEM;
- val = le64_to_cpu(cdptr[0]);
+ val = le64_to_cpu(cdptr->data[0]);
cd_live = !!(val & CTXDESC_CD_0_V);
if (!cd) { /* (5) */
@@ -1289,9 +1291,9 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
* this substream's traffic
*/
} else { /* (1) and (2) */
- cdptr[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
- cdptr[2] = 0;
- cdptr[3] = cpu_to_le64(cd->mair);
+ cdptr->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
+ cdptr->data[2] = 0;
+ cdptr->data[3] = cpu_to_le64(cd->mair);
/*
* STE may be live, and the SMMU might read dwords of this CD in any
@@ -1323,7 +1325,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
* field within an aligned 64-bit span of a structure can be altered
* without first making the structure invalid.
*/
- WRITE_ONCE(cdptr[0], cpu_to_le64(val));
+ WRITE_ONCE(cdptr->data[0], cpu_to_le64(val));
arm_smmu_sync_cd(master, ssid, true);
return 0;
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 23baf117e7e4b5..7078ed569fd4d3 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -282,6 +282,11 @@ struct arm_smmu_ste {
#define CTXDESC_L1_DESC_L2PTR_MASK GENMASK_ULL(51, 12)
#define CTXDESC_CD_DWORDS 8
+
+struct arm_smmu_cd {
+ __le64 data[CTXDESC_CD_DWORDS];
+};
+
#define CTXDESC_CD_0_TCR_T0SZ GENMASK_ULL(5, 0)
#define CTXDESC_CD_0_TCR_TG0 GENMASK_ULL(7, 6)
#define CTXDESC_CD_0_TCR_IRGN0 GENMASK_ULL(9, 8)
@@ -591,7 +596,7 @@ struct arm_smmu_ctx_desc {
};
struct arm_smmu_l1_ctx_desc {
- __le64 *l2ptr;
+ struct arm_smmu_cd *l2ptr;
dma_addr_t l2ptr_dma;
};
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 03/27] iommu/arm-smmu-v3: Add a type for the CD entry
@ 2024-03-04 23:43 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Instead of passing a naked __le16 * around to represent a CD table entry
wrap it in a "struct arm_smmu_cd" with an array of the correct size. This
makes it much clearer which functions will comprise the "CD API".
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 +++++++++++---------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 ++++++-
2 files changed, 17 insertions(+), 10 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 9e9233331c4636..c60b067c1f553e 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1219,7 +1219,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
WRITE_ONCE(*dst, cpu_to_le64(val));
}
-static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
+static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
+ u32 ssid)
{
__le64 *l1ptr;
unsigned int idx;
@@ -1228,7 +1229,8 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
- return cd_table->cdtab + ssid * CTXDESC_CD_DWORDS;
+ return (struct arm_smmu_cd *)(cd_table->cdtab +
+ ssid * CTXDESC_CD_DWORDS);
idx = ssid >> CTXDESC_SPLIT;
l1_desc = &cd_table->l1_desc[idx];
@@ -1242,7 +1244,7 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
arm_smmu_sync_cd(master, ssid, false);
}
idx = ssid & (CTXDESC_L2_ENTRIES - 1);
- return l1_desc->l2ptr + idx * CTXDESC_CD_DWORDS;
+ return &l1_desc->l2ptr[idx];
}
int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
@@ -1261,7 +1263,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
*/
u64 val;
bool cd_live;
- __le64 *cdptr;
+ struct arm_smmu_cd *cdptr;
struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
struct arm_smmu_device *smmu = master->smmu;
@@ -1272,7 +1274,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
if (!cdptr)
return -ENOMEM;
- val = le64_to_cpu(cdptr[0]);
+ val = le64_to_cpu(cdptr->data[0]);
cd_live = !!(val & CTXDESC_CD_0_V);
if (!cd) { /* (5) */
@@ -1289,9 +1291,9 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
* this substream's traffic
*/
} else { /* (1) and (2) */
- cdptr[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
- cdptr[2] = 0;
- cdptr[3] = cpu_to_le64(cd->mair);
+ cdptr->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
+ cdptr->data[2] = 0;
+ cdptr->data[3] = cpu_to_le64(cd->mair);
/*
* STE may be live, and the SMMU might read dwords of this CD in any
@@ -1323,7 +1325,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
* field within an aligned 64-bit span of a structure can be altered
* without first making the structure invalid.
*/
- WRITE_ONCE(cdptr[0], cpu_to_le64(val));
+ WRITE_ONCE(cdptr->data[0], cpu_to_le64(val));
arm_smmu_sync_cd(master, ssid, true);
return 0;
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 23baf117e7e4b5..7078ed569fd4d3 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -282,6 +282,11 @@ struct arm_smmu_ste {
#define CTXDESC_L1_DESC_L2PTR_MASK GENMASK_ULL(51, 12)
#define CTXDESC_CD_DWORDS 8
+
+struct arm_smmu_cd {
+ __le64 data[CTXDESC_CD_DWORDS];
+};
+
#define CTXDESC_CD_0_TCR_T0SZ GENMASK_ULL(5, 0)
#define CTXDESC_CD_0_TCR_TG0 GENMASK_ULL(7, 6)
#define CTXDESC_CD_0_TCR_IRGN0 GENMASK_ULL(9, 8)
@@ -591,7 +596,7 @@ struct arm_smmu_ctx_desc {
};
struct arm_smmu_l1_ctx_desc {
- __le64 *l2ptr;
+ struct arm_smmu_cd *l2ptr;
dma_addr_t l2ptr_dma;
};
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:43 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Prepare to put the CD code into the same mechanism. Add an ops indirection
around all the STE specific code and make the worker functions independent
of the entry content being processed.
get_used and sync ops are provided to hook the correct code.
Signed-off-by: Michael Shavit <mshavit@google.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 172 ++++++++++++--------
1 file changed, 104 insertions(+), 68 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index c60b067c1f553e..b7f947e36f596f 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -48,8 +48,20 @@ enum arm_smmu_msi_index {
ARM_SMMU_MAX_MSIS,
};
-static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu,
- ioasid_t sid);
+struct arm_smmu_entry_writer_ops;
+struct arm_smmu_entry_writer {
+ const struct arm_smmu_entry_writer_ops *ops;
+ struct arm_smmu_master *master;
+};
+
+struct arm_smmu_entry_writer_ops {
+ unsigned int num_entry_qwords;
+ __le64 v_bit;
+ void (*get_used)(const __le64 *entry, __le64 *used);
+ void (*sync)(struct arm_smmu_entry_writer *writer);
+};
+
+#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
[EVTQ_MSI_INDEX] = {
@@ -982,43 +994,42 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid)
* would be nice if this was complete according to the spec, but minimally it
* has to capture the bits this driver uses.
*/
-static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent,
- struct arm_smmu_ste *used_bits)
+static void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits)
{
- unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent->data[0]));
+ unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent[0]));
- used_bits->data[0] = cpu_to_le64(STRTAB_STE_0_V);
- if (!(ent->data[0] & cpu_to_le64(STRTAB_STE_0_V)))
+ used_bits[0] = cpu_to_le64(STRTAB_STE_0_V);
+ if (!(ent[0] & cpu_to_le64(STRTAB_STE_0_V)))
return;
- used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_CFG);
+ used_bits[0] |= cpu_to_le64(STRTAB_STE_0_CFG);
/* S1 translates */
if (cfg & BIT(0)) {
- used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT |
- STRTAB_STE_0_S1CTXPTR_MASK |
- STRTAB_STE_0_S1CDMAX);
- used_bits->data[1] |=
+ used_bits[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT |
+ STRTAB_STE_0_S1CTXPTR_MASK |
+ STRTAB_STE_0_S1CDMAX);
+ used_bits[1] |=
cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR |
STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH |
STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW |
STRTAB_STE_1_EATS);
- used_bits->data[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);
+ used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);
}
/* S2 translates */
if (cfg & BIT(1)) {
- used_bits->data[1] |=
+ used_bits[1] |=
cpu_to_le64(STRTAB_STE_1_EATS | STRTAB_STE_1_SHCFG);
- used_bits->data[2] |=
+ used_bits[2] |=
cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR |
STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI |
STRTAB_STE_2_S2PTW | STRTAB_STE_2_S2R);
- used_bits->data[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK);
+ used_bits[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK);
}
if (cfg == STRTAB_STE_0_CFG_BYPASS)
- used_bits->data[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG);
+ used_bits[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG);
}
/*
@@ -1027,57 +1038,55 @@ static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent,
* unused_update is an intermediate value of entry that has unused bits set to
* their new values.
*/
-static u8 arm_smmu_entry_qword_diff(const struct arm_smmu_ste *entry,
- const struct arm_smmu_ste *target,
- struct arm_smmu_ste *unused_update)
+static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
+ const __le64 *entry, const __le64 *target,
+ __le64 *unused_update)
{
- struct arm_smmu_ste target_used = {};
- struct arm_smmu_ste cur_used = {};
+ __le64 target_used[NUM_ENTRY_QWORDS] = {};
+ __le64 cur_used[NUM_ENTRY_QWORDS] = {};
u8 used_qword_diff = 0;
unsigned int i;
- arm_smmu_get_ste_used(entry, &cur_used);
- arm_smmu_get_ste_used(target, &target_used);
+ writer->ops->get_used(entry, cur_used);
+ writer->ops->get_used(target, target_used);
- for (i = 0; i != ARRAY_SIZE(target_used.data); i++) {
+ for (i = 0; i != writer->ops->num_entry_qwords; i++) {
/*
* Check that masks are up to date, the make functions are not
* allowed to set a bit to 1 if the used function doesn't say it
* is used.
*/
- WARN_ON_ONCE(target->data[i] & ~target_used.data[i]);
+ WARN_ON_ONCE(target[i] & ~target_used[i]);
/* Bits can change because they are not currently being used */
- unused_update->data[i] = (entry->data[i] & cur_used.data[i]) |
- (target->data[i] & ~cur_used.data[i]);
+ unused_update[i] = (entry[i] & cur_used[i]) |
+ (target[i] & ~cur_used[i]);
/*
* Each bit indicates that a used bit in a qword needs to be
* changed after unused_update is applied.
*/
- if ((unused_update->data[i] & target_used.data[i]) !=
- target->data[i])
+ if ((unused_update[i] & target_used[i]) != target[i])
used_qword_diff |= 1 << i;
}
return used_qword_diff;
}
-static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid,
- struct arm_smmu_ste *entry,
- const struct arm_smmu_ste *target, unsigned int start,
+static bool entry_set(struct arm_smmu_entry_writer *writer, __le64 *entry,
+ const __le64 *target, unsigned int start,
unsigned int len)
{
bool changed = false;
unsigned int i;
for (i = start; len != 0; len--, i++) {
- if (entry->data[i] != target->data[i]) {
- WRITE_ONCE(entry->data[i], target->data[i]);
+ if (entry[i] != target[i]) {
+ WRITE_ONCE(entry[i], target[i]);
changed = true;
}
}
if (changed)
- arm_smmu_sync_ste_for_sid(smmu, sid);
+ writer->ops->sync(writer);
return changed;
}
@@ -1107,17 +1116,15 @@ static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid,
* V=0 process. This relies on the IGNORED behavior described in the
* specification.
*/
-static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
- struct arm_smmu_ste *entry,
- const struct arm_smmu_ste *target)
+static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer,
+ __le64 *entry, const __le64 *target)
{
- unsigned int num_entry_qwords = ARRAY_SIZE(target->data);
- struct arm_smmu_device *smmu = master->smmu;
- struct arm_smmu_ste unused_update;
+ unsigned int num_entry_qwords = writer->ops->num_entry_qwords;
+ __le64 unused_update[NUM_ENTRY_QWORDS];
u8 used_qword_diff;
used_qword_diff =
- arm_smmu_entry_qword_diff(entry, target, &unused_update);
+ arm_smmu_entry_qword_diff(writer, entry, target, unused_update);
if (hweight8(used_qword_diff) == 1) {
/*
* Only one qword needs its used bits to be changed. This is a
@@ -1133,22 +1140,21 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
* writing it in the next step anyways. This can save a sync
* when the only change is in that qword.
*/
- unused_update.data[critical_qword_index] =
- entry->data[critical_qword_index];
- entry_set(smmu, sid, entry, &unused_update, 0, num_entry_qwords);
- entry_set(smmu, sid, entry, target, critical_qword_index, 1);
- entry_set(smmu, sid, entry, target, 0, num_entry_qwords);
+ unused_update[critical_qword_index] =
+ entry[critical_qword_index];
+ entry_set(writer, entry, unused_update, 0, num_entry_qwords);
+ entry_set(writer, entry, target, critical_qword_index, 1);
+ entry_set(writer, entry, target, 0, num_entry_qwords);
} else if (used_qword_diff) {
/*
* At least two qwords need their inuse bits to be changed. This
* requires a breaking update, zero the V bit, write all qwords
* but 0, then set qword 0
*/
- unused_update.data[0] = entry->data[0] &
- cpu_to_le64(~STRTAB_STE_0_V);
- entry_set(smmu, sid, entry, &unused_update, 0, 1);
- entry_set(smmu, sid, entry, target, 1, num_entry_qwords - 1);
- entry_set(smmu, sid, entry, target, 0, 1);
+ unused_update[0] = entry[0] & (~writer->ops->v_bit);
+ entry_set(writer, entry, unused_update, 0, 1);
+ entry_set(writer, entry, target, 1, num_entry_qwords - 1);
+ entry_set(writer, entry, target, 0, 1);
} else {
/*
* No inuse bit changed. Sanity check that all unused bits are 0
@@ -1156,18 +1162,7 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
* compute_qword_diff().
*/
WARN_ON_ONCE(
- entry_set(smmu, sid, entry, target, 0, num_entry_qwords));
- }
-
- /* It's likely that we'll want to use the new STE soon */
- if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) {
- struct arm_smmu_cmdq_ent
- prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG,
- .prefetch = {
- .sid = sid,
- } };
-
- arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
+ entry_set(writer, entry, target, 0, num_entry_qwords));
}
}
@@ -1440,17 +1435,58 @@ arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc)
WRITE_ONCE(*dst, cpu_to_le64(val));
}
-static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu, u32 sid)
+struct arm_smmu_ste_writer {
+ struct arm_smmu_entry_writer writer;
+ u32 sid;
+};
+
+static void arm_smmu_ste_writer_sync_entry(struct arm_smmu_entry_writer *writer)
{
+ struct arm_smmu_ste_writer *ste_writer =
+ container_of(writer, struct arm_smmu_ste_writer, writer);
struct arm_smmu_cmdq_ent cmd = {
.opcode = CMDQ_OP_CFGI_STE,
.cfgi = {
- .sid = sid,
+ .sid = ste_writer->sid,
.leaf = true,
},
};
- arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
+ arm_smmu_cmdq_issue_cmd_with_sync(writer->master->smmu, &cmd);
+}
+
+static const struct arm_smmu_entry_writer_ops arm_smmu_ste_writer_ops = {
+ .sync = arm_smmu_ste_writer_sync_entry,
+ .get_used = arm_smmu_get_ste_used,
+ .v_bit = cpu_to_le64(STRTAB_STE_0_V),
+ .num_entry_qwords = sizeof(struct arm_smmu_ste) / sizeof(u64),
+};
+
+static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
+ struct arm_smmu_ste *ste,
+ const struct arm_smmu_ste *target)
+{
+ struct arm_smmu_device *smmu = master->smmu;
+ struct arm_smmu_ste_writer ste_writer = {
+ .writer = {
+ .ops = &arm_smmu_ste_writer_ops,
+ .master = master,
+ },
+ .sid = sid,
+ };
+
+ arm_smmu_write_entry(&ste_writer.writer, ste->data, target->data);
+
+ /* It's likely that we'll want to use the new STE soon */
+ if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) {
+ struct arm_smmu_cmdq_ent
+ prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG,
+ .prefetch = {
+ .sid = sid,
+ } };
+
+ arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
+ }
}
static void arm_smmu_make_abort_ste(struct arm_smmu_ste *target)
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code
@ 2024-03-04 23:43 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Prepare to put the CD code into the same mechanism. Add an ops indirection
around all the STE specific code and make the worker functions independent
of the entry content being processed.
get_used and sync ops are provided to hook the correct code.
Signed-off-by: Michael Shavit <mshavit@google.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 172 ++++++++++++--------
1 file changed, 104 insertions(+), 68 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index c60b067c1f553e..b7f947e36f596f 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -48,8 +48,20 @@ enum arm_smmu_msi_index {
ARM_SMMU_MAX_MSIS,
};
-static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu,
- ioasid_t sid);
+struct arm_smmu_entry_writer_ops;
+struct arm_smmu_entry_writer {
+ const struct arm_smmu_entry_writer_ops *ops;
+ struct arm_smmu_master *master;
+};
+
+struct arm_smmu_entry_writer_ops {
+ unsigned int num_entry_qwords;
+ __le64 v_bit;
+ void (*get_used)(const __le64 *entry, __le64 *used);
+ void (*sync)(struct arm_smmu_entry_writer *writer);
+};
+
+#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
[EVTQ_MSI_INDEX] = {
@@ -982,43 +994,42 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid)
* would be nice if this was complete according to the spec, but minimally it
* has to capture the bits this driver uses.
*/
-static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent,
- struct arm_smmu_ste *used_bits)
+static void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits)
{
- unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent->data[0]));
+ unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent[0]));
- used_bits->data[0] = cpu_to_le64(STRTAB_STE_0_V);
- if (!(ent->data[0] & cpu_to_le64(STRTAB_STE_0_V)))
+ used_bits[0] = cpu_to_le64(STRTAB_STE_0_V);
+ if (!(ent[0] & cpu_to_le64(STRTAB_STE_0_V)))
return;
- used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_CFG);
+ used_bits[0] |= cpu_to_le64(STRTAB_STE_0_CFG);
/* S1 translates */
if (cfg & BIT(0)) {
- used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT |
- STRTAB_STE_0_S1CTXPTR_MASK |
- STRTAB_STE_0_S1CDMAX);
- used_bits->data[1] |=
+ used_bits[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT |
+ STRTAB_STE_0_S1CTXPTR_MASK |
+ STRTAB_STE_0_S1CDMAX);
+ used_bits[1] |=
cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR |
STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH |
STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW |
STRTAB_STE_1_EATS);
- used_bits->data[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);
+ used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);
}
/* S2 translates */
if (cfg & BIT(1)) {
- used_bits->data[1] |=
+ used_bits[1] |=
cpu_to_le64(STRTAB_STE_1_EATS | STRTAB_STE_1_SHCFG);
- used_bits->data[2] |=
+ used_bits[2] |=
cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR |
STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI |
STRTAB_STE_2_S2PTW | STRTAB_STE_2_S2R);
- used_bits->data[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK);
+ used_bits[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK);
}
if (cfg == STRTAB_STE_0_CFG_BYPASS)
- used_bits->data[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG);
+ used_bits[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG);
}
/*
@@ -1027,57 +1038,55 @@ static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent,
* unused_update is an intermediate value of entry that has unused bits set to
* their new values.
*/
-static u8 arm_smmu_entry_qword_diff(const struct arm_smmu_ste *entry,
- const struct arm_smmu_ste *target,
- struct arm_smmu_ste *unused_update)
+static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
+ const __le64 *entry, const __le64 *target,
+ __le64 *unused_update)
{
- struct arm_smmu_ste target_used = {};
- struct arm_smmu_ste cur_used = {};
+ __le64 target_used[NUM_ENTRY_QWORDS] = {};
+ __le64 cur_used[NUM_ENTRY_QWORDS] = {};
u8 used_qword_diff = 0;
unsigned int i;
- arm_smmu_get_ste_used(entry, &cur_used);
- arm_smmu_get_ste_used(target, &target_used);
+ writer->ops->get_used(entry, cur_used);
+ writer->ops->get_used(target, target_used);
- for (i = 0; i != ARRAY_SIZE(target_used.data); i++) {
+ for (i = 0; i != writer->ops->num_entry_qwords; i++) {
/*
* Check that masks are up to date, the make functions are not
* allowed to set a bit to 1 if the used function doesn't say it
* is used.
*/
- WARN_ON_ONCE(target->data[i] & ~target_used.data[i]);
+ WARN_ON_ONCE(target[i] & ~target_used[i]);
/* Bits can change because they are not currently being used */
- unused_update->data[i] = (entry->data[i] & cur_used.data[i]) |
- (target->data[i] & ~cur_used.data[i]);
+ unused_update[i] = (entry[i] & cur_used[i]) |
+ (target[i] & ~cur_used[i]);
/*
* Each bit indicates that a used bit in a qword needs to be
* changed after unused_update is applied.
*/
- if ((unused_update->data[i] & target_used.data[i]) !=
- target->data[i])
+ if ((unused_update[i] & target_used[i]) != target[i])
used_qword_diff |= 1 << i;
}
return used_qword_diff;
}
-static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid,
- struct arm_smmu_ste *entry,
- const struct arm_smmu_ste *target, unsigned int start,
+static bool entry_set(struct arm_smmu_entry_writer *writer, __le64 *entry,
+ const __le64 *target, unsigned int start,
unsigned int len)
{
bool changed = false;
unsigned int i;
for (i = start; len != 0; len--, i++) {
- if (entry->data[i] != target->data[i]) {
- WRITE_ONCE(entry->data[i], target->data[i]);
+ if (entry[i] != target[i]) {
+ WRITE_ONCE(entry[i], target[i]);
changed = true;
}
}
if (changed)
- arm_smmu_sync_ste_for_sid(smmu, sid);
+ writer->ops->sync(writer);
return changed;
}
@@ -1107,17 +1116,15 @@ static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid,
* V=0 process. This relies on the IGNORED behavior described in the
* specification.
*/
-static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
- struct arm_smmu_ste *entry,
- const struct arm_smmu_ste *target)
+static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer,
+ __le64 *entry, const __le64 *target)
{
- unsigned int num_entry_qwords = ARRAY_SIZE(target->data);
- struct arm_smmu_device *smmu = master->smmu;
- struct arm_smmu_ste unused_update;
+ unsigned int num_entry_qwords = writer->ops->num_entry_qwords;
+ __le64 unused_update[NUM_ENTRY_QWORDS];
u8 used_qword_diff;
used_qword_diff =
- arm_smmu_entry_qword_diff(entry, target, &unused_update);
+ arm_smmu_entry_qword_diff(writer, entry, target, unused_update);
if (hweight8(used_qword_diff) == 1) {
/*
* Only one qword needs its used bits to be changed. This is a
@@ -1133,22 +1140,21 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
* writing it in the next step anyways. This can save a sync
* when the only change is in that qword.
*/
- unused_update.data[critical_qword_index] =
- entry->data[critical_qword_index];
- entry_set(smmu, sid, entry, &unused_update, 0, num_entry_qwords);
- entry_set(smmu, sid, entry, target, critical_qword_index, 1);
- entry_set(smmu, sid, entry, target, 0, num_entry_qwords);
+ unused_update[critical_qword_index] =
+ entry[critical_qword_index];
+ entry_set(writer, entry, unused_update, 0, num_entry_qwords);
+ entry_set(writer, entry, target, critical_qword_index, 1);
+ entry_set(writer, entry, target, 0, num_entry_qwords);
} else if (used_qword_diff) {
/*
* At least two qwords need their inuse bits to be changed. This
* requires a breaking update, zero the V bit, write all qwords
* but 0, then set qword 0
*/
- unused_update.data[0] = entry->data[0] &
- cpu_to_le64(~STRTAB_STE_0_V);
- entry_set(smmu, sid, entry, &unused_update, 0, 1);
- entry_set(smmu, sid, entry, target, 1, num_entry_qwords - 1);
- entry_set(smmu, sid, entry, target, 0, 1);
+ unused_update[0] = entry[0] & (~writer->ops->v_bit);
+ entry_set(writer, entry, unused_update, 0, 1);
+ entry_set(writer, entry, target, 1, num_entry_qwords - 1);
+ entry_set(writer, entry, target, 0, 1);
} else {
/*
* No inuse bit changed. Sanity check that all unused bits are 0
@@ -1156,18 +1162,7 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
* compute_qword_diff().
*/
WARN_ON_ONCE(
- entry_set(smmu, sid, entry, target, 0, num_entry_qwords));
- }
-
- /* It's likely that we'll want to use the new STE soon */
- if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) {
- struct arm_smmu_cmdq_ent
- prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG,
- .prefetch = {
- .sid = sid,
- } };
-
- arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
+ entry_set(writer, entry, target, 0, num_entry_qwords));
}
}
@@ -1440,17 +1435,58 @@ arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc)
WRITE_ONCE(*dst, cpu_to_le64(val));
}
-static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu, u32 sid)
+struct arm_smmu_ste_writer {
+ struct arm_smmu_entry_writer writer;
+ u32 sid;
+};
+
+static void arm_smmu_ste_writer_sync_entry(struct arm_smmu_entry_writer *writer)
{
+ struct arm_smmu_ste_writer *ste_writer =
+ container_of(writer, struct arm_smmu_ste_writer, writer);
struct arm_smmu_cmdq_ent cmd = {
.opcode = CMDQ_OP_CFGI_STE,
.cfgi = {
- .sid = sid,
+ .sid = ste_writer->sid,
.leaf = true,
},
};
- arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
+ arm_smmu_cmdq_issue_cmd_with_sync(writer->master->smmu, &cmd);
+}
+
+static const struct arm_smmu_entry_writer_ops arm_smmu_ste_writer_ops = {
+ .sync = arm_smmu_ste_writer_sync_entry,
+ .get_used = arm_smmu_get_ste_used,
+ .v_bit = cpu_to_le64(STRTAB_STE_0_V),
+ .num_entry_qwords = sizeof(struct arm_smmu_ste) / sizeof(u64),
+};
+
+static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
+ struct arm_smmu_ste *ste,
+ const struct arm_smmu_ste *target)
+{
+ struct arm_smmu_device *smmu = master->smmu;
+ struct arm_smmu_ste_writer ste_writer = {
+ .writer = {
+ .ops = &arm_smmu_ste_writer_ops,
+ .master = master,
+ },
+ .sid = sid,
+ };
+
+ arm_smmu_write_entry(&ste_writer.writer, ste->data, target->data);
+
+ /* It's likely that we'll want to use the new STE soon */
+ if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) {
+ struct arm_smmu_cmdq_ent
+ prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG,
+ .prefetch = {
+ .sid = sid,
+ } };
+
+ arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
+ }
}
static void arm_smmu_make_abort_ste(struct arm_smmu_ste *target)
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:43 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
CD table entries and STE's have the same essential programming sequence,
just with different types and sizes.
Have arm_smmu_write_ctx_desc() generate a target CD and call
arm_smmu_write_entry() to do the programming. Due to the way the
target CD is generated by modifying the existing CD this alone is not
enough for the CD callers to be freed of the ordering requirements.
The following patches will make the rest of the CD flow mirror the STE
flow with precise CD contents generated in all cases.
Currently the logic can't ensure that the CD always conforms to the used
requirements until all the CD generation is moved to the new method. Add a
temporary no_used_check to disable the assertions.
Signed-off-by: Michael Shavit <mshavit@google.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 101 ++++++++++++++------
1 file changed, 74 insertions(+), 27 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index b7f947e36f596f..237fd6d92c880b 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -57,11 +57,14 @@ struct arm_smmu_entry_writer {
struct arm_smmu_entry_writer_ops {
unsigned int num_entry_qwords;
__le64 v_bit;
+ bool no_used_check;
void (*get_used)(const __le64 *entry, __le64 *used);
void (*sync)(struct arm_smmu_entry_writer *writer);
};
-#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
+#define NUM_ENTRY_QWORDS \
+ (max(sizeof(struct arm_smmu_ste), sizeof(struct arm_smmu_cd)) / \
+ sizeof(u64))
static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
[EVTQ_MSI_INDEX] = {
@@ -1056,7 +1059,8 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
* allowed to set a bit to 1 if the used function doesn't say it
* is used.
*/
- WARN_ON_ONCE(target[i] & ~target_used[i]);
+ if (!writer->ops->no_used_check)
+ WARN_ON_ONCE(target[i] & ~target_used[i]);
/* Bits can change because they are not currently being used */
unused_update[i] = (entry[i] & cur_used[i]) |
@@ -1065,7 +1069,8 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
* Each bit indicates that a used bit in a qword needs to be
* changed after unused_update is applied.
*/
- if ((unused_update[i] & target_used[i]) != target[i])
+ if ((unused_update[i] & target_used[i]) !=
+ (target[i] & target_used[i]))
used_qword_diff |= 1 << i;
}
return used_qword_diff;
@@ -1161,8 +1166,11 @@ static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer,
* in the entry. The target was already sanity checked by
* compute_qword_diff().
*/
- WARN_ON_ONCE(
- entry_set(writer, entry, target, 0, num_entry_qwords));
+ if (writer->ops->no_used_check)
+ entry_set(writer, entry, target, 0, num_entry_qwords);
+ else
+ WARN_ON_ONCE(entry_set(writer, entry, target, 0,
+ num_entry_qwords));
}
}
@@ -1242,6 +1250,59 @@ static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
return &l1_desc->l2ptr[idx];
}
+struct arm_smmu_cd_writer {
+ struct arm_smmu_entry_writer writer;
+ unsigned int ssid;
+};
+
+static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
+{
+ used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
+ if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
+ return;
+ memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
+
+ /* EPD0 means T0SZ/TG0/IR0/OR0/SH0/TTB0 are IGNORED */
+ if (ent[0] & cpu_to_le64(CTXDESC_CD_0_TCR_EPD0)) {
+ used_bits[0] &= ~cpu_to_le64(
+ CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 |
+ CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 |
+ CTXDESC_CD_0_TCR_SH0);
+ used_bits[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK);
+ }
+}
+
+static void arm_smmu_cd_writer_sync_entry(struct arm_smmu_entry_writer *writer)
+{
+ struct arm_smmu_cd_writer *cd_writer =
+ container_of(writer, struct arm_smmu_cd_writer, writer);
+
+ arm_smmu_sync_cd(writer->master, cd_writer->ssid, true);
+}
+
+static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
+ .sync = arm_smmu_cd_writer_sync_entry,
+ .get_used = arm_smmu_get_cd_used,
+ .v_bit = cpu_to_le64(CTXDESC_CD_0_V),
+ .no_used_check = true,
+ .num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
+};
+
+static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
+ struct arm_smmu_cd *cdptr,
+ const struct arm_smmu_cd *target)
+{
+ struct arm_smmu_cd_writer cd_writer = {
+ .writer = {
+ .ops = &arm_smmu_cd_writer_ops,
+ .master = master,
+ },
+ .ssid = ssid,
+ };
+
+ arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
+}
+
int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
struct arm_smmu_ctx_desc *cd)
{
@@ -1258,17 +1319,20 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
*/
u64 val;
bool cd_live;
- struct arm_smmu_cd *cdptr;
+ struct arm_smmu_cd target;
+ struct arm_smmu_cd *cdptr = ⌖
+ struct arm_smmu_cd *cd_table_entry;
struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
struct arm_smmu_device *smmu = master->smmu;
if (WARN_ON(ssid >= (1 << cd_table->s1cdmax)))
return -E2BIG;
- cdptr = arm_smmu_get_cd_ptr(master, ssid);
- if (!cdptr)
+ cd_table_entry = arm_smmu_get_cd_ptr(master, ssid);
+ if (!cd_table_entry)
return -ENOMEM;
+ target = *cd_table_entry;
val = le64_to_cpu(cdptr->data[0]);
cd_live = !!(val & CTXDESC_CD_0_V);
@@ -1290,13 +1354,6 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
cdptr->data[2] = 0;
cdptr->data[3] = cpu_to_le64(cd->mair);
- /*
- * STE may be live, and the SMMU might read dwords of this CD in any
- * order. Ensure that it observes valid values before reading
- * V=1.
- */
- arm_smmu_sync_cd(master, ssid, true);
-
val = cd->tcr |
#ifdef __BIG_ENDIAN
CTXDESC_CD_0_ENDI |
@@ -1310,18 +1367,8 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
if (cd_table->stall_enabled)
val |= CTXDESC_CD_0_S;
}
-
- /*
- * The SMMU accesses 64-bit values atomically. See IHI0070Ca 3.21.3
- * "Configuration structures and configuration invalidation completion"
- *
- * The size of single-copy atomic reads made by the SMMU is
- * IMPLEMENTATION DEFINED but must be at least 64 bits. Any single
- * field within an aligned 64-bit span of a structure can be altered
- * without first making the structure invalid.
- */
- WRITE_ONCE(cdptr->data[0], cpu_to_le64(val));
- arm_smmu_sync_cd(master, ssid, true);
+ cdptr->data[0] = cpu_to_le64(val);
+ arm_smmu_write_cd_entry(master, ssid, cd_table_entry, &target);
return 0;
}
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
@ 2024-03-04 23:43 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
CD table entries and STE's have the same essential programming sequence,
just with different types and sizes.
Have arm_smmu_write_ctx_desc() generate a target CD and call
arm_smmu_write_entry() to do the programming. Due to the way the
target CD is generated by modifying the existing CD this alone is not
enough for the CD callers to be freed of the ordering requirements.
The following patches will make the rest of the CD flow mirror the STE
flow with precise CD contents generated in all cases.
Currently the logic can't ensure that the CD always conforms to the used
requirements until all the CD generation is moved to the new method. Add a
temporary no_used_check to disable the assertions.
Signed-off-by: Michael Shavit <mshavit@google.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 101 ++++++++++++++------
1 file changed, 74 insertions(+), 27 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index b7f947e36f596f..237fd6d92c880b 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -57,11 +57,14 @@ struct arm_smmu_entry_writer {
struct arm_smmu_entry_writer_ops {
unsigned int num_entry_qwords;
__le64 v_bit;
+ bool no_used_check;
void (*get_used)(const __le64 *entry, __le64 *used);
void (*sync)(struct arm_smmu_entry_writer *writer);
};
-#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
+#define NUM_ENTRY_QWORDS \
+ (max(sizeof(struct arm_smmu_ste), sizeof(struct arm_smmu_cd)) / \
+ sizeof(u64))
static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
[EVTQ_MSI_INDEX] = {
@@ -1056,7 +1059,8 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
* allowed to set a bit to 1 if the used function doesn't say it
* is used.
*/
- WARN_ON_ONCE(target[i] & ~target_used[i]);
+ if (!writer->ops->no_used_check)
+ WARN_ON_ONCE(target[i] & ~target_used[i]);
/* Bits can change because they are not currently being used */
unused_update[i] = (entry[i] & cur_used[i]) |
@@ -1065,7 +1069,8 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
* Each bit indicates that a used bit in a qword needs to be
* changed after unused_update is applied.
*/
- if ((unused_update[i] & target_used[i]) != target[i])
+ if ((unused_update[i] & target_used[i]) !=
+ (target[i] & target_used[i]))
used_qword_diff |= 1 << i;
}
return used_qword_diff;
@@ -1161,8 +1166,11 @@ static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer,
* in the entry. The target was already sanity checked by
* compute_qword_diff().
*/
- WARN_ON_ONCE(
- entry_set(writer, entry, target, 0, num_entry_qwords));
+ if (writer->ops->no_used_check)
+ entry_set(writer, entry, target, 0, num_entry_qwords);
+ else
+ WARN_ON_ONCE(entry_set(writer, entry, target, 0,
+ num_entry_qwords));
}
}
@@ -1242,6 +1250,59 @@ static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
return &l1_desc->l2ptr[idx];
}
+struct arm_smmu_cd_writer {
+ struct arm_smmu_entry_writer writer;
+ unsigned int ssid;
+};
+
+static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
+{
+ used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
+ if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
+ return;
+ memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
+
+ /* EPD0 means T0SZ/TG0/IR0/OR0/SH0/TTB0 are IGNORED */
+ if (ent[0] & cpu_to_le64(CTXDESC_CD_0_TCR_EPD0)) {
+ used_bits[0] &= ~cpu_to_le64(
+ CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 |
+ CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 |
+ CTXDESC_CD_0_TCR_SH0);
+ used_bits[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK);
+ }
+}
+
+static void arm_smmu_cd_writer_sync_entry(struct arm_smmu_entry_writer *writer)
+{
+ struct arm_smmu_cd_writer *cd_writer =
+ container_of(writer, struct arm_smmu_cd_writer, writer);
+
+ arm_smmu_sync_cd(writer->master, cd_writer->ssid, true);
+}
+
+static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
+ .sync = arm_smmu_cd_writer_sync_entry,
+ .get_used = arm_smmu_get_cd_used,
+ .v_bit = cpu_to_le64(CTXDESC_CD_0_V),
+ .no_used_check = true,
+ .num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
+};
+
+static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
+ struct arm_smmu_cd *cdptr,
+ const struct arm_smmu_cd *target)
+{
+ struct arm_smmu_cd_writer cd_writer = {
+ .writer = {
+ .ops = &arm_smmu_cd_writer_ops,
+ .master = master,
+ },
+ .ssid = ssid,
+ };
+
+ arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
+}
+
int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
struct arm_smmu_ctx_desc *cd)
{
@@ -1258,17 +1319,20 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
*/
u64 val;
bool cd_live;
- struct arm_smmu_cd *cdptr;
+ struct arm_smmu_cd target;
+ struct arm_smmu_cd *cdptr = ⌖
+ struct arm_smmu_cd *cd_table_entry;
struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
struct arm_smmu_device *smmu = master->smmu;
if (WARN_ON(ssid >= (1 << cd_table->s1cdmax)))
return -E2BIG;
- cdptr = arm_smmu_get_cd_ptr(master, ssid);
- if (!cdptr)
+ cd_table_entry = arm_smmu_get_cd_ptr(master, ssid);
+ if (!cd_table_entry)
return -ENOMEM;
+ target = *cd_table_entry;
val = le64_to_cpu(cdptr->data[0]);
cd_live = !!(val & CTXDESC_CD_0_V);
@@ -1290,13 +1354,6 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
cdptr->data[2] = 0;
cdptr->data[3] = cpu_to_le64(cd->mair);
- /*
- * STE may be live, and the SMMU might read dwords of this CD in any
- * order. Ensure that it observes valid values before reading
- * V=1.
- */
- arm_smmu_sync_cd(master, ssid, true);
-
val = cd->tcr |
#ifdef __BIG_ENDIAN
CTXDESC_CD_0_ENDI |
@@ -1310,18 +1367,8 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
if (cd_table->stall_enabled)
val |= CTXDESC_CD_0_S;
}
-
- /*
- * The SMMU accesses 64-bit values atomically. See IHI0070Ca 3.21.3
- * "Configuration structures and configuration invalidation completion"
- *
- * The size of single-copy atomic reads made by the SMMU is
- * IMPLEMENTATION DEFINED but must be at least 64 bits. Any single
- * field within an aligned 64-bit span of a structure can be altered
- * without first making the structure invalid.
- */
- WRITE_ONCE(cdptr->data[0], cpu_to_le64(val));
- arm_smmu_sync_cd(master, ssid, true);
+ cdptr->data[0] = cpu_to_le64(val);
+ arm_smmu_write_cd_entry(master, ssid, cd_table_entry, &target);
return 0;
}
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 06/27] iommu/arm-smmu-v3: Consolidate clearing a CD table entry
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:43 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
A cleared entry is all 0's. Make arm_smmu_clear_cd() do this sequence.
If we are clearing an entry and for some reason it is not already
allocated in the CD table then something has gone wrong.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 ++++++++++++++-----
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 ++
3 files changed, 18 insertions(+), 6 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 347c2fdd865c1a..bb9bb6fd7914ce 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -558,7 +558,7 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
mutex_lock(&sva_lock);
- arm_smmu_write_ctx_desc(master, id, NULL);
+ arm_smmu_clear_cd(master, id);
list_for_each_entry(t, &master->bonds, list) {
if (t->mm == mm) {
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 237fd6d92c880b..3fb4a1523d1d3f 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1303,6 +1303,19 @@ static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
}
+void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
+{
+ struct arm_smmu_cd target = {};
+ struct arm_smmu_cd *cdptr;
+
+ if (!master->cd_table.cdtab)
+ return;
+ cdptr = arm_smmu_get_cd_ptr(master, ssid);
+ if (WARN_ON(!cdptr))
+ return;
+ arm_smmu_write_cd_entry(master, ssid, cdptr, &target);
+}
+
int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
struct arm_smmu_ctx_desc *cd)
{
@@ -2702,9 +2715,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
case ARM_SMMU_DOMAIN_S2:
arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
arm_smmu_install_ste_for_dev(master, &target);
- if (master->cd_table.cdtab)
- arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID,
- NULL);
+ arm_smmu_clear_cd(master, IOMMU_NO_PASID);
break;
}
@@ -2752,8 +2763,7 @@ static int arm_smmu_attach_dev_ste(struct device *dev,
* arm_smmu_domain->devices to avoid races updating the same context
* descriptor from arm_smmu_share_asid().
*/
- if (master->cd_table.cdtab)
- arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, NULL);
+ arm_smmu_clear_cd(master, IOMMU_NO_PASID);
return 0;
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 7078ed569fd4d3..87a7b57f566fbc 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -749,6 +749,8 @@ extern struct xarray arm_smmu_asid_xa;
extern struct mutex arm_smmu_asid_lock;
extern struct arm_smmu_ctx_desc quiet_cd;
+void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
+
int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid,
struct arm_smmu_ctx_desc *cd);
void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 06/27] iommu/arm-smmu-v3: Consolidate clearing a CD table entry
@ 2024-03-04 23:43 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
A cleared entry is all 0's. Make arm_smmu_clear_cd() do this sequence.
If we are clearing an entry and for some reason it is not already
allocated in the CD table then something has gone wrong.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 ++++++++++++++-----
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 ++
3 files changed, 18 insertions(+), 6 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 347c2fdd865c1a..bb9bb6fd7914ce 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -558,7 +558,7 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
mutex_lock(&sva_lock);
- arm_smmu_write_ctx_desc(master, id, NULL);
+ arm_smmu_clear_cd(master, id);
list_for_each_entry(t, &master->bonds, list) {
if (t->mm == mm) {
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 237fd6d92c880b..3fb4a1523d1d3f 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1303,6 +1303,19 @@ static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
}
+void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
+{
+ struct arm_smmu_cd target = {};
+ struct arm_smmu_cd *cdptr;
+
+ if (!master->cd_table.cdtab)
+ return;
+ cdptr = arm_smmu_get_cd_ptr(master, ssid);
+ if (WARN_ON(!cdptr))
+ return;
+ arm_smmu_write_cd_entry(master, ssid, cdptr, &target);
+}
+
int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
struct arm_smmu_ctx_desc *cd)
{
@@ -2702,9 +2715,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
case ARM_SMMU_DOMAIN_S2:
arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
arm_smmu_install_ste_for_dev(master, &target);
- if (master->cd_table.cdtab)
- arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID,
- NULL);
+ arm_smmu_clear_cd(master, IOMMU_NO_PASID);
break;
}
@@ -2752,8 +2763,7 @@ static int arm_smmu_attach_dev_ste(struct device *dev,
* arm_smmu_domain->devices to avoid races updating the same context
* descriptor from arm_smmu_share_asid().
*/
- if (master->cd_table.cdtab)
- arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, NULL);
+ arm_smmu_clear_cd(master, IOMMU_NO_PASID);
return 0;
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 7078ed569fd4d3..87a7b57f566fbc 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -749,6 +749,8 @@ extern struct xarray arm_smmu_asid_xa;
extern struct mutex arm_smmu_asid_lock;
extern struct arm_smmu_ctx_desc quiet_cd;
+void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
+
int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid,
struct arm_smmu_ctx_desc *cd);
void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 07/27] iommu/arm-smmu-v3: Move the CD generation for S1 domains into a function
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:43 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Introduce arm_smmu_make_s1_cd() to build the CD from the paging S1 domain,
and reorganize all the places programming S1 domain CD table entries to
call it.
Split arm_smmu_update_s1_domain_cd_entry() from
arm_smmu_update_ctx_desc_devices() so that the S1 path has its own call
chain separate from the unrelated SVA path.
arm_smmu_update_s1_domain_cd_entry() only works on S1 domains
attached to RIDs and refreshes all their CDs.
Remove the forced clear of the CD during S1 domain attach,
arm_smmu_write_cd_entry() will do this automatically if necessary.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 25 +++++++-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 60 +++++++++++++------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 +++
3 files changed, 75 insertions(+), 18 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index bb9bb6fd7914ce..6acc65f6d00a71 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -54,6 +54,29 @@ static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
}
+static void
+arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
+{
+ struct arm_smmu_master *master;
+ struct arm_smmu_cd target_cd;
+ unsigned long flags;
+
+ spin_lock_irqsave(&smmu_domain->devices_lock, flags);
+ list_for_each_entry(master, &smmu_domain->devices, domain_head) {
+ struct arm_smmu_cd *cdptr;
+
+ /* S1 domains only support RID attachment right now */
+ cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
+ if (WARN_ON(!cdptr))
+ continue;
+
+ arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
+ arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
+ &target_cd);
+ }
+ spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
+}
+
/*
* Check if the CPU ASID is available on the SMMU side. If a private context
* descriptor is using it, try to replace it.
@@ -97,7 +120,7 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid)
* be some overlap between use of both ASIDs, until we invalidate the
* TLB.
*/
- arm_smmu_update_ctx_desc_devices(smmu_domain, IOMMU_NO_PASID, cd);
+ arm_smmu_update_s1_domain_cd_entry(smmu_domain);
/* Invalidate TLB entries previously associated with that context */
arm_smmu_tlb_inv_asid(smmu, asid);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 3fb4a1523d1d3f..e25dbb982feeee 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1222,8 +1222,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
WRITE_ONCE(*dst, cpu_to_le64(val));
}
-static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
- u32 ssid)
+struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
+ u32 ssid)
{
__le64 *l1ptr;
unsigned int idx;
@@ -1288,9 +1288,9 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
.num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
};
-static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
- struct arm_smmu_cd *cdptr,
- const struct arm_smmu_cd *target)
+void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
+ struct arm_smmu_cd *cdptr,
+ const struct arm_smmu_cd *target)
{
struct arm_smmu_cd_writer cd_writer = {
.writer = {
@@ -1303,6 +1303,32 @@ static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
}
+void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
+ struct arm_smmu_master *master,
+ struct arm_smmu_domain *smmu_domain)
+{
+ struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
+
+ memset(target, 0, sizeof(*target));
+
+ target->data[0] = cpu_to_le64(
+ cd->tcr |
+#ifdef __BIG_ENDIAN
+ CTXDESC_CD_0_ENDI |
+#endif
+ CTXDESC_CD_0_V |
+ CTXDESC_CD_0_AA64 |
+ (master->stall_enabled ? CTXDESC_CD_0_S : 0) |
+ CTXDESC_CD_0_R |
+ CTXDESC_CD_0_A |
+ CTXDESC_CD_0_ASET |
+ FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid)
+ );
+
+ target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
+ target->data[3] = cpu_to_le64(cd->mair);
+}
+
void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
{
struct arm_smmu_cd target = {};
@@ -2689,29 +2715,29 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
switch (smmu_domain->stage) {
- case ARM_SMMU_DOMAIN_S1:
+ case ARM_SMMU_DOMAIN_S1: {
+ struct arm_smmu_cd target_cd;
+ struct arm_smmu_cd *cdptr;
+
if (!master->cd_table.cdtab) {
ret = arm_smmu_alloc_cd_tables(master);
if (ret)
goto out_list_del;
- } else {
- /*
- * arm_smmu_write_ctx_desc() relies on the entry being
- * invalid to work, clear any existing entry.
- */
- ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID,
- NULL);
- if (ret)
- goto out_list_del;
}
- ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, &smmu_domain->cd);
- if (ret)
+ cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
+ if (!cdptr) {
+ ret = -ENOMEM;
goto out_list_del;
+ }
+ arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
+ arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
+ &target_cd);
arm_smmu_make_cdtable_ste(&target, master);
arm_smmu_install_ste_for_dev(master, &target);
break;
+ }
case ARM_SMMU_DOMAIN_S2:
arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
arm_smmu_install_ste_for_dev(master, &target);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 87a7b57f566fbc..d32da11058aab6 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -750,6 +750,14 @@ extern struct mutex arm_smmu_asid_lock;
extern struct arm_smmu_ctx_desc quiet_cd;
void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
+struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
+ u32 ssid);
+void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
+ struct arm_smmu_master *master,
+ struct arm_smmu_domain *smmu_domain);
+void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
+ struct arm_smmu_cd *cdptr,
+ const struct arm_smmu_cd *target);
int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid,
struct arm_smmu_ctx_desc *cd);
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 07/27] iommu/arm-smmu-v3: Move the CD generation for S1 domains into a function
@ 2024-03-04 23:43 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Introduce arm_smmu_make_s1_cd() to build the CD from the paging S1 domain,
and reorganize all the places programming S1 domain CD table entries to
call it.
Split arm_smmu_update_s1_domain_cd_entry() from
arm_smmu_update_ctx_desc_devices() so that the S1 path has its own call
chain separate from the unrelated SVA path.
arm_smmu_update_s1_domain_cd_entry() only works on S1 domains
attached to RIDs and refreshes all their CDs.
Remove the forced clear of the CD during S1 domain attach,
arm_smmu_write_cd_entry() will do this automatically if necessary.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 25 +++++++-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 60 +++++++++++++------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 +++
3 files changed, 75 insertions(+), 18 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index bb9bb6fd7914ce..6acc65f6d00a71 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -54,6 +54,29 @@ static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
}
+static void
+arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
+{
+ struct arm_smmu_master *master;
+ struct arm_smmu_cd target_cd;
+ unsigned long flags;
+
+ spin_lock_irqsave(&smmu_domain->devices_lock, flags);
+ list_for_each_entry(master, &smmu_domain->devices, domain_head) {
+ struct arm_smmu_cd *cdptr;
+
+ /* S1 domains only support RID attachment right now */
+ cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
+ if (WARN_ON(!cdptr))
+ continue;
+
+ arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
+ arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
+ &target_cd);
+ }
+ spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
+}
+
/*
* Check if the CPU ASID is available on the SMMU side. If a private context
* descriptor is using it, try to replace it.
@@ -97,7 +120,7 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid)
* be some overlap between use of both ASIDs, until we invalidate the
* TLB.
*/
- arm_smmu_update_ctx_desc_devices(smmu_domain, IOMMU_NO_PASID, cd);
+ arm_smmu_update_s1_domain_cd_entry(smmu_domain);
/* Invalidate TLB entries previously associated with that context */
arm_smmu_tlb_inv_asid(smmu, asid);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 3fb4a1523d1d3f..e25dbb982feeee 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1222,8 +1222,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
WRITE_ONCE(*dst, cpu_to_le64(val));
}
-static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
- u32 ssid)
+struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
+ u32 ssid)
{
__le64 *l1ptr;
unsigned int idx;
@@ -1288,9 +1288,9 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
.num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
};
-static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
- struct arm_smmu_cd *cdptr,
- const struct arm_smmu_cd *target)
+void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
+ struct arm_smmu_cd *cdptr,
+ const struct arm_smmu_cd *target)
{
struct arm_smmu_cd_writer cd_writer = {
.writer = {
@@ -1303,6 +1303,32 @@ static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
}
+void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
+ struct arm_smmu_master *master,
+ struct arm_smmu_domain *smmu_domain)
+{
+ struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
+
+ memset(target, 0, sizeof(*target));
+
+ target->data[0] = cpu_to_le64(
+ cd->tcr |
+#ifdef __BIG_ENDIAN
+ CTXDESC_CD_0_ENDI |
+#endif
+ CTXDESC_CD_0_V |
+ CTXDESC_CD_0_AA64 |
+ (master->stall_enabled ? CTXDESC_CD_0_S : 0) |
+ CTXDESC_CD_0_R |
+ CTXDESC_CD_0_A |
+ CTXDESC_CD_0_ASET |
+ FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid)
+ );
+
+ target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
+ target->data[3] = cpu_to_le64(cd->mair);
+}
+
void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
{
struct arm_smmu_cd target = {};
@@ -2689,29 +2715,29 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
switch (smmu_domain->stage) {
- case ARM_SMMU_DOMAIN_S1:
+ case ARM_SMMU_DOMAIN_S1: {
+ struct arm_smmu_cd target_cd;
+ struct arm_smmu_cd *cdptr;
+
if (!master->cd_table.cdtab) {
ret = arm_smmu_alloc_cd_tables(master);
if (ret)
goto out_list_del;
- } else {
- /*
- * arm_smmu_write_ctx_desc() relies on the entry being
- * invalid to work, clear any existing entry.
- */
- ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID,
- NULL);
- if (ret)
- goto out_list_del;
}
- ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, &smmu_domain->cd);
- if (ret)
+ cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
+ if (!cdptr) {
+ ret = -ENOMEM;
goto out_list_del;
+ }
+ arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
+ arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
+ &target_cd);
arm_smmu_make_cdtable_ste(&target, master);
arm_smmu_install_ste_for_dev(master, &target);
break;
+ }
case ARM_SMMU_DOMAIN_S2:
arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
arm_smmu_install_ste_for_dev(master, &target);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 87a7b57f566fbc..d32da11058aab6 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -750,6 +750,14 @@ extern struct mutex arm_smmu_asid_lock;
extern struct arm_smmu_ctx_desc quiet_cd;
void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
+struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
+ u32 ssid);
+void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
+ struct arm_smmu_master *master,
+ struct arm_smmu_domain *smmu_domain);
+void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
+ struct arm_smmu_cd *cdptr,
+ const struct arm_smmu_cd *target);
int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid,
struct arm_smmu_ctx_desc *cd);
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 08/27] iommu/arm-smmu-v3: Move allocation of the cdtable into arm_smmu_get_cd_ptr()
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:43 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
No reason to force callers to do two steps. Make arm_smmu_get_cd_ptr()
able to return an entry in all cases except OOM.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index e25dbb982feeee..2dd6cb17112e98 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -106,6 +106,7 @@ static struct arm_smmu_option_prop arm_smmu_options[] = {
static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
struct arm_smmu_device *smmu);
+static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master);
static void parse_driver_options(struct arm_smmu_device *smmu)
{
@@ -1231,6 +1232,11 @@ struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
struct arm_smmu_device *smmu = master->smmu;
struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
+ if (!master->cd_table.cdtab) {
+ if (arm_smmu_alloc_cd_tables(master))
+ return NULL;
+ }
+
if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
return (struct arm_smmu_cd *)(cd_table->cdtab +
ssid * CTXDESC_CD_DWORDS);
@@ -2719,12 +2725,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
struct arm_smmu_cd target_cd;
struct arm_smmu_cd *cdptr;
- if (!master->cd_table.cdtab) {
- ret = arm_smmu_alloc_cd_tables(master);
- if (ret)
- goto out_list_del;
- }
-
cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
if (!cdptr) {
ret = -ENOMEM;
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 08/27] iommu/arm-smmu-v3: Move allocation of the cdtable into arm_smmu_get_cd_ptr()
@ 2024-03-04 23:43 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
No reason to force callers to do two steps. Make arm_smmu_get_cd_ptr()
able to return an entry in all cases except OOM.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index e25dbb982feeee..2dd6cb17112e98 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -106,6 +106,7 @@ static struct arm_smmu_option_prop arm_smmu_options[] = {
static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
struct arm_smmu_device *smmu);
+static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master);
static void parse_driver_options(struct arm_smmu_device *smmu)
{
@@ -1231,6 +1232,11 @@ struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
struct arm_smmu_device *smmu = master->smmu;
struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
+ if (!master->cd_table.cdtab) {
+ if (arm_smmu_alloc_cd_tables(master))
+ return NULL;
+ }
+
if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
return (struct arm_smmu_cd *)(cd_table->cdtab +
ssid * CTXDESC_CD_DWORDS);
@@ -2719,12 +2725,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
struct arm_smmu_cd target_cd;
struct arm_smmu_cd *cdptr;
- if (!master->cd_table.cdtab) {
- ret = arm_smmu_alloc_cd_tables(master);
- if (ret)
- goto out_list_del;
- }
-
cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
if (!cdptr) {
ret = -ENOMEM;
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 09/27] iommu/arm-smmu-v3: Allocate the CD table entry in advance
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:43 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Avoid arm_smmu_attach_dev() having to undo the changes to the
smmu_domain->devices list, acquire the cdptr earlier so we don't need to
handle that error.
Now there is a clear break in arm_smmu_attach_dev() where all the
prep-work has been done non-disruptively and we commit to making the HW
change, which cannot fail.
This completes transforming arm_smmu_attach_dev() so that it does not
disturb the HW if it fails.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 24 +++++++--------------
1 file changed, 8 insertions(+), 16 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 2dd6cb17112e98..39081d828a2132 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2676,6 +2676,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
struct arm_smmu_device *smmu;
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
struct arm_smmu_master *master;
+ struct arm_smmu_cd *cdptr;
if (!fwspec)
return -ENOENT;
@@ -2704,6 +2705,12 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
if (ret)
return ret;
+ if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
+ cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
+ if (!cdptr)
+ return -ENOMEM;
+ }
+
/*
* Prevent arm_smmu_share_asid() from trying to change the ASID
* of either the old or new domain while we are working on it.
@@ -2723,13 +2730,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
switch (smmu_domain->stage) {
case ARM_SMMU_DOMAIN_S1: {
struct arm_smmu_cd target_cd;
- struct arm_smmu_cd *cdptr;
-
- cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
- if (!cdptr) {
- ret = -ENOMEM;
- goto out_list_del;
- }
arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
@@ -2746,16 +2746,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
}
arm_smmu_enable_ats(master, smmu_domain);
- goto out_unlock;
-
-out_list_del:
- spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- list_del_init(&master->domain_head);
- spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
-
-out_unlock:
mutex_unlock(&arm_smmu_asid_lock);
- return ret;
+ return 0;
}
static int arm_smmu_attach_dev_ste(struct device *dev,
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 09/27] iommu/arm-smmu-v3: Allocate the CD table entry in advance
@ 2024-03-04 23:43 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Avoid arm_smmu_attach_dev() having to undo the changes to the
smmu_domain->devices list, acquire the cdptr earlier so we don't need to
handle that error.
Now there is a clear break in arm_smmu_attach_dev() where all the
prep-work has been done non-disruptively and we commit to making the HW
change, which cannot fail.
This completes transforming arm_smmu_attach_dev() so that it does not
disturb the HW if it fails.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 24 +++++++--------------
1 file changed, 8 insertions(+), 16 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 2dd6cb17112e98..39081d828a2132 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2676,6 +2676,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
struct arm_smmu_device *smmu;
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
struct arm_smmu_master *master;
+ struct arm_smmu_cd *cdptr;
if (!fwspec)
return -ENOENT;
@@ -2704,6 +2705,12 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
if (ret)
return ret;
+ if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
+ cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
+ if (!cdptr)
+ return -ENOMEM;
+ }
+
/*
* Prevent arm_smmu_share_asid() from trying to change the ASID
* of either the old or new domain while we are working on it.
@@ -2723,13 +2730,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
switch (smmu_domain->stage) {
case ARM_SMMU_DOMAIN_S1: {
struct arm_smmu_cd target_cd;
- struct arm_smmu_cd *cdptr;
-
- cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
- if (!cdptr) {
- ret = -ENOMEM;
- goto out_list_del;
- }
arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
@@ -2746,16 +2746,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
}
arm_smmu_enable_ats(master, smmu_domain);
- goto out_unlock;
-
-out_list_del:
- spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- list_del_init(&master->domain_head);
- spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
-
-out_unlock:
mutex_unlock(&arm_smmu_asid_lock);
- return ret;
+ return 0;
}
static int arm_smmu_attach_dev_ste(struct device *dev,
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 10/27] iommu/arm-smmu-v3: Move the CD generation for SVA into a function
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:43 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Pull all the calculations for building the CD table entry for a mmu_struct
into arm_smmu_make_sva_cd().
Call it in the two places installing the SVA CD table entry.
Open code the last caller of arm_smmu_update_ctx_desc_devices() and remove
the function.
Remove arm_smmu_write_ctx_desc() since all callers are gone.
Remove quiet_cd since all users are gone, arm_smmu_make_sva_cd() creates
the same value.
The behavior of quiet_cd changes slightly, the old implementation edited
the CD in place to set CTXDESC_CD_0_TCR_EPD0 assuming it was a SVA CD
entry. This version generates a full CD entry with a 0 TTB0 and relies on
arm_smmu_write_cd_entry() to install it hitlessly.
Remove no_used_check since this was the flow that could result in an used
bit inconsistent CD.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 156 +++++++++++-------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 95 +----------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 -
3 files changed, 105 insertions(+), 151 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 6acc65f6d00a71..159350a78d4cf9 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -35,25 +35,6 @@ struct arm_smmu_bond {
static DEFINE_MUTEX(sva_lock);
-/*
- * Write the CD to the CD tables for all masters that this domain is attached
- * to. Note that this is only used to update existing CD entries in the target
- * CD table, for which it's assumed that arm_smmu_write_ctx_desc can't fail.
- */
-static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain,
- int ssid,
- struct arm_smmu_ctx_desc *cd)
-{
- struct arm_smmu_master *master;
- unsigned long flags;
-
- spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- list_for_each_entry(master, &smmu_domain->devices, domain_head) {
- arm_smmu_write_ctx_desc(master, ssid, cd);
- }
- spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
-}
-
static void
arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
{
@@ -129,11 +110,86 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid)
return NULL;
}
+static u64 page_size_to_cd(void)
+{
+ static_assert(PAGE_SIZE == SZ_4K || PAGE_SIZE == SZ_16K ||
+ PAGE_SIZE == SZ_64K);
+ if (PAGE_SIZE == SZ_64K)
+ return ARM_LPAE_TCR_TG0_64K;
+ if (PAGE_SIZE == SZ_16K)
+ return ARM_LPAE_TCR_TG0_16K;
+ return ARM_LPAE_TCR_TG0_4K;
+}
+
+static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target,
+ struct arm_smmu_master *master,
+ struct mm_struct *mm, u16 asid)
+{
+ u64 par;
+
+ memset(target, 0, sizeof(*target));
+
+ par = cpuid_feature_extract_unsigned_field(
+ read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1),
+ ID_AA64MMFR0_EL1_PARANGE_SHIFT);
+
+ target->data[0] = cpu_to_le64(
+ CTXDESC_CD_0_TCR_EPD1 |
+#ifdef __BIG_ENDIAN
+ CTXDESC_CD_0_ENDI |
+#endif
+ CTXDESC_CD_0_V |
+ FIELD_PREP(CTXDESC_CD_0_TCR_IPS, par) |
+ CTXDESC_CD_0_AA64 |
+ (master->stall_enabled ? CTXDESC_CD_0_S : 0) |
+ CTXDESC_CD_0_R |
+ CTXDESC_CD_0_A |
+ CTXDESC_CD_0_ASET |
+ FIELD_PREP(CTXDESC_CD_0_ASID, asid));
+
+ /*
+ * If no MM is passed then this creates a SVA entry that faults
+ * everything. arm_smmu_write_cd_entry() can hitlessly go between these
+ * two entries types since TTB0 is ignored by HW when EPD0 is set.
+ */
+ if (mm) {
+ target->data[0] |= cpu_to_le64(
+ FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ,
+ 64ULL - vabits_actual) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_TG0, page_size_to_cd()) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0,
+ ARM_LPAE_TCR_RGN_WBWA) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0,
+ ARM_LPAE_TCR_RGN_WBWA) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS));
+
+ target->data[1] = cpu_to_le64(virt_to_phys(mm->pgd) &
+ CTXDESC_CD_1_TTB0_MASK);
+ } else {
+ target->data[0] |= cpu_to_le64(CTXDESC_CD_0_TCR_EPD0);
+
+ /*
+ * Disable stall and immediately generate an abort if stall
+ * disable is permitted. This speeds up cleanup for an unclean
+ * exit if the device is still doing a lot of DMA.
+ */
+ if (master->stall_enabled &&
+ !(master->smmu->features & ARM_SMMU_FEAT_STALL_FORCE))
+ target->data[0] &=
+ cpu_to_le64(~(CTXDESC_CD_0_S | CTXDESC_CD_0_R));
+ }
+
+ /*
+ * MAIR value is pretty much constant and global, so we can just get it
+ * from the current CPU register
+ */
+ target->data[3] = cpu_to_le64(read_sysreg(mair_el1));
+}
+
static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm)
{
u16 asid;
int err = 0;
- u64 tcr, par, reg;
struct arm_smmu_ctx_desc *cd;
struct arm_smmu_ctx_desc *ret = NULL;
@@ -167,39 +223,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm)
if (err)
goto out_free_asid;
- tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - vabits_actual) |
- FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) |
- FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) |
- FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS) |
- CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64;
-
- switch (PAGE_SIZE) {
- case SZ_4K:
- tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_4K);
- break;
- case SZ_16K:
- tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_16K);
- break;
- case SZ_64K:
- tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_64K);
- break;
- default:
- WARN_ON(1);
- err = -EINVAL;
- goto out_free_asid;
- }
-
- reg = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);
- par = cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR0_EL1_PARANGE_SHIFT);
- tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_IPS, par);
-
- cd->ttbr = virt_to_phys(mm->pgd);
- cd->tcr = tcr;
- /*
- * MAIR value is pretty much constant and global, so we can just get it
- * from the current CPU register
- */
- cd->mair = read_sysreg(mair_el1);
cd->asid = asid;
cd->mm = mm;
@@ -277,6 +300,8 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
{
struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn);
struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
+ struct arm_smmu_master *master;
+ unsigned long flags;
mutex_lock(&sva_lock);
if (smmu_mn->cleared) {
@@ -288,8 +313,19 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
* DMA may still be running. Keep the cd valid to avoid C_BAD_CD events,
* but disable translation.
*/
- arm_smmu_update_ctx_desc_devices(smmu_domain, mm_get_enqcmd_pasid(mm),
- &quiet_cd);
+ spin_lock_irqsave(&smmu_domain->devices_lock, flags);
+ list_for_each_entry(master, &smmu_domain->devices, domain_head) {
+ struct arm_smmu_cd target;
+ struct arm_smmu_cd *cdptr;
+
+ cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm));
+ if (WARN_ON(!cdptr))
+ continue;
+ arm_smmu_make_sva_cd(&target, master, NULL, smmu_mn->cd->asid);
+ arm_smmu_write_cd_entry(master, mm_get_enqcmd_pasid(mm), cdptr,
+ &target);
+ }
+ spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid);
arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0);
@@ -384,6 +420,8 @@ static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid,
struct mm_struct *mm)
{
int ret;
+ struct arm_smmu_cd target;
+ struct arm_smmu_cd *cdptr;
struct arm_smmu_bond *bond;
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
@@ -410,9 +448,13 @@ static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid,
goto err_free_bond;
}
- ret = arm_smmu_write_ctx_desc(master, pasid, bond->smmu_mn->cd);
- if (ret)
+ cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm));
+ if (!cdptr) {
+ ret = -ENOMEM;
goto err_put_notifier;
+ }
+ arm_smmu_make_sva_cd(&target, master, mm, bond->smmu_mn->cd->asid);
+ arm_smmu_write_cd_entry(master, pasid, cdptr, &target);
list_add(&bond->list, &master->bonds);
return 0;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 39081d828a2132..50d17e3ce0a956 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -57,7 +57,6 @@ struct arm_smmu_entry_writer {
struct arm_smmu_entry_writer_ops {
unsigned int num_entry_qwords;
__le64 v_bit;
- bool no_used_check;
void (*get_used)(const __le64 *entry, __le64 *used);
void (*sync)(struct arm_smmu_entry_writer *writer);
};
@@ -92,12 +91,6 @@ struct arm_smmu_option_prop {
DEFINE_XARRAY_ALLOC1(arm_smmu_asid_xa);
DEFINE_MUTEX(arm_smmu_asid_lock);
-/*
- * Special value used by SVA when a process dies, to quiesce a CD without
- * disabling it.
- */
-struct arm_smmu_ctx_desc quiet_cd = { 0 };
-
static struct arm_smmu_option_prop arm_smmu_options[] = {
{ ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" },
{ ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"},
@@ -1060,8 +1053,7 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
* allowed to set a bit to 1 if the used function doesn't say it
* is used.
*/
- if (!writer->ops->no_used_check)
- WARN_ON_ONCE(target[i] & ~target_used[i]);
+ WARN_ON_ONCE(target[i] & ~target_used[i]);
/* Bits can change because they are not currently being used */
unused_update[i] = (entry[i] & cur_used[i]) |
@@ -1070,8 +1062,7 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
* Each bit indicates that a used bit in a qword needs to be
* changed after unused_update is applied.
*/
- if ((unused_update[i] & target_used[i]) !=
- (target[i] & target_used[i]))
+ if ((unused_update[i] & target_used[i]) != target[i])
used_qword_diff |= 1 << i;
}
return used_qword_diff;
@@ -1167,11 +1158,8 @@ static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer,
* in the entry. The target was already sanity checked by
* compute_qword_diff().
*/
- if (writer->ops->no_used_check)
- entry_set(writer, entry, target, 0, num_entry_qwords);
- else
- WARN_ON_ONCE(entry_set(writer, entry, target, 0,
- num_entry_qwords));
+ WARN_ON_ONCE(
+ entry_set(writer, entry, target, 0, num_entry_qwords));
}
}
@@ -1219,7 +1207,7 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
u64 val = (l1_desc->l2ptr_dma & CTXDESC_L1_DESC_L2PTR_MASK) |
CTXDESC_L1_DESC_V;
- /* See comment in arm_smmu_write_ctx_desc() */
+ /* The HW has 64 bit atomicity with stores to the L2 CD table */
WRITE_ONCE(*dst, cpu_to_le64(val));
}
@@ -1290,7 +1278,6 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
.sync = arm_smmu_cd_writer_sync_entry,
.get_used = arm_smmu_get_cd_used,
.v_bit = cpu_to_le64(CTXDESC_CD_0_V),
- .no_used_check = true,
.num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
};
@@ -1348,75 +1335,6 @@ void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
arm_smmu_write_cd_entry(master, ssid, cdptr, &target);
}
-int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
- struct arm_smmu_ctx_desc *cd)
-{
- /*
- * This function handles the following cases:
- *
- * (1) Install primary CD, for normal DMA traffic (SSID = IOMMU_NO_PASID = 0).
- * (2) Install a secondary CD, for SID+SSID traffic.
- * (3) Update ASID of a CD. Atomically write the first 64 bits of the
- * CD, then invalidate the old entry and mappings.
- * (4) Quiesce the context without clearing the valid bit. Disable
- * translation, and ignore any translation fault.
- * (5) Remove a secondary CD.
- */
- u64 val;
- bool cd_live;
- struct arm_smmu_cd target;
- struct arm_smmu_cd *cdptr = ⌖
- struct arm_smmu_cd *cd_table_entry;
- struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
- struct arm_smmu_device *smmu = master->smmu;
-
- if (WARN_ON(ssid >= (1 << cd_table->s1cdmax)))
- return -E2BIG;
-
- cd_table_entry = arm_smmu_get_cd_ptr(master, ssid);
- if (!cd_table_entry)
- return -ENOMEM;
-
- target = *cd_table_entry;
- val = le64_to_cpu(cdptr->data[0]);
- cd_live = !!(val & CTXDESC_CD_0_V);
-
- if (!cd) { /* (5) */
- val = 0;
- } else if (cd == &quiet_cd) { /* (4) */
- if (!(smmu->features & ARM_SMMU_FEAT_STALL_FORCE))
- val &= ~(CTXDESC_CD_0_S | CTXDESC_CD_0_R);
- val |= CTXDESC_CD_0_TCR_EPD0;
- } else if (cd_live) { /* (3) */
- val &= ~CTXDESC_CD_0_ASID;
- val |= FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid);
- /*
- * Until CD+TLB invalidation, both ASIDs may be used for tagging
- * this substream's traffic
- */
- } else { /* (1) and (2) */
- cdptr->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
- cdptr->data[2] = 0;
- cdptr->data[3] = cpu_to_le64(cd->mair);
-
- val = cd->tcr |
-#ifdef __BIG_ENDIAN
- CTXDESC_CD_0_ENDI |
-#endif
- CTXDESC_CD_0_R | CTXDESC_CD_0_A |
- (cd->mm ? 0 : CTXDESC_CD_0_ASET) |
- CTXDESC_CD_0_AA64 |
- FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) |
- CTXDESC_CD_0_V;
-
- if (cd_table->stall_enabled)
- val |= CTXDESC_CD_0_S;
- }
- cdptr->data[0] = cpu_to_le64(val);
- arm_smmu_write_cd_entry(master, ssid, cd_table_entry, &target);
- return 0;
-}
-
static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master)
{
int ret;
@@ -1425,7 +1343,6 @@ static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master)
struct arm_smmu_device *smmu = master->smmu;
struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
- cd_table->stall_enabled = master->stall_enabled;
cd_table->s1cdmax = master->ssid_bits;
max_contexts = 1 << cd_table->s1cdmax;
@@ -1523,7 +1440,7 @@ arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc)
val |= FIELD_PREP(STRTAB_L1_DESC_SPAN, desc->span);
val |= desc->l2ptr_dma & STRTAB_L1_DESC_L2PTR_MASK;
- /* See comment in arm_smmu_write_ctx_desc() */
+ /* The HW has 64 bit atomicity with stores to the L2 STE table */
WRITE_ONCE(*dst, cpu_to_le64(val));
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index d32da11058aab6..8eabcccb9420ba 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -608,8 +608,6 @@ struct arm_smmu_ctx_desc_cfg {
u8 s1fmt;
/* log2 of the maximum number of CDs supported by this table */
u8 s1cdmax;
- /* Whether CD entries in this table have the stall bit set. */
- u8 stall_enabled:1;
};
struct arm_smmu_s2_cfg {
@@ -747,7 +745,6 @@ static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
extern struct xarray arm_smmu_asid_xa;
extern struct mutex arm_smmu_asid_lock;
-extern struct arm_smmu_ctx_desc quiet_cd;
void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
@@ -759,8 +756,6 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
struct arm_smmu_cd *cdptr,
const struct arm_smmu_cd *target);
-int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid,
- struct arm_smmu_ctx_desc *cd);
void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
size_t granule, bool leaf,
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 10/27] iommu/arm-smmu-v3: Move the CD generation for SVA into a function
@ 2024-03-04 23:43 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Pull all the calculations for building the CD table entry for a mmu_struct
into arm_smmu_make_sva_cd().
Call it in the two places installing the SVA CD table entry.
Open code the last caller of arm_smmu_update_ctx_desc_devices() and remove
the function.
Remove arm_smmu_write_ctx_desc() since all callers are gone.
Remove quiet_cd since all users are gone, arm_smmu_make_sva_cd() creates
the same value.
The behavior of quiet_cd changes slightly, the old implementation edited
the CD in place to set CTXDESC_CD_0_TCR_EPD0 assuming it was a SVA CD
entry. This version generates a full CD entry with a 0 TTB0 and relies on
arm_smmu_write_cd_entry() to install it hitlessly.
Remove no_used_check since this was the flow that could result in an used
bit inconsistent CD.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 156 +++++++++++-------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 95 +----------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 -
3 files changed, 105 insertions(+), 151 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 6acc65f6d00a71..159350a78d4cf9 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -35,25 +35,6 @@ struct arm_smmu_bond {
static DEFINE_MUTEX(sva_lock);
-/*
- * Write the CD to the CD tables for all masters that this domain is attached
- * to. Note that this is only used to update existing CD entries in the target
- * CD table, for which it's assumed that arm_smmu_write_ctx_desc can't fail.
- */
-static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain,
- int ssid,
- struct arm_smmu_ctx_desc *cd)
-{
- struct arm_smmu_master *master;
- unsigned long flags;
-
- spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- list_for_each_entry(master, &smmu_domain->devices, domain_head) {
- arm_smmu_write_ctx_desc(master, ssid, cd);
- }
- spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
-}
-
static void
arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
{
@@ -129,11 +110,86 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid)
return NULL;
}
+static u64 page_size_to_cd(void)
+{
+ static_assert(PAGE_SIZE == SZ_4K || PAGE_SIZE == SZ_16K ||
+ PAGE_SIZE == SZ_64K);
+ if (PAGE_SIZE == SZ_64K)
+ return ARM_LPAE_TCR_TG0_64K;
+ if (PAGE_SIZE == SZ_16K)
+ return ARM_LPAE_TCR_TG0_16K;
+ return ARM_LPAE_TCR_TG0_4K;
+}
+
+static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target,
+ struct arm_smmu_master *master,
+ struct mm_struct *mm, u16 asid)
+{
+ u64 par;
+
+ memset(target, 0, sizeof(*target));
+
+ par = cpuid_feature_extract_unsigned_field(
+ read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1),
+ ID_AA64MMFR0_EL1_PARANGE_SHIFT);
+
+ target->data[0] = cpu_to_le64(
+ CTXDESC_CD_0_TCR_EPD1 |
+#ifdef __BIG_ENDIAN
+ CTXDESC_CD_0_ENDI |
+#endif
+ CTXDESC_CD_0_V |
+ FIELD_PREP(CTXDESC_CD_0_TCR_IPS, par) |
+ CTXDESC_CD_0_AA64 |
+ (master->stall_enabled ? CTXDESC_CD_0_S : 0) |
+ CTXDESC_CD_0_R |
+ CTXDESC_CD_0_A |
+ CTXDESC_CD_0_ASET |
+ FIELD_PREP(CTXDESC_CD_0_ASID, asid));
+
+ /*
+ * If no MM is passed then this creates a SVA entry that faults
+ * everything. arm_smmu_write_cd_entry() can hitlessly go between these
+ * two entries types since TTB0 is ignored by HW when EPD0 is set.
+ */
+ if (mm) {
+ target->data[0] |= cpu_to_le64(
+ FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ,
+ 64ULL - vabits_actual) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_TG0, page_size_to_cd()) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0,
+ ARM_LPAE_TCR_RGN_WBWA) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0,
+ ARM_LPAE_TCR_RGN_WBWA) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS));
+
+ target->data[1] = cpu_to_le64(virt_to_phys(mm->pgd) &
+ CTXDESC_CD_1_TTB0_MASK);
+ } else {
+ target->data[0] |= cpu_to_le64(CTXDESC_CD_0_TCR_EPD0);
+
+ /*
+ * Disable stall and immediately generate an abort if stall
+ * disable is permitted. This speeds up cleanup for an unclean
+ * exit if the device is still doing a lot of DMA.
+ */
+ if (master->stall_enabled &&
+ !(master->smmu->features & ARM_SMMU_FEAT_STALL_FORCE))
+ target->data[0] &=
+ cpu_to_le64(~(CTXDESC_CD_0_S | CTXDESC_CD_0_R));
+ }
+
+ /*
+ * MAIR value is pretty much constant and global, so we can just get it
+ * from the current CPU register
+ */
+ target->data[3] = cpu_to_le64(read_sysreg(mair_el1));
+}
+
static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm)
{
u16 asid;
int err = 0;
- u64 tcr, par, reg;
struct arm_smmu_ctx_desc *cd;
struct arm_smmu_ctx_desc *ret = NULL;
@@ -167,39 +223,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm)
if (err)
goto out_free_asid;
- tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - vabits_actual) |
- FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) |
- FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) |
- FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS) |
- CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64;
-
- switch (PAGE_SIZE) {
- case SZ_4K:
- tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_4K);
- break;
- case SZ_16K:
- tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_16K);
- break;
- case SZ_64K:
- tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_64K);
- break;
- default:
- WARN_ON(1);
- err = -EINVAL;
- goto out_free_asid;
- }
-
- reg = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);
- par = cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR0_EL1_PARANGE_SHIFT);
- tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_IPS, par);
-
- cd->ttbr = virt_to_phys(mm->pgd);
- cd->tcr = tcr;
- /*
- * MAIR value is pretty much constant and global, so we can just get it
- * from the current CPU register
- */
- cd->mair = read_sysreg(mair_el1);
cd->asid = asid;
cd->mm = mm;
@@ -277,6 +300,8 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
{
struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn);
struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
+ struct arm_smmu_master *master;
+ unsigned long flags;
mutex_lock(&sva_lock);
if (smmu_mn->cleared) {
@@ -288,8 +313,19 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
* DMA may still be running. Keep the cd valid to avoid C_BAD_CD events,
* but disable translation.
*/
- arm_smmu_update_ctx_desc_devices(smmu_domain, mm_get_enqcmd_pasid(mm),
- &quiet_cd);
+ spin_lock_irqsave(&smmu_domain->devices_lock, flags);
+ list_for_each_entry(master, &smmu_domain->devices, domain_head) {
+ struct arm_smmu_cd target;
+ struct arm_smmu_cd *cdptr;
+
+ cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm));
+ if (WARN_ON(!cdptr))
+ continue;
+ arm_smmu_make_sva_cd(&target, master, NULL, smmu_mn->cd->asid);
+ arm_smmu_write_cd_entry(master, mm_get_enqcmd_pasid(mm), cdptr,
+ &target);
+ }
+ spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid);
arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0);
@@ -384,6 +420,8 @@ static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid,
struct mm_struct *mm)
{
int ret;
+ struct arm_smmu_cd target;
+ struct arm_smmu_cd *cdptr;
struct arm_smmu_bond *bond;
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
@@ -410,9 +448,13 @@ static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid,
goto err_free_bond;
}
- ret = arm_smmu_write_ctx_desc(master, pasid, bond->smmu_mn->cd);
- if (ret)
+ cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm));
+ if (!cdptr) {
+ ret = -ENOMEM;
goto err_put_notifier;
+ }
+ arm_smmu_make_sva_cd(&target, master, mm, bond->smmu_mn->cd->asid);
+ arm_smmu_write_cd_entry(master, pasid, cdptr, &target);
list_add(&bond->list, &master->bonds);
return 0;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 39081d828a2132..50d17e3ce0a956 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -57,7 +57,6 @@ struct arm_smmu_entry_writer {
struct arm_smmu_entry_writer_ops {
unsigned int num_entry_qwords;
__le64 v_bit;
- bool no_used_check;
void (*get_used)(const __le64 *entry, __le64 *used);
void (*sync)(struct arm_smmu_entry_writer *writer);
};
@@ -92,12 +91,6 @@ struct arm_smmu_option_prop {
DEFINE_XARRAY_ALLOC1(arm_smmu_asid_xa);
DEFINE_MUTEX(arm_smmu_asid_lock);
-/*
- * Special value used by SVA when a process dies, to quiesce a CD without
- * disabling it.
- */
-struct arm_smmu_ctx_desc quiet_cd = { 0 };
-
static struct arm_smmu_option_prop arm_smmu_options[] = {
{ ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" },
{ ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"},
@@ -1060,8 +1053,7 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
* allowed to set a bit to 1 if the used function doesn't say it
* is used.
*/
- if (!writer->ops->no_used_check)
- WARN_ON_ONCE(target[i] & ~target_used[i]);
+ WARN_ON_ONCE(target[i] & ~target_used[i]);
/* Bits can change because they are not currently being used */
unused_update[i] = (entry[i] & cur_used[i]) |
@@ -1070,8 +1062,7 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
* Each bit indicates that a used bit in a qword needs to be
* changed after unused_update is applied.
*/
- if ((unused_update[i] & target_used[i]) !=
- (target[i] & target_used[i]))
+ if ((unused_update[i] & target_used[i]) != target[i])
used_qword_diff |= 1 << i;
}
return used_qword_diff;
@@ -1167,11 +1158,8 @@ static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer,
* in the entry. The target was already sanity checked by
* compute_qword_diff().
*/
- if (writer->ops->no_used_check)
- entry_set(writer, entry, target, 0, num_entry_qwords);
- else
- WARN_ON_ONCE(entry_set(writer, entry, target, 0,
- num_entry_qwords));
+ WARN_ON_ONCE(
+ entry_set(writer, entry, target, 0, num_entry_qwords));
}
}
@@ -1219,7 +1207,7 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
u64 val = (l1_desc->l2ptr_dma & CTXDESC_L1_DESC_L2PTR_MASK) |
CTXDESC_L1_DESC_V;
- /* See comment in arm_smmu_write_ctx_desc() */
+ /* The HW has 64 bit atomicity with stores to the L2 CD table */
WRITE_ONCE(*dst, cpu_to_le64(val));
}
@@ -1290,7 +1278,6 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
.sync = arm_smmu_cd_writer_sync_entry,
.get_used = arm_smmu_get_cd_used,
.v_bit = cpu_to_le64(CTXDESC_CD_0_V),
- .no_used_check = true,
.num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
};
@@ -1348,75 +1335,6 @@ void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
arm_smmu_write_cd_entry(master, ssid, cdptr, &target);
}
-int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
- struct arm_smmu_ctx_desc *cd)
-{
- /*
- * This function handles the following cases:
- *
- * (1) Install primary CD, for normal DMA traffic (SSID = IOMMU_NO_PASID = 0).
- * (2) Install a secondary CD, for SID+SSID traffic.
- * (3) Update ASID of a CD. Atomically write the first 64 bits of the
- * CD, then invalidate the old entry and mappings.
- * (4) Quiesce the context without clearing the valid bit. Disable
- * translation, and ignore any translation fault.
- * (5) Remove a secondary CD.
- */
- u64 val;
- bool cd_live;
- struct arm_smmu_cd target;
- struct arm_smmu_cd *cdptr = ⌖
- struct arm_smmu_cd *cd_table_entry;
- struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
- struct arm_smmu_device *smmu = master->smmu;
-
- if (WARN_ON(ssid >= (1 << cd_table->s1cdmax)))
- return -E2BIG;
-
- cd_table_entry = arm_smmu_get_cd_ptr(master, ssid);
- if (!cd_table_entry)
- return -ENOMEM;
-
- target = *cd_table_entry;
- val = le64_to_cpu(cdptr->data[0]);
- cd_live = !!(val & CTXDESC_CD_0_V);
-
- if (!cd) { /* (5) */
- val = 0;
- } else if (cd == &quiet_cd) { /* (4) */
- if (!(smmu->features & ARM_SMMU_FEAT_STALL_FORCE))
- val &= ~(CTXDESC_CD_0_S | CTXDESC_CD_0_R);
- val |= CTXDESC_CD_0_TCR_EPD0;
- } else if (cd_live) { /* (3) */
- val &= ~CTXDESC_CD_0_ASID;
- val |= FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid);
- /*
- * Until CD+TLB invalidation, both ASIDs may be used for tagging
- * this substream's traffic
- */
- } else { /* (1) and (2) */
- cdptr->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
- cdptr->data[2] = 0;
- cdptr->data[3] = cpu_to_le64(cd->mair);
-
- val = cd->tcr |
-#ifdef __BIG_ENDIAN
- CTXDESC_CD_0_ENDI |
-#endif
- CTXDESC_CD_0_R | CTXDESC_CD_0_A |
- (cd->mm ? 0 : CTXDESC_CD_0_ASET) |
- CTXDESC_CD_0_AA64 |
- FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) |
- CTXDESC_CD_0_V;
-
- if (cd_table->stall_enabled)
- val |= CTXDESC_CD_0_S;
- }
- cdptr->data[0] = cpu_to_le64(val);
- arm_smmu_write_cd_entry(master, ssid, cd_table_entry, &target);
- return 0;
-}
-
static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master)
{
int ret;
@@ -1425,7 +1343,6 @@ static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master)
struct arm_smmu_device *smmu = master->smmu;
struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
- cd_table->stall_enabled = master->stall_enabled;
cd_table->s1cdmax = master->ssid_bits;
max_contexts = 1 << cd_table->s1cdmax;
@@ -1523,7 +1440,7 @@ arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc)
val |= FIELD_PREP(STRTAB_L1_DESC_SPAN, desc->span);
val |= desc->l2ptr_dma & STRTAB_L1_DESC_L2PTR_MASK;
- /* See comment in arm_smmu_write_ctx_desc() */
+ /* The HW has 64 bit atomicity with stores to the L2 STE table */
WRITE_ONCE(*dst, cpu_to_le64(val));
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index d32da11058aab6..8eabcccb9420ba 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -608,8 +608,6 @@ struct arm_smmu_ctx_desc_cfg {
u8 s1fmt;
/* log2 of the maximum number of CDs supported by this table */
u8 s1cdmax;
- /* Whether CD entries in this table have the stall bit set. */
- u8 stall_enabled:1;
};
struct arm_smmu_s2_cfg {
@@ -747,7 +745,6 @@ static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
extern struct xarray arm_smmu_asid_xa;
extern struct mutex arm_smmu_asid_lock;
-extern struct arm_smmu_ctx_desc quiet_cd;
void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
@@ -759,8 +756,6 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
struct arm_smmu_cd *cdptr,
const struct arm_smmu_cd *target);
-int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid,
- struct arm_smmu_ctx_desc *cd);
void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
size_t granule, bool leaf,
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 11/27] iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd()
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:43 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Half the code was living in arm_smmu_domain_finalise_s1(), just move it
here and take the values directly from the pgtbl_ops instead of storing
copies.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 47 ++++++++-------------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 --
2 files changed, 18 insertions(+), 32 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 50d17e3ce0a956..dfdd48cf217c4e 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1301,15 +1301,25 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
struct arm_smmu_domain *smmu_domain)
{
struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
+ const struct io_pgtable_cfg *pgtbl_cfg =
+ &io_pgtable_ops_to_pgtable(smmu_domain->pgtbl_ops)->cfg;
+ typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr =
+ &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
memset(target, 0, sizeof(*target));
target->data[0] = cpu_to_le64(
- cd->tcr |
+ FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) |
+ CTXDESC_CD_0_TCR_EPD1 |
#ifdef __BIG_ENDIAN
CTXDESC_CD_0_ENDI |
#endif
CTXDESC_CD_0_V |
+ FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) |
CTXDESC_CD_0_AA64 |
(master->stall_enabled ? CTXDESC_CD_0_S : 0) |
CTXDESC_CD_0_R |
@@ -1317,9 +1327,9 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
CTXDESC_CD_0_ASET |
FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid)
);
-
- target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
- target->data[3] = cpu_to_le64(cd->mair);
+ target->data[1] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.ttbr &
+ CTXDESC_CD_1_TTB0_MASK);
+ target->data[3] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.mair);
}
void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
@@ -2305,13 +2315,11 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
}
static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
- struct arm_smmu_domain *smmu_domain,
- struct io_pgtable_cfg *pgtbl_cfg)
+ struct arm_smmu_domain *smmu_domain)
{
int ret;
u32 asid;
struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
- typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
refcount_set(&cd->refs, 1);
@@ -2319,31 +2327,13 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
mutex_lock(&arm_smmu_asid_lock);
ret = xa_alloc(&arm_smmu_asid_xa, &asid, cd,
XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
- if (ret)
- goto out_unlock;
-
cd->asid = (u16)asid;
- cd->ttbr = pgtbl_cfg->arm_lpae_s1_cfg.ttbr;
- cd->tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) |
- FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) |
- FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) |
- FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) |
- FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) |
- FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) |
- CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64;
- cd->mair = pgtbl_cfg->arm_lpae_s1_cfg.mair;
-
- mutex_unlock(&arm_smmu_asid_lock);
- return 0;
-
-out_unlock:
mutex_unlock(&arm_smmu_asid_lock);
return ret;
}
static int arm_smmu_domain_finalise_s2(struct arm_smmu_device *smmu,
- struct arm_smmu_domain *smmu_domain,
- struct io_pgtable_cfg *pgtbl_cfg)
+ struct arm_smmu_domain *smmu_domain)
{
int vmid;
struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
@@ -2367,8 +2357,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
struct io_pgtable_cfg pgtbl_cfg;
struct io_pgtable_ops *pgtbl_ops;
int (*finalise_stage_fn)(struct arm_smmu_device *smmu,
- struct arm_smmu_domain *smmu_domain,
- struct io_pgtable_cfg *pgtbl_cfg);
+ struct arm_smmu_domain *smmu_domain);
/* Restrict the stage to what we can actually support */
if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S1))
@@ -2411,7 +2400,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
smmu_domain->domain.geometry.aperture_end = (1UL << pgtbl_cfg.ias) - 1;
smmu_domain->domain.geometry.force_aperture = true;
- ret = finalise_stage_fn(smmu, smmu_domain, &pgtbl_cfg);
+ ret = finalise_stage_fn(smmu, smmu_domain);
if (ret < 0) {
free_io_pgtable_ops(pgtbl_ops);
return ret;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 8eabcccb9420ba..468cd33b80ac35 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -587,9 +587,6 @@ struct arm_smmu_strtab_l1_desc {
struct arm_smmu_ctx_desc {
u16 asid;
- u64 ttbr;
- u64 tcr;
- u64 mair;
refcount_t refs;
struct mm_struct *mm;
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 11/27] iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd()
@ 2024-03-04 23:43 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:43 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Half the code was living in arm_smmu_domain_finalise_s1(), just move it
here and take the values directly from the pgtbl_ops instead of storing
copies.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 47 ++++++++-------------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 --
2 files changed, 18 insertions(+), 32 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 50d17e3ce0a956..dfdd48cf217c4e 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1301,15 +1301,25 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
struct arm_smmu_domain *smmu_domain)
{
struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
+ const struct io_pgtable_cfg *pgtbl_cfg =
+ &io_pgtable_ops_to_pgtable(smmu_domain->pgtbl_ops)->cfg;
+ typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr =
+ &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
memset(target, 0, sizeof(*target));
target->data[0] = cpu_to_le64(
- cd->tcr |
+ FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) |
+ FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) |
+ CTXDESC_CD_0_TCR_EPD1 |
#ifdef __BIG_ENDIAN
CTXDESC_CD_0_ENDI |
#endif
CTXDESC_CD_0_V |
+ FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) |
CTXDESC_CD_0_AA64 |
(master->stall_enabled ? CTXDESC_CD_0_S : 0) |
CTXDESC_CD_0_R |
@@ -1317,9 +1327,9 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
CTXDESC_CD_0_ASET |
FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid)
);
-
- target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
- target->data[3] = cpu_to_le64(cd->mair);
+ target->data[1] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.ttbr &
+ CTXDESC_CD_1_TTB0_MASK);
+ target->data[3] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.mair);
}
void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
@@ -2305,13 +2315,11 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
}
static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
- struct arm_smmu_domain *smmu_domain,
- struct io_pgtable_cfg *pgtbl_cfg)
+ struct arm_smmu_domain *smmu_domain)
{
int ret;
u32 asid;
struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
- typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
refcount_set(&cd->refs, 1);
@@ -2319,31 +2327,13 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
mutex_lock(&arm_smmu_asid_lock);
ret = xa_alloc(&arm_smmu_asid_xa, &asid, cd,
XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
- if (ret)
- goto out_unlock;
-
cd->asid = (u16)asid;
- cd->ttbr = pgtbl_cfg->arm_lpae_s1_cfg.ttbr;
- cd->tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) |
- FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) |
- FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) |
- FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) |
- FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) |
- FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) |
- CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64;
- cd->mair = pgtbl_cfg->arm_lpae_s1_cfg.mair;
-
- mutex_unlock(&arm_smmu_asid_lock);
- return 0;
-
-out_unlock:
mutex_unlock(&arm_smmu_asid_lock);
return ret;
}
static int arm_smmu_domain_finalise_s2(struct arm_smmu_device *smmu,
- struct arm_smmu_domain *smmu_domain,
- struct io_pgtable_cfg *pgtbl_cfg)
+ struct arm_smmu_domain *smmu_domain)
{
int vmid;
struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
@@ -2367,8 +2357,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
struct io_pgtable_cfg pgtbl_cfg;
struct io_pgtable_ops *pgtbl_ops;
int (*finalise_stage_fn)(struct arm_smmu_device *smmu,
- struct arm_smmu_domain *smmu_domain,
- struct io_pgtable_cfg *pgtbl_cfg);
+ struct arm_smmu_domain *smmu_domain);
/* Restrict the stage to what we can actually support */
if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S1))
@@ -2411,7 +2400,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
smmu_domain->domain.geometry.aperture_end = (1UL << pgtbl_cfg.ias) - 1;
smmu_domain->domain.geometry.force_aperture = true;
- ret = finalise_stage_fn(smmu, smmu_domain, &pgtbl_cfg);
+ ret = finalise_stage_fn(smmu, smmu_domain);
if (ret < 0) {
free_io_pgtable_ops(pgtbl_ops);
return ret;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 8eabcccb9420ba..468cd33b80ac35 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -587,9 +587,6 @@ struct arm_smmu_strtab_l1_desc {
struct arm_smmu_ctx_desc {
u16 asid;
- u64 ttbr;
- u64 tcr;
- u64 mair;
refcount_t refs;
struct mm_struct *mm;
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 12/27] iommu/arm-smmu-v3: Start building a generic PASID layer
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Add arm_smmu_set_pasid()/arm_smmu_remove_pasid() which are to be used by
callers that already constructed the arm_smmu_cd they wish to program.
These functions will encapsulate the shared logic to setup a CD entry that
will be shared by SVA and S1 domain cases.
Prior fixes had already moved this logic most up into
__arm_smmu_sva_bind(), move it to it's final home.
This does not yet relieve the SVA restrictions:
- The RID domain is a S1 domain
- The programmed PASID is the mm_get_enqcmd_pasid()
- Nothing changes while SVA is running (sva_enable)
Also, SVA invalidation will still iterate over the S1 domain's master
list.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 39 +++++++++----------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 29 ++++++++++++++
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 +++
3 files changed, 53 insertions(+), 21 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 159350a78d4cf9..e1fa2074c0b37b 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -416,12 +416,10 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn)
arm_smmu_free_shared_cd(cd);
}
-static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid,
- struct mm_struct *mm)
+static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm,
+ struct arm_smmu_cd *target)
{
int ret;
- struct arm_smmu_cd target;
- struct arm_smmu_cd *cdptr;
struct arm_smmu_bond *bond;
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
@@ -448,19 +446,10 @@ static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid,
goto err_free_bond;
}
- cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm));
- if (!cdptr) {
- ret = -ENOMEM;
- goto err_put_notifier;
- }
- arm_smmu_make_sva_cd(&target, master, mm, bond->smmu_mn->cd->asid);
- arm_smmu_write_cd_entry(master, pasid, cdptr, &target);
-
+ arm_smmu_make_sva_cd(target, master, mm, bond->smmu_mn->cd->asid);
list_add(&bond->list, &master->bonds);
return 0;
-err_put_notifier:
- arm_smmu_mmu_notifier_put(bond->smmu_mn);
err_free_bond:
kfree(bond);
return ret;
@@ -621,10 +610,9 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
struct arm_smmu_bond *bond = NULL, *t;
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
+ arm_smmu_remove_pasid(master, to_smmu_domain(domain), id);
+
mutex_lock(&sva_lock);
-
- arm_smmu_clear_cd(master, id);
-
list_for_each_entry(t, &master->bonds, list) {
if (t->mm == mm) {
bond = t;
@@ -643,17 +631,26 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
struct device *dev, ioasid_t id)
{
+ struct arm_smmu_master *master = dev_iommu_priv_get(dev);
int ret = 0;
struct mm_struct *mm = domain->mm;
+ struct arm_smmu_cd target;
if (mm_get_enqcmd_pasid(mm) != id)
return -EINVAL;
- mutex_lock(&sva_lock);
- ret = __arm_smmu_sva_bind(dev, id, mm);
- mutex_unlock(&sva_lock);
+ if (!arm_smmu_get_cd_ptr(master, id))
+ return -ENOMEM;
- return ret;
+ mutex_lock(&sva_lock);
+ ret = __arm_smmu_sva_bind(dev, mm, &target);
+ mutex_unlock(&sva_lock);
+ if (ret)
+ return ret;
+
+ /* This cannot fail since we preallocated the cdptr */
+ arm_smmu_set_pasid(master, to_smmu_domain(domain), id, &target);
+ return 0;
}
static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index dfdd48cf217c4e..7a0be7dd719793 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2656,6 +2656,35 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
return 0;
}
+static bool arm_smmu_is_s1_domain(struct iommu_domain *domain)
+{
+ if (!domain || !(domain->type & __IOMMU_DOMAIN_PAGING))
+ return false;
+ return to_smmu_domain(domain)->stage == ARM_SMMU_DOMAIN_S1;
+}
+
+int arm_smmu_set_pasid(struct arm_smmu_master *master,
+ struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
+ const struct arm_smmu_cd *cd)
+{
+ struct arm_smmu_cd *cdptr;
+
+ if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)))
+ return -ENODEV;
+
+ cdptr = arm_smmu_get_cd_ptr(master, pasid);
+ if (!cdptr)
+ return -ENOMEM;
+ arm_smmu_write_cd_entry(master, pasid, cdptr, cd);
+ return 0;
+}
+
+void arm_smmu_remove_pasid(struct arm_smmu_master *master,
+ struct arm_smmu_domain *smmu_domain, ioasid_t pasid)
+{
+ arm_smmu_clear_cd(master, pasid);
+}
+
static int arm_smmu_attach_dev_ste(struct device *dev,
struct arm_smmu_ste *ste)
{
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 468cd33b80ac35..ab0a6587e07ec8 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -753,6 +753,12 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
struct arm_smmu_cd *cdptr,
const struct arm_smmu_cd *target);
+int arm_smmu_set_pasid(struct arm_smmu_master *master,
+ struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
+ const struct arm_smmu_cd *cd);
+void arm_smmu_remove_pasid(struct arm_smmu_master *master,
+ struct arm_smmu_domain *smmu_domain, ioasid_t pasid);
+
void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
size_t granule, bool leaf,
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 12/27] iommu/arm-smmu-v3: Start building a generic PASID layer
@ 2024-03-04 23:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Add arm_smmu_set_pasid()/arm_smmu_remove_pasid() which are to be used by
callers that already constructed the arm_smmu_cd they wish to program.
These functions will encapsulate the shared logic to setup a CD entry that
will be shared by SVA and S1 domain cases.
Prior fixes had already moved this logic most up into
__arm_smmu_sva_bind(), move it to it's final home.
This does not yet relieve the SVA restrictions:
- The RID domain is a S1 domain
- The programmed PASID is the mm_get_enqcmd_pasid()
- Nothing changes while SVA is running (sva_enable)
Also, SVA invalidation will still iterate over the S1 domain's master
list.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 39 +++++++++----------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 29 ++++++++++++++
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 +++
3 files changed, 53 insertions(+), 21 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 159350a78d4cf9..e1fa2074c0b37b 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -416,12 +416,10 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn)
arm_smmu_free_shared_cd(cd);
}
-static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid,
- struct mm_struct *mm)
+static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm,
+ struct arm_smmu_cd *target)
{
int ret;
- struct arm_smmu_cd target;
- struct arm_smmu_cd *cdptr;
struct arm_smmu_bond *bond;
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
@@ -448,19 +446,10 @@ static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid,
goto err_free_bond;
}
- cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm));
- if (!cdptr) {
- ret = -ENOMEM;
- goto err_put_notifier;
- }
- arm_smmu_make_sva_cd(&target, master, mm, bond->smmu_mn->cd->asid);
- arm_smmu_write_cd_entry(master, pasid, cdptr, &target);
-
+ arm_smmu_make_sva_cd(target, master, mm, bond->smmu_mn->cd->asid);
list_add(&bond->list, &master->bonds);
return 0;
-err_put_notifier:
- arm_smmu_mmu_notifier_put(bond->smmu_mn);
err_free_bond:
kfree(bond);
return ret;
@@ -621,10 +610,9 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
struct arm_smmu_bond *bond = NULL, *t;
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
+ arm_smmu_remove_pasid(master, to_smmu_domain(domain), id);
+
mutex_lock(&sva_lock);
-
- arm_smmu_clear_cd(master, id);
-
list_for_each_entry(t, &master->bonds, list) {
if (t->mm == mm) {
bond = t;
@@ -643,17 +631,26 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
struct device *dev, ioasid_t id)
{
+ struct arm_smmu_master *master = dev_iommu_priv_get(dev);
int ret = 0;
struct mm_struct *mm = domain->mm;
+ struct arm_smmu_cd target;
if (mm_get_enqcmd_pasid(mm) != id)
return -EINVAL;
- mutex_lock(&sva_lock);
- ret = __arm_smmu_sva_bind(dev, id, mm);
- mutex_unlock(&sva_lock);
+ if (!arm_smmu_get_cd_ptr(master, id))
+ return -ENOMEM;
- return ret;
+ mutex_lock(&sva_lock);
+ ret = __arm_smmu_sva_bind(dev, mm, &target);
+ mutex_unlock(&sva_lock);
+ if (ret)
+ return ret;
+
+ /* This cannot fail since we preallocated the cdptr */
+ arm_smmu_set_pasid(master, to_smmu_domain(domain), id, &target);
+ return 0;
}
static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index dfdd48cf217c4e..7a0be7dd719793 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2656,6 +2656,35 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
return 0;
}
+static bool arm_smmu_is_s1_domain(struct iommu_domain *domain)
+{
+ if (!domain || !(domain->type & __IOMMU_DOMAIN_PAGING))
+ return false;
+ return to_smmu_domain(domain)->stage == ARM_SMMU_DOMAIN_S1;
+}
+
+int arm_smmu_set_pasid(struct arm_smmu_master *master,
+ struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
+ const struct arm_smmu_cd *cd)
+{
+ struct arm_smmu_cd *cdptr;
+
+ if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)))
+ return -ENODEV;
+
+ cdptr = arm_smmu_get_cd_ptr(master, pasid);
+ if (!cdptr)
+ return -ENOMEM;
+ arm_smmu_write_cd_entry(master, pasid, cdptr, cd);
+ return 0;
+}
+
+void arm_smmu_remove_pasid(struct arm_smmu_master *master,
+ struct arm_smmu_domain *smmu_domain, ioasid_t pasid)
+{
+ arm_smmu_clear_cd(master, pasid);
+}
+
static int arm_smmu_attach_dev_ste(struct device *dev,
struct arm_smmu_ste *ste)
{
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 468cd33b80ac35..ab0a6587e07ec8 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -753,6 +753,12 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
struct arm_smmu_cd *cdptr,
const struct arm_smmu_cd *target);
+int arm_smmu_set_pasid(struct arm_smmu_master *master,
+ struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
+ const struct arm_smmu_cd *cd);
+void arm_smmu_remove_pasid(struct arm_smmu_master *master,
+ struct arm_smmu_domain *smmu_domain, ioasid_t pasid);
+
void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
size_t granule, bool leaf,
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 13/27] iommu/arm-smmu-v3: Make smmu_domain->devices into an allocated list
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
The next patch will need to store the same master twice (with different
SSIDs), so allocate memory for each list element.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 11 ++++--
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 39 ++++++++++++++++---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 +++-
3 files changed, 47 insertions(+), 10 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index e1fa2074c0b37b..9a285b828e8b12 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -38,12 +38,13 @@ static DEFINE_MUTEX(sva_lock);
static void
arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
{
- struct arm_smmu_master *master;
+ struct arm_smmu_master_domain *master_domain;
struct arm_smmu_cd target_cd;
unsigned long flags;
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- list_for_each_entry(master, &smmu_domain->devices, domain_head) {
+ list_for_each_entry(master_domain, &smmu_domain->devices, devices_elm) {
+ struct arm_smmu_master *master = master_domain->master;
struct arm_smmu_cd *cdptr;
/* S1 domains only support RID attachment right now */
@@ -300,7 +301,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
{
struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn);
struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
- struct arm_smmu_master *master;
+ struct arm_smmu_master_domain *master_domain;
unsigned long flags;
mutex_lock(&sva_lock);
@@ -314,7 +315,9 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
* but disable translation.
*/
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- list_for_each_entry(master, &smmu_domain->devices, domain_head) {
+ list_for_each_entry(master_domain, &smmu_domain->devices,
+ devices_elm) {
+ struct arm_smmu_master *master = master_domain->master;
struct arm_smmu_cd target;
struct arm_smmu_cd *cdptr;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 7a0be7dd719793..51a1e7198fd1af 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2031,10 +2031,10 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master)
int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
unsigned long iova, size_t size)
{
+ struct arm_smmu_master_domain *master_domain;
int i;
unsigned long flags;
struct arm_smmu_cmdq_ent cmd;
- struct arm_smmu_master *master;
struct arm_smmu_cmdq_batch cmds;
if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_ATS))
@@ -2062,7 +2062,10 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
cmds.num = 0;
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- list_for_each_entry(master, &smmu_domain->devices, domain_head) {
+ list_for_each_entry(master_domain, &smmu_domain->devices,
+ devices_elm) {
+ struct arm_smmu_master *master = master_domain->master;
+
if (!master->ats_enabled)
continue;
@@ -2554,9 +2557,26 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master)
pci_disable_pasid(pdev);
}
+static struct arm_smmu_master_domain *
+arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
+ struct arm_smmu_master *master)
+{
+ struct arm_smmu_master_domain *master_domain;
+
+ lockdep_assert_held(&smmu_domain->devices_lock);
+
+ list_for_each_entry(master_domain, &smmu_domain->devices,
+ devices_elm) {
+ if (master_domain->master == master)
+ return master_domain;
+ }
+ return NULL;
+}
+
static void arm_smmu_detach_dev(struct arm_smmu_master *master)
{
struct iommu_domain *domain = iommu_get_domain_for_dev(master->dev);
+ struct arm_smmu_master_domain *master_domain;
struct arm_smmu_domain *smmu_domain;
unsigned long flags;
@@ -2567,7 +2587,11 @@ static void arm_smmu_detach_dev(struct arm_smmu_master *master)
arm_smmu_disable_ats(master, smmu_domain);
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- list_del_init(&master->domain_head);
+ master_domain = arm_smmu_find_master_domain(smmu_domain, master);
+ if (master_domain) {
+ list_del(&master_domain->devices_elm);
+ kfree(master_domain);
+ }
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
master->ats_enabled = false;
@@ -2581,6 +2605,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
struct arm_smmu_device *smmu;
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct arm_smmu_master_domain *master_domain;
struct arm_smmu_master *master;
struct arm_smmu_cd *cdptr;
@@ -2617,6 +2642,11 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
return -ENOMEM;
}
+ master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL);
+ if (!master_domain)
+ return -ENOMEM;
+ master_domain->master = master;
+
/*
* Prevent arm_smmu_share_asid() from trying to change the ASID
* of either the old or new domain while we are working on it.
@@ -2630,7 +2660,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
master->ats_enabled = arm_smmu_ats_supported(master);
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- list_add(&master->domain_head, &smmu_domain->devices);
+ list_add(&master_domain->devices_elm, &smmu_domain->devices);
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
switch (smmu_domain->stage) {
@@ -2949,7 +2979,6 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev)
master->dev = dev;
master->smmu = smmu;
INIT_LIST_HEAD(&master->bonds);
- INIT_LIST_HEAD(&master->domain_head);
dev_iommu_priv_set(dev, master);
ret = arm_smmu_insert_master(smmu, master);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index ab0a6587e07ec8..c4b79bc52991be 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -695,7 +695,6 @@ struct arm_smmu_stream {
struct arm_smmu_master {
struct arm_smmu_device *smmu;
struct device *dev;
- struct list_head domain_head;
struct arm_smmu_stream *streams;
/* Locked by the iommu core using the group mutex */
struct arm_smmu_ctx_desc_cfg cd_table;
@@ -729,12 +728,18 @@ struct arm_smmu_domain {
struct iommu_domain domain;
+ /* List of struct arm_smmu_master_domain */
struct list_head devices;
spinlock_t devices_lock;
struct list_head mmu_notifiers;
};
+struct arm_smmu_master_domain {
+ struct list_head devices_elm;
+ struct arm_smmu_master *master;
+};
+
static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
{
return container_of(dom, struct arm_smmu_domain, domain);
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 13/27] iommu/arm-smmu-v3: Make smmu_domain->devices into an allocated list
@ 2024-03-04 23:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
The next patch will need to store the same master twice (with different
SSIDs), so allocate memory for each list element.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 11 ++++--
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 39 ++++++++++++++++---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 +++-
3 files changed, 47 insertions(+), 10 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index e1fa2074c0b37b..9a285b828e8b12 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -38,12 +38,13 @@ static DEFINE_MUTEX(sva_lock);
static void
arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
{
- struct arm_smmu_master *master;
+ struct arm_smmu_master_domain *master_domain;
struct arm_smmu_cd target_cd;
unsigned long flags;
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- list_for_each_entry(master, &smmu_domain->devices, domain_head) {
+ list_for_each_entry(master_domain, &smmu_domain->devices, devices_elm) {
+ struct arm_smmu_master *master = master_domain->master;
struct arm_smmu_cd *cdptr;
/* S1 domains only support RID attachment right now */
@@ -300,7 +301,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
{
struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn);
struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
- struct arm_smmu_master *master;
+ struct arm_smmu_master_domain *master_domain;
unsigned long flags;
mutex_lock(&sva_lock);
@@ -314,7 +315,9 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
* but disable translation.
*/
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- list_for_each_entry(master, &smmu_domain->devices, domain_head) {
+ list_for_each_entry(master_domain, &smmu_domain->devices,
+ devices_elm) {
+ struct arm_smmu_master *master = master_domain->master;
struct arm_smmu_cd target;
struct arm_smmu_cd *cdptr;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 7a0be7dd719793..51a1e7198fd1af 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2031,10 +2031,10 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master)
int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
unsigned long iova, size_t size)
{
+ struct arm_smmu_master_domain *master_domain;
int i;
unsigned long flags;
struct arm_smmu_cmdq_ent cmd;
- struct arm_smmu_master *master;
struct arm_smmu_cmdq_batch cmds;
if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_ATS))
@@ -2062,7 +2062,10 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
cmds.num = 0;
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- list_for_each_entry(master, &smmu_domain->devices, domain_head) {
+ list_for_each_entry(master_domain, &smmu_domain->devices,
+ devices_elm) {
+ struct arm_smmu_master *master = master_domain->master;
+
if (!master->ats_enabled)
continue;
@@ -2554,9 +2557,26 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master)
pci_disable_pasid(pdev);
}
+static struct arm_smmu_master_domain *
+arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
+ struct arm_smmu_master *master)
+{
+ struct arm_smmu_master_domain *master_domain;
+
+ lockdep_assert_held(&smmu_domain->devices_lock);
+
+ list_for_each_entry(master_domain, &smmu_domain->devices,
+ devices_elm) {
+ if (master_domain->master == master)
+ return master_domain;
+ }
+ return NULL;
+}
+
static void arm_smmu_detach_dev(struct arm_smmu_master *master)
{
struct iommu_domain *domain = iommu_get_domain_for_dev(master->dev);
+ struct arm_smmu_master_domain *master_domain;
struct arm_smmu_domain *smmu_domain;
unsigned long flags;
@@ -2567,7 +2587,11 @@ static void arm_smmu_detach_dev(struct arm_smmu_master *master)
arm_smmu_disable_ats(master, smmu_domain);
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- list_del_init(&master->domain_head);
+ master_domain = arm_smmu_find_master_domain(smmu_domain, master);
+ if (master_domain) {
+ list_del(&master_domain->devices_elm);
+ kfree(master_domain);
+ }
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
master->ats_enabled = false;
@@ -2581,6 +2605,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
struct arm_smmu_device *smmu;
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct arm_smmu_master_domain *master_domain;
struct arm_smmu_master *master;
struct arm_smmu_cd *cdptr;
@@ -2617,6 +2642,11 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
return -ENOMEM;
}
+ master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL);
+ if (!master_domain)
+ return -ENOMEM;
+ master_domain->master = master;
+
/*
* Prevent arm_smmu_share_asid() from trying to change the ASID
* of either the old or new domain while we are working on it.
@@ -2630,7 +2660,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
master->ats_enabled = arm_smmu_ats_supported(master);
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- list_add(&master->domain_head, &smmu_domain->devices);
+ list_add(&master_domain->devices_elm, &smmu_domain->devices);
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
switch (smmu_domain->stage) {
@@ -2949,7 +2979,6 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev)
master->dev = dev;
master->smmu = smmu;
INIT_LIST_HEAD(&master->bonds);
- INIT_LIST_HEAD(&master->domain_head);
dev_iommu_priv_set(dev, master);
ret = arm_smmu_insert_master(smmu, master);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index ab0a6587e07ec8..c4b79bc52991be 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -695,7 +695,6 @@ struct arm_smmu_stream {
struct arm_smmu_master {
struct arm_smmu_device *smmu;
struct device *dev;
- struct list_head domain_head;
struct arm_smmu_stream *streams;
/* Locked by the iommu core using the group mutex */
struct arm_smmu_ctx_desc_cfg cd_table;
@@ -729,12 +728,18 @@ struct arm_smmu_domain {
struct iommu_domain domain;
+ /* List of struct arm_smmu_master_domain */
struct list_head devices;
spinlock_t devices_lock;
struct list_head mmu_notifiers;
};
+struct arm_smmu_master_domain {
+ struct list_head devices_elm;
+ struct arm_smmu_master *master;
+};
+
static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
{
return container_of(dom, struct arm_smmu_domain, domain);
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
The core code allows the domain to be changed on the fly without a forced
stop in BLOCKED/IDENTITY. In this flow the driver should just continually
maintain the ATS with no change while the STE is updated.
ATS relies on a linked list smmu_domain->devices to keep track of which
masters have the domain programmed, but this list is also used by
arm_smmu_share_asid(), unrelated to ats.
Create two new functions to encapsulate this combined logic:
arm_smmu_attach_prepare()
<caller generates and sets the STE>
arm_smmu_attach_commit()
The two functions can sequence both enabling ATS and disabling across
the STE store. Have every update of the STE use this sequence.
Installing a S1/S2 domain always enables the ATS if the PCIe device
supports it.
The enable flow is now ordered differently to allow it to be hitless:
1) Add the master to the new smmu_domain->devices list
2) Program the STE
3) Enable ATS at PCIe
4) Remove the master from the old smmu_domain
This flow ensures that invalidations to either domain will generate an ATC
invalidation to the device while the STE is being switched. Thus we don't
need to turn off the ATS anymore for correctness.
The disable flow is the reverse:
1) Disable ATS at PCIe
2) Program the STE
3) Invalidate the ATC
4) Remove the master from the old smmu_domain
Move the nr_ats_masters adjustments to be close to the list
manipulations. It is a count of the number of ATS enabled masters
currently in the list. This is stricly before and after the STE/CD are
revised, and done under the list's spin_lock.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 198 ++++++++++++++------
1 file changed, 140 insertions(+), 58 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 51a1e7198fd1af..45f2190fc31786 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1527,7 +1527,8 @@ static void arm_smmu_make_bypass_ste(struct arm_smmu_ste *target)
}
static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
- struct arm_smmu_master *master)
+ struct arm_smmu_master *master,
+ bool ats_enabled)
{
struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
struct arm_smmu_device *smmu = master->smmu;
@@ -1550,7 +1551,7 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
STRTAB_STE_1_S1STALLD :
0) |
FIELD_PREP(STRTAB_STE_1_EATS,
- master->ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0));
+ ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0));
if (smmu->features & ARM_SMMU_FEAT_E2H) {
/*
@@ -1578,7 +1579,8 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
static void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target,
struct arm_smmu_master *master,
- struct arm_smmu_domain *smmu_domain)
+ struct arm_smmu_domain *smmu_domain,
+ bool ats_enabled)
{
struct arm_smmu_s2_cfg *s2_cfg = &smmu_domain->s2_cfg;
const struct io_pgtable_cfg *pgtbl_cfg =
@@ -1594,7 +1596,7 @@ static void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target,
target->data[1] = cpu_to_le64(
FIELD_PREP(STRTAB_STE_1_EATS,
- master->ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0) |
+ ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0) |
FIELD_PREP(STRTAB_STE_1_SHCFG,
STRTAB_STE_1_SHCFG_INCOMING));
@@ -2470,22 +2472,16 @@ static bool arm_smmu_ats_supported(struct arm_smmu_master *master)
return dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev));
}
-static void arm_smmu_enable_ats(struct arm_smmu_master *master,
- struct arm_smmu_domain *smmu_domain)
+static void arm_smmu_enable_ats(struct arm_smmu_master *master)
{
size_t stu;
struct pci_dev *pdev;
struct arm_smmu_device *smmu = master->smmu;
- /* Don't enable ATS at the endpoint if it's not enabled in the STE */
- if (!master->ats_enabled)
- return;
-
/* Smallest Translation Unit: log2 of the smallest supported granule */
stu = __ffs(smmu->pgsize_bitmap);
pdev = to_pci_dev(master->dev);
- atomic_inc(&smmu_domain->nr_ats_masters);
/*
* ATC invalidation of PASID 0 causes the entire ATC to be flushed.
*/
@@ -2494,22 +2490,6 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master,
dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu);
}
-static void arm_smmu_disable_ats(struct arm_smmu_master *master,
- struct arm_smmu_domain *smmu_domain)
-{
- if (!master->ats_enabled)
- return;
-
- pci_disable_ats(to_pci_dev(master->dev));
- /*
- * Ensure ATS is disabled at the endpoint before we issue the
- * ATC invalidation via the SMMU.
- */
- wmb();
- arm_smmu_atc_inv_master(master);
- atomic_dec(&smmu_domain->nr_ats_masters);
-}
-
static int arm_smmu_enable_pasid(struct arm_smmu_master *master)
{
int ret;
@@ -2573,39 +2553,145 @@ arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
return NULL;
}
-static void arm_smmu_detach_dev(struct arm_smmu_master *master)
+/*
+ * If the domain uses the smmu_domain->devices list return the arm_smmu_domain
+ * structure, otherwise NULL. These domains track attached devices so they can
+ * issue invalidations.
+ */
+static struct arm_smmu_domain *
+to_smmu_domain_devices(struct iommu_domain *domain)
{
- struct iommu_domain *domain = iommu_get_domain_for_dev(master->dev);
+ /* The domain can be NULL only when processing the first attach */
+ if (!domain)
+ return NULL;
+ if (domain->type & __IOMMU_DOMAIN_PAGING)
+ return to_smmu_domain(domain);
+ return NULL;
+}
+
+static void arm_smmu_remove_master_domain(struct arm_smmu_master *master,
+ struct iommu_domain *domain)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain_devices(domain);
struct arm_smmu_master_domain *master_domain;
- struct arm_smmu_domain *smmu_domain;
unsigned long flags;
- if (!domain || !(domain->type & __IOMMU_DOMAIN_PAGING))
+ if (!smmu_domain)
return;
- smmu_domain = to_smmu_domain(domain);
- arm_smmu_disable_ats(master, smmu_domain);
-
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
master_domain = arm_smmu_find_master_domain(smmu_domain, master);
if (master_domain) {
list_del(&master_domain->devices_elm);
kfree(master_domain);
+ if (master->ats_enabled)
+ atomic_dec(&smmu_domain->nr_ats_masters);
}
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
+}
- master->ats_enabled = false;
+struct attach_state {
+ bool want_ats;
+ bool disable_ats;
+};
+
+/*
+ * Prepare to attach a domain to a master. If disable_ats is not set this will
+ * turn on ATS if supported. smmu_domain can be NULL if the domain being
+ * attached does not have a page table and does not require invalidation
+ * tracking.
+ */
+static int arm_smmu_attach_prepare(struct arm_smmu_master *master,
+ struct iommu_domain *domain,
+ struct attach_state *state)
+{
+ struct arm_smmu_domain *smmu_domain =
+ to_smmu_domain_devices(domain);
+ struct arm_smmu_master_domain *master_domain;
+ unsigned long flags;
+
+ /*
+ * arm_smmu_share_asid() must not see two domains pointing to the same
+ * arm_smmu_master_domain contents otherwise it could randomly write one
+ * or the other to the CD.
+ */
+ lockdep_assert_held(&arm_smmu_asid_lock);
+
+ state->want_ats = !state->disable_ats && arm_smmu_ats_supported(master);
+
+ if (smmu_domain) {
+ master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL);
+ if (!master_domain)
+ return -ENOMEM;
+ master_domain->master = master;
+
+ /*
+ * During prepare we want the current smmu_domain and new
+ * smmu_domain to be in the devices list before we change any
+ * HW. This ensures that both domains will send ATS
+ * invalidations to the master until we are done.
+ *
+ * It is tempting to make this list only track masters that are
+ * using ATS, but arm_smmu_share_asid() also uses this to change
+ * the ASID of a domain, unrelated to ATS.
+ *
+ * Notice if we are re-attaching the same domain then the list
+ * will have two identical entries and commit will remove only
+ * one of them.
+ */
+ spin_lock_irqsave(&smmu_domain->devices_lock, flags);
+ if (state->want_ats)
+ atomic_inc(&smmu_domain->nr_ats_masters);
+ list_add(&master_domain->devices_elm, &smmu_domain->devices);
+ spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
+ }
+
+ if (!state->want_ats && master->ats_enabled) {
+ pci_disable_ats(to_pci_dev(master->dev));
+ /*
+ * This is probably overkill, but the config write for disabling
+ * ATS should complete before the STE is configured to generate
+ * UR to avoid AER noise.
+ */
+ wmb();
+ }
+ return 0;
+}
+
+/*
+ * Commit is done after the STE/CD are configured with the EATS setting. It
+ * completes synchronizing the PCI device's ATC and finishes manipulating the
+ * smmu_domain->devices list.
+ */
+static void arm_smmu_attach_commit(struct arm_smmu_master *master,
+ struct attach_state *state)
+{
+ lockdep_assert_held(&arm_smmu_asid_lock);
+
+ if (state->want_ats && !master->ats_enabled) {
+ arm_smmu_enable_ats(master);
+ } else if (master->ats_enabled) {
+ /*
+ * The translation has changed, flush the ATC. At this point the
+ * SMMU is translating for the new domain and both the old&new
+ * domain will issue invalidations.
+ */
+ arm_smmu_atc_inv_master(master);
+ }
+ master->ats_enabled = state->want_ats;
+
+ arm_smmu_remove_master_domain(master,
+ iommu_get_domain_for_dev(master->dev));
}
static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
{
int ret = 0;
- unsigned long flags;
struct arm_smmu_ste target;
struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
struct arm_smmu_device *smmu;
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
- struct arm_smmu_master_domain *master_domain;
+ struct attach_state state = {};
struct arm_smmu_master *master;
struct arm_smmu_cd *cdptr;
@@ -2642,11 +2728,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
return -ENOMEM;
}
- master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL);
- if (!master_domain)
- return -ENOMEM;
- master_domain->master = master;
-
/*
* Prevent arm_smmu_share_asid() from trying to change the ASID
* of either the old or new domain while we are working on it.
@@ -2655,13 +2736,11 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
*/
mutex_lock(&arm_smmu_asid_lock);
- arm_smmu_detach_dev(master);
-
- master->ats_enabled = arm_smmu_ats_supported(master);
-
- spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- list_add(&master_domain->devices_elm, &smmu_domain->devices);
- spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
+ ret = arm_smmu_attach_prepare(master, domain, &state);
+ if (ret) {
+ mutex_unlock(&arm_smmu_asid_lock);
+ return ret;
+ }
switch (smmu_domain->stage) {
case ARM_SMMU_DOMAIN_S1: {
@@ -2670,18 +2749,19 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
&target_cd);
- arm_smmu_make_cdtable_ste(&target, master);
+ arm_smmu_make_cdtable_ste(&target, master, state.want_ats);
arm_smmu_install_ste_for_dev(master, &target);
break;
}
case ARM_SMMU_DOMAIN_S2:
- arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
+ arm_smmu_make_s2_domain_ste(&target, master, smmu_domain,
+ state.want_ats);
arm_smmu_install_ste_for_dev(master, &target);
arm_smmu_clear_cd(master, IOMMU_NO_PASID);
break;
}
- arm_smmu_enable_ats(master, smmu_domain);
+ arm_smmu_attach_commit(master, &state);
mutex_unlock(&arm_smmu_asid_lock);
return 0;
}
@@ -2715,10 +2795,11 @@ void arm_smmu_remove_pasid(struct arm_smmu_master *master,
arm_smmu_clear_cd(master, pasid);
}
-static int arm_smmu_attach_dev_ste(struct device *dev,
- struct arm_smmu_ste *ste)
+static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
+ struct device *dev, struct arm_smmu_ste *ste)
{
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
+ struct attach_state state = {};
if (arm_smmu_master_sva_enabled(master))
return -EBUSY;
@@ -2736,9 +2817,10 @@ static int arm_smmu_attach_dev_ste(struct device *dev,
* the stream (STE.EATS == 0b00), causing F_BAD_ATS_TREQ and
* F_TRANSL_FORBIDDEN events (IHI0070Ea 5.2 Stream Table Entry).
*/
- arm_smmu_detach_dev(master);
-
+ state.disable_ats = true;
+ arm_smmu_attach_prepare(master, domain, &state);
arm_smmu_install_ste_for_dev(master, ste);
+ arm_smmu_attach_commit(master, &state);
mutex_unlock(&arm_smmu_asid_lock);
/*
@@ -2756,7 +2838,7 @@ static int arm_smmu_attach_dev_identity(struct iommu_domain *domain,
struct arm_smmu_ste ste;
arm_smmu_make_bypass_ste(&ste);
- return arm_smmu_attach_dev_ste(dev, &ste);
+ return arm_smmu_attach_dev_ste(domain, dev, &ste);
}
static const struct iommu_domain_ops arm_smmu_identity_ops = {
@@ -2774,7 +2856,7 @@ static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain,
struct arm_smmu_ste ste;
arm_smmu_make_abort_ste(&ste);
- return arm_smmu_attach_dev_ste(dev, &ste);
+ return arm_smmu_attach_dev_ste(domain, dev, &ste);
}
static const struct iommu_domain_ops arm_smmu_blocked_ops = {
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS
@ 2024-03-04 23:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
The core code allows the domain to be changed on the fly without a forced
stop in BLOCKED/IDENTITY. In this flow the driver should just continually
maintain the ATS with no change while the STE is updated.
ATS relies on a linked list smmu_domain->devices to keep track of which
masters have the domain programmed, but this list is also used by
arm_smmu_share_asid(), unrelated to ats.
Create two new functions to encapsulate this combined logic:
arm_smmu_attach_prepare()
<caller generates and sets the STE>
arm_smmu_attach_commit()
The two functions can sequence both enabling ATS and disabling across
the STE store. Have every update of the STE use this sequence.
Installing a S1/S2 domain always enables the ATS if the PCIe device
supports it.
The enable flow is now ordered differently to allow it to be hitless:
1) Add the master to the new smmu_domain->devices list
2) Program the STE
3) Enable ATS at PCIe
4) Remove the master from the old smmu_domain
This flow ensures that invalidations to either domain will generate an ATC
invalidation to the device while the STE is being switched. Thus we don't
need to turn off the ATS anymore for correctness.
The disable flow is the reverse:
1) Disable ATS at PCIe
2) Program the STE
3) Invalidate the ATC
4) Remove the master from the old smmu_domain
Move the nr_ats_masters adjustments to be close to the list
manipulations. It is a count of the number of ATS enabled masters
currently in the list. This is stricly before and after the STE/CD are
revised, and done under the list's spin_lock.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 198 ++++++++++++++------
1 file changed, 140 insertions(+), 58 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 51a1e7198fd1af..45f2190fc31786 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1527,7 +1527,8 @@ static void arm_smmu_make_bypass_ste(struct arm_smmu_ste *target)
}
static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
- struct arm_smmu_master *master)
+ struct arm_smmu_master *master,
+ bool ats_enabled)
{
struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
struct arm_smmu_device *smmu = master->smmu;
@@ -1550,7 +1551,7 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
STRTAB_STE_1_S1STALLD :
0) |
FIELD_PREP(STRTAB_STE_1_EATS,
- master->ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0));
+ ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0));
if (smmu->features & ARM_SMMU_FEAT_E2H) {
/*
@@ -1578,7 +1579,8 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
static void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target,
struct arm_smmu_master *master,
- struct arm_smmu_domain *smmu_domain)
+ struct arm_smmu_domain *smmu_domain,
+ bool ats_enabled)
{
struct arm_smmu_s2_cfg *s2_cfg = &smmu_domain->s2_cfg;
const struct io_pgtable_cfg *pgtbl_cfg =
@@ -1594,7 +1596,7 @@ static void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target,
target->data[1] = cpu_to_le64(
FIELD_PREP(STRTAB_STE_1_EATS,
- master->ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0) |
+ ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0) |
FIELD_PREP(STRTAB_STE_1_SHCFG,
STRTAB_STE_1_SHCFG_INCOMING));
@@ -2470,22 +2472,16 @@ static bool arm_smmu_ats_supported(struct arm_smmu_master *master)
return dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev));
}
-static void arm_smmu_enable_ats(struct arm_smmu_master *master,
- struct arm_smmu_domain *smmu_domain)
+static void arm_smmu_enable_ats(struct arm_smmu_master *master)
{
size_t stu;
struct pci_dev *pdev;
struct arm_smmu_device *smmu = master->smmu;
- /* Don't enable ATS at the endpoint if it's not enabled in the STE */
- if (!master->ats_enabled)
- return;
-
/* Smallest Translation Unit: log2 of the smallest supported granule */
stu = __ffs(smmu->pgsize_bitmap);
pdev = to_pci_dev(master->dev);
- atomic_inc(&smmu_domain->nr_ats_masters);
/*
* ATC invalidation of PASID 0 causes the entire ATC to be flushed.
*/
@@ -2494,22 +2490,6 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master,
dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu);
}
-static void arm_smmu_disable_ats(struct arm_smmu_master *master,
- struct arm_smmu_domain *smmu_domain)
-{
- if (!master->ats_enabled)
- return;
-
- pci_disable_ats(to_pci_dev(master->dev));
- /*
- * Ensure ATS is disabled at the endpoint before we issue the
- * ATC invalidation via the SMMU.
- */
- wmb();
- arm_smmu_atc_inv_master(master);
- atomic_dec(&smmu_domain->nr_ats_masters);
-}
-
static int arm_smmu_enable_pasid(struct arm_smmu_master *master)
{
int ret;
@@ -2573,39 +2553,145 @@ arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
return NULL;
}
-static void arm_smmu_detach_dev(struct arm_smmu_master *master)
+/*
+ * If the domain uses the smmu_domain->devices list return the arm_smmu_domain
+ * structure, otherwise NULL. These domains track attached devices so they can
+ * issue invalidations.
+ */
+static struct arm_smmu_domain *
+to_smmu_domain_devices(struct iommu_domain *domain)
{
- struct iommu_domain *domain = iommu_get_domain_for_dev(master->dev);
+ /* The domain can be NULL only when processing the first attach */
+ if (!domain)
+ return NULL;
+ if (domain->type & __IOMMU_DOMAIN_PAGING)
+ return to_smmu_domain(domain);
+ return NULL;
+}
+
+static void arm_smmu_remove_master_domain(struct arm_smmu_master *master,
+ struct iommu_domain *domain)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain_devices(domain);
struct arm_smmu_master_domain *master_domain;
- struct arm_smmu_domain *smmu_domain;
unsigned long flags;
- if (!domain || !(domain->type & __IOMMU_DOMAIN_PAGING))
+ if (!smmu_domain)
return;
- smmu_domain = to_smmu_domain(domain);
- arm_smmu_disable_ats(master, smmu_domain);
-
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
master_domain = arm_smmu_find_master_domain(smmu_domain, master);
if (master_domain) {
list_del(&master_domain->devices_elm);
kfree(master_domain);
+ if (master->ats_enabled)
+ atomic_dec(&smmu_domain->nr_ats_masters);
}
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
+}
- master->ats_enabled = false;
+struct attach_state {
+ bool want_ats;
+ bool disable_ats;
+};
+
+/*
+ * Prepare to attach a domain to a master. If disable_ats is not set this will
+ * turn on ATS if supported. smmu_domain can be NULL if the domain being
+ * attached does not have a page table and does not require invalidation
+ * tracking.
+ */
+static int arm_smmu_attach_prepare(struct arm_smmu_master *master,
+ struct iommu_domain *domain,
+ struct attach_state *state)
+{
+ struct arm_smmu_domain *smmu_domain =
+ to_smmu_domain_devices(domain);
+ struct arm_smmu_master_domain *master_domain;
+ unsigned long flags;
+
+ /*
+ * arm_smmu_share_asid() must not see two domains pointing to the same
+ * arm_smmu_master_domain contents otherwise it could randomly write one
+ * or the other to the CD.
+ */
+ lockdep_assert_held(&arm_smmu_asid_lock);
+
+ state->want_ats = !state->disable_ats && arm_smmu_ats_supported(master);
+
+ if (smmu_domain) {
+ master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL);
+ if (!master_domain)
+ return -ENOMEM;
+ master_domain->master = master;
+
+ /*
+ * During prepare we want the current smmu_domain and new
+ * smmu_domain to be in the devices list before we change any
+ * HW. This ensures that both domains will send ATS
+ * invalidations to the master until we are done.
+ *
+ * It is tempting to make this list only track masters that are
+ * using ATS, but arm_smmu_share_asid() also uses this to change
+ * the ASID of a domain, unrelated to ATS.
+ *
+ * Notice if we are re-attaching the same domain then the list
+ * will have two identical entries and commit will remove only
+ * one of them.
+ */
+ spin_lock_irqsave(&smmu_domain->devices_lock, flags);
+ if (state->want_ats)
+ atomic_inc(&smmu_domain->nr_ats_masters);
+ list_add(&master_domain->devices_elm, &smmu_domain->devices);
+ spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
+ }
+
+ if (!state->want_ats && master->ats_enabled) {
+ pci_disable_ats(to_pci_dev(master->dev));
+ /*
+ * This is probably overkill, but the config write for disabling
+ * ATS should complete before the STE is configured to generate
+ * UR to avoid AER noise.
+ */
+ wmb();
+ }
+ return 0;
+}
+
+/*
+ * Commit is done after the STE/CD are configured with the EATS setting. It
+ * completes synchronizing the PCI device's ATC and finishes manipulating the
+ * smmu_domain->devices list.
+ */
+static void arm_smmu_attach_commit(struct arm_smmu_master *master,
+ struct attach_state *state)
+{
+ lockdep_assert_held(&arm_smmu_asid_lock);
+
+ if (state->want_ats && !master->ats_enabled) {
+ arm_smmu_enable_ats(master);
+ } else if (master->ats_enabled) {
+ /*
+ * The translation has changed, flush the ATC. At this point the
+ * SMMU is translating for the new domain and both the old&new
+ * domain will issue invalidations.
+ */
+ arm_smmu_atc_inv_master(master);
+ }
+ master->ats_enabled = state->want_ats;
+
+ arm_smmu_remove_master_domain(master,
+ iommu_get_domain_for_dev(master->dev));
}
static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
{
int ret = 0;
- unsigned long flags;
struct arm_smmu_ste target;
struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
struct arm_smmu_device *smmu;
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
- struct arm_smmu_master_domain *master_domain;
+ struct attach_state state = {};
struct arm_smmu_master *master;
struct arm_smmu_cd *cdptr;
@@ -2642,11 +2728,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
return -ENOMEM;
}
- master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL);
- if (!master_domain)
- return -ENOMEM;
- master_domain->master = master;
-
/*
* Prevent arm_smmu_share_asid() from trying to change the ASID
* of either the old or new domain while we are working on it.
@@ -2655,13 +2736,11 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
*/
mutex_lock(&arm_smmu_asid_lock);
- arm_smmu_detach_dev(master);
-
- master->ats_enabled = arm_smmu_ats_supported(master);
-
- spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- list_add(&master_domain->devices_elm, &smmu_domain->devices);
- spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
+ ret = arm_smmu_attach_prepare(master, domain, &state);
+ if (ret) {
+ mutex_unlock(&arm_smmu_asid_lock);
+ return ret;
+ }
switch (smmu_domain->stage) {
case ARM_SMMU_DOMAIN_S1: {
@@ -2670,18 +2749,19 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
&target_cd);
- arm_smmu_make_cdtable_ste(&target, master);
+ arm_smmu_make_cdtable_ste(&target, master, state.want_ats);
arm_smmu_install_ste_for_dev(master, &target);
break;
}
case ARM_SMMU_DOMAIN_S2:
- arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
+ arm_smmu_make_s2_domain_ste(&target, master, smmu_domain,
+ state.want_ats);
arm_smmu_install_ste_for_dev(master, &target);
arm_smmu_clear_cd(master, IOMMU_NO_PASID);
break;
}
- arm_smmu_enable_ats(master, smmu_domain);
+ arm_smmu_attach_commit(master, &state);
mutex_unlock(&arm_smmu_asid_lock);
return 0;
}
@@ -2715,10 +2795,11 @@ void arm_smmu_remove_pasid(struct arm_smmu_master *master,
arm_smmu_clear_cd(master, pasid);
}
-static int arm_smmu_attach_dev_ste(struct device *dev,
- struct arm_smmu_ste *ste)
+static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
+ struct device *dev, struct arm_smmu_ste *ste)
{
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
+ struct attach_state state = {};
if (arm_smmu_master_sva_enabled(master))
return -EBUSY;
@@ -2736,9 +2817,10 @@ static int arm_smmu_attach_dev_ste(struct device *dev,
* the stream (STE.EATS == 0b00), causing F_BAD_ATS_TREQ and
* F_TRANSL_FORBIDDEN events (IHI0070Ea 5.2 Stream Table Entry).
*/
- arm_smmu_detach_dev(master);
-
+ state.disable_ats = true;
+ arm_smmu_attach_prepare(master, domain, &state);
arm_smmu_install_ste_for_dev(master, ste);
+ arm_smmu_attach_commit(master, &state);
mutex_unlock(&arm_smmu_asid_lock);
/*
@@ -2756,7 +2838,7 @@ static int arm_smmu_attach_dev_identity(struct iommu_domain *domain,
struct arm_smmu_ste ste;
arm_smmu_make_bypass_ste(&ste);
- return arm_smmu_attach_dev_ste(dev, &ste);
+ return arm_smmu_attach_dev_ste(domain, dev, &ste);
}
static const struct iommu_domain_ops arm_smmu_identity_ops = {
@@ -2774,7 +2856,7 @@ static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain,
struct arm_smmu_ste ste;
arm_smmu_make_abort_ste(&ste);
- return arm_smmu_attach_dev_ste(dev, &ste);
+ return arm_smmu_attach_dev_ste(domain, dev, &ste);
}
static const struct iommu_domain_ops arm_smmu_blocked_ops = {
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 15/27] iommu/arm-smmu-v3: Add ssid to struct arm_smmu_master_domain
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Prepare to allow a S1 domain to be attached to a PASID as well. Keep track
of the SSID the domain is using on each master in the
arm_smmu_master_domain.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 15 ++++---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 42 +++++++++++++++----
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 ++-
3 files changed, 43 insertions(+), 19 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 9a285b828e8b12..ab9de8e36c45f5 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -47,13 +47,12 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
struct arm_smmu_master *master = master_domain->master;
struct arm_smmu_cd *cdptr;
- /* S1 domains only support RID attachment right now */
- cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
+ cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid);
if (WARN_ON(!cdptr))
continue;
arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
- arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
+ arm_smmu_write_cd_entry(master, master_domain->ssid, cdptr,
&target_cd);
}
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
@@ -293,8 +292,8 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
smmu_domain);
}
- arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), start,
- size);
+ arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), start,
+ size);
}
static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
@@ -331,7 +330,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid);
- arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0);
+ arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0);
smmu_mn->cleared = true;
mutex_unlock(&sva_lock);
@@ -410,8 +409,8 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn)
*/
if (!smmu_mn->cleared) {
arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid);
- arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), 0,
- 0);
+ arm_smmu_atc_inv_domain_sva(smmu_domain,
+ mm_get_enqcmd_pasid(mm), 0, 0);
}
/* Frees smmu_mn */
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 45f2190fc31786..26c6b9f6f34fd3 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2030,8 +2030,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master)
return arm_smmu_cmdq_batch_submit(master->smmu, &cmds);
}
-int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
- unsigned long iova, size_t size)
+static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
+ ioasid_t ssid, unsigned long iova, size_t size)
{
struct arm_smmu_master_domain *master_domain;
int i;
@@ -2059,8 +2059,6 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
if (!atomic_read(&smmu_domain->nr_ats_masters))
return 0;
- arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd);
-
cmds.num = 0;
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
@@ -2071,6 +2069,16 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
if (!master->ats_enabled)
continue;
+ /*
+ * Non-zero ssid means SVA is co-opting the S1 domain to issue
+ * invalidations for SVA PASIDs.
+ */
+ if (ssid != IOMMU_NO_PASID)
+ arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd);
+ else
+ arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size,
+ &cmd);
+
for (i = 0; i < master->num_streams; i++) {
cmd.atc.sid = master->streams[i].id;
arm_smmu_cmdq_batch_add(smmu_domain->smmu, &cmds, &cmd);
@@ -2081,6 +2089,19 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
return arm_smmu_cmdq_batch_submit(smmu_domain->smmu, &cmds);
}
+static int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
+ unsigned long iova, size_t size)
+{
+ return __arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova,
+ size);
+}
+
+int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain,
+ ioasid_t ssid, unsigned long iova, size_t size)
+{
+ return __arm_smmu_atc_inv_domain(smmu_domain, ssid, iova, size);
+}
+
/* IO_PGTABLE API */
static void arm_smmu_tlb_inv_context(void *cookie)
{
@@ -2102,7 +2123,7 @@ static void arm_smmu_tlb_inv_context(void *cookie)
cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid;
arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
}
- arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, 0, 0);
+ arm_smmu_atc_inv_domain(smmu_domain, 0, 0);
}
static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd,
@@ -2200,7 +2221,7 @@ static void arm_smmu_tlb_inv_range_domain(unsigned long iova, size_t size,
* Unfortunately, this can't be leaf-only since we may have
* zapped an entire table.
*/
- arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova, size);
+ arm_smmu_atc_inv_domain(smmu_domain, iova, size);
}
void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
@@ -2539,7 +2560,8 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master)
static struct arm_smmu_master_domain *
arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
- struct arm_smmu_master *master)
+ struct arm_smmu_master *master,
+ ioasid_t ssid)
{
struct arm_smmu_master_domain *master_domain;
@@ -2547,7 +2569,8 @@ arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
list_for_each_entry(master_domain, &smmu_domain->devices,
devices_elm) {
- if (master_domain->master == master)
+ if (master_domain->master == master &&
+ master_domain->ssid == ssid)
return master_domain;
}
return NULL;
@@ -2580,7 +2603,8 @@ static void arm_smmu_remove_master_domain(struct arm_smmu_master *master,
return;
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- master_domain = arm_smmu_find_master_domain(smmu_domain, master);
+ master_domain = arm_smmu_find_master_domain(smmu_domain, master,
+ IOMMU_NO_PASID);
if (master_domain) {
list_del(&master_domain->devices_elm);
kfree(master_domain);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index c4b79bc52991be..98dc5885c48655 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -738,6 +738,7 @@ struct arm_smmu_domain {
struct arm_smmu_master_domain {
struct list_head devices_elm;
struct arm_smmu_master *master;
+ u16 ssid;
};
static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
@@ -769,8 +770,8 @@ void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
size_t granule, bool leaf,
struct arm_smmu_domain *smmu_domain);
bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd);
-int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
- unsigned long iova, size_t size);
+int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain,
+ ioasid_t ssid, unsigned long iova, size_t size);
#ifdef CONFIG_ARM_SMMU_V3_SVA
bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 15/27] iommu/arm-smmu-v3: Add ssid to struct arm_smmu_master_domain
@ 2024-03-04 23:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Prepare to allow a S1 domain to be attached to a PASID as well. Keep track
of the SSID the domain is using on each master in the
arm_smmu_master_domain.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 15 ++++---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 42 +++++++++++++++----
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 ++-
3 files changed, 43 insertions(+), 19 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 9a285b828e8b12..ab9de8e36c45f5 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -47,13 +47,12 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
struct arm_smmu_master *master = master_domain->master;
struct arm_smmu_cd *cdptr;
- /* S1 domains only support RID attachment right now */
- cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
+ cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid);
if (WARN_ON(!cdptr))
continue;
arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
- arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
+ arm_smmu_write_cd_entry(master, master_domain->ssid, cdptr,
&target_cd);
}
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
@@ -293,8 +292,8 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
smmu_domain);
}
- arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), start,
- size);
+ arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), start,
+ size);
}
static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
@@ -331,7 +330,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid);
- arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0);
+ arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0);
smmu_mn->cleared = true;
mutex_unlock(&sva_lock);
@@ -410,8 +409,8 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn)
*/
if (!smmu_mn->cleared) {
arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid);
- arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), 0,
- 0);
+ arm_smmu_atc_inv_domain_sva(smmu_domain,
+ mm_get_enqcmd_pasid(mm), 0, 0);
}
/* Frees smmu_mn */
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 45f2190fc31786..26c6b9f6f34fd3 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2030,8 +2030,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master)
return arm_smmu_cmdq_batch_submit(master->smmu, &cmds);
}
-int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
- unsigned long iova, size_t size)
+static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
+ ioasid_t ssid, unsigned long iova, size_t size)
{
struct arm_smmu_master_domain *master_domain;
int i;
@@ -2059,8 +2059,6 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
if (!atomic_read(&smmu_domain->nr_ats_masters))
return 0;
- arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd);
-
cmds.num = 0;
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
@@ -2071,6 +2069,16 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
if (!master->ats_enabled)
continue;
+ /*
+ * Non-zero ssid means SVA is co-opting the S1 domain to issue
+ * invalidations for SVA PASIDs.
+ */
+ if (ssid != IOMMU_NO_PASID)
+ arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd);
+ else
+ arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size,
+ &cmd);
+
for (i = 0; i < master->num_streams; i++) {
cmd.atc.sid = master->streams[i].id;
arm_smmu_cmdq_batch_add(smmu_domain->smmu, &cmds, &cmd);
@@ -2081,6 +2089,19 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
return arm_smmu_cmdq_batch_submit(smmu_domain->smmu, &cmds);
}
+static int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
+ unsigned long iova, size_t size)
+{
+ return __arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova,
+ size);
+}
+
+int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain,
+ ioasid_t ssid, unsigned long iova, size_t size)
+{
+ return __arm_smmu_atc_inv_domain(smmu_domain, ssid, iova, size);
+}
+
/* IO_PGTABLE API */
static void arm_smmu_tlb_inv_context(void *cookie)
{
@@ -2102,7 +2123,7 @@ static void arm_smmu_tlb_inv_context(void *cookie)
cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid;
arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
}
- arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, 0, 0);
+ arm_smmu_atc_inv_domain(smmu_domain, 0, 0);
}
static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd,
@@ -2200,7 +2221,7 @@ static void arm_smmu_tlb_inv_range_domain(unsigned long iova, size_t size,
* Unfortunately, this can't be leaf-only since we may have
* zapped an entire table.
*/
- arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova, size);
+ arm_smmu_atc_inv_domain(smmu_domain, iova, size);
}
void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
@@ -2539,7 +2560,8 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master)
static struct arm_smmu_master_domain *
arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
- struct arm_smmu_master *master)
+ struct arm_smmu_master *master,
+ ioasid_t ssid)
{
struct arm_smmu_master_domain *master_domain;
@@ -2547,7 +2569,8 @@ arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
list_for_each_entry(master_domain, &smmu_domain->devices,
devices_elm) {
- if (master_domain->master == master)
+ if (master_domain->master == master &&
+ master_domain->ssid == ssid)
return master_domain;
}
return NULL;
@@ -2580,7 +2603,8 @@ static void arm_smmu_remove_master_domain(struct arm_smmu_master *master,
return;
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- master_domain = arm_smmu_find_master_domain(smmu_domain, master);
+ master_domain = arm_smmu_find_master_domain(smmu_domain, master,
+ IOMMU_NO_PASID);
if (master_domain) {
list_del(&master_domain->devices_elm);
kfree(master_domain);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index c4b79bc52991be..98dc5885c48655 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -738,6 +738,7 @@ struct arm_smmu_domain {
struct arm_smmu_master_domain {
struct list_head devices_elm;
struct arm_smmu_master *master;
+ u16 ssid;
};
static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
@@ -769,8 +770,8 @@ void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
size_t granule, bool leaf,
struct arm_smmu_domain *smmu_domain);
bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd);
-int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
- unsigned long iova, size_t size);
+int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain,
+ ioasid_t ssid, unsigned long iova, size_t size);
#ifdef CONFIG_ARM_SMMU_V3_SVA
bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 16/27] iommu/arm-smmu-v3: Keep track of valid CD entries in the cd_table
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
We no longer need a master->sva_enable to control what attaches are
allowed.
Instead keep track inside the cd_table how many valid CD entries exist,
and if the RID has a valid entry.
Replace all the attach focused master->sva_enabled tests with a check if
the CD has valid entries (or not). If there are any valid entries then the
CD table must be currently programmed to the STE.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 5 +---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 ++++++++++---------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 10 +++++++
3 files changed, 25 insertions(+), 16 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index ab9de8e36c45f5..82b9c4d4061c3d 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -433,9 +433,6 @@ static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm,
if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1)
return -ENODEV;
- if (!master || !master->sva_enabled)
- return -ENODEV;
-
bond = kzalloc(sizeof(*bond), GFP_KERNEL);
if (!bond)
return -ENOMEM;
@@ -638,7 +635,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
struct mm_struct *mm = domain->mm;
struct arm_smmu_cd target;
- if (mm_get_enqcmd_pasid(mm) != id)
+ if (mm_get_enqcmd_pasid(mm) != id || !master->cd_table.used_sid)
return -EINVAL;
if (!arm_smmu_get_cd_ptr(master, id))
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 26c6b9f6f34fd3..fc19585bf192c6 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1285,6 +1285,8 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
struct arm_smmu_cd *cdptr,
const struct arm_smmu_cd *target)
{
+ bool target_valid = target->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
+ bool cur_valid = cdptr->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
struct arm_smmu_cd_writer cd_writer = {
.writer = {
.ops = &arm_smmu_cd_writer_ops,
@@ -1293,6 +1295,15 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
.ssid = ssid,
};
+ if (cur_valid != target_valid) {
+ if (cur_valid)
+ master->cd_table.used_ssids--;
+ else
+ master->cd_table.used_ssids++;
+ }
+ if (ssid == IOMMU_NO_PASID)
+ master->cd_table.used_sid = target_valid;
+
arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
}
@@ -2725,16 +2736,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
master = dev_iommu_priv_get(dev);
smmu = master->smmu;
- /*
- * Checking that SVA is disabled ensures that this device isn't bound to
- * any mm, and can be safely detached from its old domain. Bonds cannot
- * be removed concurrently since we're holding the group mutex.
- */
- if (arm_smmu_master_sva_enabled(master)) {
- dev_err(dev, "cannot attach - SVA enabled\n");
- return -EBUSY;
- }
-
mutex_lock(&smmu_domain->init_mutex);
if (!smmu_domain->smmu) {
@@ -2750,7 +2751,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
if (!cdptr)
return -ENOMEM;
- }
+ } else if (arm_smmu_ssids_in_use(&master->cd_table))
+ return -EBUSY;
/*
* Prevent arm_smmu_share_asid() from trying to change the ASID
@@ -2825,7 +2827,7 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct attach_state state = {};
- if (arm_smmu_master_sva_enabled(master))
+ if (arm_smmu_ssids_in_use(&master->cd_table))
return -EBUSY;
/*
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 98dc5885c48655..7e1f6af4ce4e79 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -602,11 +602,21 @@ struct arm_smmu_ctx_desc_cfg {
dma_addr_t cdtab_dma;
struct arm_smmu_l1_ctx_desc *l1_desc;
unsigned int num_l1_ents;
+ unsigned int used_ssids;
+ bool used_sid;
u8 s1fmt;
/* log2 of the maximum number of CDs supported by this table */
u8 s1cdmax;
};
+/* True if the cd table has SSIDS > 0 in use. */
+static inline bool arm_smmu_ssids_in_use(struct arm_smmu_ctx_desc_cfg *cd_table)
+{
+ if (cd_table->used_sid)
+ return cd_table->used_ssids > 1;
+ return cd_table->used_ssids;
+}
+
struct arm_smmu_s2_cfg {
u16 vmid;
};
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 16/27] iommu/arm-smmu-v3: Keep track of valid CD entries in the cd_table
@ 2024-03-04 23:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
We no longer need a master->sva_enable to control what attaches are
allowed.
Instead keep track inside the cd_table how many valid CD entries exist,
and if the RID has a valid entry.
Replace all the attach focused master->sva_enabled tests with a check if
the CD has valid entries (or not). If there are any valid entries then the
CD table must be currently programmed to the STE.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 5 +---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 ++++++++++---------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 10 +++++++
3 files changed, 25 insertions(+), 16 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index ab9de8e36c45f5..82b9c4d4061c3d 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -433,9 +433,6 @@ static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm,
if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1)
return -ENODEV;
- if (!master || !master->sva_enabled)
- return -ENODEV;
-
bond = kzalloc(sizeof(*bond), GFP_KERNEL);
if (!bond)
return -ENOMEM;
@@ -638,7 +635,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
struct mm_struct *mm = domain->mm;
struct arm_smmu_cd target;
- if (mm_get_enqcmd_pasid(mm) != id)
+ if (mm_get_enqcmd_pasid(mm) != id || !master->cd_table.used_sid)
return -EINVAL;
if (!arm_smmu_get_cd_ptr(master, id))
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 26c6b9f6f34fd3..fc19585bf192c6 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1285,6 +1285,8 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
struct arm_smmu_cd *cdptr,
const struct arm_smmu_cd *target)
{
+ bool target_valid = target->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
+ bool cur_valid = cdptr->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
struct arm_smmu_cd_writer cd_writer = {
.writer = {
.ops = &arm_smmu_cd_writer_ops,
@@ -1293,6 +1295,15 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
.ssid = ssid,
};
+ if (cur_valid != target_valid) {
+ if (cur_valid)
+ master->cd_table.used_ssids--;
+ else
+ master->cd_table.used_ssids++;
+ }
+ if (ssid == IOMMU_NO_PASID)
+ master->cd_table.used_sid = target_valid;
+
arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
}
@@ -2725,16 +2736,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
master = dev_iommu_priv_get(dev);
smmu = master->smmu;
- /*
- * Checking that SVA is disabled ensures that this device isn't bound to
- * any mm, and can be safely detached from its old domain. Bonds cannot
- * be removed concurrently since we're holding the group mutex.
- */
- if (arm_smmu_master_sva_enabled(master)) {
- dev_err(dev, "cannot attach - SVA enabled\n");
- return -EBUSY;
- }
-
mutex_lock(&smmu_domain->init_mutex);
if (!smmu_domain->smmu) {
@@ -2750,7 +2751,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
if (!cdptr)
return -ENOMEM;
- }
+ } else if (arm_smmu_ssids_in_use(&master->cd_table))
+ return -EBUSY;
/*
* Prevent arm_smmu_share_asid() from trying to change the ASID
@@ -2825,7 +2827,7 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct attach_state state = {};
- if (arm_smmu_master_sva_enabled(master))
+ if (arm_smmu_ssids_in_use(&master->cd_table))
return -EBUSY;
/*
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 98dc5885c48655..7e1f6af4ce4e79 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -602,11 +602,21 @@ struct arm_smmu_ctx_desc_cfg {
dma_addr_t cdtab_dma;
struct arm_smmu_l1_ctx_desc *l1_desc;
unsigned int num_l1_ents;
+ unsigned int used_ssids;
+ bool used_sid;
u8 s1fmt;
/* log2 of the maximum number of CDs supported by this table */
u8 s1cdmax;
};
+/* True if the cd table has SSIDS > 0 in use. */
+static inline bool arm_smmu_ssids_in_use(struct arm_smmu_ctx_desc_cfg *cd_table)
+{
+ if (cd_table->used_sid)
+ return cd_table->used_ssids > 1;
+ return cd_table->used_ssids;
+}
+
struct arm_smmu_s2_cfg {
u16 vmid;
};
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 17/27] iommu/arm-smmu-v3: Thread SSID through the arm_smmu_attach_*() interface
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Allow creating and managing arm_smmu_mater_domain's with a non-zero SSID
through the arm_smmu_attach_*() family of functions. This triggers ATC
invalidation for the correct SSID in PASID cases and tracks the
per-attachment SSID in the struct arm_smmu_master_domain.
Generalize arm_smmu_attach_remove() to be able to remove SSID's as well by
ensuring the ATC for the PASID is flushed properly.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 28 +++++++++++++--------
1 file changed, 17 insertions(+), 11 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index fc19585bf192c6..dd7f841cd19b3c 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2024,13 +2024,14 @@ arm_smmu_atc_inv_to_cmd(int ssid, unsigned long iova, size_t size,
cmd->atc.size = log2_span;
}
-static int arm_smmu_atc_inv_master(struct arm_smmu_master *master)
+static int arm_smmu_atc_inv_master(struct arm_smmu_master *master,
+ ioasid_t ssid)
{
int i;
struct arm_smmu_cmdq_ent cmd;
struct arm_smmu_cmdq_batch cmds;
- arm_smmu_atc_inv_to_cmd(IOMMU_NO_PASID, 0, 0, &cmd);
+ arm_smmu_atc_inv_to_cmd(ssid, 0, 0, &cmd);
cmds.num = 0;
for (i = 0; i < master->num_streams; i++) {
@@ -2517,7 +2518,7 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master)
/*
* ATC invalidation of PASID 0 causes the entire ATC to be flushed.
*/
- arm_smmu_atc_inv_master(master);
+ arm_smmu_atc_inv_master(master, IOMMU_NO_PASID);
if (pci_enable_ats(pdev, stu))
dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu);
}
@@ -2604,7 +2605,8 @@ to_smmu_domain_devices(struct iommu_domain *domain)
}
static void arm_smmu_remove_master_domain(struct arm_smmu_master *master,
- struct iommu_domain *domain)
+ struct iommu_domain *domain,
+ ioasid_t ssid)
{
struct arm_smmu_domain *smmu_domain = to_smmu_domain_devices(domain);
struct arm_smmu_master_domain *master_domain;
@@ -2614,8 +2616,7 @@ static void arm_smmu_remove_master_domain(struct arm_smmu_master *master,
return;
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- master_domain = arm_smmu_find_master_domain(smmu_domain, master,
- IOMMU_NO_PASID);
+ master_domain = arm_smmu_find_master_domain(smmu_domain, master, ssid);
if (master_domain) {
list_del(&master_domain->devices_elm);
kfree(master_domain);
@@ -2628,6 +2629,7 @@ static void arm_smmu_remove_master_domain(struct arm_smmu_master *master,
struct attach_state {
bool want_ats;
bool disable_ats;
+ ioasid_t ssid;
};
/*
@@ -2659,6 +2661,7 @@ static int arm_smmu_attach_prepare(struct arm_smmu_master *master,
if (!master_domain)
return -ENOMEM;
master_domain->master = master;
+ master_domain->ssid = state->ssid;
/*
* During prepare we want the current smmu_domain and new
@@ -2711,12 +2714,15 @@ static void arm_smmu_attach_commit(struct arm_smmu_master *master,
* SMMU is translating for the new domain and both the old&new
* domain will issue invalidations.
*/
- arm_smmu_atc_inv_master(master);
+ if (state->want_ats)
+ arm_smmu_atc_inv_master(master, state->ssid);
+ else
+ arm_smmu_atc_inv_master(master, IOMMU_NO_PASID);
}
master->ats_enabled = state->want_ats;
- arm_smmu_remove_master_domain(master,
- iommu_get_domain_for_dev(master->dev));
+ arm_smmu_remove_master_domain(
+ master, iommu_get_domain_for_dev(master->dev), state->ssid);
}
static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
@@ -2726,7 +2732,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
struct arm_smmu_device *smmu;
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
- struct attach_state state = {};
+ struct attach_state state = {.ssid = IOMMU_NO_PASID};
struct arm_smmu_master *master;
struct arm_smmu_cd *cdptr;
@@ -2825,7 +2831,7 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
struct device *dev, struct arm_smmu_ste *ste)
{
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
- struct attach_state state = {};
+ struct attach_state state = {.ssid = IOMMU_NO_PASID};
if (arm_smmu_ssids_in_use(&master->cd_table))
return -EBUSY;
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 17/27] iommu/arm-smmu-v3: Thread SSID through the arm_smmu_attach_*() interface
@ 2024-03-04 23:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Allow creating and managing arm_smmu_mater_domain's with a non-zero SSID
through the arm_smmu_attach_*() family of functions. This triggers ATC
invalidation for the correct SSID in PASID cases and tracks the
per-attachment SSID in the struct arm_smmu_master_domain.
Generalize arm_smmu_attach_remove() to be able to remove SSID's as well by
ensuring the ATC for the PASID is flushed properly.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 28 +++++++++++++--------
1 file changed, 17 insertions(+), 11 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index fc19585bf192c6..dd7f841cd19b3c 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2024,13 +2024,14 @@ arm_smmu_atc_inv_to_cmd(int ssid, unsigned long iova, size_t size,
cmd->atc.size = log2_span;
}
-static int arm_smmu_atc_inv_master(struct arm_smmu_master *master)
+static int arm_smmu_atc_inv_master(struct arm_smmu_master *master,
+ ioasid_t ssid)
{
int i;
struct arm_smmu_cmdq_ent cmd;
struct arm_smmu_cmdq_batch cmds;
- arm_smmu_atc_inv_to_cmd(IOMMU_NO_PASID, 0, 0, &cmd);
+ arm_smmu_atc_inv_to_cmd(ssid, 0, 0, &cmd);
cmds.num = 0;
for (i = 0; i < master->num_streams; i++) {
@@ -2517,7 +2518,7 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master)
/*
* ATC invalidation of PASID 0 causes the entire ATC to be flushed.
*/
- arm_smmu_atc_inv_master(master);
+ arm_smmu_atc_inv_master(master, IOMMU_NO_PASID);
if (pci_enable_ats(pdev, stu))
dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu);
}
@@ -2604,7 +2605,8 @@ to_smmu_domain_devices(struct iommu_domain *domain)
}
static void arm_smmu_remove_master_domain(struct arm_smmu_master *master,
- struct iommu_domain *domain)
+ struct iommu_domain *domain,
+ ioasid_t ssid)
{
struct arm_smmu_domain *smmu_domain = to_smmu_domain_devices(domain);
struct arm_smmu_master_domain *master_domain;
@@ -2614,8 +2616,7 @@ static void arm_smmu_remove_master_domain(struct arm_smmu_master *master,
return;
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
- master_domain = arm_smmu_find_master_domain(smmu_domain, master,
- IOMMU_NO_PASID);
+ master_domain = arm_smmu_find_master_domain(smmu_domain, master, ssid);
if (master_domain) {
list_del(&master_domain->devices_elm);
kfree(master_domain);
@@ -2628,6 +2629,7 @@ static void arm_smmu_remove_master_domain(struct arm_smmu_master *master,
struct attach_state {
bool want_ats;
bool disable_ats;
+ ioasid_t ssid;
};
/*
@@ -2659,6 +2661,7 @@ static int arm_smmu_attach_prepare(struct arm_smmu_master *master,
if (!master_domain)
return -ENOMEM;
master_domain->master = master;
+ master_domain->ssid = state->ssid;
/*
* During prepare we want the current smmu_domain and new
@@ -2711,12 +2714,15 @@ static void arm_smmu_attach_commit(struct arm_smmu_master *master,
* SMMU is translating for the new domain and both the old&new
* domain will issue invalidations.
*/
- arm_smmu_atc_inv_master(master);
+ if (state->want_ats)
+ arm_smmu_atc_inv_master(master, state->ssid);
+ else
+ arm_smmu_atc_inv_master(master, IOMMU_NO_PASID);
}
master->ats_enabled = state->want_ats;
- arm_smmu_remove_master_domain(master,
- iommu_get_domain_for_dev(master->dev));
+ arm_smmu_remove_master_domain(
+ master, iommu_get_domain_for_dev(master->dev), state->ssid);
}
static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
@@ -2726,7 +2732,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
struct arm_smmu_device *smmu;
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
- struct attach_state state = {};
+ struct attach_state state = {.ssid = IOMMU_NO_PASID};
struct arm_smmu_master *master;
struct arm_smmu_cd *cdptr;
@@ -2825,7 +2831,7 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
struct device *dev, struct arm_smmu_ste *ste)
{
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
- struct attach_state state = {};
+ struct attach_state state = {.ssid = IOMMU_NO_PASID};
if (arm_smmu_ssids_in_use(&master->cd_table))
return -EBUSY;
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 18/27] iommu/arm-smmu-v3: Make SVA allocate a normal arm_smmu_domain
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Currently the SVA domain is a naked struct iommu_domain, allocate a struct
arm_smmu_domain instead.
This is necessary to be able to use the struct arm_master_domain
mechanism.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 19 ++++++-----
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 34 +++++++++++--------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++-
3 files changed, 33 insertions(+), 24 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 82b9c4d4061c3d..d633316f2e45bc 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -654,7 +654,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
{
- kfree(domain);
+ kfree(to_smmu_domain(domain));
}
static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
@@ -662,14 +662,17 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
.free = arm_smmu_sva_domain_free
};
-struct iommu_domain *arm_smmu_sva_domain_alloc(void)
+struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned type)
{
- struct iommu_domain *domain;
+ struct arm_smmu_domain *smmu_domain;
- domain = kzalloc(sizeof(*domain), GFP_KERNEL);
- if (!domain)
- return NULL;
- domain->ops = &arm_smmu_sva_domain_ops;
+ if (type != IOMMU_DOMAIN_SVA)
+ return ERR_PTR(-EOPNOTSUPP);
- return domain;
+ smmu_domain = arm_smmu_domain_alloc();
+ if (IS_ERR(smmu_domain))
+ return ERR_CAST(smmu_domain);
+ smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
+
+ return &smmu_domain->domain;
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index dd7f841cd19b3c..2db2b822292a87 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2291,23 +2291,10 @@ static bool arm_smmu_capable(struct device *dev, enum iommu_cap cap)
}
}
-static struct iommu_domain *arm_smmu_domain_alloc(unsigned type)
-{
-
- if (type == IOMMU_DOMAIN_SVA)
- return arm_smmu_sva_domain_alloc();
- return ERR_PTR(-EOPNOTSUPP);
-}
-
-static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
+struct arm_smmu_domain *arm_smmu_domain_alloc(void)
{
struct arm_smmu_domain *smmu_domain;
- /*
- * Allocate the domain and initialise some of its data structures.
- * We can't really do anything meaningful until we've added a
- * master.
- */
smmu_domain = kzalloc(sizeof(*smmu_domain), GFP_KERNEL);
if (!smmu_domain)
return ERR_PTR(-ENOMEM);
@@ -2317,6 +2304,23 @@ static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
spin_lock_init(&smmu_domain->devices_lock);
INIT_LIST_HEAD(&smmu_domain->mmu_notifiers);
+ return smmu_domain;
+}
+
+static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
+{
+ struct arm_smmu_domain *smmu_domain;
+
+ smmu_domain = arm_smmu_domain_alloc();
+ if (IS_ERR(smmu_domain))
+ return ERR_CAST(smmu_domain);
+
+ /*
+ * Allocate the domain and initialise some of its data structures.
+ * We can't really do anything meaningful until we've added a
+ * master.
+ */
+
if (dev) {
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
int ret;
@@ -3288,7 +3292,7 @@ static struct iommu_ops arm_smmu_ops = {
.identity_domain = &arm_smmu_identity_domain,
.blocked_domain = &arm_smmu_blocked_domain,
.capable = arm_smmu_capable,
- .domain_alloc = arm_smmu_domain_alloc,
+ .domain_alloc = arm_smmu_sva_domain_alloc,
.domain_alloc_paging = arm_smmu_domain_alloc_paging,
.probe_device = arm_smmu_probe_device,
.release_device = arm_smmu_release_device,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 7e1f6af4ce4e79..c47e07d695bef2 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -759,6 +759,8 @@ static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
extern struct xarray arm_smmu_asid_xa;
extern struct mutex arm_smmu_asid_lock;
+struct arm_smmu_domain *arm_smmu_domain_alloc(void);
+
void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
u32 ssid);
@@ -791,7 +793,7 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master);
int arm_smmu_master_disable_sva(struct arm_smmu_master *master);
bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master);
void arm_smmu_sva_notifier_synchronize(void);
-struct iommu_domain *arm_smmu_sva_domain_alloc(void);
+struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned int type);
void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
struct device *dev, ioasid_t id);
#else /* CONFIG_ARM_SMMU_V3_SVA */
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 18/27] iommu/arm-smmu-v3: Make SVA allocate a normal arm_smmu_domain
@ 2024-03-04 23:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Currently the SVA domain is a naked struct iommu_domain, allocate a struct
arm_smmu_domain instead.
This is necessary to be able to use the struct arm_master_domain
mechanism.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 19 ++++++-----
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 34 +++++++++++--------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++-
3 files changed, 33 insertions(+), 24 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 82b9c4d4061c3d..d633316f2e45bc 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -654,7 +654,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
{
- kfree(domain);
+ kfree(to_smmu_domain(domain));
}
static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
@@ -662,14 +662,17 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
.free = arm_smmu_sva_domain_free
};
-struct iommu_domain *arm_smmu_sva_domain_alloc(void)
+struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned type)
{
- struct iommu_domain *domain;
+ struct arm_smmu_domain *smmu_domain;
- domain = kzalloc(sizeof(*domain), GFP_KERNEL);
- if (!domain)
- return NULL;
- domain->ops = &arm_smmu_sva_domain_ops;
+ if (type != IOMMU_DOMAIN_SVA)
+ return ERR_PTR(-EOPNOTSUPP);
- return domain;
+ smmu_domain = arm_smmu_domain_alloc();
+ if (IS_ERR(smmu_domain))
+ return ERR_CAST(smmu_domain);
+ smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
+
+ return &smmu_domain->domain;
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index dd7f841cd19b3c..2db2b822292a87 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2291,23 +2291,10 @@ static bool arm_smmu_capable(struct device *dev, enum iommu_cap cap)
}
}
-static struct iommu_domain *arm_smmu_domain_alloc(unsigned type)
-{
-
- if (type == IOMMU_DOMAIN_SVA)
- return arm_smmu_sva_domain_alloc();
- return ERR_PTR(-EOPNOTSUPP);
-}
-
-static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
+struct arm_smmu_domain *arm_smmu_domain_alloc(void)
{
struct arm_smmu_domain *smmu_domain;
- /*
- * Allocate the domain and initialise some of its data structures.
- * We can't really do anything meaningful until we've added a
- * master.
- */
smmu_domain = kzalloc(sizeof(*smmu_domain), GFP_KERNEL);
if (!smmu_domain)
return ERR_PTR(-ENOMEM);
@@ -2317,6 +2304,23 @@ static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
spin_lock_init(&smmu_domain->devices_lock);
INIT_LIST_HEAD(&smmu_domain->mmu_notifiers);
+ return smmu_domain;
+}
+
+static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
+{
+ struct arm_smmu_domain *smmu_domain;
+
+ smmu_domain = arm_smmu_domain_alloc();
+ if (IS_ERR(smmu_domain))
+ return ERR_CAST(smmu_domain);
+
+ /*
+ * Allocate the domain and initialise some of its data structures.
+ * We can't really do anything meaningful until we've added a
+ * master.
+ */
+
if (dev) {
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
int ret;
@@ -3288,7 +3292,7 @@ static struct iommu_ops arm_smmu_ops = {
.identity_domain = &arm_smmu_identity_domain,
.blocked_domain = &arm_smmu_blocked_domain,
.capable = arm_smmu_capable,
- .domain_alloc = arm_smmu_domain_alloc,
+ .domain_alloc = arm_smmu_sva_domain_alloc,
.domain_alloc_paging = arm_smmu_domain_alloc_paging,
.probe_device = arm_smmu_probe_device,
.release_device = arm_smmu_release_device,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 7e1f6af4ce4e79..c47e07d695bef2 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -759,6 +759,8 @@ static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
extern struct xarray arm_smmu_asid_xa;
extern struct mutex arm_smmu_asid_lock;
+struct arm_smmu_domain *arm_smmu_domain_alloc(void);
+
void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
u32 ssid);
@@ -791,7 +793,7 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master);
int arm_smmu_master_disable_sva(struct arm_smmu_master *master);
bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master);
void arm_smmu_sva_notifier_synchronize(void);
-struct iommu_domain *arm_smmu_sva_domain_alloc(void);
+struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned int type);
void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
struct device *dev, ioasid_t id);
#else /* CONFIG_ARM_SMMU_V3_SVA */
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 19/27] iommu/arm-smmu-v3: Keep track of arm_smmu_master_domain for SVA
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Currently the smmu_domain->devices list is unused for SVA domains.
Fill it in with the SSID and master of every arm_smmu_set_pasid()
using the same logic as the RID attach.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 23 +++++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 2db2b822292a87..6d15fe3a160acf 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2603,7 +2603,8 @@ to_smmu_domain_devices(struct iommu_domain *domain)
/* The domain can be NULL only when processing the first attach */
if (!domain)
return NULL;
- if (domain->type & __IOMMU_DOMAIN_PAGING)
+ if ((domain->type & __IOMMU_DOMAIN_PAGING) ||
+ domain->type == IOMMU_DOMAIN_SVA)
return to_smmu_domain(domain);
return NULL;
}
@@ -2813,7 +2814,9 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
const struct arm_smmu_cd *cd)
{
+ struct attach_state state = {.ssid = pasid};
struct arm_smmu_cd *cdptr;
+ int ret;
if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)))
return -ENODEV;
@@ -2821,14 +2824,30 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
cdptr = arm_smmu_get_cd_ptr(master, pasid);
if (!cdptr)
return -ENOMEM;
+
+ mutex_lock(&arm_smmu_asid_lock);
+ ret = arm_smmu_attach_prepare(master, &smmu_domain->domain, &state);
+ if (ret)
+ goto out_unlock;
+
arm_smmu_write_cd_entry(master, pasid, cdptr, cd);
- return 0;
+
+ arm_smmu_attach_commit(master, &state);
+
+out_unlock:
+ mutex_unlock(&arm_smmu_asid_lock);
+ return ret;
}
void arm_smmu_remove_pasid(struct arm_smmu_master *master,
struct arm_smmu_domain *smmu_domain, ioasid_t pasid)
{
+ mutex_lock(&arm_smmu_asid_lock);
arm_smmu_clear_cd(master, pasid);
+ if (master->ats_enabled)
+ arm_smmu_atc_inv_master(master, pasid);
+ arm_smmu_remove_master_domain(master, &smmu_domain->domain, pasid);
+ mutex_unlock(&arm_smmu_asid_lock);
}
static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 19/27] iommu/arm-smmu-v3: Keep track of arm_smmu_master_domain for SVA
@ 2024-03-04 23:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Currently the smmu_domain->devices list is unused for SVA domains.
Fill it in with the SSID and master of every arm_smmu_set_pasid()
using the same logic as the RID attach.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 23 +++++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 2db2b822292a87..6d15fe3a160acf 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2603,7 +2603,8 @@ to_smmu_domain_devices(struct iommu_domain *domain)
/* The domain can be NULL only when processing the first attach */
if (!domain)
return NULL;
- if (domain->type & __IOMMU_DOMAIN_PAGING)
+ if ((domain->type & __IOMMU_DOMAIN_PAGING) ||
+ domain->type == IOMMU_DOMAIN_SVA)
return to_smmu_domain(domain);
return NULL;
}
@@ -2813,7 +2814,9 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
const struct arm_smmu_cd *cd)
{
+ struct attach_state state = {.ssid = pasid};
struct arm_smmu_cd *cdptr;
+ int ret;
if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)))
return -ENODEV;
@@ -2821,14 +2824,30 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
cdptr = arm_smmu_get_cd_ptr(master, pasid);
if (!cdptr)
return -ENOMEM;
+
+ mutex_lock(&arm_smmu_asid_lock);
+ ret = arm_smmu_attach_prepare(master, &smmu_domain->domain, &state);
+ if (ret)
+ goto out_unlock;
+
arm_smmu_write_cd_entry(master, pasid, cdptr, cd);
- return 0;
+
+ arm_smmu_attach_commit(master, &state);
+
+out_unlock:
+ mutex_unlock(&arm_smmu_asid_lock);
+ return ret;
}
void arm_smmu_remove_pasid(struct arm_smmu_master *master,
struct arm_smmu_domain *smmu_domain, ioasid_t pasid)
{
+ mutex_lock(&arm_smmu_asid_lock);
arm_smmu_clear_cd(master, pasid);
+ if (master->ats_enabled)
+ arm_smmu_atc_inv_master(master, pasid);
+ arm_smmu_remove_master_domain(master, &smmu_domain->domain, pasid);
+ mutex_unlock(&arm_smmu_asid_lock);
}
static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 20/27] iommu: Add ops->domain_alloc_sva()
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Make a new op that receives the device and the mm_struct that the SVA
domain should be created for. Unlike domain_alloc_paging() the dev
argument is never NULL here.
This allows drivers to fully initialize the SVA domain and allocate the
mmu_notifier during allocation. It allows the notifier lifetime to follow
the lifetime of the iommu_domain.
Since we have only one call site, upgrade the new op to return ERR_PTR
instead of NULL.
Change SMMUv3 to use the new op.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 11 +++++++----
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 2 +-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 +++++-
drivers/iommu/iommu-sva.c | 4 ++--
drivers/iommu/iommu.c | 12 +++++++++---
include/linux/iommu.h | 3 +++
6 files changed, 27 insertions(+), 11 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index d633316f2e45bc..30b1cf587efffb 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -662,17 +662,20 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
.free = arm_smmu_sva_domain_free
};
-struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned type)
+struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
+ struct mm_struct *mm)
{
+ struct arm_smmu_master *master = dev_iommu_priv_get(dev);
+ struct arm_smmu_device *smmu = master->smmu;
struct arm_smmu_domain *smmu_domain;
- if (type != IOMMU_DOMAIN_SVA)
- return ERR_PTR(-EOPNOTSUPP);
-
smmu_domain = arm_smmu_domain_alloc();
if (IS_ERR(smmu_domain))
return ERR_CAST(smmu_domain);
+
+ smmu_domain->domain.type = IOMMU_DOMAIN_SVA;
smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
+ smmu_domain->smmu = smmu;
return &smmu_domain->domain;
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 6d15fe3a160acf..a255a02a5fc8a9 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -3311,8 +3311,8 @@ static struct iommu_ops arm_smmu_ops = {
.identity_domain = &arm_smmu_identity_domain,
.blocked_domain = &arm_smmu_blocked_domain,
.capable = arm_smmu_capable,
- .domain_alloc = arm_smmu_sva_domain_alloc,
.domain_alloc_paging = arm_smmu_domain_alloc_paging,
+ .domain_alloc_sva = arm_smmu_sva_domain_alloc,
.probe_device = arm_smmu_probe_device,
.release_device = arm_smmu_release_device,
.device_group = arm_smmu_device_group,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index c47e07d695bef2..f62bd2ef36a18d 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -793,7 +793,8 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master);
int arm_smmu_master_disable_sva(struct arm_smmu_master *master);
bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master);
void arm_smmu_sva_notifier_synchronize(void);
-struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned int type);
+struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
+ struct mm_struct *mm);
void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
struct device *dev, ioasid_t id);
#else /* CONFIG_ARM_SMMU_V3_SVA */
@@ -839,5 +840,8 @@ static inline void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
ioasid_t id)
{
}
+
+#define arm_smmu_sva_domain_alloc NULL
+
#endif /* CONFIG_ARM_SMMU_V3_SVA */
#endif /* _ARM_SMMU_V3_H */
diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c
index 65814cbc840200..aa20ee6a1d8ae7 100644
--- a/drivers/iommu/iommu-sva.c
+++ b/drivers/iommu/iommu-sva.c
@@ -108,8 +108,8 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm
/* Allocate a new domain and set it on device pasid. */
domain = iommu_sva_domain_alloc(dev, mm);
- if (!domain) {
- ret = -ENOMEM;
+ if (IS_ERR(domain)) {
+ ret = PTR_ERR(domain);
goto out_free_handle;
}
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index d14413916f93a0..f9a2b8363ad941 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -3629,9 +3629,15 @@ struct iommu_domain *iommu_sva_domain_alloc(struct device *dev,
const struct iommu_ops *ops = dev_iommu_ops(dev);
struct iommu_domain *domain;
- domain = ops->domain_alloc(IOMMU_DOMAIN_SVA);
- if (!domain)
- return NULL;
+ if (ops->domain_alloc_sva) {
+ domain = ops->domain_alloc_sva(dev, mm);
+ if (IS_ERR(domain))
+ return domain;
+ } else {
+ domain = ops->domain_alloc(IOMMU_DOMAIN_SVA);
+ if (!domain)
+ return ERR_PTR(-ENOMEM);
+ }
domain->type = IOMMU_DOMAIN_SVA;
mmgrab(mm);
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index 5e27cb3a3be99b..0274044f1fb084 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -419,6 +419,7 @@ static inline int __iommu_copy_struct_from_user_array(
* Upon failure, ERR_PTR must be returned.
* @domain_alloc_paging: Allocate an iommu_domain that can be used for
* UNMANAGED, DMA, and DMA_FQ domain types.
+ * @domain_alloc_sva: Allocate an iommu_domain for Shared Virtual Addressing.
* @probe_device: Add device to iommu driver handling
* @release_device: Remove device from iommu driver handling
* @probe_finalize: Do final setup work after the device is added to an IOMMU
@@ -459,6 +460,8 @@ struct iommu_ops {
struct device *dev, u32 flags, struct iommu_domain *parent,
const struct iommu_user_data *user_data);
struct iommu_domain *(*domain_alloc_paging)(struct device *dev);
+ struct iommu_domain *(*domain_alloc_sva)(struct device *dev,
+ struct mm_struct *mm);
struct iommu_device *(*probe_device)(struct device *dev);
void (*release_device)(struct device *dev);
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 20/27] iommu: Add ops->domain_alloc_sva()
@ 2024-03-04 23:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Make a new op that receives the device and the mm_struct that the SVA
domain should be created for. Unlike domain_alloc_paging() the dev
argument is never NULL here.
This allows drivers to fully initialize the SVA domain and allocate the
mmu_notifier during allocation. It allows the notifier lifetime to follow
the lifetime of the iommu_domain.
Since we have only one call site, upgrade the new op to return ERR_PTR
instead of NULL.
Change SMMUv3 to use the new op.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 11 +++++++----
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 2 +-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 +++++-
drivers/iommu/iommu-sva.c | 4 ++--
drivers/iommu/iommu.c | 12 +++++++++---
include/linux/iommu.h | 3 +++
6 files changed, 27 insertions(+), 11 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index d633316f2e45bc..30b1cf587efffb 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -662,17 +662,20 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
.free = arm_smmu_sva_domain_free
};
-struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned type)
+struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
+ struct mm_struct *mm)
{
+ struct arm_smmu_master *master = dev_iommu_priv_get(dev);
+ struct arm_smmu_device *smmu = master->smmu;
struct arm_smmu_domain *smmu_domain;
- if (type != IOMMU_DOMAIN_SVA)
- return ERR_PTR(-EOPNOTSUPP);
-
smmu_domain = arm_smmu_domain_alloc();
if (IS_ERR(smmu_domain))
return ERR_CAST(smmu_domain);
+
+ smmu_domain->domain.type = IOMMU_DOMAIN_SVA;
smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
+ smmu_domain->smmu = smmu;
return &smmu_domain->domain;
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 6d15fe3a160acf..a255a02a5fc8a9 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -3311,8 +3311,8 @@ static struct iommu_ops arm_smmu_ops = {
.identity_domain = &arm_smmu_identity_domain,
.blocked_domain = &arm_smmu_blocked_domain,
.capable = arm_smmu_capable,
- .domain_alloc = arm_smmu_sva_domain_alloc,
.domain_alloc_paging = arm_smmu_domain_alloc_paging,
+ .domain_alloc_sva = arm_smmu_sva_domain_alloc,
.probe_device = arm_smmu_probe_device,
.release_device = arm_smmu_release_device,
.device_group = arm_smmu_device_group,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index c47e07d695bef2..f62bd2ef36a18d 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -793,7 +793,8 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master);
int arm_smmu_master_disable_sva(struct arm_smmu_master *master);
bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master);
void arm_smmu_sva_notifier_synchronize(void);
-struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned int type);
+struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
+ struct mm_struct *mm);
void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
struct device *dev, ioasid_t id);
#else /* CONFIG_ARM_SMMU_V3_SVA */
@@ -839,5 +840,8 @@ static inline void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
ioasid_t id)
{
}
+
+#define arm_smmu_sva_domain_alloc NULL
+
#endif /* CONFIG_ARM_SMMU_V3_SVA */
#endif /* _ARM_SMMU_V3_H */
diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c
index 65814cbc840200..aa20ee6a1d8ae7 100644
--- a/drivers/iommu/iommu-sva.c
+++ b/drivers/iommu/iommu-sva.c
@@ -108,8 +108,8 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm
/* Allocate a new domain and set it on device pasid. */
domain = iommu_sva_domain_alloc(dev, mm);
- if (!domain) {
- ret = -ENOMEM;
+ if (IS_ERR(domain)) {
+ ret = PTR_ERR(domain);
goto out_free_handle;
}
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index d14413916f93a0..f9a2b8363ad941 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -3629,9 +3629,15 @@ struct iommu_domain *iommu_sva_domain_alloc(struct device *dev,
const struct iommu_ops *ops = dev_iommu_ops(dev);
struct iommu_domain *domain;
- domain = ops->domain_alloc(IOMMU_DOMAIN_SVA);
- if (!domain)
- return NULL;
+ if (ops->domain_alloc_sva) {
+ domain = ops->domain_alloc_sva(dev, mm);
+ if (IS_ERR(domain))
+ return domain;
+ } else {
+ domain = ops->domain_alloc(IOMMU_DOMAIN_SVA);
+ if (!domain)
+ return ERR_PTR(-ENOMEM);
+ }
domain->type = IOMMU_DOMAIN_SVA;
mmgrab(mm);
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index 5e27cb3a3be99b..0274044f1fb084 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -419,6 +419,7 @@ static inline int __iommu_copy_struct_from_user_array(
* Upon failure, ERR_PTR must be returned.
* @domain_alloc_paging: Allocate an iommu_domain that can be used for
* UNMANAGED, DMA, and DMA_FQ domain types.
+ * @domain_alloc_sva: Allocate an iommu_domain for Shared Virtual Addressing.
* @probe_device: Add device to iommu driver handling
* @release_device: Remove device from iommu driver handling
* @probe_finalize: Do final setup work after the device is added to an IOMMU
@@ -459,6 +460,8 @@ struct iommu_ops {
struct device *dev, u32 flags, struct iommu_domain *parent,
const struct iommu_user_data *user_data);
struct iommu_domain *(*domain_alloc_paging)(struct device *dev);
+ struct iommu_domain *(*domain_alloc_sva)(struct device *dev,
+ struct mm_struct *mm);
struct iommu_device *(*probe_device)(struct device *dev);
void (*release_device)(struct device *dev);
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 21/27] iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
This removes all the notifier de-duplication logic in the driver and
relies on the core code to de-duplicate and allocate only one SVA domain
per mm per smmu instance. This naturally gives a 1:1 relationship between
SVA domain and mmu notifier.
Remove all of the previous mmu_notifier, bond, shared cd, and cd refcount
logic entirely.
For the purpose of organizing patches lightly remove BTM support. The next
patches will add it back in. BTM is a performance optimization so this is
bisection friendly functionally invisible change. Note that the current
code never even enables ARM_SMMU_FEAT_BTM so none of this code is ever
even used.
The bond/shared_cd/btm/asid allocator are tightly wound together and
changing them all at once would make this patch too big. The core issue is
that having a single SVA domain per-smmu instance conflicts with the
design of having a global ASID table that BTM currently needs, as we would
end up having to assign multiple SVA domains to the same ASID.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 389 ++++--------------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 81 +---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 14 +-
3 files changed, 101 insertions(+), 383 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 30b1cf587efffb..cb0da4e5a5517a 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -13,29 +13,9 @@
#include "../../iommu-sva.h"
#include "../../io-pgtable-arm.h"
-struct arm_smmu_mmu_notifier {
- struct mmu_notifier mn;
- struct arm_smmu_ctx_desc *cd;
- bool cleared;
- refcount_t refs;
- struct list_head list;
- struct arm_smmu_domain *domain;
-};
-
-#define mn_to_smmu(mn) container_of(mn, struct arm_smmu_mmu_notifier, mn)
-
-struct arm_smmu_bond {
- struct mm_struct *mm;
- struct arm_smmu_mmu_notifier *smmu_mn;
- struct list_head list;
-};
-
-#define sva_to_bond(handle) \
- container_of(handle, struct arm_smmu_bond, sva)
-
static DEFINE_MUTEX(sva_lock);
-static void
+static void __maybe_unused
arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
{
struct arm_smmu_master_domain *master_domain;
@@ -58,58 +38,6 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
}
-/*
- * Check if the CPU ASID is available on the SMMU side. If a private context
- * descriptor is using it, try to replace it.
- */
-static struct arm_smmu_ctx_desc *
-arm_smmu_share_asid(struct mm_struct *mm, u16 asid)
-{
- int ret;
- u32 new_asid;
- struct arm_smmu_ctx_desc *cd;
- struct arm_smmu_device *smmu;
- struct arm_smmu_domain *smmu_domain;
-
- cd = xa_load(&arm_smmu_asid_xa, asid);
- if (!cd)
- return NULL;
-
- if (cd->mm) {
- if (WARN_ON(cd->mm != mm))
- return ERR_PTR(-EINVAL);
- /* All devices bound to this mm use the same cd struct. */
- refcount_inc(&cd->refs);
- return cd;
- }
-
- smmu_domain = container_of(cd, struct arm_smmu_domain, cd);
- smmu = smmu_domain->smmu;
-
- ret = xa_alloc(&arm_smmu_asid_xa, &new_asid, cd,
- XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
- if (ret)
- return ERR_PTR(-ENOSPC);
- /*
- * Race with unmap: TLB invalidations will start targeting the new ASID,
- * which isn't assigned yet. We'll do an invalidate-all on the old ASID
- * later, so it doesn't matter.
- */
- cd->asid = new_asid;
- /*
- * Update ASID and invalidate CD in all associated masters. There will
- * be some overlap between use of both ASIDs, until we invalidate the
- * TLB.
- */
- arm_smmu_update_s1_domain_cd_entry(smmu_domain);
-
- /* Invalidate TLB entries previously associated with that context */
- arm_smmu_tlb_inv_asid(smmu, asid);
-
- xa_erase(&arm_smmu_asid_xa, asid);
- return NULL;
-}
-
static u64 page_size_to_cd(void)
{
static_assert(PAGE_SIZE == SZ_4K || PAGE_SIZE == SZ_16K ||
@@ -123,7 +51,8 @@ static u64 page_size_to_cd(void)
static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target,
struct arm_smmu_master *master,
- struct mm_struct *mm, u16 asid)
+ struct mm_struct *mm, u16 asid,
+ bool btm_invalidation)
{
u64 par;
@@ -144,7 +73,7 @@ static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target,
(master->stall_enabled ? CTXDESC_CD_0_S : 0) |
CTXDESC_CD_0_R |
CTXDESC_CD_0_A |
- CTXDESC_CD_0_ASET |
+ (btm_invalidation ? 0 : CTXDESC_CD_0_ASET) |
FIELD_PREP(CTXDESC_CD_0_ASID, asid));
/*
@@ -186,69 +115,6 @@ static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target,
target->data[3] = cpu_to_le64(read_sysreg(mair_el1));
}
-static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm)
-{
- u16 asid;
- int err = 0;
- struct arm_smmu_ctx_desc *cd;
- struct arm_smmu_ctx_desc *ret = NULL;
-
- /* Don't free the mm until we release the ASID */
- mmgrab(mm);
-
- asid = arm64_mm_context_get(mm);
- if (!asid) {
- err = -ESRCH;
- goto out_drop_mm;
- }
-
- cd = kzalloc(sizeof(*cd), GFP_KERNEL);
- if (!cd) {
- err = -ENOMEM;
- goto out_put_context;
- }
-
- refcount_set(&cd->refs, 1);
-
- mutex_lock(&arm_smmu_asid_lock);
- ret = arm_smmu_share_asid(mm, asid);
- if (ret) {
- mutex_unlock(&arm_smmu_asid_lock);
- goto out_free_cd;
- }
-
- err = xa_insert(&arm_smmu_asid_xa, asid, cd, GFP_KERNEL);
- mutex_unlock(&arm_smmu_asid_lock);
-
- if (err)
- goto out_free_asid;
-
- cd->asid = asid;
- cd->mm = mm;
-
- return cd;
-
-out_free_asid:
- arm_smmu_free_asid(cd);
-out_free_cd:
- kfree(cd);
-out_put_context:
- arm64_mm_context_put(mm);
-out_drop_mm:
- mmdrop(mm);
- return err < 0 ? ERR_PTR(err) : ret;
-}
-
-static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd)
-{
- if (arm_smmu_free_asid(cd)) {
- /* Unpin ASID */
- arm64_mm_context_put(cd->mm);
- mmdrop(cd->mm);
- kfree(cd);
- }
-}
-
/*
* Cloned from the MAX_TLBI_OPS in arch/arm64/include/asm/tlbflush.h, this
* is used as a threshold to replace per-page TLBI commands to issue in the
@@ -263,8 +129,8 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
unsigned long start,
unsigned long end)
{
- struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn);
- struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
+ struct arm_smmu_domain *smmu_domain =
+ container_of(mn, struct arm_smmu_domain, mmu_notifier);
size_t size;
/*
@@ -281,34 +147,27 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
size = 0;
}
- if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM)) {
+ if (!smmu_domain->btm_invalidation) {
if (!size)
arm_smmu_tlb_inv_asid(smmu_domain->smmu,
- smmu_mn->cd->asid);
+ smmu_domain->cd.asid);
else
arm_smmu_tlb_inv_range_asid(start, size,
- smmu_mn->cd->asid,
+ smmu_domain->cd.asid,
PAGE_SIZE, false,
smmu_domain);
}
- arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), start,
- size);
+ arm_smmu_atc_inv_domain(smmu_domain, start, size);
}
static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
{
- struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn);
- struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
+ struct arm_smmu_domain *smmu_domain =
+ container_of(mn, struct arm_smmu_domain, mmu_notifier);
struct arm_smmu_master_domain *master_domain;
unsigned long flags;
- mutex_lock(&sva_lock);
- if (smmu_mn->cleared) {
- mutex_unlock(&sva_lock);
- return;
- }
-
/*
* DMA may still be running. Keep the cd valid to avoid C_BAD_CD events,
* but disable translation.
@@ -320,25 +179,27 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
struct arm_smmu_cd target;
struct arm_smmu_cd *cdptr;
- cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm));
+ cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid);
if (WARN_ON(!cdptr))
continue;
- arm_smmu_make_sva_cd(&target, master, NULL, smmu_mn->cd->asid);
- arm_smmu_write_cd_entry(master, mm_get_enqcmd_pasid(mm), cdptr,
+ arm_smmu_make_sva_cd(&target, master, NULL,
+ smmu_domain->cd.asid,
+ smmu_domain->btm_invalidation);
+ arm_smmu_write_cd_entry(master, master_domain->ssid, cdptr,
&target);
}
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
- arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid);
- arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0);
-
- smmu_mn->cleared = true;
- mutex_unlock(&sva_lock);
+ arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
+ arm_smmu_atc_inv_domain(smmu_domain, 0, 0);
}
static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn)
{
- kfree(mn_to_smmu(mn));
+ struct arm_smmu_domain *smmu_domain =
+ container_of(mn, struct arm_smmu_domain, mmu_notifier);
+
+ kfree(smmu_domain);
}
static const struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = {
@@ -347,113 +208,6 @@ static const struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = {
.free_notifier = arm_smmu_mmu_notifier_free,
};
-/* Allocate or get existing MMU notifier for this {domain, mm} pair */
-static struct arm_smmu_mmu_notifier *
-arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain,
- struct mm_struct *mm)
-{
- int ret;
- struct arm_smmu_ctx_desc *cd;
- struct arm_smmu_mmu_notifier *smmu_mn;
-
- list_for_each_entry(smmu_mn, &smmu_domain->mmu_notifiers, list) {
- if (smmu_mn->mn.mm == mm) {
- refcount_inc(&smmu_mn->refs);
- return smmu_mn;
- }
- }
-
- cd = arm_smmu_alloc_shared_cd(mm);
- if (IS_ERR(cd))
- return ERR_CAST(cd);
-
- smmu_mn = kzalloc(sizeof(*smmu_mn), GFP_KERNEL);
- if (!smmu_mn) {
- ret = -ENOMEM;
- goto err_free_cd;
- }
-
- refcount_set(&smmu_mn->refs, 1);
- smmu_mn->cd = cd;
- smmu_mn->domain = smmu_domain;
- smmu_mn->mn.ops = &arm_smmu_mmu_notifier_ops;
-
- ret = mmu_notifier_register(&smmu_mn->mn, mm);
- if (ret) {
- kfree(smmu_mn);
- goto err_free_cd;
- }
-
- list_add(&smmu_mn->list, &smmu_domain->mmu_notifiers);
- return smmu_mn;
-
-err_free_cd:
- arm_smmu_free_shared_cd(cd);
- return ERR_PTR(ret);
-}
-
-static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn)
-{
- struct mm_struct *mm = smmu_mn->mn.mm;
- struct arm_smmu_ctx_desc *cd = smmu_mn->cd;
- struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
-
- if (!refcount_dec_and_test(&smmu_mn->refs))
- return;
-
- list_del(&smmu_mn->list);
-
- /*
- * If we went through clear(), we've already invalidated, and no
- * new TLB entry can have been formed.
- */
- if (!smmu_mn->cleared) {
- arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid);
- arm_smmu_atc_inv_domain_sva(smmu_domain,
- mm_get_enqcmd_pasid(mm), 0, 0);
- }
-
- /* Frees smmu_mn */
- mmu_notifier_put(&smmu_mn->mn);
- arm_smmu_free_shared_cd(cd);
-}
-
-static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm,
- struct arm_smmu_cd *target)
-{
- int ret;
- struct arm_smmu_bond *bond;
- struct arm_smmu_master *master = dev_iommu_priv_get(dev);
- struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
- struct arm_smmu_domain *smmu_domain;
-
- if (!(domain->type & __IOMMU_DOMAIN_PAGING))
- return -ENODEV;
- smmu_domain = to_smmu_domain(domain);
- if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1)
- return -ENODEV;
-
- bond = kzalloc(sizeof(*bond), GFP_KERNEL);
- if (!bond)
- return -ENOMEM;
-
- bond->mm = mm;
-
- bond->smmu_mn = arm_smmu_mmu_notifier_get(smmu_domain, mm);
- if (IS_ERR(bond->smmu_mn)) {
- ret = PTR_ERR(bond->smmu_mn);
- goto err_free_bond;
- }
-
- arm_smmu_make_sva_cd(target, master, mm, bond->smmu_mn->cd->asid);
- list_add(&bond->list, &master->bonds);
- return 0;
-
-err_free_bond:
- kfree(bond);
- return ret;
-}
-
bool arm_smmu_sva_supported(struct arm_smmu_device *smmu)
{
unsigned long reg, fld;
@@ -581,11 +335,6 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master)
int arm_smmu_master_disable_sva(struct arm_smmu_master *master)
{
mutex_lock(&sva_lock);
- if (!list_empty(&master->bonds)) {
- dev_err(master->dev, "cannot disable SVA, device is bound\n");
- mutex_unlock(&sva_lock);
- return -EBUSY;
- }
arm_smmu_master_sva_disable_iopf(master);
master->sva_enabled = false;
mutex_unlock(&sva_lock);
@@ -602,59 +351,54 @@ void arm_smmu_sva_notifier_synchronize(void)
mmu_notifier_synchronize();
}
-void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
- struct device *dev, ioasid_t id)
-{
- struct mm_struct *mm = domain->mm;
- struct arm_smmu_bond *bond = NULL, *t;
- struct arm_smmu_master *master = dev_iommu_priv_get(dev);
-
- arm_smmu_remove_pasid(master, to_smmu_domain(domain), id);
-
- mutex_lock(&sva_lock);
- list_for_each_entry(t, &master->bonds, list) {
- if (t->mm == mm) {
- bond = t;
- break;
- }
- }
-
- if (!WARN_ON(!bond)) {
- list_del(&bond->list);
- arm_smmu_mmu_notifier_put(bond->smmu_mn);
- kfree(bond);
- }
- mutex_unlock(&sva_lock);
-}
-
static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
struct device *dev, ioasid_t id)
{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
- int ret = 0;
- struct mm_struct *mm = domain->mm;
struct arm_smmu_cd target;
+ int ret;
- if (mm_get_enqcmd_pasid(mm) != id || !master->cd_table.used_sid)
+ /* Prevent arm_smmu_mm_release from being called while we are attaching */
+ if (!mmget_not_zero(domain->mm))
return -EINVAL;
- if (!arm_smmu_get_cd_ptr(master, id))
- return -ENOMEM;
+ /*
+ * This does not need the arm_smmu_asid_lock because SVA domains never
+ * get reassigned
+ */
+ arm_smmu_make_sva_cd(&target, master, smmu_domain->domain.mm,
+ smmu_domain->cd.asid,
+ smmu_domain->btm_invalidation);
- mutex_lock(&sva_lock);
- ret = __arm_smmu_sva_bind(dev, mm, &target);
- mutex_unlock(&sva_lock);
- if (ret)
- return ret;
+ ret = arm_smmu_set_pasid(master, smmu_domain, id, &target);
- /* This cannot fail since we preallocated the cdptr */
- arm_smmu_set_pasid(master, to_smmu_domain(domain), id, &target);
- return 0;
+ mmput(domain->mm);
+ return ret;
}
static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
{
- kfree(to_smmu_domain(domain));
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+
+ /*
+ * Ensure the ASID is empty in the iommu cache before allowing reuse.
+ */
+ arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
+
+ /*
+ * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can
+ * still be called/running at this point. We allow the ASID to be
+ * reused, and if there is a race then it just suffers harmless
+ * unnecessary invalidation.
+ */
+ xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
+
+ /*
+ * Actual free is defered to the SRCU callback
+ * arm_smmu_mmu_notifier_free()
+ */
+ mmu_notifier_put(&smmu_domain->mmu_notifier);
}
static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
@@ -668,6 +412,8 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct arm_smmu_device *smmu = master->smmu;
struct arm_smmu_domain *smmu_domain;
+ u32 asid;
+ int ret;
smmu_domain = arm_smmu_domain_alloc();
if (IS_ERR(smmu_domain))
@@ -677,5 +423,22 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
smmu_domain->smmu = smmu;
+ ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
+ XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
+ if (ret)
+ goto err_free;
+
+ smmu_domain->cd.asid = asid;
+ smmu_domain->mmu_notifier.ops = &arm_smmu_mmu_notifier_ops;
+ ret = mmu_notifier_register(&smmu_domain->mmu_notifier, mm);
+ if (ret)
+ goto err_asid;
+
return &smmu_domain->domain;
+
+err_asid:
+ xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
+err_free:
+ kfree(smmu_domain);
+ return ERR_PTR(ret);
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index a255a02a5fc8a9..5642321b2124d9 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1436,22 +1436,6 @@ static void arm_smmu_free_cd_tables(struct arm_smmu_master *master)
cd_table->cdtab = NULL;
}
-bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd)
-{
- bool free;
- struct arm_smmu_ctx_desc *old_cd;
-
- if (!cd->asid)
- return false;
-
- free = refcount_dec_and_test(&cd->refs);
- if (free) {
- old_cd = xa_erase(&arm_smmu_asid_xa, cd->asid);
- WARN_ON(old_cd != cd);
- }
- return free;
-}
-
/* Stream table manipulation functions */
static void
arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc)
@@ -2042,8 +2026,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master,
return arm_smmu_cmdq_batch_submit(master->smmu, &cmds);
}
-static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
- ioasid_t ssid, unsigned long iova, size_t size)
+int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
+ unsigned long iova, size_t size)
{
struct arm_smmu_master_domain *master_domain;
int i;
@@ -2081,15 +2065,7 @@ static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
if (!master->ats_enabled)
continue;
- /*
- * Non-zero ssid means SVA is co-opting the S1 domain to issue
- * invalidations for SVA PASIDs.
- */
- if (ssid != IOMMU_NO_PASID)
- arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd);
- else
- arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size,
- &cmd);
+ arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size, &cmd);
for (i = 0; i < master->num_streams; i++) {
cmd.atc.sid = master->streams[i].id;
@@ -2101,19 +2077,6 @@ static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
return arm_smmu_cmdq_batch_submit(smmu_domain->smmu, &cmds);
}
-static int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
- unsigned long iova, size_t size)
-{
- return __arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova,
- size);
-}
-
-int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain,
- ioasid_t ssid, unsigned long iova, size_t size)
-{
- return __arm_smmu_atc_inv_domain(smmu_domain, ssid, iova, size);
-}
-
/* IO_PGTABLE API */
static void arm_smmu_tlb_inv_context(void *cookie)
{
@@ -2302,7 +2265,6 @@ struct arm_smmu_domain *arm_smmu_domain_alloc(void)
mutex_init(&smmu_domain->init_mutex);
INIT_LIST_HEAD(&smmu_domain->devices);
spin_lock_init(&smmu_domain->devices_lock);
- INIT_LIST_HEAD(&smmu_domain->mmu_notifiers);
return smmu_domain;
}
@@ -2345,7 +2307,7 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
/* Prevent SVA from touching the CD while we're freeing it */
mutex_lock(&arm_smmu_asid_lock);
- arm_smmu_free_asid(&smmu_domain->cd);
+ xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
mutex_unlock(&arm_smmu_asid_lock);
} else {
struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
@@ -2363,11 +2325,9 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
u32 asid;
struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
- refcount_set(&cd->refs, 1);
-
/* Prevent SVA from modifying the ASID until it is written to the CD */
mutex_lock(&arm_smmu_asid_lock);
- ret = xa_alloc(&arm_smmu_asid_xa, &asid, cd,
+ ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
cd->asid = (u16)asid;
mutex_unlock(&arm_smmu_asid_lock);
@@ -2818,7 +2778,11 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
struct arm_smmu_cd *cdptr;
int ret;
- if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)))
+ if (smmu_domain->smmu != master->smmu)
+ return -EINVAL;
+
+ if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)) ||
+ !master->cd_table.used_sid)
return -ENODEV;
cdptr = arm_smmu_get_cd_ptr(master, pasid);
@@ -2839,9 +2803,18 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
return ret;
}
-void arm_smmu_remove_pasid(struct arm_smmu_master *master,
- struct arm_smmu_domain *smmu_domain, ioasid_t pasid)
+static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
{
+ struct arm_smmu_master *master = dev_iommu_priv_get(dev);
+ struct arm_smmu_domain *smmu_domain;
+ struct iommu_domain *domain;
+
+ domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA);
+ if (WARN_ON(IS_ERR(domain)) || !domain)
+ return;
+
+ smmu_domain = to_smmu_domain(domain);
+
mutex_lock(&arm_smmu_asid_lock);
arm_smmu_clear_cd(master, pasid);
if (master->ats_enabled)
@@ -3115,7 +3088,6 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev)
master->dev = dev;
master->smmu = smmu;
- INIT_LIST_HEAD(&master->bonds);
dev_iommu_priv_set(dev, master);
ret = arm_smmu_insert_master(smmu, master);
@@ -3296,17 +3268,6 @@ static int arm_smmu_def_domain_type(struct device *dev)
return 0;
}
-static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
-{
- struct iommu_domain *domain;
-
- domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA);
- if (WARN_ON(IS_ERR(domain)) || !domain)
- return;
-
- arm_smmu_sva_remove_dev_pasid(domain, dev, pasid);
-}
-
static struct iommu_ops arm_smmu_ops = {
.identity_domain = &arm_smmu_identity_domain,
.blocked_domain = &arm_smmu_blocked_domain,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index f62bd2ef36a18d..cfae4d69cd810c 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -587,9 +587,6 @@ struct arm_smmu_strtab_l1_desc {
struct arm_smmu_ctx_desc {
u16 asid;
-
- refcount_t refs;
- struct mm_struct *mm;
};
struct arm_smmu_l1_ctx_desc {
@@ -713,7 +710,6 @@ struct arm_smmu_master {
bool stall_enabled;
bool sva_enabled;
bool iopf_enabled;
- struct list_head bonds;
unsigned int ssid_bits;
};
@@ -742,7 +738,8 @@ struct arm_smmu_domain {
struct list_head devices;
spinlock_t devices_lock;
- struct list_head mmu_notifiers;
+ struct mmu_notifier mmu_notifier;
+ bool btm_invalidation;
};
struct arm_smmu_master_domain {
@@ -781,9 +778,8 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
size_t granule, bool leaf,
struct arm_smmu_domain *smmu_domain);
-bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd);
-int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain,
- ioasid_t ssid, unsigned long iova, size_t size);
+int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
+ unsigned long iova, size_t size);
#ifdef CONFIG_ARM_SMMU_V3_SVA
bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
@@ -795,8 +791,6 @@ bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master);
void arm_smmu_sva_notifier_synchronize(void);
struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
struct mm_struct *mm);
-void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
- struct device *dev, ioasid_t id);
#else /* CONFIG_ARM_SMMU_V3_SVA */
static inline bool arm_smmu_sva_supported(struct arm_smmu_device *smmu)
{
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 21/27] iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain
@ 2024-03-04 23:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
This removes all the notifier de-duplication logic in the driver and
relies on the core code to de-duplicate and allocate only one SVA domain
per mm per smmu instance. This naturally gives a 1:1 relationship between
SVA domain and mmu notifier.
Remove all of the previous mmu_notifier, bond, shared cd, and cd refcount
logic entirely.
For the purpose of organizing patches lightly remove BTM support. The next
patches will add it back in. BTM is a performance optimization so this is
bisection friendly functionally invisible change. Note that the current
code never even enables ARM_SMMU_FEAT_BTM so none of this code is ever
even used.
The bond/shared_cd/btm/asid allocator are tightly wound together and
changing them all at once would make this patch too big. The core issue is
that having a single SVA domain per-smmu instance conflicts with the
design of having a global ASID table that BTM currently needs, as we would
end up having to assign multiple SVA domains to the same ASID.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 389 ++++--------------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 81 +---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 14 +-
3 files changed, 101 insertions(+), 383 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 30b1cf587efffb..cb0da4e5a5517a 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -13,29 +13,9 @@
#include "../../iommu-sva.h"
#include "../../io-pgtable-arm.h"
-struct arm_smmu_mmu_notifier {
- struct mmu_notifier mn;
- struct arm_smmu_ctx_desc *cd;
- bool cleared;
- refcount_t refs;
- struct list_head list;
- struct arm_smmu_domain *domain;
-};
-
-#define mn_to_smmu(mn) container_of(mn, struct arm_smmu_mmu_notifier, mn)
-
-struct arm_smmu_bond {
- struct mm_struct *mm;
- struct arm_smmu_mmu_notifier *smmu_mn;
- struct list_head list;
-};
-
-#define sva_to_bond(handle) \
- container_of(handle, struct arm_smmu_bond, sva)
-
static DEFINE_MUTEX(sva_lock);
-static void
+static void __maybe_unused
arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
{
struct arm_smmu_master_domain *master_domain;
@@ -58,58 +38,6 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
}
-/*
- * Check if the CPU ASID is available on the SMMU side. If a private context
- * descriptor is using it, try to replace it.
- */
-static struct arm_smmu_ctx_desc *
-arm_smmu_share_asid(struct mm_struct *mm, u16 asid)
-{
- int ret;
- u32 new_asid;
- struct arm_smmu_ctx_desc *cd;
- struct arm_smmu_device *smmu;
- struct arm_smmu_domain *smmu_domain;
-
- cd = xa_load(&arm_smmu_asid_xa, asid);
- if (!cd)
- return NULL;
-
- if (cd->mm) {
- if (WARN_ON(cd->mm != mm))
- return ERR_PTR(-EINVAL);
- /* All devices bound to this mm use the same cd struct. */
- refcount_inc(&cd->refs);
- return cd;
- }
-
- smmu_domain = container_of(cd, struct arm_smmu_domain, cd);
- smmu = smmu_domain->smmu;
-
- ret = xa_alloc(&arm_smmu_asid_xa, &new_asid, cd,
- XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
- if (ret)
- return ERR_PTR(-ENOSPC);
- /*
- * Race with unmap: TLB invalidations will start targeting the new ASID,
- * which isn't assigned yet. We'll do an invalidate-all on the old ASID
- * later, so it doesn't matter.
- */
- cd->asid = new_asid;
- /*
- * Update ASID and invalidate CD in all associated masters. There will
- * be some overlap between use of both ASIDs, until we invalidate the
- * TLB.
- */
- arm_smmu_update_s1_domain_cd_entry(smmu_domain);
-
- /* Invalidate TLB entries previously associated with that context */
- arm_smmu_tlb_inv_asid(smmu, asid);
-
- xa_erase(&arm_smmu_asid_xa, asid);
- return NULL;
-}
-
static u64 page_size_to_cd(void)
{
static_assert(PAGE_SIZE == SZ_4K || PAGE_SIZE == SZ_16K ||
@@ -123,7 +51,8 @@ static u64 page_size_to_cd(void)
static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target,
struct arm_smmu_master *master,
- struct mm_struct *mm, u16 asid)
+ struct mm_struct *mm, u16 asid,
+ bool btm_invalidation)
{
u64 par;
@@ -144,7 +73,7 @@ static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target,
(master->stall_enabled ? CTXDESC_CD_0_S : 0) |
CTXDESC_CD_0_R |
CTXDESC_CD_0_A |
- CTXDESC_CD_0_ASET |
+ (btm_invalidation ? 0 : CTXDESC_CD_0_ASET) |
FIELD_PREP(CTXDESC_CD_0_ASID, asid));
/*
@@ -186,69 +115,6 @@ static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target,
target->data[3] = cpu_to_le64(read_sysreg(mair_el1));
}
-static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm)
-{
- u16 asid;
- int err = 0;
- struct arm_smmu_ctx_desc *cd;
- struct arm_smmu_ctx_desc *ret = NULL;
-
- /* Don't free the mm until we release the ASID */
- mmgrab(mm);
-
- asid = arm64_mm_context_get(mm);
- if (!asid) {
- err = -ESRCH;
- goto out_drop_mm;
- }
-
- cd = kzalloc(sizeof(*cd), GFP_KERNEL);
- if (!cd) {
- err = -ENOMEM;
- goto out_put_context;
- }
-
- refcount_set(&cd->refs, 1);
-
- mutex_lock(&arm_smmu_asid_lock);
- ret = arm_smmu_share_asid(mm, asid);
- if (ret) {
- mutex_unlock(&arm_smmu_asid_lock);
- goto out_free_cd;
- }
-
- err = xa_insert(&arm_smmu_asid_xa, asid, cd, GFP_KERNEL);
- mutex_unlock(&arm_smmu_asid_lock);
-
- if (err)
- goto out_free_asid;
-
- cd->asid = asid;
- cd->mm = mm;
-
- return cd;
-
-out_free_asid:
- arm_smmu_free_asid(cd);
-out_free_cd:
- kfree(cd);
-out_put_context:
- arm64_mm_context_put(mm);
-out_drop_mm:
- mmdrop(mm);
- return err < 0 ? ERR_PTR(err) : ret;
-}
-
-static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd)
-{
- if (arm_smmu_free_asid(cd)) {
- /* Unpin ASID */
- arm64_mm_context_put(cd->mm);
- mmdrop(cd->mm);
- kfree(cd);
- }
-}
-
/*
* Cloned from the MAX_TLBI_OPS in arch/arm64/include/asm/tlbflush.h, this
* is used as a threshold to replace per-page TLBI commands to issue in the
@@ -263,8 +129,8 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
unsigned long start,
unsigned long end)
{
- struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn);
- struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
+ struct arm_smmu_domain *smmu_domain =
+ container_of(mn, struct arm_smmu_domain, mmu_notifier);
size_t size;
/*
@@ -281,34 +147,27 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
size = 0;
}
- if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM)) {
+ if (!smmu_domain->btm_invalidation) {
if (!size)
arm_smmu_tlb_inv_asid(smmu_domain->smmu,
- smmu_mn->cd->asid);
+ smmu_domain->cd.asid);
else
arm_smmu_tlb_inv_range_asid(start, size,
- smmu_mn->cd->asid,
+ smmu_domain->cd.asid,
PAGE_SIZE, false,
smmu_domain);
}
- arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), start,
- size);
+ arm_smmu_atc_inv_domain(smmu_domain, start, size);
}
static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
{
- struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn);
- struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
+ struct arm_smmu_domain *smmu_domain =
+ container_of(mn, struct arm_smmu_domain, mmu_notifier);
struct arm_smmu_master_domain *master_domain;
unsigned long flags;
- mutex_lock(&sva_lock);
- if (smmu_mn->cleared) {
- mutex_unlock(&sva_lock);
- return;
- }
-
/*
* DMA may still be running. Keep the cd valid to avoid C_BAD_CD events,
* but disable translation.
@@ -320,25 +179,27 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
struct arm_smmu_cd target;
struct arm_smmu_cd *cdptr;
- cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm));
+ cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid);
if (WARN_ON(!cdptr))
continue;
- arm_smmu_make_sva_cd(&target, master, NULL, smmu_mn->cd->asid);
- arm_smmu_write_cd_entry(master, mm_get_enqcmd_pasid(mm), cdptr,
+ arm_smmu_make_sva_cd(&target, master, NULL,
+ smmu_domain->cd.asid,
+ smmu_domain->btm_invalidation);
+ arm_smmu_write_cd_entry(master, master_domain->ssid, cdptr,
&target);
}
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
- arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid);
- arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0);
-
- smmu_mn->cleared = true;
- mutex_unlock(&sva_lock);
+ arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
+ arm_smmu_atc_inv_domain(smmu_domain, 0, 0);
}
static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn)
{
- kfree(mn_to_smmu(mn));
+ struct arm_smmu_domain *smmu_domain =
+ container_of(mn, struct arm_smmu_domain, mmu_notifier);
+
+ kfree(smmu_domain);
}
static const struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = {
@@ -347,113 +208,6 @@ static const struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = {
.free_notifier = arm_smmu_mmu_notifier_free,
};
-/* Allocate or get existing MMU notifier for this {domain, mm} pair */
-static struct arm_smmu_mmu_notifier *
-arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain,
- struct mm_struct *mm)
-{
- int ret;
- struct arm_smmu_ctx_desc *cd;
- struct arm_smmu_mmu_notifier *smmu_mn;
-
- list_for_each_entry(smmu_mn, &smmu_domain->mmu_notifiers, list) {
- if (smmu_mn->mn.mm == mm) {
- refcount_inc(&smmu_mn->refs);
- return smmu_mn;
- }
- }
-
- cd = arm_smmu_alloc_shared_cd(mm);
- if (IS_ERR(cd))
- return ERR_CAST(cd);
-
- smmu_mn = kzalloc(sizeof(*smmu_mn), GFP_KERNEL);
- if (!smmu_mn) {
- ret = -ENOMEM;
- goto err_free_cd;
- }
-
- refcount_set(&smmu_mn->refs, 1);
- smmu_mn->cd = cd;
- smmu_mn->domain = smmu_domain;
- smmu_mn->mn.ops = &arm_smmu_mmu_notifier_ops;
-
- ret = mmu_notifier_register(&smmu_mn->mn, mm);
- if (ret) {
- kfree(smmu_mn);
- goto err_free_cd;
- }
-
- list_add(&smmu_mn->list, &smmu_domain->mmu_notifiers);
- return smmu_mn;
-
-err_free_cd:
- arm_smmu_free_shared_cd(cd);
- return ERR_PTR(ret);
-}
-
-static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn)
-{
- struct mm_struct *mm = smmu_mn->mn.mm;
- struct arm_smmu_ctx_desc *cd = smmu_mn->cd;
- struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
-
- if (!refcount_dec_and_test(&smmu_mn->refs))
- return;
-
- list_del(&smmu_mn->list);
-
- /*
- * If we went through clear(), we've already invalidated, and no
- * new TLB entry can have been formed.
- */
- if (!smmu_mn->cleared) {
- arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid);
- arm_smmu_atc_inv_domain_sva(smmu_domain,
- mm_get_enqcmd_pasid(mm), 0, 0);
- }
-
- /* Frees smmu_mn */
- mmu_notifier_put(&smmu_mn->mn);
- arm_smmu_free_shared_cd(cd);
-}
-
-static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm,
- struct arm_smmu_cd *target)
-{
- int ret;
- struct arm_smmu_bond *bond;
- struct arm_smmu_master *master = dev_iommu_priv_get(dev);
- struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
- struct arm_smmu_domain *smmu_domain;
-
- if (!(domain->type & __IOMMU_DOMAIN_PAGING))
- return -ENODEV;
- smmu_domain = to_smmu_domain(domain);
- if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1)
- return -ENODEV;
-
- bond = kzalloc(sizeof(*bond), GFP_KERNEL);
- if (!bond)
- return -ENOMEM;
-
- bond->mm = mm;
-
- bond->smmu_mn = arm_smmu_mmu_notifier_get(smmu_domain, mm);
- if (IS_ERR(bond->smmu_mn)) {
- ret = PTR_ERR(bond->smmu_mn);
- goto err_free_bond;
- }
-
- arm_smmu_make_sva_cd(target, master, mm, bond->smmu_mn->cd->asid);
- list_add(&bond->list, &master->bonds);
- return 0;
-
-err_free_bond:
- kfree(bond);
- return ret;
-}
-
bool arm_smmu_sva_supported(struct arm_smmu_device *smmu)
{
unsigned long reg, fld;
@@ -581,11 +335,6 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master)
int arm_smmu_master_disable_sva(struct arm_smmu_master *master)
{
mutex_lock(&sva_lock);
- if (!list_empty(&master->bonds)) {
- dev_err(master->dev, "cannot disable SVA, device is bound\n");
- mutex_unlock(&sva_lock);
- return -EBUSY;
- }
arm_smmu_master_sva_disable_iopf(master);
master->sva_enabled = false;
mutex_unlock(&sva_lock);
@@ -602,59 +351,54 @@ void arm_smmu_sva_notifier_synchronize(void)
mmu_notifier_synchronize();
}
-void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
- struct device *dev, ioasid_t id)
-{
- struct mm_struct *mm = domain->mm;
- struct arm_smmu_bond *bond = NULL, *t;
- struct arm_smmu_master *master = dev_iommu_priv_get(dev);
-
- arm_smmu_remove_pasid(master, to_smmu_domain(domain), id);
-
- mutex_lock(&sva_lock);
- list_for_each_entry(t, &master->bonds, list) {
- if (t->mm == mm) {
- bond = t;
- break;
- }
- }
-
- if (!WARN_ON(!bond)) {
- list_del(&bond->list);
- arm_smmu_mmu_notifier_put(bond->smmu_mn);
- kfree(bond);
- }
- mutex_unlock(&sva_lock);
-}
-
static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
struct device *dev, ioasid_t id)
{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
- int ret = 0;
- struct mm_struct *mm = domain->mm;
struct arm_smmu_cd target;
+ int ret;
- if (mm_get_enqcmd_pasid(mm) != id || !master->cd_table.used_sid)
+ /* Prevent arm_smmu_mm_release from being called while we are attaching */
+ if (!mmget_not_zero(domain->mm))
return -EINVAL;
- if (!arm_smmu_get_cd_ptr(master, id))
- return -ENOMEM;
+ /*
+ * This does not need the arm_smmu_asid_lock because SVA domains never
+ * get reassigned
+ */
+ arm_smmu_make_sva_cd(&target, master, smmu_domain->domain.mm,
+ smmu_domain->cd.asid,
+ smmu_domain->btm_invalidation);
- mutex_lock(&sva_lock);
- ret = __arm_smmu_sva_bind(dev, mm, &target);
- mutex_unlock(&sva_lock);
- if (ret)
- return ret;
+ ret = arm_smmu_set_pasid(master, smmu_domain, id, &target);
- /* This cannot fail since we preallocated the cdptr */
- arm_smmu_set_pasid(master, to_smmu_domain(domain), id, &target);
- return 0;
+ mmput(domain->mm);
+ return ret;
}
static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
{
- kfree(to_smmu_domain(domain));
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+
+ /*
+ * Ensure the ASID is empty in the iommu cache before allowing reuse.
+ */
+ arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
+
+ /*
+ * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can
+ * still be called/running at this point. We allow the ASID to be
+ * reused, and if there is a race then it just suffers harmless
+ * unnecessary invalidation.
+ */
+ xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
+
+ /*
+ * Actual free is defered to the SRCU callback
+ * arm_smmu_mmu_notifier_free()
+ */
+ mmu_notifier_put(&smmu_domain->mmu_notifier);
}
static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
@@ -668,6 +412,8 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct arm_smmu_device *smmu = master->smmu;
struct arm_smmu_domain *smmu_domain;
+ u32 asid;
+ int ret;
smmu_domain = arm_smmu_domain_alloc();
if (IS_ERR(smmu_domain))
@@ -677,5 +423,22 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
smmu_domain->smmu = smmu;
+ ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
+ XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
+ if (ret)
+ goto err_free;
+
+ smmu_domain->cd.asid = asid;
+ smmu_domain->mmu_notifier.ops = &arm_smmu_mmu_notifier_ops;
+ ret = mmu_notifier_register(&smmu_domain->mmu_notifier, mm);
+ if (ret)
+ goto err_asid;
+
return &smmu_domain->domain;
+
+err_asid:
+ xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
+err_free:
+ kfree(smmu_domain);
+ return ERR_PTR(ret);
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index a255a02a5fc8a9..5642321b2124d9 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1436,22 +1436,6 @@ static void arm_smmu_free_cd_tables(struct arm_smmu_master *master)
cd_table->cdtab = NULL;
}
-bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd)
-{
- bool free;
- struct arm_smmu_ctx_desc *old_cd;
-
- if (!cd->asid)
- return false;
-
- free = refcount_dec_and_test(&cd->refs);
- if (free) {
- old_cd = xa_erase(&arm_smmu_asid_xa, cd->asid);
- WARN_ON(old_cd != cd);
- }
- return free;
-}
-
/* Stream table manipulation functions */
static void
arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc)
@@ -2042,8 +2026,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master,
return arm_smmu_cmdq_batch_submit(master->smmu, &cmds);
}
-static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
- ioasid_t ssid, unsigned long iova, size_t size)
+int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
+ unsigned long iova, size_t size)
{
struct arm_smmu_master_domain *master_domain;
int i;
@@ -2081,15 +2065,7 @@ static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
if (!master->ats_enabled)
continue;
- /*
- * Non-zero ssid means SVA is co-opting the S1 domain to issue
- * invalidations for SVA PASIDs.
- */
- if (ssid != IOMMU_NO_PASID)
- arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd);
- else
- arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size,
- &cmd);
+ arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size, &cmd);
for (i = 0; i < master->num_streams; i++) {
cmd.atc.sid = master->streams[i].id;
@@ -2101,19 +2077,6 @@ static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
return arm_smmu_cmdq_batch_submit(smmu_domain->smmu, &cmds);
}
-static int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
- unsigned long iova, size_t size)
-{
- return __arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova,
- size);
-}
-
-int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain,
- ioasid_t ssid, unsigned long iova, size_t size)
-{
- return __arm_smmu_atc_inv_domain(smmu_domain, ssid, iova, size);
-}
-
/* IO_PGTABLE API */
static void arm_smmu_tlb_inv_context(void *cookie)
{
@@ -2302,7 +2265,6 @@ struct arm_smmu_domain *arm_smmu_domain_alloc(void)
mutex_init(&smmu_domain->init_mutex);
INIT_LIST_HEAD(&smmu_domain->devices);
spin_lock_init(&smmu_domain->devices_lock);
- INIT_LIST_HEAD(&smmu_domain->mmu_notifiers);
return smmu_domain;
}
@@ -2345,7 +2307,7 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
/* Prevent SVA from touching the CD while we're freeing it */
mutex_lock(&arm_smmu_asid_lock);
- arm_smmu_free_asid(&smmu_domain->cd);
+ xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
mutex_unlock(&arm_smmu_asid_lock);
} else {
struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
@@ -2363,11 +2325,9 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
u32 asid;
struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
- refcount_set(&cd->refs, 1);
-
/* Prevent SVA from modifying the ASID until it is written to the CD */
mutex_lock(&arm_smmu_asid_lock);
- ret = xa_alloc(&arm_smmu_asid_xa, &asid, cd,
+ ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
cd->asid = (u16)asid;
mutex_unlock(&arm_smmu_asid_lock);
@@ -2818,7 +2778,11 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
struct arm_smmu_cd *cdptr;
int ret;
- if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)))
+ if (smmu_domain->smmu != master->smmu)
+ return -EINVAL;
+
+ if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)) ||
+ !master->cd_table.used_sid)
return -ENODEV;
cdptr = arm_smmu_get_cd_ptr(master, pasid);
@@ -2839,9 +2803,18 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
return ret;
}
-void arm_smmu_remove_pasid(struct arm_smmu_master *master,
- struct arm_smmu_domain *smmu_domain, ioasid_t pasid)
+static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
{
+ struct arm_smmu_master *master = dev_iommu_priv_get(dev);
+ struct arm_smmu_domain *smmu_domain;
+ struct iommu_domain *domain;
+
+ domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA);
+ if (WARN_ON(IS_ERR(domain)) || !domain)
+ return;
+
+ smmu_domain = to_smmu_domain(domain);
+
mutex_lock(&arm_smmu_asid_lock);
arm_smmu_clear_cd(master, pasid);
if (master->ats_enabled)
@@ -3115,7 +3088,6 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev)
master->dev = dev;
master->smmu = smmu;
- INIT_LIST_HEAD(&master->bonds);
dev_iommu_priv_set(dev, master);
ret = arm_smmu_insert_master(smmu, master);
@@ -3296,17 +3268,6 @@ static int arm_smmu_def_domain_type(struct device *dev)
return 0;
}
-static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
-{
- struct iommu_domain *domain;
-
- domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA);
- if (WARN_ON(IS_ERR(domain)) || !domain)
- return;
-
- arm_smmu_sva_remove_dev_pasid(domain, dev, pasid);
-}
-
static struct iommu_ops arm_smmu_ops = {
.identity_domain = &arm_smmu_identity_domain,
.blocked_domain = &arm_smmu_blocked_domain,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index f62bd2ef36a18d..cfae4d69cd810c 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -587,9 +587,6 @@ struct arm_smmu_strtab_l1_desc {
struct arm_smmu_ctx_desc {
u16 asid;
-
- refcount_t refs;
- struct mm_struct *mm;
};
struct arm_smmu_l1_ctx_desc {
@@ -713,7 +710,6 @@ struct arm_smmu_master {
bool stall_enabled;
bool sva_enabled;
bool iopf_enabled;
- struct list_head bonds;
unsigned int ssid_bits;
};
@@ -742,7 +738,8 @@ struct arm_smmu_domain {
struct list_head devices;
spinlock_t devices_lock;
- struct list_head mmu_notifiers;
+ struct mmu_notifier mmu_notifier;
+ bool btm_invalidation;
};
struct arm_smmu_master_domain {
@@ -781,9 +778,8 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
size_t granule, bool leaf,
struct arm_smmu_domain *smmu_domain);
-bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd);
-int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain,
- ioasid_t ssid, unsigned long iova, size_t size);
+int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
+ unsigned long iova, size_t size);
#ifdef CONFIG_ARM_SMMU_V3_SVA
bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
@@ -795,8 +791,6 @@ bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master);
void arm_smmu_sva_notifier_synchronize(void);
struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
struct mm_struct *mm);
-void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
- struct device *dev, ioasid_t id);
#else /* CONFIG_ARM_SMMU_V3_SVA */
static inline bool arm_smmu_sva_supported(struct arm_smmu_device *smmu)
{
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 22/27] iommu/arm-smmu-v3: Consolidate freeing the ASID/VMID
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
The SMMUv3 IOTLB is tagged with a VMID/ASID cache tag. Any time the
underlying translation is changed these need to be invalidated. At boot
time the IOTLB starts out empty and all cache tags are available for
allocation.
When a tag is taken out of the allocator the code assumes the IOTLB
doesn't reference it, and immediately programs it into a STE/CD. If the
cache is referencing the tag then it will have stale data and IOMMU will
become incoherent.
Thus, whenever an ASID/VMID is freed back to the allocator we need to know
that the IOTLB doesn't have any references to it. The SVA code correctly
had an invalidation here, but the paging code does not.
Consolidate freeing the VMID/ASID to one place and consistently flush both
ID types before returning to their allocators.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 9 ++---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 36 +++++++++++++------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 +
3 files changed, 29 insertions(+), 17 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index cb0da4e5a5517a..3a9f4ef47c6b6f 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -381,18 +381,13 @@ static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
{
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
- /*
- * Ensure the ASID is empty in the iommu cache before allowing reuse.
- */
- arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
-
/*
* Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can
* still be called/running at this point. We allow the ASID to be
* reused, and if there is a race then it just suffers harmless
* unnecessary invalidation.
*/
- xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
+ arm_smmu_domain_free_id(smmu_domain);
/*
* Actual free is defered to the SRCU callback
@@ -437,7 +432,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
return &smmu_domain->domain;
err_asid:
- xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
+ arm_smmu_domain_free_id(smmu_domain);
err_free:
kfree(smmu_domain);
return ERR_PTR(ret);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 5642321b2124d9..4f22eb810c8dbd 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2296,25 +2296,41 @@ static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
return &smmu_domain->domain;
}
-static void arm_smmu_domain_free(struct iommu_domain *domain)
+/*
+ * Return the domain's ASID or VMID back to the allocator. All IDs in the
+ * allocator do not have an IOTLB entries referencing them.
+ */
+void arm_smmu_domain_free_id(struct arm_smmu_domain *smmu_domain)
{
- struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
struct arm_smmu_device *smmu = smmu_domain->smmu;
- free_io_pgtable_ops(smmu_domain->pgtbl_ops);
+ if ((smmu_domain->stage == ARM_SMMU_DOMAIN_S1 ||
+ smmu_domain->domain.type == IOMMU_DOMAIN_SVA) &&
+ smmu_domain->cd.asid) {
+ arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid);
- /* Free the ASID or VMID */
- if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
/* Prevent SVA from touching the CD while we're freeing it */
mutex_lock(&arm_smmu_asid_lock);
xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
mutex_unlock(&arm_smmu_asid_lock);
- } else {
- struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
- if (cfg->vmid)
- ida_free(&smmu->vmid_map, cfg->vmid);
- }
+ } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S2 &&
+ smmu_domain->s2_cfg.vmid) {
+ struct arm_smmu_cmdq_ent cmd = {
+ .opcode = CMDQ_OP_TLBI_S12_VMALL,
+ .tlbi.vmid = smmu_domain->s2_cfg.vmid
+ };
+ arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
+ ida_free(&smmu->vmid_map, smmu_domain->s2_cfg.vmid);
+ }
+}
+
+static void arm_smmu_domain_free(struct iommu_domain *domain)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+
+ free_io_pgtable_ops(smmu_domain->pgtbl_ops);
+ arm_smmu_domain_free_id(smmu_domain);
kfree(smmu_domain);
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index cfae4d69cd810c..4631f0ac396dc3 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -774,6 +774,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
void arm_smmu_remove_pasid(struct arm_smmu_master *master,
struct arm_smmu_domain *smmu_domain, ioasid_t pasid);
+void arm_smmu_domain_free_id(struct arm_smmu_domain *smmu_domain);
void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
size_t granule, bool leaf,
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 22/27] iommu/arm-smmu-v3: Consolidate freeing the ASID/VMID
@ 2024-03-04 23:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
The SMMUv3 IOTLB is tagged with a VMID/ASID cache tag. Any time the
underlying translation is changed these need to be invalidated. At boot
time the IOTLB starts out empty and all cache tags are available for
allocation.
When a tag is taken out of the allocator the code assumes the IOTLB
doesn't reference it, and immediately programs it into a STE/CD. If the
cache is referencing the tag then it will have stale data and IOMMU will
become incoherent.
Thus, whenever an ASID/VMID is freed back to the allocator we need to know
that the IOTLB doesn't have any references to it. The SVA code correctly
had an invalidation here, but the paging code does not.
Consolidate freeing the VMID/ASID to one place and consistently flush both
ID types before returning to their allocators.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 9 ++---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 36 +++++++++++++------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 +
3 files changed, 29 insertions(+), 17 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index cb0da4e5a5517a..3a9f4ef47c6b6f 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -381,18 +381,13 @@ static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
{
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
- /*
- * Ensure the ASID is empty in the iommu cache before allowing reuse.
- */
- arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
-
/*
* Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can
* still be called/running at this point. We allow the ASID to be
* reused, and if there is a race then it just suffers harmless
* unnecessary invalidation.
*/
- xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
+ arm_smmu_domain_free_id(smmu_domain);
/*
* Actual free is defered to the SRCU callback
@@ -437,7 +432,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
return &smmu_domain->domain;
err_asid:
- xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
+ arm_smmu_domain_free_id(smmu_domain);
err_free:
kfree(smmu_domain);
return ERR_PTR(ret);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 5642321b2124d9..4f22eb810c8dbd 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2296,25 +2296,41 @@ static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
return &smmu_domain->domain;
}
-static void arm_smmu_domain_free(struct iommu_domain *domain)
+/*
+ * Return the domain's ASID or VMID back to the allocator. All IDs in the
+ * allocator do not have an IOTLB entries referencing them.
+ */
+void arm_smmu_domain_free_id(struct arm_smmu_domain *smmu_domain)
{
- struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
struct arm_smmu_device *smmu = smmu_domain->smmu;
- free_io_pgtable_ops(smmu_domain->pgtbl_ops);
+ if ((smmu_domain->stage == ARM_SMMU_DOMAIN_S1 ||
+ smmu_domain->domain.type == IOMMU_DOMAIN_SVA) &&
+ smmu_domain->cd.asid) {
+ arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid);
- /* Free the ASID or VMID */
- if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
/* Prevent SVA from touching the CD while we're freeing it */
mutex_lock(&arm_smmu_asid_lock);
xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
mutex_unlock(&arm_smmu_asid_lock);
- } else {
- struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
- if (cfg->vmid)
- ida_free(&smmu->vmid_map, cfg->vmid);
- }
+ } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S2 &&
+ smmu_domain->s2_cfg.vmid) {
+ struct arm_smmu_cmdq_ent cmd = {
+ .opcode = CMDQ_OP_TLBI_S12_VMALL,
+ .tlbi.vmid = smmu_domain->s2_cfg.vmid
+ };
+ arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
+ ida_free(&smmu->vmid_map, smmu_domain->s2_cfg.vmid);
+ }
+}
+
+static void arm_smmu_domain_free(struct iommu_domain *domain)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+
+ free_io_pgtable_ops(smmu_domain->pgtbl_ops);
+ arm_smmu_domain_free_id(smmu_domain);
kfree(smmu_domain);
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index cfae4d69cd810c..4631f0ac396dc3 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -774,6 +774,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
void arm_smmu_remove_pasid(struct arm_smmu_master *master,
struct arm_smmu_domain *smmu_domain, ioasid_t pasid);
+void arm_smmu_domain_free_id(struct arm_smmu_domain *smmu_domain);
void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
size_t granule, bool leaf,
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 23/27] iommu/arm-smmu-v3: Move the arm_smmu_asid_xa to per-smmu like vmid
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
The SVA BTM and shared cd code was the only thing keeping this as a global
array. Now that is out of the way we can move it to per-smmu.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 39 +++++++++----------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 +--
3 files changed, 22 insertions(+), 24 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 3a9f4ef47c6b6f..3acd699433b7d8 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -418,7 +418,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
smmu_domain->smmu = smmu;
- ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
+ ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain,
XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
if (ret)
goto err_free;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 4f22eb810c8dbd..1a72dd63e0ca14 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -88,9 +88,6 @@ struct arm_smmu_option_prop {
const char *prop;
};
-DEFINE_XARRAY_ALLOC1(arm_smmu_asid_xa);
-DEFINE_MUTEX(arm_smmu_asid_lock);
-
static struct arm_smmu_option_prop arm_smmu_options[] = {
{ ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" },
{ ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"},
@@ -2310,9 +2307,9 @@ void arm_smmu_domain_free_id(struct arm_smmu_domain *smmu_domain)
arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid);
/* Prevent SVA from touching the CD while we're freeing it */
- mutex_lock(&arm_smmu_asid_lock);
- xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
- mutex_unlock(&arm_smmu_asid_lock);
+ mutex_lock(&smmu->asid_lock);
+ xa_erase(&smmu->asid_map, smmu_domain->cd.asid);
+ mutex_unlock(&smmu->asid_lock);
} else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S2 &&
smmu_domain->s2_cfg.vmid) {
struct arm_smmu_cmdq_ent cmd = {
@@ -2342,11 +2339,11 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
/* Prevent SVA from modifying the ASID until it is written to the CD */
- mutex_lock(&arm_smmu_asid_lock);
- ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
+ mutex_lock(&smmu->asid_lock);
+ ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain,
XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
cd->asid = (u16)asid;
- mutex_unlock(&arm_smmu_asid_lock);
+ mutex_unlock(&smmu->asid_lock);
return ret;
}
@@ -2633,7 +2630,7 @@ static int arm_smmu_attach_prepare(struct arm_smmu_master *master,
* arm_smmu_master_domain contents otherwise it could randomly write one
* or the other to the CD.
*/
- lockdep_assert_held(&arm_smmu_asid_lock);
+ lockdep_assert_held(&master->smmu->asid_lock);
state->want_ats = !state->disable_ats && arm_smmu_ats_supported(master);
@@ -2685,7 +2682,7 @@ static int arm_smmu_attach_prepare(struct arm_smmu_master *master,
static void arm_smmu_attach_commit(struct arm_smmu_master *master,
struct attach_state *state)
{
- lockdep_assert_held(&arm_smmu_asid_lock);
+ lockdep_assert_held(&master->smmu->asid_lock);
if (state->want_ats && !master->ats_enabled) {
arm_smmu_enable_ats(master);
@@ -2747,11 +2744,11 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
* This allows the STE and the smmu_domain->devices list to
* be inconsistent during this routine.
*/
- mutex_lock(&arm_smmu_asid_lock);
+ mutex_lock(&smmu->asid_lock);
ret = arm_smmu_attach_prepare(master, domain, &state);
if (ret) {
- mutex_unlock(&arm_smmu_asid_lock);
+ mutex_unlock(&smmu->asid_lock);
return ret;
}
@@ -2775,7 +2772,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
}
arm_smmu_attach_commit(master, &state);
- mutex_unlock(&arm_smmu_asid_lock);
+ mutex_unlock(&smmu->asid_lock);
return 0;
}
@@ -2805,7 +2802,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
if (!cdptr)
return -ENOMEM;
- mutex_lock(&arm_smmu_asid_lock);
+ mutex_lock(&master->smmu->asid_lock);
ret = arm_smmu_attach_prepare(master, &smmu_domain->domain, &state);
if (ret)
goto out_unlock;
@@ -2815,7 +2812,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
arm_smmu_attach_commit(master, &state);
out_unlock:
- mutex_unlock(&arm_smmu_asid_lock);
+ mutex_unlock(&master->smmu->asid_lock);
return ret;
}
@@ -2831,12 +2828,12 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
smmu_domain = to_smmu_domain(domain);
- mutex_lock(&arm_smmu_asid_lock);
+ mutex_lock(&master->smmu->asid_lock);
arm_smmu_clear_cd(master, pasid);
if (master->ats_enabled)
arm_smmu_atc_inv_master(master, pasid);
arm_smmu_remove_master_domain(master, &smmu_domain->domain, pasid);
- mutex_unlock(&arm_smmu_asid_lock);
+ mutex_unlock(&master->smmu->asid_lock);
}
static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
@@ -2852,7 +2849,7 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
* Do not allow any ASID to be changed while are working on the STE,
* otherwise we could miss invalidations.
*/
- mutex_lock(&arm_smmu_asid_lock);
+ mutex_lock(&master->smmu->asid_lock);
/*
* The SMMU does not support enabling ATS with bypass/abort. When the
@@ -2865,7 +2862,7 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
arm_smmu_attach_prepare(master, domain, &state);
arm_smmu_install_ste_for_dev(master, ste);
arm_smmu_attach_commit(master, &state);
- mutex_unlock(&arm_smmu_asid_lock);
+ mutex_unlock(&master->smmu->asid_lock);
/*
* This has to be done after removing the master from the
@@ -3516,6 +3513,8 @@ static int arm_smmu_init_strtab(struct arm_smmu_device *smmu)
smmu->strtab_cfg.strtab_base = reg;
ida_init(&smmu->vmid_map);
+ xa_init_flags(&smmu->asid_map, XA_FLAGS_ALLOC1);
+ mutex_init(&smmu->asid_lock);
return 0;
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 4631f0ac396dc3..6becdbae905598 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -675,6 +675,8 @@ struct arm_smmu_device {
#define ARM_SMMU_MAX_ASIDS (1 << 16)
unsigned int asid_bits;
+ struct xarray asid_map;
+ struct mutex asid_lock;
#define ARM_SMMU_MAX_VMIDS (1 << 16)
unsigned int vmid_bits;
@@ -753,9 +755,6 @@ static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
return container_of(dom, struct arm_smmu_domain, domain);
}
-extern struct xarray arm_smmu_asid_xa;
-extern struct mutex arm_smmu_asid_lock;
-
struct arm_smmu_domain *arm_smmu_domain_alloc(void);
void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 23/27] iommu/arm-smmu-v3: Move the arm_smmu_asid_xa to per-smmu like vmid
@ 2024-03-04 23:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
The SVA BTM and shared cd code was the only thing keeping this as a global
array. Now that is out of the way we can move it to per-smmu.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 39 +++++++++----------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 +--
3 files changed, 22 insertions(+), 24 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 3a9f4ef47c6b6f..3acd699433b7d8 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -418,7 +418,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
smmu_domain->smmu = smmu;
- ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
+ ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain,
XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
if (ret)
goto err_free;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 4f22eb810c8dbd..1a72dd63e0ca14 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -88,9 +88,6 @@ struct arm_smmu_option_prop {
const char *prop;
};
-DEFINE_XARRAY_ALLOC1(arm_smmu_asid_xa);
-DEFINE_MUTEX(arm_smmu_asid_lock);
-
static struct arm_smmu_option_prop arm_smmu_options[] = {
{ ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" },
{ ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"},
@@ -2310,9 +2307,9 @@ void arm_smmu_domain_free_id(struct arm_smmu_domain *smmu_domain)
arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid);
/* Prevent SVA from touching the CD while we're freeing it */
- mutex_lock(&arm_smmu_asid_lock);
- xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
- mutex_unlock(&arm_smmu_asid_lock);
+ mutex_lock(&smmu->asid_lock);
+ xa_erase(&smmu->asid_map, smmu_domain->cd.asid);
+ mutex_unlock(&smmu->asid_lock);
} else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S2 &&
smmu_domain->s2_cfg.vmid) {
struct arm_smmu_cmdq_ent cmd = {
@@ -2342,11 +2339,11 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
/* Prevent SVA from modifying the ASID until it is written to the CD */
- mutex_lock(&arm_smmu_asid_lock);
- ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
+ mutex_lock(&smmu->asid_lock);
+ ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain,
XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
cd->asid = (u16)asid;
- mutex_unlock(&arm_smmu_asid_lock);
+ mutex_unlock(&smmu->asid_lock);
return ret;
}
@@ -2633,7 +2630,7 @@ static int arm_smmu_attach_prepare(struct arm_smmu_master *master,
* arm_smmu_master_domain contents otherwise it could randomly write one
* or the other to the CD.
*/
- lockdep_assert_held(&arm_smmu_asid_lock);
+ lockdep_assert_held(&master->smmu->asid_lock);
state->want_ats = !state->disable_ats && arm_smmu_ats_supported(master);
@@ -2685,7 +2682,7 @@ static int arm_smmu_attach_prepare(struct arm_smmu_master *master,
static void arm_smmu_attach_commit(struct arm_smmu_master *master,
struct attach_state *state)
{
- lockdep_assert_held(&arm_smmu_asid_lock);
+ lockdep_assert_held(&master->smmu->asid_lock);
if (state->want_ats && !master->ats_enabled) {
arm_smmu_enable_ats(master);
@@ -2747,11 +2744,11 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
* This allows the STE and the smmu_domain->devices list to
* be inconsistent during this routine.
*/
- mutex_lock(&arm_smmu_asid_lock);
+ mutex_lock(&smmu->asid_lock);
ret = arm_smmu_attach_prepare(master, domain, &state);
if (ret) {
- mutex_unlock(&arm_smmu_asid_lock);
+ mutex_unlock(&smmu->asid_lock);
return ret;
}
@@ -2775,7 +2772,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
}
arm_smmu_attach_commit(master, &state);
- mutex_unlock(&arm_smmu_asid_lock);
+ mutex_unlock(&smmu->asid_lock);
return 0;
}
@@ -2805,7 +2802,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
if (!cdptr)
return -ENOMEM;
- mutex_lock(&arm_smmu_asid_lock);
+ mutex_lock(&master->smmu->asid_lock);
ret = arm_smmu_attach_prepare(master, &smmu_domain->domain, &state);
if (ret)
goto out_unlock;
@@ -2815,7 +2812,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
arm_smmu_attach_commit(master, &state);
out_unlock:
- mutex_unlock(&arm_smmu_asid_lock);
+ mutex_unlock(&master->smmu->asid_lock);
return ret;
}
@@ -2831,12 +2828,12 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
smmu_domain = to_smmu_domain(domain);
- mutex_lock(&arm_smmu_asid_lock);
+ mutex_lock(&master->smmu->asid_lock);
arm_smmu_clear_cd(master, pasid);
if (master->ats_enabled)
arm_smmu_atc_inv_master(master, pasid);
arm_smmu_remove_master_domain(master, &smmu_domain->domain, pasid);
- mutex_unlock(&arm_smmu_asid_lock);
+ mutex_unlock(&master->smmu->asid_lock);
}
static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
@@ -2852,7 +2849,7 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
* Do not allow any ASID to be changed while are working on the STE,
* otherwise we could miss invalidations.
*/
- mutex_lock(&arm_smmu_asid_lock);
+ mutex_lock(&master->smmu->asid_lock);
/*
* The SMMU does not support enabling ATS with bypass/abort. When the
@@ -2865,7 +2862,7 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
arm_smmu_attach_prepare(master, domain, &state);
arm_smmu_install_ste_for_dev(master, ste);
arm_smmu_attach_commit(master, &state);
- mutex_unlock(&arm_smmu_asid_lock);
+ mutex_unlock(&master->smmu->asid_lock);
/*
* This has to be done after removing the master from the
@@ -3516,6 +3513,8 @@ static int arm_smmu_init_strtab(struct arm_smmu_device *smmu)
smmu->strtab_cfg.strtab_base = reg;
ida_init(&smmu->vmid_map);
+ xa_init_flags(&smmu->asid_map, XA_FLAGS_ALLOC1);
+ mutex_init(&smmu->asid_lock);
return 0;
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 4631f0ac396dc3..6becdbae905598 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -675,6 +675,8 @@ struct arm_smmu_device {
#define ARM_SMMU_MAX_ASIDS (1 << 16)
unsigned int asid_bits;
+ struct xarray asid_map;
+ struct mutex asid_lock;
#define ARM_SMMU_MAX_VMIDS (1 << 16)
unsigned int vmid_bits;
@@ -753,9 +755,6 @@ static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
return container_of(dom, struct arm_smmu_domain, domain);
}
-extern struct xarray arm_smmu_asid_xa;
-extern struct mutex arm_smmu_asid_lock;
-
struct arm_smmu_domain *arm_smmu_domain_alloc(void);
void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 24/27] iommu/arm-smmu-v3: Bring back SVA BTM support
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
BTM support is a feature where the CPU TLB invalidation can be forwarded
to the IOMMU and also invalidate the IOTLB. For this to work the CPU and
IOMMU ASID must be the same.
Retain the prior SVA design here of keeping the ASID allocator for the
IOMMU private to SMMU and force SVA domains to set an ASID that matches
the CPU ASID.
This requires changing the ASID assigned to a S1 domain if it happens to
be overlapping with the required CPU ASID. We hold on to the CPU ASID so
long as the SVA iommu_domain exists, so SVA domain conflict is not
possible.
With the asid per-smmu we no longer have a problem that two per-smmu
iommu_domain's would need to share a CPU ASID entry in the IOMMU's xarray.
Use the same ASID move algorithm for the S1 domains as before with some
streamlining around how the xarray is being used. Do not synchronize the
ASID's if BTM mode is not supported. Just leave BTM features off
everywhere.
Audit all the places that touch cd->asid and think carefully about how the
locking works with the change to the cd->asid by the move algorithm. Use
xarray internal locking during xa_alloc() instead of double locking. Add a
note that concurrent S1 invalidation doesn't fully work.
Note that this is all still dead code, ARM_SMMU_FEAT_BTM is never set.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 133 ++++++++++++++++--
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 15 +-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 +-
3 files changed, 129 insertions(+), 21 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 3acd699433b7d8..0b5aeaa3a85575 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -15,12 +15,33 @@
static DEFINE_MUTEX(sva_lock);
-static void __maybe_unused
-arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
+static int arm_smmu_realloc_s1_domain_asid(struct arm_smmu_device *smmu,
+ struct arm_smmu_domain *smmu_domain)
{
struct arm_smmu_master_domain *master_domain;
+ u32 old_asid = smmu_domain->cd.asid;
struct arm_smmu_cd target_cd;
unsigned long flags;
+ int ret;
+
+ lockdep_assert_held(&smmu->asid_lock);
+
+ /*
+ * FIXME: The unmap and invalidation path doesn't take any locks but
+ * this is not fully safe. Since updating the CD tables is not atomic
+ * there is always a hole where invalidating only one ASID of two active
+ * ASIDs during unmap will cause the IOTLB to become stale.
+ *
+ * This approach is to hopefully shift the racing CPUs to the new ASID
+ * before we start programming the HW. This increases the chance that
+ * racing IOPTE changes will pick up an invalidation for the new ASID
+ * and we achieve eventual consistency. For the brief period where the
+ * old ASID is still in the CD entries it will become incoherent.
+ */
+ ret = xa_alloc(&smmu->asid_map, &smmu_domain->cd.asid, smmu_domain,
+ XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
+ if (ret)
+ return ret;
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
list_for_each_entry(master_domain, &smmu_domain->devices, devices_elm) {
@@ -36,6 +57,10 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
&target_cd);
}
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
+
+ /* Clean the ASID we are about to assign to a new translation */
+ arm_smmu_tlb_inv_asid(smmu, old_asid);
+ return 0;
}
static u64 page_size_to_cd(void)
@@ -148,12 +173,12 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
}
if (!smmu_domain->btm_invalidation) {
+ ioasid_t asid = READ_ONCE(smmu_domain->cd.asid);
+
if (!size)
- arm_smmu_tlb_inv_asid(smmu_domain->smmu,
- smmu_domain->cd.asid);
+ arm_smmu_tlb_inv_asid(smmu_domain->smmu, asid);
else
- arm_smmu_tlb_inv_range_asid(start, size,
- smmu_domain->cd.asid,
+ arm_smmu_tlb_inv_range_asid(start, size, asid,
PAGE_SIZE, false,
smmu_domain);
}
@@ -182,6 +207,8 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid);
if (WARN_ON(!cdptr))
continue;
+
+ /* An SVA ASID never changes, no asid_lock required */
arm_smmu_make_sva_cd(&target, master, NULL,
smmu_domain->cd.asid,
smmu_domain->btm_invalidation);
@@ -388,6 +415,8 @@ static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
* unnecessary invalidation.
*/
arm_smmu_domain_free_id(smmu_domain);
+ if (smmu_domain->btm_invalidation)
+ arm64_mm_context_put(domain->mm);
/*
* Actual free is defered to the SRCU callback
@@ -401,13 +430,97 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
.free = arm_smmu_sva_domain_free
};
+static int arm_smmu_share_asid(struct arm_smmu_device *smmu,
+ struct arm_smmu_domain *smmu_domain,
+ struct mm_struct *mm)
+{
+ struct arm_smmu_domain *old_s1_domain;
+ int ret;
+
+ /*
+ * Notice that BTM is never currently enabled, this is all dead code.
+ * The specification cautions:
+ *
+ * Note: Arm expects that SMMU stage 2 address spaces are generally
+ * shared with their respective PE virtual machine stage 2
+ * configuration. If broadcast invalidation is required to be avoided
+ * for a particular SMMU stage 2 address space, Arm recommends that a
+ * hypervisor configures the STE with a VMID that is not allocated for
+ * virtual machine use on the PEs
+ *
+ * However, in Linux, both KVM and SMMU think they own the VMID pool.
+ * Unfortunately the ARM design is problematic for Linux as we do not
+ * currently share the S2 table with KVM. This creates a situation where
+ * the S2 needs to have the same VMID as KVM in order to allow the guest
+ * to use BTM, however we must still invalidate the S2 directly since it
+ * is a different radix tree. What Linux would like is something like
+ * ASET for the STE to disable BTM only for the S2.
+ *
+ * Arguably in a system with BTM the driver should prefer to use a S1
+ * table in all cases execpt when explicitly asked to create a nesting
+ * parent. Then it should use the VMID of KVM to enable BTM in the
+ * guest. We cannot optimize away the resulting double invalidation of
+ * the S2 :( Or we simply ignore BTM entirely as we are doing now.
+ */
+ if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM))
+ return xa_alloc(&smmu->asid_map, &smmu_domain->cd.asid,
+ smmu_domain,
+ XA_LIMIT(1, (1 << smmu->asid_bits) - 1),
+ GFP_KERNEL);
+
+ /* At this point the caller ensures we have a mmget() */
+ smmu_domain->cd.asid = arm64_mm_context_get(mm);
+
+ mutex_lock(&smmu->asid_lock);
+ old_s1_domain = xa_store(&smmu->asid_map, smmu_domain->cd.asid,
+ smmu_domain, GFP_KERNEL);
+ if (xa_err(old_s1_domain)) {
+ ret = xa_err(old_s1_domain);
+ goto out_put_asid;
+ }
+
+ /*
+ * In BTM mode the CPU ASID and the IOMMU ASID have to be the same.
+ * Unfortunately we run separate allocators for this and the IOMMU
+ * ASID can already have been assigned to a S1 domain. SVA domains
+ * always align to their CPU ASIDs. In this case we change
+ * the S1 domain's ASID, update the CD entry and flush the caches.
+ *
+ * This is a bit tricky, all the places writing to a S1 CD, reading the
+ * S1 ASID, or doing xa_erase must hold the asid_lock or xa_lock to
+ * avoid IOTLB incoherence.
+ */
+ if (old_s1_domain) {
+ if (WARN_ON(old_s1_domain->domain.type == IOMMU_DOMAIN_SVA)) {
+ ret = -EINVAL;
+ goto out_restore_s1;
+ }
+ ret = arm_smmu_realloc_s1_domain_asid(smmu, old_s1_domain);
+ if (ret)
+ goto out_restore_s1;
+ }
+
+ smmu_domain->btm_invalidation = true;
+
+ ret = 0;
+ goto out_unlock;
+
+out_restore_s1:
+ xa_store(&smmu->asid_map, smmu_domain->cd.asid, old_s1_domain,
+ GFP_KERNEL);
+out_put_asid:
+ arm64_mm_context_put(mm);
+out_unlock:
+ mutex_unlock(&smmu->asid_lock);
+ return ret;
+}
+
struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
struct mm_struct *mm)
{
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct arm_smmu_device *smmu = master->smmu;
struct arm_smmu_domain *smmu_domain;
- u32 asid;
int ret;
smmu_domain = arm_smmu_domain_alloc();
@@ -418,12 +531,10 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
smmu_domain->smmu = smmu;
- ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain,
- XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
+ ret = arm_smmu_share_asid(smmu, smmu_domain, mm);
if (ret)
goto err_free;
- smmu_domain->cd.asid = asid;
smmu_domain->mmu_notifier.ops = &arm_smmu_mmu_notifier_ops;
ret = mmu_notifier_register(&smmu_domain->mmu_notifier, mm);
if (ret)
@@ -433,6 +544,8 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
err_asid:
arm_smmu_domain_free_id(smmu_domain);
+ if (smmu_domain->btm_invalidation)
+ arm64_mm_context_put(mm);
err_free:
kfree(smmu_domain);
return ERR_PTR(ret);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 1a72dd63e0ca14..01737a0fc82828 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1314,6 +1314,8 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr =
&pgtbl_cfg->arm_lpae_s1_cfg.tcr;
+ lockdep_assert_held(&master->smmu->asid_lock);
+
memset(target, 0, sizeof(*target));
target->data[0] = cpu_to_le64(
@@ -2089,7 +2091,7 @@ static void arm_smmu_tlb_inv_context(void *cookie)
* careful, 007.
*/
if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
- arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid);
+ arm_smmu_tlb_inv_asid(smmu, READ_ONCE(smmu_domain->cd.asid));
} else {
cmd.opcode = CMDQ_OP_TLBI_S12_VMALL;
cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid;
@@ -2334,17 +2336,10 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
struct arm_smmu_domain *smmu_domain)
{
- int ret;
- u32 asid;
struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
- /* Prevent SVA from modifying the ASID until it is written to the CD */
- mutex_lock(&smmu->asid_lock);
- ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain,
- XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
- cd->asid = (u16)asid;
- mutex_unlock(&smmu->asid_lock);
- return ret;
+ return xa_alloc(&smmu->asid_map, &cd->asid, smmu_domain,
+ XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
}
static int arm_smmu_domain_finalise_s2(struct arm_smmu_device *smmu,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 6becdbae905598..a23593f1830106 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -586,7 +586,7 @@ struct arm_smmu_strtab_l1_desc {
};
struct arm_smmu_ctx_desc {
- u16 asid;
+ u32 asid;
};
struct arm_smmu_l1_ctx_desc {
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 24/27] iommu/arm-smmu-v3: Bring back SVA BTM support
@ 2024-03-04 23:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
BTM support is a feature where the CPU TLB invalidation can be forwarded
to the IOMMU and also invalidate the IOTLB. For this to work the CPU and
IOMMU ASID must be the same.
Retain the prior SVA design here of keeping the ASID allocator for the
IOMMU private to SMMU and force SVA domains to set an ASID that matches
the CPU ASID.
This requires changing the ASID assigned to a S1 domain if it happens to
be overlapping with the required CPU ASID. We hold on to the CPU ASID so
long as the SVA iommu_domain exists, so SVA domain conflict is not
possible.
With the asid per-smmu we no longer have a problem that two per-smmu
iommu_domain's would need to share a CPU ASID entry in the IOMMU's xarray.
Use the same ASID move algorithm for the S1 domains as before with some
streamlining around how the xarray is being used. Do not synchronize the
ASID's if BTM mode is not supported. Just leave BTM features off
everywhere.
Audit all the places that touch cd->asid and think carefully about how the
locking works with the change to the cd->asid by the move algorithm. Use
xarray internal locking during xa_alloc() instead of double locking. Add a
note that concurrent S1 invalidation doesn't fully work.
Note that this is all still dead code, ARM_SMMU_FEAT_BTM is never set.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 133 ++++++++++++++++--
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 15 +-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 +-
3 files changed, 129 insertions(+), 21 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 3acd699433b7d8..0b5aeaa3a85575 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -15,12 +15,33 @@
static DEFINE_MUTEX(sva_lock);
-static void __maybe_unused
-arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
+static int arm_smmu_realloc_s1_domain_asid(struct arm_smmu_device *smmu,
+ struct arm_smmu_domain *smmu_domain)
{
struct arm_smmu_master_domain *master_domain;
+ u32 old_asid = smmu_domain->cd.asid;
struct arm_smmu_cd target_cd;
unsigned long flags;
+ int ret;
+
+ lockdep_assert_held(&smmu->asid_lock);
+
+ /*
+ * FIXME: The unmap and invalidation path doesn't take any locks but
+ * this is not fully safe. Since updating the CD tables is not atomic
+ * there is always a hole where invalidating only one ASID of two active
+ * ASIDs during unmap will cause the IOTLB to become stale.
+ *
+ * This approach is to hopefully shift the racing CPUs to the new ASID
+ * before we start programming the HW. This increases the chance that
+ * racing IOPTE changes will pick up an invalidation for the new ASID
+ * and we achieve eventual consistency. For the brief period where the
+ * old ASID is still in the CD entries it will become incoherent.
+ */
+ ret = xa_alloc(&smmu->asid_map, &smmu_domain->cd.asid, smmu_domain,
+ XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
+ if (ret)
+ return ret;
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
list_for_each_entry(master_domain, &smmu_domain->devices, devices_elm) {
@@ -36,6 +57,10 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
&target_cd);
}
spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
+
+ /* Clean the ASID we are about to assign to a new translation */
+ arm_smmu_tlb_inv_asid(smmu, old_asid);
+ return 0;
}
static u64 page_size_to_cd(void)
@@ -148,12 +173,12 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
}
if (!smmu_domain->btm_invalidation) {
+ ioasid_t asid = READ_ONCE(smmu_domain->cd.asid);
+
if (!size)
- arm_smmu_tlb_inv_asid(smmu_domain->smmu,
- smmu_domain->cd.asid);
+ arm_smmu_tlb_inv_asid(smmu_domain->smmu, asid);
else
- arm_smmu_tlb_inv_range_asid(start, size,
- smmu_domain->cd.asid,
+ arm_smmu_tlb_inv_range_asid(start, size, asid,
PAGE_SIZE, false,
smmu_domain);
}
@@ -182,6 +207,8 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid);
if (WARN_ON(!cdptr))
continue;
+
+ /* An SVA ASID never changes, no asid_lock required */
arm_smmu_make_sva_cd(&target, master, NULL,
smmu_domain->cd.asid,
smmu_domain->btm_invalidation);
@@ -388,6 +415,8 @@ static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
* unnecessary invalidation.
*/
arm_smmu_domain_free_id(smmu_domain);
+ if (smmu_domain->btm_invalidation)
+ arm64_mm_context_put(domain->mm);
/*
* Actual free is defered to the SRCU callback
@@ -401,13 +430,97 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
.free = arm_smmu_sva_domain_free
};
+static int arm_smmu_share_asid(struct arm_smmu_device *smmu,
+ struct arm_smmu_domain *smmu_domain,
+ struct mm_struct *mm)
+{
+ struct arm_smmu_domain *old_s1_domain;
+ int ret;
+
+ /*
+ * Notice that BTM is never currently enabled, this is all dead code.
+ * The specification cautions:
+ *
+ * Note: Arm expects that SMMU stage 2 address spaces are generally
+ * shared with their respective PE virtual machine stage 2
+ * configuration. If broadcast invalidation is required to be avoided
+ * for a particular SMMU stage 2 address space, Arm recommends that a
+ * hypervisor configures the STE with a VMID that is not allocated for
+ * virtual machine use on the PEs
+ *
+ * However, in Linux, both KVM and SMMU think they own the VMID pool.
+ * Unfortunately the ARM design is problematic for Linux as we do not
+ * currently share the S2 table with KVM. This creates a situation where
+ * the S2 needs to have the same VMID as KVM in order to allow the guest
+ * to use BTM, however we must still invalidate the S2 directly since it
+ * is a different radix tree. What Linux would like is something like
+ * ASET for the STE to disable BTM only for the S2.
+ *
+ * Arguably in a system with BTM the driver should prefer to use a S1
+ * table in all cases execpt when explicitly asked to create a nesting
+ * parent. Then it should use the VMID of KVM to enable BTM in the
+ * guest. We cannot optimize away the resulting double invalidation of
+ * the S2 :( Or we simply ignore BTM entirely as we are doing now.
+ */
+ if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM))
+ return xa_alloc(&smmu->asid_map, &smmu_domain->cd.asid,
+ smmu_domain,
+ XA_LIMIT(1, (1 << smmu->asid_bits) - 1),
+ GFP_KERNEL);
+
+ /* At this point the caller ensures we have a mmget() */
+ smmu_domain->cd.asid = arm64_mm_context_get(mm);
+
+ mutex_lock(&smmu->asid_lock);
+ old_s1_domain = xa_store(&smmu->asid_map, smmu_domain->cd.asid,
+ smmu_domain, GFP_KERNEL);
+ if (xa_err(old_s1_domain)) {
+ ret = xa_err(old_s1_domain);
+ goto out_put_asid;
+ }
+
+ /*
+ * In BTM mode the CPU ASID and the IOMMU ASID have to be the same.
+ * Unfortunately we run separate allocators for this and the IOMMU
+ * ASID can already have been assigned to a S1 domain. SVA domains
+ * always align to their CPU ASIDs. In this case we change
+ * the S1 domain's ASID, update the CD entry and flush the caches.
+ *
+ * This is a bit tricky, all the places writing to a S1 CD, reading the
+ * S1 ASID, or doing xa_erase must hold the asid_lock or xa_lock to
+ * avoid IOTLB incoherence.
+ */
+ if (old_s1_domain) {
+ if (WARN_ON(old_s1_domain->domain.type == IOMMU_DOMAIN_SVA)) {
+ ret = -EINVAL;
+ goto out_restore_s1;
+ }
+ ret = arm_smmu_realloc_s1_domain_asid(smmu, old_s1_domain);
+ if (ret)
+ goto out_restore_s1;
+ }
+
+ smmu_domain->btm_invalidation = true;
+
+ ret = 0;
+ goto out_unlock;
+
+out_restore_s1:
+ xa_store(&smmu->asid_map, smmu_domain->cd.asid, old_s1_domain,
+ GFP_KERNEL);
+out_put_asid:
+ arm64_mm_context_put(mm);
+out_unlock:
+ mutex_unlock(&smmu->asid_lock);
+ return ret;
+}
+
struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
struct mm_struct *mm)
{
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct arm_smmu_device *smmu = master->smmu;
struct arm_smmu_domain *smmu_domain;
- u32 asid;
int ret;
smmu_domain = arm_smmu_domain_alloc();
@@ -418,12 +531,10 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
smmu_domain->smmu = smmu;
- ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain,
- XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
+ ret = arm_smmu_share_asid(smmu, smmu_domain, mm);
if (ret)
goto err_free;
- smmu_domain->cd.asid = asid;
smmu_domain->mmu_notifier.ops = &arm_smmu_mmu_notifier_ops;
ret = mmu_notifier_register(&smmu_domain->mmu_notifier, mm);
if (ret)
@@ -433,6 +544,8 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
err_asid:
arm_smmu_domain_free_id(smmu_domain);
+ if (smmu_domain->btm_invalidation)
+ arm64_mm_context_put(mm);
err_free:
kfree(smmu_domain);
return ERR_PTR(ret);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 1a72dd63e0ca14..01737a0fc82828 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1314,6 +1314,8 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr =
&pgtbl_cfg->arm_lpae_s1_cfg.tcr;
+ lockdep_assert_held(&master->smmu->asid_lock);
+
memset(target, 0, sizeof(*target));
target->data[0] = cpu_to_le64(
@@ -2089,7 +2091,7 @@ static void arm_smmu_tlb_inv_context(void *cookie)
* careful, 007.
*/
if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
- arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid);
+ arm_smmu_tlb_inv_asid(smmu, READ_ONCE(smmu_domain->cd.asid));
} else {
cmd.opcode = CMDQ_OP_TLBI_S12_VMALL;
cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid;
@@ -2334,17 +2336,10 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
struct arm_smmu_domain *smmu_domain)
{
- int ret;
- u32 asid;
struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
- /* Prevent SVA from modifying the ASID until it is written to the CD */
- mutex_lock(&smmu->asid_lock);
- ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain,
- XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
- cd->asid = (u16)asid;
- mutex_unlock(&smmu->asid_lock);
- return ret;
+ return xa_alloc(&smmu->asid_map, &cd->asid, smmu_domain,
+ XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
}
static int arm_smmu_domain_finalise_s2(struct arm_smmu_device *smmu,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 6becdbae905598..a23593f1830106 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -586,7 +586,7 @@ struct arm_smmu_strtab_l1_desc {
};
struct arm_smmu_ctx_desc {
- u16 asid;
+ u32 asid;
};
struct arm_smmu_l1_ctx_desc {
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 25/27] iommu/arm-smmu-v3: Allow IDENTITY/BLOCKED to be set while PASID is used
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
The HW supports this, use the S1DSS bits to configure the behavior
of SSID=0 which is the RID's translation.
If SSID's are currently being used in the CD table then just update the
S1DSS bits in the STE, remove the master_domain and leave ATS alone.
For iommufd the driver design has a small problem that all the unused CD
table entries are set with V=0 which will generate an event if VFIO
userspace tries to use the CD entry. This patch extends this problem to
include the RID as well if PASID is being used.
For BLOCKED with used PASIDs the
F_STREAM_DISABLED (STRTAB_STE_1_S1DSS_TERMINATE) event is generated on
untagged traffic and a substream CD table entry with V=0 (removed pasid)
will generate C_BAD_CD. Arguably there is no advantage to using S1DSS over
the CD entry 0 with V=0.
As we don't yet support PASID in iommufd this is a problem to resolve
later, possibly by using EPD0 for unused CD table entries instead of V=0,
and not using S1DSS for BLOCKED.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 65 +++++++++++++++------
1 file changed, 47 insertions(+), 18 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 01737a0fc82828..6b5a30c86dda2d 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1009,6 +1009,14 @@ static void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits)
STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW |
STRTAB_STE_1_EATS);
used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);
+
+ /*
+ * See 13.5 Summary of attribute/permission configuration fields
+ * for the SHCFG behavior.
+ */
+ if (FIELD_GET(STRTAB_STE_1_S1DSS, le64_to_cpu(ent[1])) ==
+ STRTAB_STE_1_S1DSS_BYPASS)
+ used_bits[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG);
}
/* S2 translates */
@@ -1522,7 +1530,7 @@ static void arm_smmu_make_bypass_ste(struct arm_smmu_ste *target)
static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
struct arm_smmu_master *master,
- bool ats_enabled)
+ bool ats_enabled, unsigned int s1dss)
{
struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
struct arm_smmu_device *smmu = master->smmu;
@@ -1536,7 +1544,7 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
FIELD_PREP(STRTAB_STE_0_S1CDMAX, cd_table->s1cdmax));
target->data[1] = cpu_to_le64(
- FIELD_PREP(STRTAB_STE_1_S1DSS, STRTAB_STE_1_S1DSS_SSID0) |
+ FIELD_PREP(STRTAB_STE_1_S1DSS, s1dss) |
FIELD_PREP(STRTAB_STE_1_S1CIR, STRTAB_STE_1_S1C_CACHE_WBRA) |
FIELD_PREP(STRTAB_STE_1_S1COR, STRTAB_STE_1_S1C_CACHE_WBRA) |
FIELD_PREP(STRTAB_STE_1_S1CSH, ARM_SMMU_SH_ISH) |
@@ -1547,6 +1555,10 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
FIELD_PREP(STRTAB_STE_1_EATS,
ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0));
+ if (s1dss == STRTAB_STE_1_S1DSS_BYPASS)
+ target->data[1] |= cpu_to_le64(FIELD_PREP(
+ STRTAB_STE_1_SHCFG, STRTAB_STE_1_SHCFG_INCOMING));
+
if (smmu->features & ARM_SMMU_FEAT_E2H) {
/*
* To support BTM the streamworld needs to match the
@@ -2754,7 +2766,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
&target_cd);
- arm_smmu_make_cdtable_ste(&target, master, state.want_ats);
+ arm_smmu_make_cdtable_ste(&target, master, state.want_ats,
+ STRTAB_STE_1_S1DSS_SSID0);
arm_smmu_install_ste_for_dev(master, &target);
break;
}
@@ -2831,15 +2844,14 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
mutex_unlock(&master->smmu->asid_lock);
}
-static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
- struct device *dev, struct arm_smmu_ste *ste)
+static void arm_smmu_attach_dev_ste(struct iommu_domain *domain,
+ struct device *dev,
+ struct arm_smmu_ste *ste,
+ unsigned int s1dss)
{
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct attach_state state = {.ssid = IOMMU_NO_PASID};
- if (arm_smmu_ssids_in_use(&master->cd_table))
- return -EBUSY;
-
/*
* Do not allow any ASID to be changed while are working on the STE,
* otherwise we could miss invalidations.
@@ -2847,14 +2859,29 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
mutex_lock(&master->smmu->asid_lock);
/*
- * The SMMU does not support enabling ATS with bypass/abort. When the
- * STE is in bypass (STE.Config[2:0] == 0b100), ATS Translation Requests
- * and Translated transactions are denied as though ATS is disabled for
- * the stream (STE.EATS == 0b00), causing F_BAD_ATS_TREQ and
- * F_TRANSL_FORBIDDEN events (IHI0070Ea 5.2 Stream Table Entry).
+ * If the CD table is not in use we can use the provided STE, otherwise
+ * we use a cdtable STE with the provided S1DSS.
*/
- state.disable_ats = true;
- arm_smmu_attach_prepare(master, domain, &state);
+ if (arm_smmu_ssids_in_use(&master->cd_table)) {
+ /*
+ * If a CD table has to be present then we need to run with ATS
+ * on even though the RID will fail ATS queries with UR. This is
+ * because we have no idea what the PASID's need.
+ */
+ arm_smmu_attach_prepare(master, domain, &state);
+ arm_smmu_make_cdtable_ste(ste, master, state.want_ats, s1dss);
+ } else {
+ /*
+ * The SMMU does not support enabling ATS with bypass/abort.
+ * When the STE is in bypass (STE.Config[2:0] == 0b100), ATS
+ * Translation Requests and Translated transactions are denied
+ * as though ATS is disabled for the stream (STE.EATS == 0b00),
+ * causing F_BAD_ATS_TREQ and F_TRANSL_FORBIDDEN events
+ * (IHI0070Ea 5.2 Stream Table Entry).
+ */
+ state.disable_ats = true;
+ arm_smmu_attach_prepare(master, domain, &state);
+ }
arm_smmu_install_ste_for_dev(master, ste);
arm_smmu_attach_commit(master, &state);
mutex_unlock(&master->smmu->asid_lock);
@@ -2865,7 +2892,6 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
* descriptor from arm_smmu_share_asid().
*/
arm_smmu_clear_cd(master, IOMMU_NO_PASID);
- return 0;
}
static int arm_smmu_attach_dev_identity(struct iommu_domain *domain,
@@ -2874,7 +2900,8 @@ static int arm_smmu_attach_dev_identity(struct iommu_domain *domain,
struct arm_smmu_ste ste;
arm_smmu_make_bypass_ste(&ste);
- return arm_smmu_attach_dev_ste(domain, dev, &ste);
+ arm_smmu_attach_dev_ste(domain, dev, &ste, STRTAB_STE_1_S1DSS_BYPASS);
+ return 0;
}
static const struct iommu_domain_ops arm_smmu_identity_ops = {
@@ -2892,7 +2919,9 @@ static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain,
struct arm_smmu_ste ste;
arm_smmu_make_abort_ste(&ste);
- return arm_smmu_attach_dev_ste(domain, dev, &ste);
+ arm_smmu_attach_dev_ste(domain, dev, &ste,
+ STRTAB_STE_1_S1DSS_TERMINATE);
+ return 0;
}
static const struct iommu_domain_ops arm_smmu_blocked_ops = {
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 25/27] iommu/arm-smmu-v3: Allow IDENTITY/BLOCKED to be set while PASID is used
@ 2024-03-04 23:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
The HW supports this, use the S1DSS bits to configure the behavior
of SSID=0 which is the RID's translation.
If SSID's are currently being used in the CD table then just update the
S1DSS bits in the STE, remove the master_domain and leave ATS alone.
For iommufd the driver design has a small problem that all the unused CD
table entries are set with V=0 which will generate an event if VFIO
userspace tries to use the CD entry. This patch extends this problem to
include the RID as well if PASID is being used.
For BLOCKED with used PASIDs the
F_STREAM_DISABLED (STRTAB_STE_1_S1DSS_TERMINATE) event is generated on
untagged traffic and a substream CD table entry with V=0 (removed pasid)
will generate C_BAD_CD. Arguably there is no advantage to using S1DSS over
the CD entry 0 with V=0.
As we don't yet support PASID in iommufd this is a problem to resolve
later, possibly by using EPD0 for unused CD table entries instead of V=0,
and not using S1DSS for BLOCKED.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 65 +++++++++++++++------
1 file changed, 47 insertions(+), 18 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 01737a0fc82828..6b5a30c86dda2d 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1009,6 +1009,14 @@ static void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits)
STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW |
STRTAB_STE_1_EATS);
used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);
+
+ /*
+ * See 13.5 Summary of attribute/permission configuration fields
+ * for the SHCFG behavior.
+ */
+ if (FIELD_GET(STRTAB_STE_1_S1DSS, le64_to_cpu(ent[1])) ==
+ STRTAB_STE_1_S1DSS_BYPASS)
+ used_bits[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG);
}
/* S2 translates */
@@ -1522,7 +1530,7 @@ static void arm_smmu_make_bypass_ste(struct arm_smmu_ste *target)
static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
struct arm_smmu_master *master,
- bool ats_enabled)
+ bool ats_enabled, unsigned int s1dss)
{
struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
struct arm_smmu_device *smmu = master->smmu;
@@ -1536,7 +1544,7 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
FIELD_PREP(STRTAB_STE_0_S1CDMAX, cd_table->s1cdmax));
target->data[1] = cpu_to_le64(
- FIELD_PREP(STRTAB_STE_1_S1DSS, STRTAB_STE_1_S1DSS_SSID0) |
+ FIELD_PREP(STRTAB_STE_1_S1DSS, s1dss) |
FIELD_PREP(STRTAB_STE_1_S1CIR, STRTAB_STE_1_S1C_CACHE_WBRA) |
FIELD_PREP(STRTAB_STE_1_S1COR, STRTAB_STE_1_S1C_CACHE_WBRA) |
FIELD_PREP(STRTAB_STE_1_S1CSH, ARM_SMMU_SH_ISH) |
@@ -1547,6 +1555,10 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
FIELD_PREP(STRTAB_STE_1_EATS,
ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0));
+ if (s1dss == STRTAB_STE_1_S1DSS_BYPASS)
+ target->data[1] |= cpu_to_le64(FIELD_PREP(
+ STRTAB_STE_1_SHCFG, STRTAB_STE_1_SHCFG_INCOMING));
+
if (smmu->features & ARM_SMMU_FEAT_E2H) {
/*
* To support BTM the streamworld needs to match the
@@ -2754,7 +2766,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
&target_cd);
- arm_smmu_make_cdtable_ste(&target, master, state.want_ats);
+ arm_smmu_make_cdtable_ste(&target, master, state.want_ats,
+ STRTAB_STE_1_S1DSS_SSID0);
arm_smmu_install_ste_for_dev(master, &target);
break;
}
@@ -2831,15 +2844,14 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
mutex_unlock(&master->smmu->asid_lock);
}
-static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
- struct device *dev, struct arm_smmu_ste *ste)
+static void arm_smmu_attach_dev_ste(struct iommu_domain *domain,
+ struct device *dev,
+ struct arm_smmu_ste *ste,
+ unsigned int s1dss)
{
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct attach_state state = {.ssid = IOMMU_NO_PASID};
- if (arm_smmu_ssids_in_use(&master->cd_table))
- return -EBUSY;
-
/*
* Do not allow any ASID to be changed while are working on the STE,
* otherwise we could miss invalidations.
@@ -2847,14 +2859,29 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
mutex_lock(&master->smmu->asid_lock);
/*
- * The SMMU does not support enabling ATS with bypass/abort. When the
- * STE is in bypass (STE.Config[2:0] == 0b100), ATS Translation Requests
- * and Translated transactions are denied as though ATS is disabled for
- * the stream (STE.EATS == 0b00), causing F_BAD_ATS_TREQ and
- * F_TRANSL_FORBIDDEN events (IHI0070Ea 5.2 Stream Table Entry).
+ * If the CD table is not in use we can use the provided STE, otherwise
+ * we use a cdtable STE with the provided S1DSS.
*/
- state.disable_ats = true;
- arm_smmu_attach_prepare(master, domain, &state);
+ if (arm_smmu_ssids_in_use(&master->cd_table)) {
+ /*
+ * If a CD table has to be present then we need to run with ATS
+ * on even though the RID will fail ATS queries with UR. This is
+ * because we have no idea what the PASID's need.
+ */
+ arm_smmu_attach_prepare(master, domain, &state);
+ arm_smmu_make_cdtable_ste(ste, master, state.want_ats, s1dss);
+ } else {
+ /*
+ * The SMMU does not support enabling ATS with bypass/abort.
+ * When the STE is in bypass (STE.Config[2:0] == 0b100), ATS
+ * Translation Requests and Translated transactions are denied
+ * as though ATS is disabled for the stream (STE.EATS == 0b00),
+ * causing F_BAD_ATS_TREQ and F_TRANSL_FORBIDDEN events
+ * (IHI0070Ea 5.2 Stream Table Entry).
+ */
+ state.disable_ats = true;
+ arm_smmu_attach_prepare(master, domain, &state);
+ }
arm_smmu_install_ste_for_dev(master, ste);
arm_smmu_attach_commit(master, &state);
mutex_unlock(&master->smmu->asid_lock);
@@ -2865,7 +2892,6 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
* descriptor from arm_smmu_share_asid().
*/
arm_smmu_clear_cd(master, IOMMU_NO_PASID);
- return 0;
}
static int arm_smmu_attach_dev_identity(struct iommu_domain *domain,
@@ -2874,7 +2900,8 @@ static int arm_smmu_attach_dev_identity(struct iommu_domain *domain,
struct arm_smmu_ste ste;
arm_smmu_make_bypass_ste(&ste);
- return arm_smmu_attach_dev_ste(domain, dev, &ste);
+ arm_smmu_attach_dev_ste(domain, dev, &ste, STRTAB_STE_1_S1DSS_BYPASS);
+ return 0;
}
static const struct iommu_domain_ops arm_smmu_identity_ops = {
@@ -2892,7 +2919,9 @@ static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain,
struct arm_smmu_ste ste;
arm_smmu_make_abort_ste(&ste);
- return arm_smmu_attach_dev_ste(domain, dev, &ste);
+ arm_smmu_attach_dev_ste(domain, dev, &ste,
+ STRTAB_STE_1_S1DSS_TERMINATE);
+ return 0;
}
static const struct iommu_domain_ops arm_smmu_blocked_ops = {
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 26/27] iommu/arm-smmu-v3: Allow a PASID to be set when RID is IDENTITY/BLOCKED
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
If the STE doesn't point to the CD table we can upgrade it by
reprogramming the STE with the appropriate S1DSS. We may also need to turn
on ATS at the same time.
Keep track if the installed STE is pointing at the cd_table and the ATS
state to trigger this path.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 56 ++++++++++++++++++---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 ++-
2 files changed, 52 insertions(+), 10 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 6b5a30c86dda2d..535b28d13da5c4 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2458,6 +2458,13 @@ static void arm_smmu_install_ste_for_dev(struct arm_smmu_master *master,
int i, j;
struct arm_smmu_device *smmu = master->smmu;
+ master->cd_table.in_ste =
+ FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(target->data[0])) ==
+ STRTAB_STE_0_CFG_S1_TRANS;
+ master->ste_ats_enabled =
+ FIELD_GET(STRTAB_STE_1_EATS, le64_to_cpu(target->data[1])) ==
+ STRTAB_STE_1_EATS_TRANS;
+
for (i = 0; i < master->num_streams; ++i) {
u32 sid = master->streams[i].id;
struct arm_smmu_ste *step =
@@ -2784,27 +2791,47 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
return 0;
}
-static bool arm_smmu_is_s1_domain(struct iommu_domain *domain)
+static void arm_smmu_update_ste(struct arm_smmu_master *master,
+ struct iommu_domain *sid_domain,
+ bool want_ats)
{
- if (!domain || !(domain->type & __IOMMU_DOMAIN_PAGING))
- return false;
- return to_smmu_domain(domain)->stage == ARM_SMMU_DOMAIN_S1;
+ unsigned int s1dss = STRTAB_STE_1_S1DSS_TERMINATE;
+ struct arm_smmu_ste ste;
+
+ if (master->cd_table.in_ste && master->ste_ats_enabled == want_ats)
+ return;
+
+ if (sid_domain->type == IOMMU_DOMAIN_IDENTITY)
+ s1dss = STRTAB_STE_1_S1DSS_BYPASS;
+ else
+ WARN_ON(sid_domain->type != IOMMU_DOMAIN_BLOCKED);
+
+ /*
+ * Change the STE into a cdtable one with SID IDENTITY/BLOCKED behavior
+ * using s1dss if necessary. The cd_table is already installed then
+ * the S1DSS is correct and this will just update the EATS. Otherwise
+ * it installs the entire thing. This will be hitless.
+ */
+ arm_smmu_make_cdtable_ste(&ste, master, want_ats, s1dss);
+ arm_smmu_install_ste_for_dev(master, &ste);
}
int arm_smmu_set_pasid(struct arm_smmu_master *master,
struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
const struct arm_smmu_cd *cd)
{
+ struct iommu_domain *sid_domain = iommu_get_domain_for_dev(master->dev);
struct attach_state state = {.ssid = pasid};
struct arm_smmu_cd *cdptr;
int ret;
- if (smmu_domain->smmu != master->smmu)
+ if (smmu_domain->smmu != master->smmu || pasid == IOMMU_NO_PASID)
return -EINVAL;
- if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)) ||
- !master->cd_table.used_sid)
- return -ENODEV;
+ if (!master->cd_table.in_ste &&
+ sid_domain->type != IOMMU_DOMAIN_IDENTITY &&
+ sid_domain->type != IOMMU_DOMAIN_BLOCKED)
+ return -EINVAL;
cdptr = arm_smmu_get_cd_ptr(master, pasid);
if (!cdptr)
@@ -2816,6 +2843,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
goto out_unlock;
arm_smmu_write_cd_entry(master, pasid, cdptr, cd);
+ arm_smmu_update_ste(master, sid_domain, state.want_ats);
arm_smmu_attach_commit(master, &state);
@@ -2829,6 +2857,7 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct arm_smmu_domain *smmu_domain;
struct iommu_domain *domain;
+ bool last_ssid = master->cd_table.used_ssids == 1;
domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA);
if (WARN_ON(IS_ERR(domain)) || !domain)
@@ -2842,6 +2871,17 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
arm_smmu_atc_inv_master(master, pasid);
arm_smmu_remove_master_domain(master, &smmu_domain->domain, pasid);
mutex_unlock(&master->smmu->asid_lock);
+
+ /*
+ * When the last user of the CD table goes away downgrade the STE back
+ * to a non-cd_table one.
+ */
+ if (last_ssid && !master->cd_table.used_sid) {
+ struct iommu_domain *sid_domain =
+ iommu_get_domain_for_dev(master->dev);
+
+ sid_domain->ops->attach_dev(sid_domain, master->dev);
+ }
}
static void arm_smmu_attach_dev_ste(struct iommu_domain *domain,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index a23593f1830106..92bff64eef644b 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -600,7 +600,8 @@ struct arm_smmu_ctx_desc_cfg {
struct arm_smmu_l1_ctx_desc *l1_desc;
unsigned int num_l1_ents;
unsigned int used_ssids;
- bool used_sid;
+ u8 used_sid;
+ u8 in_ste;
u8 s1fmt;
/* log2 of the maximum number of CDs supported by this table */
u8 s1cdmax;
@@ -708,7 +709,8 @@ struct arm_smmu_master {
/* Locked by the iommu core using the group mutex */
struct arm_smmu_ctx_desc_cfg cd_table;
unsigned int num_streams;
- bool ats_enabled;
+ bool ats_enabled : 1;
+ bool ste_ats_enabled : 1;
bool stall_enabled;
bool sva_enabled;
bool iopf_enabled;
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 26/27] iommu/arm-smmu-v3: Allow a PASID to be set when RID is IDENTITY/BLOCKED
@ 2024-03-04 23:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
If the STE doesn't point to the CD table we can upgrade it by
reprogramming the STE with the appropriate S1DSS. We may also need to turn
on ATS at the same time.
Keep track if the installed STE is pointing at the cd_table and the ATS
state to trigger this path.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 56 ++++++++++++++++++---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 ++-
2 files changed, 52 insertions(+), 10 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 6b5a30c86dda2d..535b28d13da5c4 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2458,6 +2458,13 @@ static void arm_smmu_install_ste_for_dev(struct arm_smmu_master *master,
int i, j;
struct arm_smmu_device *smmu = master->smmu;
+ master->cd_table.in_ste =
+ FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(target->data[0])) ==
+ STRTAB_STE_0_CFG_S1_TRANS;
+ master->ste_ats_enabled =
+ FIELD_GET(STRTAB_STE_1_EATS, le64_to_cpu(target->data[1])) ==
+ STRTAB_STE_1_EATS_TRANS;
+
for (i = 0; i < master->num_streams; ++i) {
u32 sid = master->streams[i].id;
struct arm_smmu_ste *step =
@@ -2784,27 +2791,47 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
return 0;
}
-static bool arm_smmu_is_s1_domain(struct iommu_domain *domain)
+static void arm_smmu_update_ste(struct arm_smmu_master *master,
+ struct iommu_domain *sid_domain,
+ bool want_ats)
{
- if (!domain || !(domain->type & __IOMMU_DOMAIN_PAGING))
- return false;
- return to_smmu_domain(domain)->stage == ARM_SMMU_DOMAIN_S1;
+ unsigned int s1dss = STRTAB_STE_1_S1DSS_TERMINATE;
+ struct arm_smmu_ste ste;
+
+ if (master->cd_table.in_ste && master->ste_ats_enabled == want_ats)
+ return;
+
+ if (sid_domain->type == IOMMU_DOMAIN_IDENTITY)
+ s1dss = STRTAB_STE_1_S1DSS_BYPASS;
+ else
+ WARN_ON(sid_domain->type != IOMMU_DOMAIN_BLOCKED);
+
+ /*
+ * Change the STE into a cdtable one with SID IDENTITY/BLOCKED behavior
+ * using s1dss if necessary. The cd_table is already installed then
+ * the S1DSS is correct and this will just update the EATS. Otherwise
+ * it installs the entire thing. This will be hitless.
+ */
+ arm_smmu_make_cdtable_ste(&ste, master, want_ats, s1dss);
+ arm_smmu_install_ste_for_dev(master, &ste);
}
int arm_smmu_set_pasid(struct arm_smmu_master *master,
struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
const struct arm_smmu_cd *cd)
{
+ struct iommu_domain *sid_domain = iommu_get_domain_for_dev(master->dev);
struct attach_state state = {.ssid = pasid};
struct arm_smmu_cd *cdptr;
int ret;
- if (smmu_domain->smmu != master->smmu)
+ if (smmu_domain->smmu != master->smmu || pasid == IOMMU_NO_PASID)
return -EINVAL;
- if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)) ||
- !master->cd_table.used_sid)
- return -ENODEV;
+ if (!master->cd_table.in_ste &&
+ sid_domain->type != IOMMU_DOMAIN_IDENTITY &&
+ sid_domain->type != IOMMU_DOMAIN_BLOCKED)
+ return -EINVAL;
cdptr = arm_smmu_get_cd_ptr(master, pasid);
if (!cdptr)
@@ -2816,6 +2843,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
goto out_unlock;
arm_smmu_write_cd_entry(master, pasid, cdptr, cd);
+ arm_smmu_update_ste(master, sid_domain, state.want_ats);
arm_smmu_attach_commit(master, &state);
@@ -2829,6 +2857,7 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct arm_smmu_domain *smmu_domain;
struct iommu_domain *domain;
+ bool last_ssid = master->cd_table.used_ssids == 1;
domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA);
if (WARN_ON(IS_ERR(domain)) || !domain)
@@ -2842,6 +2871,17 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
arm_smmu_atc_inv_master(master, pasid);
arm_smmu_remove_master_domain(master, &smmu_domain->domain, pasid);
mutex_unlock(&master->smmu->asid_lock);
+
+ /*
+ * When the last user of the CD table goes away downgrade the STE back
+ * to a non-cd_table one.
+ */
+ if (last_ssid && !master->cd_table.used_sid) {
+ struct iommu_domain *sid_domain =
+ iommu_get_domain_for_dev(master->dev);
+
+ sid_domain->ops->attach_dev(sid_domain, master->dev);
+ }
}
static void arm_smmu_attach_dev_ste(struct iommu_domain *domain,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index a23593f1830106..92bff64eef644b 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -600,7 +600,8 @@ struct arm_smmu_ctx_desc_cfg {
struct arm_smmu_l1_ctx_desc *l1_desc;
unsigned int num_l1_ents;
unsigned int used_ssids;
- bool used_sid;
+ u8 used_sid;
+ u8 in_ste;
u8 s1fmt;
/* log2 of the maximum number of CDs supported by this table */
u8 s1cdmax;
@@ -708,7 +709,8 @@ struct arm_smmu_master {
/* Locked by the iommu core using the group mutex */
struct arm_smmu_ctx_desc_cfg cd_table;
unsigned int num_streams;
- bool ats_enabled;
+ bool ats_enabled : 1;
+ bool ste_ats_enabled : 1;
bool stall_enabled;
bool sva_enabled;
bool iopf_enabled;
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 27/27] iommu/arm-smmu-v3: Allow setting a S1 domain to a PASID
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-04 23:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
The SVA cleanup made the SSID logic entirely general so all we need to do
is call it with the correct cd table entry for a S1 domain.
This is slightly tricky because of the ASID and how the locking works, the
simple fix is to just update the ASID once we get the right locks.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 45 +++++++++++++++++++--
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 +-
2 files changed, 42 insertions(+), 5 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 535b28d13da5c4..5788998025c89d 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1322,8 +1322,6 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr =
&pgtbl_cfg->arm_lpae_s1_cfg.tcr;
- lockdep_assert_held(&master->smmu->asid_lock);
-
memset(target, 0, sizeof(*target));
target->data[0] = cpu_to_le64(
@@ -2791,6 +2789,36 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
return 0;
}
+static int arm_smmu_s1_set_dev_pasid(struct iommu_domain *domain,
+ struct device *dev, ioasid_t id)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct arm_smmu_master *master = dev_iommu_priv_get(dev);
+ struct arm_smmu_device *smmu = master->smmu;
+ struct arm_smmu_cd target_cd;
+ int ret = 0;
+
+ mutex_lock(&smmu_domain->init_mutex);
+ if (!smmu_domain->smmu)
+ ret = arm_smmu_domain_finalise(smmu_domain, smmu);
+ else if (smmu_domain->smmu != smmu)
+ ret = -EINVAL;
+ mutex_unlock(&smmu_domain->init_mutex);
+ if (ret)
+ return ret;
+
+ if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1)
+ return -EINVAL;
+
+ /*
+ * We can read cd.asid outside the lock because arm_smmu_set_pasid()
+ * will fix it
+ */
+ arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
+ return arm_smmu_set_pasid(master, to_smmu_domain(domain), id,
+ &target_cd);
+}
+
static void arm_smmu_update_ste(struct arm_smmu_master *master,
struct iommu_domain *sid_domain,
bool want_ats)
@@ -2818,7 +2846,7 @@ static void arm_smmu_update_ste(struct arm_smmu_master *master,
int arm_smmu_set_pasid(struct arm_smmu_master *master,
struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
- const struct arm_smmu_cd *cd)
+ struct arm_smmu_cd *cd)
{
struct iommu_domain *sid_domain = iommu_get_domain_for_dev(master->dev);
struct attach_state state = {.ssid = pasid};
@@ -2842,6 +2870,14 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
if (ret)
goto out_unlock;
+ /*
+ * We don't want to obtain to the asid_lock too early, so fix up the
+ * caller set ASID under the lock in case it changed.
+ */
+ cd->data[0] &= ~cpu_to_le64(CTXDESC_CD_0_ASID);
+ cd->data[0] |= cpu_to_le64(
+ FIELD_PREP(CTXDESC_CD_0_ASID, smmu_domain->cd.asid));
+
arm_smmu_write_cd_entry(master, pasid, cdptr, cd);
arm_smmu_update_ste(master, sid_domain, state.want_ats);
@@ -2859,7 +2895,7 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
struct iommu_domain *domain;
bool last_ssid = master->cd_table.used_ssids == 1;
- domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA);
+ domain = iommu_get_domain_for_dev_pasid(dev, pasid, 0);
if (WARN_ON(IS_ERR(domain)) || !domain)
return;
@@ -3365,6 +3401,7 @@ static struct iommu_ops arm_smmu_ops = {
.owner = THIS_MODULE,
.default_domain_ops = &(const struct iommu_domain_ops) {
.attach_dev = arm_smmu_attach_dev,
+ .set_dev_pasid = arm_smmu_s1_set_dev_pasid,
.map_pages = arm_smmu_map_pages,
.unmap_pages = arm_smmu_unmap_pages,
.flush_iotlb_all = arm_smmu_flush_iotlb_all,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 92bff64eef644b..1acafd5c494971 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -771,7 +771,7 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
int arm_smmu_set_pasid(struct arm_smmu_master *master,
struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
- const struct arm_smmu_cd *cd);
+ struct arm_smmu_cd *cd);
void arm_smmu_remove_pasid(struct arm_smmu_master *master,
struct arm_smmu_domain *smmu_domain, ioasid_t pasid);
--
2.43.2
^ permalink raw reply related [flat|nested] 232+ messages in thread
* [PATCH v5 27/27] iommu/arm-smmu-v3: Allow setting a S1 domain to a PASID
@ 2024-03-04 23:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-04 23:44 UTC (permalink / raw)
To: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
The SVA cleanup made the SSID logic entirely general so all we need to do
is call it with the correct cd table entry for a S1 domain.
This is slightly tricky because of the ASID and how the locking works, the
simple fix is to just update the ASID once we get the right locks.
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 45 +++++++++++++++++++--
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 +-
2 files changed, 42 insertions(+), 5 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 535b28d13da5c4..5788998025c89d 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1322,8 +1322,6 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr =
&pgtbl_cfg->arm_lpae_s1_cfg.tcr;
- lockdep_assert_held(&master->smmu->asid_lock);
-
memset(target, 0, sizeof(*target));
target->data[0] = cpu_to_le64(
@@ -2791,6 +2789,36 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
return 0;
}
+static int arm_smmu_s1_set_dev_pasid(struct iommu_domain *domain,
+ struct device *dev, ioasid_t id)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct arm_smmu_master *master = dev_iommu_priv_get(dev);
+ struct arm_smmu_device *smmu = master->smmu;
+ struct arm_smmu_cd target_cd;
+ int ret = 0;
+
+ mutex_lock(&smmu_domain->init_mutex);
+ if (!smmu_domain->smmu)
+ ret = arm_smmu_domain_finalise(smmu_domain, smmu);
+ else if (smmu_domain->smmu != smmu)
+ ret = -EINVAL;
+ mutex_unlock(&smmu_domain->init_mutex);
+ if (ret)
+ return ret;
+
+ if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1)
+ return -EINVAL;
+
+ /*
+ * We can read cd.asid outside the lock because arm_smmu_set_pasid()
+ * will fix it
+ */
+ arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
+ return arm_smmu_set_pasid(master, to_smmu_domain(domain), id,
+ &target_cd);
+}
+
static void arm_smmu_update_ste(struct arm_smmu_master *master,
struct iommu_domain *sid_domain,
bool want_ats)
@@ -2818,7 +2846,7 @@ static void arm_smmu_update_ste(struct arm_smmu_master *master,
int arm_smmu_set_pasid(struct arm_smmu_master *master,
struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
- const struct arm_smmu_cd *cd)
+ struct arm_smmu_cd *cd)
{
struct iommu_domain *sid_domain = iommu_get_domain_for_dev(master->dev);
struct attach_state state = {.ssid = pasid};
@@ -2842,6 +2870,14 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
if (ret)
goto out_unlock;
+ /*
+ * We don't want to obtain to the asid_lock too early, so fix up the
+ * caller set ASID under the lock in case it changed.
+ */
+ cd->data[0] &= ~cpu_to_le64(CTXDESC_CD_0_ASID);
+ cd->data[0] |= cpu_to_le64(
+ FIELD_PREP(CTXDESC_CD_0_ASID, smmu_domain->cd.asid));
+
arm_smmu_write_cd_entry(master, pasid, cdptr, cd);
arm_smmu_update_ste(master, sid_domain, state.want_ats);
@@ -2859,7 +2895,7 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
struct iommu_domain *domain;
bool last_ssid = master->cd_table.used_ssids == 1;
- domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA);
+ domain = iommu_get_domain_for_dev_pasid(dev, pasid, 0);
if (WARN_ON(IS_ERR(domain)) || !domain)
return;
@@ -3365,6 +3401,7 @@ static struct iommu_ops arm_smmu_ops = {
.owner = THIS_MODULE,
.default_domain_ops = &(const struct iommu_domain_ops) {
.attach_dev = arm_smmu_attach_dev,
+ .set_dev_pasid = arm_smmu_s1_set_dev_pasid,
.map_pages = arm_smmu_map_pages,
.unmap_pages = arm_smmu_unmap_pages,
.flush_iotlb_all = arm_smmu_flush_iotlb_all,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 92bff64eef644b..1acafd5c494971 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -771,7 +771,7 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
int arm_smmu_set_pasid(struct arm_smmu_master *master,
struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
- const struct arm_smmu_cd *cd);
+ struct arm_smmu_cd *cd);
void arm_smmu_remove_pasid(struct arm_smmu_master *master,
struct arm_smmu_domain *smmu_domain, ioasid_t pasid);
--
2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* Re: [PATCH v5 02/27] iommu/arm-smmu-v3: Do not ATC invalidate the entire domain
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-13 9:18 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-13 9:18 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
Reviewed-by: Michael Shavit <mshavit@google.com>
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> At this point we know which master we are going to change the PCI config
> on, this is the only device we need to invalidate. Switch
> arm_smmu_atc_inv_domain() for arm_smmu_atc_inv_master().
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index d1bc151a5dff8c..9e9233331c4636 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2447,7 +2447,10 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master,
> pdev = to_pci_dev(master->dev);
>
> atomic_inc(&smmu_domain->nr_ats_masters);
> - arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, 0, 0);
> + /*
> + * ATC invalidation of PASID 0 causes the entire ATC to be flushed.
> + */
> + arm_smmu_atc_inv_master(master);
> if (pci_enable_ats(pdev, stu))
> dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu);
> }
> --
> 2.43.2
>
>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 02/27] iommu/arm-smmu-v3: Do not ATC invalidate the entire domain
@ 2024-03-13 9:18 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-13 9:18 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
Reviewed-by: Michael Shavit <mshavit@google.com>
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> At this point we know which master we are going to change the PCI config
> on, this is the only device we need to invalidate. Switch
> arm_smmu_atc_inv_domain() for arm_smmu_atc_inv_master().
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index d1bc151a5dff8c..9e9233331c4636 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2447,7 +2447,10 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master,
> pdev = to_pci_dev(master->dev);
>
> atomic_inc(&smmu_domain->nr_ats_masters);
> - arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, 0, 0);
> + /*
> + * ATC invalidation of PASID 0 causes the entire ATC to be flushed.
> + */
> + arm_smmu_atc_inv_master(master);
> if (pci_enable_ats(pdev, stu))
> dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu);
> }
> --
> 2.43.2
>
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 03/27] iommu/arm-smmu-v3: Add a type for the CD entry
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-13 9:44 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-13 9:44 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Instead of passing a naked __le16 * around to represent a CD table entry
> wrap it in a "struct arm_smmu_cd" with an array of the correct size. This
> makes it much clearer which functions will comprise the "CD API".
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 +++++++++++---------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 ++++++-
> 2 files changed, 17 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 9e9233331c4636..c60b067c1f553e 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1219,7 +1219,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
> WRITE_ONCE(*dst, cpu_to_le64(val));
> }
>
> -static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
> +static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> + u32 ssid)
> {
> __le64 *l1ptr;
> unsigned int idx;
> @@ -1228,7 +1229,8 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
>
> if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
> - return cd_table->cdtab + ssid * CTXDESC_CD_DWORDS;
> + return (struct arm_smmu_cd *)(cd_table->cdtab +
> + ssid * CTXDESC_CD_DWORDS);
Can we define cd_table.cdtab as a union type to avoid this cast and
make the struct definition more explicit?
>
> idx = ssid >> CTXDESC_SPLIT;
> l1_desc = &cd_table->l1_desc[idx];
> @@ -1242,7 +1244,7 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
> arm_smmu_sync_cd(master, ssid, false);
> }
> idx = ssid & (CTXDESC_L2_ENTRIES - 1);
> - return l1_desc->l2ptr + idx * CTXDESC_CD_DWORDS;
> + return &l1_desc->l2ptr[idx];
> }
>
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> @@ -1261,7 +1263,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> */
> u64 val;
> bool cd_live;
> - __le64 *cdptr;
> + struct arm_smmu_cd *cdptr;
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
> struct arm_smmu_device *smmu = master->smmu;
>
> @@ -1272,7 +1274,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> if (!cdptr)
> return -ENOMEM;
>
> - val = le64_to_cpu(cdptr[0]);
> + val = le64_to_cpu(cdptr->data[0]);
> cd_live = !!(val & CTXDESC_CD_0_V);
>
> if (!cd) { /* (5) */
> @@ -1289,9 +1291,9 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> * this substream's traffic
> */
> } else { /* (1) and (2) */
> - cdptr[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> - cdptr[2] = 0;
> - cdptr[3] = cpu_to_le64(cd->mair);
> + cdptr->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> + cdptr->data[2] = 0;
> + cdptr->data[3] = cpu_to_le64(cd->mair);
>
> /*
> * STE may be live, and the SMMU might read dwords of this CD in any
> @@ -1323,7 +1325,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> * field within an aligned 64-bit span of a structure can be altered
> * without first making the structure invalid.
> */
> - WRITE_ONCE(cdptr[0], cpu_to_le64(val));
> + WRITE_ONCE(cdptr->data[0], cpu_to_le64(val));
> arm_smmu_sync_cd(master, ssid, true);
> return 0;
> }
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 23baf117e7e4b5..7078ed569fd4d3 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -282,6 +282,11 @@ struct arm_smmu_ste {
> #define CTXDESC_L1_DESC_L2PTR_MASK GENMASK_ULL(51, 12)
>
> #define CTXDESC_CD_DWORDS 8
> +
> +struct arm_smmu_cd {
> + __le64 data[CTXDESC_CD_DWORDS];
> +};
> +
> #define CTXDESC_CD_0_TCR_T0SZ GENMASK_ULL(5, 0)
> #define CTXDESC_CD_0_TCR_TG0 GENMASK_ULL(7, 6)
> #define CTXDESC_CD_0_TCR_IRGN0 GENMASK_ULL(9, 8)
> @@ -591,7 +596,7 @@ struct arm_smmu_ctx_desc {
> };
>
> struct arm_smmu_l1_ctx_desc {
> - __le64 *l2ptr;
> + struct arm_smmu_cd *l2ptr;
> dma_addr_t l2ptr_dma;
> };
>
> --
> 2.43.2
>
>
Reviewed-by: Michael Shavit <mshavit@google.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 03/27] iommu/arm-smmu-v3: Add a type for the CD entry
@ 2024-03-13 9:44 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-13 9:44 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Instead of passing a naked __le16 * around to represent a CD table entry
> wrap it in a "struct arm_smmu_cd" with an array of the correct size. This
> makes it much clearer which functions will comprise the "CD API".
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 +++++++++++---------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 ++++++-
> 2 files changed, 17 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 9e9233331c4636..c60b067c1f553e 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1219,7 +1219,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
> WRITE_ONCE(*dst, cpu_to_le64(val));
> }
>
> -static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
> +static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> + u32 ssid)
> {
> __le64 *l1ptr;
> unsigned int idx;
> @@ -1228,7 +1229,8 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
>
> if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
> - return cd_table->cdtab + ssid * CTXDESC_CD_DWORDS;
> + return (struct arm_smmu_cd *)(cd_table->cdtab +
> + ssid * CTXDESC_CD_DWORDS);
Can we define cd_table.cdtab as a union type to avoid this cast and
make the struct definition more explicit?
>
> idx = ssid >> CTXDESC_SPLIT;
> l1_desc = &cd_table->l1_desc[idx];
> @@ -1242,7 +1244,7 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
> arm_smmu_sync_cd(master, ssid, false);
> }
> idx = ssid & (CTXDESC_L2_ENTRIES - 1);
> - return l1_desc->l2ptr + idx * CTXDESC_CD_DWORDS;
> + return &l1_desc->l2ptr[idx];
> }
>
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> @@ -1261,7 +1263,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> */
> u64 val;
> bool cd_live;
> - __le64 *cdptr;
> + struct arm_smmu_cd *cdptr;
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
> struct arm_smmu_device *smmu = master->smmu;
>
> @@ -1272,7 +1274,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> if (!cdptr)
> return -ENOMEM;
>
> - val = le64_to_cpu(cdptr[0]);
> + val = le64_to_cpu(cdptr->data[0]);
> cd_live = !!(val & CTXDESC_CD_0_V);
>
> if (!cd) { /* (5) */
> @@ -1289,9 +1291,9 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> * this substream's traffic
> */
> } else { /* (1) and (2) */
> - cdptr[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> - cdptr[2] = 0;
> - cdptr[3] = cpu_to_le64(cd->mair);
> + cdptr->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> + cdptr->data[2] = 0;
> + cdptr->data[3] = cpu_to_le64(cd->mair);
>
> /*
> * STE may be live, and the SMMU might read dwords of this CD in any
> @@ -1323,7 +1325,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> * field within an aligned 64-bit span of a structure can be altered
> * without first making the structure invalid.
> */
> - WRITE_ONCE(cdptr[0], cpu_to_le64(val));
> + WRITE_ONCE(cdptr->data[0], cpu_to_le64(val));
> arm_smmu_sync_cd(master, ssid, true);
> return 0;
> }
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 23baf117e7e4b5..7078ed569fd4d3 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -282,6 +282,11 @@ struct arm_smmu_ste {
> #define CTXDESC_L1_DESC_L2PTR_MASK GENMASK_ULL(51, 12)
>
> #define CTXDESC_CD_DWORDS 8
> +
> +struct arm_smmu_cd {
> + __le64 data[CTXDESC_CD_DWORDS];
> +};
> +
> #define CTXDESC_CD_0_TCR_T0SZ GENMASK_ULL(5, 0)
> #define CTXDESC_CD_0_TCR_TG0 GENMASK_ULL(7, 6)
> #define CTXDESC_CD_0_TCR_IRGN0 GENMASK_ULL(9, 8)
> @@ -591,7 +596,7 @@ struct arm_smmu_ctx_desc {
> };
>
> struct arm_smmu_l1_ctx_desc {
> - __le64 *l2ptr;
> + struct arm_smmu_cd *l2ptr;
> dma_addr_t l2ptr_dma;
> };
>
> --
> 2.43.2
>
>
Reviewed-by: Michael Shavit <mshavit@google.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-13 11:30 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-13 11:30 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Prepare to put the CD code into the same mechanism. Add an ops indirection
> around all the STE specific code and make the worker functions independent
> of the entry content being processed.
>
> get_used and sync ops are provided to hook the correct code.
>
> Signed-off-by: Michael Shavit <mshavit@google.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 172 ++++++++++++--------
> 1 file changed, 104 insertions(+), 68 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index c60b067c1f553e..b7f947e36f596f 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -48,8 +48,20 @@ enum arm_smmu_msi_index {
> ARM_SMMU_MAX_MSIS,
> };
>
> -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu,
> - ioasid_t sid);
> +struct arm_smmu_entry_writer_ops;
> +struct arm_smmu_entry_writer {
> + const struct arm_smmu_entry_writer_ops *ops;
> + struct arm_smmu_master *master;
> +};
> +
> +struct arm_smmu_entry_writer_ops {
> + unsigned int num_entry_qwords;
> + __le64 v_bit;
> + void (*get_used)(const __le64 *entry, __le64 *used);
> + void (*sync)(struct arm_smmu_entry_writer *writer);
> +};
> +
> +#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
>
> static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
> [EVTQ_MSI_INDEX] = {
> @@ -982,43 +994,42 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid)
> * would be nice if this was complete according to the spec, but minimally it
> * has to capture the bits this driver uses.
> */
> -static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent,
> - struct arm_smmu_ste *used_bits)
> +static void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits)
> {
> - unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent->data[0]));
> + unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent[0]));
>
> - used_bits->data[0] = cpu_to_le64(STRTAB_STE_0_V);
> - if (!(ent->data[0] & cpu_to_le64(STRTAB_STE_0_V)))
> + used_bits[0] = cpu_to_le64(STRTAB_STE_0_V);
> + if (!(ent[0] & cpu_to_le64(STRTAB_STE_0_V)))
> return;
>
> - used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_CFG);
> + used_bits[0] |= cpu_to_le64(STRTAB_STE_0_CFG);
>
> /* S1 translates */
> if (cfg & BIT(0)) {
> - used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT |
> - STRTAB_STE_0_S1CTXPTR_MASK |
> - STRTAB_STE_0_S1CDMAX);
> - used_bits->data[1] |=
> + used_bits[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT |
> + STRTAB_STE_0_S1CTXPTR_MASK |
> + STRTAB_STE_0_S1CDMAX);
> + used_bits[1] |=
> cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR |
> STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH |
> STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW |
> STRTAB_STE_1_EATS);
> - used_bits->data[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);
> + used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);
> }
>
> /* S2 translates */
> if (cfg & BIT(1)) {
> - used_bits->data[1] |=
> + used_bits[1] |=
> cpu_to_le64(STRTAB_STE_1_EATS | STRTAB_STE_1_SHCFG);
> - used_bits->data[2] |=
> + used_bits[2] |=
> cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR |
> STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI |
> STRTAB_STE_2_S2PTW | STRTAB_STE_2_S2R);
> - used_bits->data[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK);
> + used_bits[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK);
> }
>
> if (cfg == STRTAB_STE_0_CFG_BYPASS)
> - used_bits->data[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG);
> + used_bits[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG);
> }
>
> /*
> @@ -1027,57 +1038,55 @@ static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent,
> * unused_update is an intermediate value of entry that has unused bits set to
> * their new values.
> */
> -static u8 arm_smmu_entry_qword_diff(const struct arm_smmu_ste *entry,
> - const struct arm_smmu_ste *target,
> - struct arm_smmu_ste *unused_update)
> +static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> + const __le64 *entry, const __le64 *target,
> + __le64 *unused_update)
> {
> - struct arm_smmu_ste target_used = {};
> - struct arm_smmu_ste cur_used = {};
> + __le64 target_used[NUM_ENTRY_QWORDS] = {};
> + __le64 cur_used[NUM_ENTRY_QWORDS] = {};
> u8 used_qword_diff = 0;
> unsigned int i;
>
> - arm_smmu_get_ste_used(entry, &cur_used);
> - arm_smmu_get_ste_used(target, &target_used);
> + writer->ops->get_used(entry, cur_used);
> + writer->ops->get_used(target, target_used);
>
> - for (i = 0; i != ARRAY_SIZE(target_used.data); i++) {
> + for (i = 0; i != writer->ops->num_entry_qwords; i++) {
> /*
> * Check that masks are up to date, the make functions are not
> * allowed to set a bit to 1 if the used function doesn't say it
> * is used.
> */
> - WARN_ON_ONCE(target->data[i] & ~target_used.data[i]);
> + WARN_ON_ONCE(target[i] & ~target_used[i]);
>
> /* Bits can change because they are not currently being used */
> - unused_update->data[i] = (entry->data[i] & cur_used.data[i]) |
> - (target->data[i] & ~cur_used.data[i]);
> + unused_update[i] = (entry[i] & cur_used[i]) |
> + (target[i] & ~cur_used[i]);
> /*
> * Each bit indicates that a used bit in a qword needs to be
> * changed after unused_update is applied.
> */
> - if ((unused_update->data[i] & target_used.data[i]) !=
> - target->data[i])
> + if ((unused_update[i] & target_used[i]) != target[i])
> used_qword_diff |= 1 << i;
> }
> return used_qword_diff;
> }
>
> -static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid,
> - struct arm_smmu_ste *entry,
> - const struct arm_smmu_ste *target, unsigned int start,
> +static bool entry_set(struct arm_smmu_entry_writer *writer, __le64 *entry,
> + const __le64 *target, unsigned int start,
> unsigned int len)
> {
> bool changed = false;
> unsigned int i;
>
> for (i = start; len != 0; len--, i++) {
> - if (entry->data[i] != target->data[i]) {
> - WRITE_ONCE(entry->data[i], target->data[i]);
> + if (entry[i] != target[i]) {
> + WRITE_ONCE(entry[i], target[i]);
> changed = true;
> }
> }
>
> if (changed)
> - arm_smmu_sync_ste_for_sid(smmu, sid);
> + writer->ops->sync(writer);
> return changed;
> }
>
> @@ -1107,17 +1116,15 @@ static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid,
> * V=0 process. This relies on the IGNORED behavior described in the
> * specification.
> */
> -static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> - struct arm_smmu_ste *entry,
> - const struct arm_smmu_ste *target)
> +static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer,
> + __le64 *entry, const __le64 *target)
> {
> - unsigned int num_entry_qwords = ARRAY_SIZE(target->data);
> - struct arm_smmu_device *smmu = master->smmu;
> - struct arm_smmu_ste unused_update;
> + unsigned int num_entry_qwords = writer->ops->num_entry_qwords;
> + __le64 unused_update[NUM_ENTRY_QWORDS];
> u8 used_qword_diff;
>
> used_qword_diff =
> - arm_smmu_entry_qword_diff(entry, target, &unused_update);
> + arm_smmu_entry_qword_diff(writer, entry, target, unused_update);
> if (hweight8(used_qword_diff) == 1) {
> /*
> * Only one qword needs its used bits to be changed. This is a
> @@ -1133,22 +1140,21 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> * writing it in the next step anyways. This can save a sync
> * when the only change is in that qword.
> */
> - unused_update.data[critical_qword_index] =
> - entry->data[critical_qword_index];
> - entry_set(smmu, sid, entry, &unused_update, 0, num_entry_qwords);
> - entry_set(smmu, sid, entry, target, critical_qword_index, 1);
> - entry_set(smmu, sid, entry, target, 0, num_entry_qwords);
> + unused_update[critical_qword_index] =
> + entry[critical_qword_index];
> + entry_set(writer, entry, unused_update, 0, num_entry_qwords);
> + entry_set(writer, entry, target, critical_qword_index, 1);
> + entry_set(writer, entry, target, 0, num_entry_qwords);
> } else if (used_qword_diff) {
> /*
> * At least two qwords need their inuse bits to be changed. This
> * requires a breaking update, zero the V bit, write all qwords
> * but 0, then set qword 0
> */
> - unused_update.data[0] = entry->data[0] &
> - cpu_to_le64(~STRTAB_STE_0_V);
> - entry_set(smmu, sid, entry, &unused_update, 0, 1);
> - entry_set(smmu, sid, entry, target, 1, num_entry_qwords - 1);
> - entry_set(smmu, sid, entry, target, 0, 1);
> + unused_update[0] = entry[0] & (~writer->ops->v_bit);
> + entry_set(writer, entry, unused_update, 0, 1);
> + entry_set(writer, entry, target, 1, num_entry_qwords - 1);
> + entry_set(writer, entry, target, 0, 1);
> } else {
> /*
> * No inuse bit changed. Sanity check that all unused bits are 0
> @@ -1156,18 +1162,7 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> * compute_qword_diff().
> */
> WARN_ON_ONCE(
> - entry_set(smmu, sid, entry, target, 0, num_entry_qwords));
> - }
> -
> - /* It's likely that we'll want to use the new STE soon */
> - if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) {
> - struct arm_smmu_cmdq_ent
> - prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG,
> - .prefetch = {
> - .sid = sid,
> - } };
> -
> - arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
> + entry_set(writer, entry, target, 0, num_entry_qwords));
> }
> }
>
> @@ -1440,17 +1435,58 @@ arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc)
> WRITE_ONCE(*dst, cpu_to_le64(val));
> }
>
> -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu, u32 sid)
> +struct arm_smmu_ste_writer {
> + struct arm_smmu_entry_writer writer;
> + u32 sid;
> +};
> +
> +static void arm_smmu_ste_writer_sync_entry(struct arm_smmu_entry_writer *writer)
> {
> + struct arm_smmu_ste_writer *ste_writer =
> + container_of(writer, struct arm_smmu_ste_writer, writer);
> struct arm_smmu_cmdq_ent cmd = {
> .opcode = CMDQ_OP_CFGI_STE,
> .cfgi = {
> - .sid = sid,
> + .sid = ste_writer->sid,
> .leaf = true,
> },
> };
>
> - arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
> + arm_smmu_cmdq_issue_cmd_with_sync(writer->master->smmu, &cmd);
> +}
> +
> +static const struct arm_smmu_entry_writer_ops arm_smmu_ste_writer_ops = {
> + .sync = arm_smmu_ste_writer_sync_entry,
> + .get_used = arm_smmu_get_ste_used,
> + .v_bit = cpu_to_le64(STRTAB_STE_0_V),
> + .num_entry_qwords = sizeof(struct arm_smmu_ste) / sizeof(u64),
> +};
> +
> +static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> + struct arm_smmu_ste *ste,
> + const struct arm_smmu_ste *target)
> +{
> + struct arm_smmu_device *smmu = master->smmu;
> + struct arm_smmu_ste_writer ste_writer = {
> + .writer = {
> + .ops = &arm_smmu_ste_writer_ops,
> + .master = master,
> + },
> + .sid = sid,
> + };
> +
> + arm_smmu_write_entry(&ste_writer.writer, ste->data, target->data);
> +
> + /* It's likely that we'll want to use the new STE soon */
> + if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) {
> + struct arm_smmu_cmdq_ent
> + prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG,
> + .prefetch = {
> + .sid = sid,
> + } };
> +
> + arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
> + }
> }
>
> static void arm_smmu_make_abort_ste(struct arm_smmu_ste *target)
> --
> 2.43.2
>
Reviewed-by: Michael Shavit <mshavit@google.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code
@ 2024-03-13 11:30 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-13 11:30 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Prepare to put the CD code into the same mechanism. Add an ops indirection
> around all the STE specific code and make the worker functions independent
> of the entry content being processed.
>
> get_used and sync ops are provided to hook the correct code.
>
> Signed-off-by: Michael Shavit <mshavit@google.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 172 ++++++++++++--------
> 1 file changed, 104 insertions(+), 68 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index c60b067c1f553e..b7f947e36f596f 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -48,8 +48,20 @@ enum arm_smmu_msi_index {
> ARM_SMMU_MAX_MSIS,
> };
>
> -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu,
> - ioasid_t sid);
> +struct arm_smmu_entry_writer_ops;
> +struct arm_smmu_entry_writer {
> + const struct arm_smmu_entry_writer_ops *ops;
> + struct arm_smmu_master *master;
> +};
> +
> +struct arm_smmu_entry_writer_ops {
> + unsigned int num_entry_qwords;
> + __le64 v_bit;
> + void (*get_used)(const __le64 *entry, __le64 *used);
> + void (*sync)(struct arm_smmu_entry_writer *writer);
> +};
> +
> +#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
>
> static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
> [EVTQ_MSI_INDEX] = {
> @@ -982,43 +994,42 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid)
> * would be nice if this was complete according to the spec, but minimally it
> * has to capture the bits this driver uses.
> */
> -static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent,
> - struct arm_smmu_ste *used_bits)
> +static void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits)
> {
> - unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent->data[0]));
> + unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent[0]));
>
> - used_bits->data[0] = cpu_to_le64(STRTAB_STE_0_V);
> - if (!(ent->data[0] & cpu_to_le64(STRTAB_STE_0_V)))
> + used_bits[0] = cpu_to_le64(STRTAB_STE_0_V);
> + if (!(ent[0] & cpu_to_le64(STRTAB_STE_0_V)))
> return;
>
> - used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_CFG);
> + used_bits[0] |= cpu_to_le64(STRTAB_STE_0_CFG);
>
> /* S1 translates */
> if (cfg & BIT(0)) {
> - used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT |
> - STRTAB_STE_0_S1CTXPTR_MASK |
> - STRTAB_STE_0_S1CDMAX);
> - used_bits->data[1] |=
> + used_bits[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT |
> + STRTAB_STE_0_S1CTXPTR_MASK |
> + STRTAB_STE_0_S1CDMAX);
> + used_bits[1] |=
> cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR |
> STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH |
> STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW |
> STRTAB_STE_1_EATS);
> - used_bits->data[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);
> + used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);
> }
>
> /* S2 translates */
> if (cfg & BIT(1)) {
> - used_bits->data[1] |=
> + used_bits[1] |=
> cpu_to_le64(STRTAB_STE_1_EATS | STRTAB_STE_1_SHCFG);
> - used_bits->data[2] |=
> + used_bits[2] |=
> cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR |
> STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI |
> STRTAB_STE_2_S2PTW | STRTAB_STE_2_S2R);
> - used_bits->data[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK);
> + used_bits[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK);
> }
>
> if (cfg == STRTAB_STE_0_CFG_BYPASS)
> - used_bits->data[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG);
> + used_bits[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG);
> }
>
> /*
> @@ -1027,57 +1038,55 @@ static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent,
> * unused_update is an intermediate value of entry that has unused bits set to
> * their new values.
> */
> -static u8 arm_smmu_entry_qword_diff(const struct arm_smmu_ste *entry,
> - const struct arm_smmu_ste *target,
> - struct arm_smmu_ste *unused_update)
> +static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> + const __le64 *entry, const __le64 *target,
> + __le64 *unused_update)
> {
> - struct arm_smmu_ste target_used = {};
> - struct arm_smmu_ste cur_used = {};
> + __le64 target_used[NUM_ENTRY_QWORDS] = {};
> + __le64 cur_used[NUM_ENTRY_QWORDS] = {};
> u8 used_qword_diff = 0;
> unsigned int i;
>
> - arm_smmu_get_ste_used(entry, &cur_used);
> - arm_smmu_get_ste_used(target, &target_used);
> + writer->ops->get_used(entry, cur_used);
> + writer->ops->get_used(target, target_used);
>
> - for (i = 0; i != ARRAY_SIZE(target_used.data); i++) {
> + for (i = 0; i != writer->ops->num_entry_qwords; i++) {
> /*
> * Check that masks are up to date, the make functions are not
> * allowed to set a bit to 1 if the used function doesn't say it
> * is used.
> */
> - WARN_ON_ONCE(target->data[i] & ~target_used.data[i]);
> + WARN_ON_ONCE(target[i] & ~target_used[i]);
>
> /* Bits can change because they are not currently being used */
> - unused_update->data[i] = (entry->data[i] & cur_used.data[i]) |
> - (target->data[i] & ~cur_used.data[i]);
> + unused_update[i] = (entry[i] & cur_used[i]) |
> + (target[i] & ~cur_used[i]);
> /*
> * Each bit indicates that a used bit in a qword needs to be
> * changed after unused_update is applied.
> */
> - if ((unused_update->data[i] & target_used.data[i]) !=
> - target->data[i])
> + if ((unused_update[i] & target_used[i]) != target[i])
> used_qword_diff |= 1 << i;
> }
> return used_qword_diff;
> }
>
> -static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid,
> - struct arm_smmu_ste *entry,
> - const struct arm_smmu_ste *target, unsigned int start,
> +static bool entry_set(struct arm_smmu_entry_writer *writer, __le64 *entry,
> + const __le64 *target, unsigned int start,
> unsigned int len)
> {
> bool changed = false;
> unsigned int i;
>
> for (i = start; len != 0; len--, i++) {
> - if (entry->data[i] != target->data[i]) {
> - WRITE_ONCE(entry->data[i], target->data[i]);
> + if (entry[i] != target[i]) {
> + WRITE_ONCE(entry[i], target[i]);
> changed = true;
> }
> }
>
> if (changed)
> - arm_smmu_sync_ste_for_sid(smmu, sid);
> + writer->ops->sync(writer);
> return changed;
> }
>
> @@ -1107,17 +1116,15 @@ static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid,
> * V=0 process. This relies on the IGNORED behavior described in the
> * specification.
> */
> -static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> - struct arm_smmu_ste *entry,
> - const struct arm_smmu_ste *target)
> +static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer,
> + __le64 *entry, const __le64 *target)
> {
> - unsigned int num_entry_qwords = ARRAY_SIZE(target->data);
> - struct arm_smmu_device *smmu = master->smmu;
> - struct arm_smmu_ste unused_update;
> + unsigned int num_entry_qwords = writer->ops->num_entry_qwords;
> + __le64 unused_update[NUM_ENTRY_QWORDS];
> u8 used_qword_diff;
>
> used_qword_diff =
> - arm_smmu_entry_qword_diff(entry, target, &unused_update);
> + arm_smmu_entry_qword_diff(writer, entry, target, unused_update);
> if (hweight8(used_qword_diff) == 1) {
> /*
> * Only one qword needs its used bits to be changed. This is a
> @@ -1133,22 +1140,21 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> * writing it in the next step anyways. This can save a sync
> * when the only change is in that qword.
> */
> - unused_update.data[critical_qword_index] =
> - entry->data[critical_qword_index];
> - entry_set(smmu, sid, entry, &unused_update, 0, num_entry_qwords);
> - entry_set(smmu, sid, entry, target, critical_qword_index, 1);
> - entry_set(smmu, sid, entry, target, 0, num_entry_qwords);
> + unused_update[critical_qword_index] =
> + entry[critical_qword_index];
> + entry_set(writer, entry, unused_update, 0, num_entry_qwords);
> + entry_set(writer, entry, target, critical_qword_index, 1);
> + entry_set(writer, entry, target, 0, num_entry_qwords);
> } else if (used_qword_diff) {
> /*
> * At least two qwords need their inuse bits to be changed. This
> * requires a breaking update, zero the V bit, write all qwords
> * but 0, then set qword 0
> */
> - unused_update.data[0] = entry->data[0] &
> - cpu_to_le64(~STRTAB_STE_0_V);
> - entry_set(smmu, sid, entry, &unused_update, 0, 1);
> - entry_set(smmu, sid, entry, target, 1, num_entry_qwords - 1);
> - entry_set(smmu, sid, entry, target, 0, 1);
> + unused_update[0] = entry[0] & (~writer->ops->v_bit);
> + entry_set(writer, entry, unused_update, 0, 1);
> + entry_set(writer, entry, target, 1, num_entry_qwords - 1);
> + entry_set(writer, entry, target, 0, 1);
> } else {
> /*
> * No inuse bit changed. Sanity check that all unused bits are 0
> @@ -1156,18 +1162,7 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> * compute_qword_diff().
> */
> WARN_ON_ONCE(
> - entry_set(smmu, sid, entry, target, 0, num_entry_qwords));
> - }
> -
> - /* It's likely that we'll want to use the new STE soon */
> - if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) {
> - struct arm_smmu_cmdq_ent
> - prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG,
> - .prefetch = {
> - .sid = sid,
> - } };
> -
> - arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
> + entry_set(writer, entry, target, 0, num_entry_qwords));
> }
> }
>
> @@ -1440,17 +1435,58 @@ arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc)
> WRITE_ONCE(*dst, cpu_to_le64(val));
> }
>
> -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu, u32 sid)
> +struct arm_smmu_ste_writer {
> + struct arm_smmu_entry_writer writer;
> + u32 sid;
> +};
> +
> +static void arm_smmu_ste_writer_sync_entry(struct arm_smmu_entry_writer *writer)
> {
> + struct arm_smmu_ste_writer *ste_writer =
> + container_of(writer, struct arm_smmu_ste_writer, writer);
> struct arm_smmu_cmdq_ent cmd = {
> .opcode = CMDQ_OP_CFGI_STE,
> .cfgi = {
> - .sid = sid,
> + .sid = ste_writer->sid,
> .leaf = true,
> },
> };
>
> - arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
> + arm_smmu_cmdq_issue_cmd_with_sync(writer->master->smmu, &cmd);
> +}
> +
> +static const struct arm_smmu_entry_writer_ops arm_smmu_ste_writer_ops = {
> + .sync = arm_smmu_ste_writer_sync_entry,
> + .get_used = arm_smmu_get_ste_used,
> + .v_bit = cpu_to_le64(STRTAB_STE_0_V),
> + .num_entry_qwords = sizeof(struct arm_smmu_ste) / sizeof(u64),
> +};
> +
> +static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> + struct arm_smmu_ste *ste,
> + const struct arm_smmu_ste *target)
> +{
> + struct arm_smmu_device *smmu = master->smmu;
> + struct arm_smmu_ste_writer ste_writer = {
> + .writer = {
> + .ops = &arm_smmu_ste_writer_ops,
> + .master = master,
> + },
> + .sid = sid,
> + };
> +
> + arm_smmu_write_entry(&ste_writer.writer, ste->data, target->data);
> +
> + /* It's likely that we'll want to use the new STE soon */
> + if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) {
> + struct arm_smmu_cmdq_ent
> + prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG,
> + .prefetch = {
> + .sid = sid,
> + } };
> +
> + arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
> + }
> }
>
> static void arm_smmu_make_abort_ste(struct arm_smmu_ste *target)
> --
> 2.43.2
>
Reviewed-by: Michael Shavit <mshavit@google.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 06/27] iommu/arm-smmu-v3: Consolidate clearing a CD table entry
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-13 11:57 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-13 11:57 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> A cleared entry is all 0's. Make arm_smmu_clear_cd() do this sequence.
>
> If we are clearing an entry and for some reason it is not already
> allocated in the CD table then something has gone wrong.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 ++++++++++++++-----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 ++
> 3 files changed, 18 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 347c2fdd865c1a..bb9bb6fd7914ce 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -558,7 +558,7 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
>
> mutex_lock(&sva_lock);
>
> - arm_smmu_write_ctx_desc(master, id, NULL);
> + arm_smmu_clear_cd(master, id);
>
> list_for_each_entry(t, &master->bonds, list) {
> if (t->mm == mm) {
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 237fd6d92c880b..3fb4a1523d1d3f 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1303,6 +1303,19 @@ static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
> }
>
> +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> +{
> + struct arm_smmu_cd target = {};
> + struct arm_smmu_cd *cdptr;
> +
> + if (!master->cd_table.cdtab)
> + return;
> + cdptr = arm_smmu_get_cd_ptr(master, ssid);
> + if (WARN_ON(!cdptr))
> + return;
> + arm_smmu_write_cd_entry(master, ssid, cdptr, &target);
> +}
> +
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_ctx_desc *cd)
> {
> @@ -2702,9 +2715,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> case ARM_SMMU_DOMAIN_S2:
> arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
> arm_smmu_install_ste_for_dev(master, &target);
> - if (master->cd_table.cdtab)
> - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID,
> - NULL);
> + arm_smmu_clear_cd(master, IOMMU_NO_PASID);
> break;
> }
>
> @@ -2752,8 +2763,7 @@ static int arm_smmu_attach_dev_ste(struct device *dev,
> * arm_smmu_domain->devices to avoid races updating the same context
> * descriptor from arm_smmu_share_asid().
> */
> - if (master->cd_table.cdtab)
> - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, NULL);
> + arm_smmu_clear_cd(master, IOMMU_NO_PASID);
> return 0;
> }
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 7078ed569fd4d3..87a7b57f566fbc 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -749,6 +749,8 @@ extern struct xarray arm_smmu_asid_xa;
> extern struct mutex arm_smmu_asid_lock;
> extern struct arm_smmu_ctx_desc quiet_cd;
>
> +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
> +
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid,
> struct arm_smmu_ctx_desc *cd);
> void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
> --
> 2.43.2
>
>
Reviewed-by: Michael Shavit <mshavit@google.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 06/27] iommu/arm-smmu-v3: Consolidate clearing a CD table entry
@ 2024-03-13 11:57 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-13 11:57 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> A cleared entry is all 0's. Make arm_smmu_clear_cd() do this sequence.
>
> If we are clearing an entry and for some reason it is not already
> allocated in the CD table then something has gone wrong.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 ++++++++++++++-----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 ++
> 3 files changed, 18 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 347c2fdd865c1a..bb9bb6fd7914ce 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -558,7 +558,7 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
>
> mutex_lock(&sva_lock);
>
> - arm_smmu_write_ctx_desc(master, id, NULL);
> + arm_smmu_clear_cd(master, id);
>
> list_for_each_entry(t, &master->bonds, list) {
> if (t->mm == mm) {
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 237fd6d92c880b..3fb4a1523d1d3f 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1303,6 +1303,19 @@ static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
> }
>
> +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> +{
> + struct arm_smmu_cd target = {};
> + struct arm_smmu_cd *cdptr;
> +
> + if (!master->cd_table.cdtab)
> + return;
> + cdptr = arm_smmu_get_cd_ptr(master, ssid);
> + if (WARN_ON(!cdptr))
> + return;
> + arm_smmu_write_cd_entry(master, ssid, cdptr, &target);
> +}
> +
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_ctx_desc *cd)
> {
> @@ -2702,9 +2715,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> case ARM_SMMU_DOMAIN_S2:
> arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
> arm_smmu_install_ste_for_dev(master, &target);
> - if (master->cd_table.cdtab)
> - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID,
> - NULL);
> + arm_smmu_clear_cd(master, IOMMU_NO_PASID);
> break;
> }
>
> @@ -2752,8 +2763,7 @@ static int arm_smmu_attach_dev_ste(struct device *dev,
> * arm_smmu_domain->devices to avoid races updating the same context
> * descriptor from arm_smmu_share_asid().
> */
> - if (master->cd_table.cdtab)
> - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, NULL);
> + arm_smmu_clear_cd(master, IOMMU_NO_PASID);
> return 0;
> }
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 7078ed569fd4d3..87a7b57f566fbc 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -749,6 +749,8 @@ extern struct xarray arm_smmu_asid_xa;
> extern struct mutex arm_smmu_asid_lock;
> extern struct arm_smmu_ctx_desc quiet_cd;
>
> +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
> +
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid,
> struct arm_smmu_ctx_desc *cd);
> void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
> --
> 2.43.2
>
>
Reviewed-by: Michael Shavit <mshavit@google.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 07/27] iommu/arm-smmu-v3: Move the CD generation for S1 domains into a function
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-13 12:13 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-13 12:13 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Introduce arm_smmu_make_s1_cd() to build the CD from the paging S1 domain,
> and reorganize all the places programming S1 domain CD table entries to
> call it.
>
> Split arm_smmu_update_s1_domain_cd_entry() from
> arm_smmu_update_ctx_desc_devices() so that the S1 path has its own call
> chain separate from the unrelated SVA path.
>
> arm_smmu_update_s1_domain_cd_entry() only works on S1 domains
> attached to RIDs and refreshes all their CDs.
>
> Remove the forced clear of the CD during S1 domain attach,
> arm_smmu_write_cd_entry() will do this automatically if necessary.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 25 +++++++-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 60 +++++++++++++------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 +++
> 3 files changed, 75 insertions(+), 18 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index bb9bb6fd7914ce..6acc65f6d00a71 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -54,6 +54,29 @@ static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> }
>
> +static void
> +arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> +{
> + struct arm_smmu_master *master;
> + struct arm_smmu_cd target_cd;
> + unsigned long flags;
> +
> + spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> + list_for_each_entry(master, &smmu_domain->devices, domain_head) {
> + struct arm_smmu_cd *cdptr;
> +
> + /* S1 domains only support RID attachment right now */
> + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> + if (WARN_ON(!cdptr))
> + continue;
> +
> + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> + arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> + &target_cd);
> + }
> + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> +}
> +
> /*
> * Check if the CPU ASID is available on the SMMU side. If a private context
> * descriptor is using it, try to replace it.
> @@ -97,7 +120,7 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid)
> * be some overlap between use of both ASIDs, until we invalidate the
> * TLB.
> */
> - arm_smmu_update_ctx_desc_devices(smmu_domain, IOMMU_NO_PASID, cd);
> + arm_smmu_update_s1_domain_cd_entry(smmu_domain);
Not to be too nitpicky, but is calling this
arm_smmu_update_s1_domain_cd_entry the right choice here? Yes the RID
domain has type "ARM_SMMU_DOMAIN_S1", but CDs are also stage 1
translations.
>
> /* Invalidate TLB entries previously associated with that context */
> arm_smmu_tlb_inv_asid(smmu, asid);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 3fb4a1523d1d3f..e25dbb982feeee 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1222,8 +1222,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
> WRITE_ONCE(*dst, cpu_to_le64(val));
> }
>
> -static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> - u32 ssid)
> +struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> + u32 ssid)
> {
> __le64 *l1ptr;
> unsigned int idx;
> @@ -1288,9 +1288,9 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
> .num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
> };
>
> -static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> - struct arm_smmu_cd *cdptr,
> - const struct arm_smmu_cd *target)
> +void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> + struct arm_smmu_cd *cdptr,
> + const struct arm_smmu_cd *target)
> {
> struct arm_smmu_cd_writer cd_writer = {
> .writer = {
> @@ -1303,6 +1303,32 @@ static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
> }
>
> +void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> + struct arm_smmu_master *master,
> + struct arm_smmu_domain *smmu_domain)
> +{
> + struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
> +
> + memset(target, 0, sizeof(*target));
> +
> + target->data[0] = cpu_to_le64(
> + cd->tcr |
> +#ifdef __BIG_ENDIAN
> + CTXDESC_CD_0_ENDI |
> +#endif
> + CTXDESC_CD_0_V |
> + CTXDESC_CD_0_AA64 |
> + (master->stall_enabled ? CTXDESC_CD_0_S : 0) |
> + CTXDESC_CD_0_R |
> + CTXDESC_CD_0_A |
> + CTXDESC_CD_0_ASET |
> + FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid)
> + );
> +
> + target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> + target->data[3] = cpu_to_le64(cd->mair);
> +}
> +
> void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> {
> struct arm_smmu_cd target = {};
> @@ -2689,29 +2715,29 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
>
> switch (smmu_domain->stage) {
> - case ARM_SMMU_DOMAIN_S1:
> + case ARM_SMMU_DOMAIN_S1: {
> + struct arm_smmu_cd target_cd;
> + struct arm_smmu_cd *cdptr;
> +
> if (!master->cd_table.cdtab) {
> ret = arm_smmu_alloc_cd_tables(master);
> if (ret)
> goto out_list_del;
> - } else {
> - /*
> - * arm_smmu_write_ctx_desc() relies on the entry being
> - * invalid to work, clear any existing entry.
> - */
> - ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID,
> - NULL);
> - if (ret)
> - goto out_list_del;
> }
>
> - ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, &smmu_domain->cd);
> - if (ret)
> + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> + if (!cdptr) {
> + ret = -ENOMEM;
> goto out_list_del;
> + }
>
> + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> + arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> + &target_cd);
> arm_smmu_make_cdtable_ste(&target, master);
> arm_smmu_install_ste_for_dev(master, &target);
> break;
> + }
> case ARM_SMMU_DOMAIN_S2:
> arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
> arm_smmu_install_ste_for_dev(master, &target);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 87a7b57f566fbc..d32da11058aab6 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -750,6 +750,14 @@ extern struct mutex arm_smmu_asid_lock;
> extern struct arm_smmu_ctx_desc quiet_cd;
>
> void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
> +struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> + u32 ssid);
> +void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> + struct arm_smmu_master *master,
> + struct arm_smmu_domain *smmu_domain);
> +void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> + struct arm_smmu_cd *cdptr,
> + const struct arm_smmu_cd *target);
>
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid,
> struct arm_smmu_ctx_desc *cd);
> --
> 2.43.2
>
Reviewed-by: Michael Shavit <mshavit@google.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 07/27] iommu/arm-smmu-v3: Move the CD generation for S1 domains into a function
@ 2024-03-13 12:13 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-13 12:13 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Introduce arm_smmu_make_s1_cd() to build the CD from the paging S1 domain,
> and reorganize all the places programming S1 domain CD table entries to
> call it.
>
> Split arm_smmu_update_s1_domain_cd_entry() from
> arm_smmu_update_ctx_desc_devices() so that the S1 path has its own call
> chain separate from the unrelated SVA path.
>
> arm_smmu_update_s1_domain_cd_entry() only works on S1 domains
> attached to RIDs and refreshes all their CDs.
>
> Remove the forced clear of the CD during S1 domain attach,
> arm_smmu_write_cd_entry() will do this automatically if necessary.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 25 +++++++-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 60 +++++++++++++------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 +++
> 3 files changed, 75 insertions(+), 18 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index bb9bb6fd7914ce..6acc65f6d00a71 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -54,6 +54,29 @@ static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> }
>
> +static void
> +arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> +{
> + struct arm_smmu_master *master;
> + struct arm_smmu_cd target_cd;
> + unsigned long flags;
> +
> + spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> + list_for_each_entry(master, &smmu_domain->devices, domain_head) {
> + struct arm_smmu_cd *cdptr;
> +
> + /* S1 domains only support RID attachment right now */
> + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> + if (WARN_ON(!cdptr))
> + continue;
> +
> + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> + arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> + &target_cd);
> + }
> + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> +}
> +
> /*
> * Check if the CPU ASID is available on the SMMU side. If a private context
> * descriptor is using it, try to replace it.
> @@ -97,7 +120,7 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid)
> * be some overlap between use of both ASIDs, until we invalidate the
> * TLB.
> */
> - arm_smmu_update_ctx_desc_devices(smmu_domain, IOMMU_NO_PASID, cd);
> + arm_smmu_update_s1_domain_cd_entry(smmu_domain);
Not to be too nitpicky, but is calling this
arm_smmu_update_s1_domain_cd_entry the right choice here? Yes the RID
domain has type "ARM_SMMU_DOMAIN_S1", but CDs are also stage 1
translations.
>
> /* Invalidate TLB entries previously associated with that context */
> arm_smmu_tlb_inv_asid(smmu, asid);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 3fb4a1523d1d3f..e25dbb982feeee 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1222,8 +1222,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
> WRITE_ONCE(*dst, cpu_to_le64(val));
> }
>
> -static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> - u32 ssid)
> +struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> + u32 ssid)
> {
> __le64 *l1ptr;
> unsigned int idx;
> @@ -1288,9 +1288,9 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
> .num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
> };
>
> -static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> - struct arm_smmu_cd *cdptr,
> - const struct arm_smmu_cd *target)
> +void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> + struct arm_smmu_cd *cdptr,
> + const struct arm_smmu_cd *target)
> {
> struct arm_smmu_cd_writer cd_writer = {
> .writer = {
> @@ -1303,6 +1303,32 @@ static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
> }
>
> +void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> + struct arm_smmu_master *master,
> + struct arm_smmu_domain *smmu_domain)
> +{
> + struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
> +
> + memset(target, 0, sizeof(*target));
> +
> + target->data[0] = cpu_to_le64(
> + cd->tcr |
> +#ifdef __BIG_ENDIAN
> + CTXDESC_CD_0_ENDI |
> +#endif
> + CTXDESC_CD_0_V |
> + CTXDESC_CD_0_AA64 |
> + (master->stall_enabled ? CTXDESC_CD_0_S : 0) |
> + CTXDESC_CD_0_R |
> + CTXDESC_CD_0_A |
> + CTXDESC_CD_0_ASET |
> + FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid)
> + );
> +
> + target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> + target->data[3] = cpu_to_le64(cd->mair);
> +}
> +
> void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> {
> struct arm_smmu_cd target = {};
> @@ -2689,29 +2715,29 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
>
> switch (smmu_domain->stage) {
> - case ARM_SMMU_DOMAIN_S1:
> + case ARM_SMMU_DOMAIN_S1: {
> + struct arm_smmu_cd target_cd;
> + struct arm_smmu_cd *cdptr;
> +
> if (!master->cd_table.cdtab) {
> ret = arm_smmu_alloc_cd_tables(master);
> if (ret)
> goto out_list_del;
> - } else {
> - /*
> - * arm_smmu_write_ctx_desc() relies on the entry being
> - * invalid to work, clear any existing entry.
> - */
> - ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID,
> - NULL);
> - if (ret)
> - goto out_list_del;
> }
>
> - ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, &smmu_domain->cd);
> - if (ret)
> + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> + if (!cdptr) {
> + ret = -ENOMEM;
> goto out_list_del;
> + }
>
> + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> + arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> + &target_cd);
> arm_smmu_make_cdtable_ste(&target, master);
> arm_smmu_install_ste_for_dev(master, &target);
> break;
> + }
> case ARM_SMMU_DOMAIN_S2:
> arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
> arm_smmu_install_ste_for_dev(master, &target);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 87a7b57f566fbc..d32da11058aab6 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -750,6 +750,14 @@ extern struct mutex arm_smmu_asid_lock;
> extern struct arm_smmu_ctx_desc quiet_cd;
>
> void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
> +struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> + u32 ssid);
> +void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> + struct arm_smmu_master *master,
> + struct arm_smmu_domain *smmu_domain);
> +void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> + struct arm_smmu_cd *cdptr,
> + const struct arm_smmu_cd *target);
>
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid,
> struct arm_smmu_ctx_desc *cd);
> --
> 2.43.2
>
Reviewed-by: Michael Shavit <mshavit@google.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 08/27] iommu/arm-smmu-v3: Move allocation of the cdtable into arm_smmu_get_cd_ptr()
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-13 12:15 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-13 12:15 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> No reason to force callers to do two steps. Make arm_smmu_get_cd_ptr()
> able to return an entry in all cases except OOM.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 12 ++++++------
> 1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index e25dbb982feeee..2dd6cb17112e98 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -106,6 +106,7 @@ static struct arm_smmu_option_prop arm_smmu_options[] = {
>
> static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
> struct arm_smmu_device *smmu);
> +static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master);
>
> static void parse_driver_options(struct arm_smmu_device *smmu)
> {
> @@ -1231,6 +1232,11 @@ struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> struct arm_smmu_device *smmu = master->smmu;
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
>
> + if (!master->cd_table.cdtab) {
> + if (arm_smmu_alloc_cd_tables(master))
> + return NULL;
> + }
> +
> if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
> return (struct arm_smmu_cd *)(cd_table->cdtab +
> ssid * CTXDESC_CD_DWORDS);
> @@ -2719,12 +2725,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> struct arm_smmu_cd target_cd;
> struct arm_smmu_cd *cdptr;
>
> - if (!master->cd_table.cdtab) {
> - ret = arm_smmu_alloc_cd_tables(master);
> - if (ret)
> - goto out_list_del;
> - }
> -
> cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> if (!cdptr) {
> ret = -ENOMEM;
> --
> 2.43.2
>
Reviewed-by: Michael Shavit <mshavit@google.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 08/27] iommu/arm-smmu-v3: Move allocation of the cdtable into arm_smmu_get_cd_ptr()
@ 2024-03-13 12:15 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-13 12:15 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> No reason to force callers to do two steps. Make arm_smmu_get_cd_ptr()
> able to return an entry in all cases except OOM.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 12 ++++++------
> 1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index e25dbb982feeee..2dd6cb17112e98 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -106,6 +106,7 @@ static struct arm_smmu_option_prop arm_smmu_options[] = {
>
> static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
> struct arm_smmu_device *smmu);
> +static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master);
>
> static void parse_driver_options(struct arm_smmu_device *smmu)
> {
> @@ -1231,6 +1232,11 @@ struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> struct arm_smmu_device *smmu = master->smmu;
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
>
> + if (!master->cd_table.cdtab) {
> + if (arm_smmu_alloc_cd_tables(master))
> + return NULL;
> + }
> +
> if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
> return (struct arm_smmu_cd *)(cd_table->cdtab +
> ssid * CTXDESC_CD_DWORDS);
> @@ -2719,12 +2725,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> struct arm_smmu_cd target_cd;
> struct arm_smmu_cd *cdptr;
>
> - if (!master->cd_table.cdtab) {
> - ret = arm_smmu_alloc_cd_tables(master);
> - if (ret)
> - goto out_list_del;
> - }
> -
> cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> if (!cdptr) {
> ret = -ENOMEM;
> --
> 2.43.2
>
Reviewed-by: Michael Shavit <mshavit@google.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 09/27] iommu/arm-smmu-v3: Allocate the CD table entry in advance
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-13 12:17 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-13 12:17 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Avoid arm_smmu_attach_dev() having to undo the changes to the
> smmu_domain->devices list, acquire the cdptr earlier so we don't need to
> handle that error.
>
> Now there is a clear break in arm_smmu_attach_dev() where all the
> prep-work has been done non-disruptively and we commit to making the HW
> change, which cannot fail.
>
> This completes transforming arm_smmu_attach_dev() so that it does not
> disturb the HW if it fails.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 24 +++++++--------------
> 1 file changed, 8 insertions(+), 16 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 2dd6cb17112e98..39081d828a2132 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2676,6 +2676,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> struct arm_smmu_device *smmu;
> struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> struct arm_smmu_master *master;
> + struct arm_smmu_cd *cdptr;
>
> if (!fwspec)
> return -ENOENT;
> @@ -2704,6 +2705,12 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> if (ret)
> return ret;
>
> + if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
> + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> + if (!cdptr)
> + return -ENOMEM;
> + }
> +
> /*
> * Prevent arm_smmu_share_asid() from trying to change the ASID
> * of either the old or new domain while we are working on it.
> @@ -2723,13 +2730,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> switch (smmu_domain->stage) {
> case ARM_SMMU_DOMAIN_S1: {
> struct arm_smmu_cd target_cd;
> - struct arm_smmu_cd *cdptr;
> -
> - cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> - if (!cdptr) {
> - ret = -ENOMEM;
> - goto out_list_del;
> - }
>
> arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> @@ -2746,16 +2746,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> }
>
> arm_smmu_enable_ats(master, smmu_domain);
> - goto out_unlock;
> -
> -out_list_del:
> - spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - list_del_init(&master->domain_head);
> - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> -
> -out_unlock:
> mutex_unlock(&arm_smmu_asid_lock);
> - return ret;
> + return 0;
> }
>
> static int arm_smmu_attach_dev_ste(struct device *dev,
> --
> 2.43.2
>
Reviewed-by: Michael Shavit <mshavit@google.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 09/27] iommu/arm-smmu-v3: Allocate the CD table entry in advance
@ 2024-03-13 12:17 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-13 12:17 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Avoid arm_smmu_attach_dev() having to undo the changes to the
> smmu_domain->devices list, acquire the cdptr earlier so we don't need to
> handle that error.
>
> Now there is a clear break in arm_smmu_attach_dev() where all the
> prep-work has been done non-disruptively and we commit to making the HW
> change, which cannot fail.
>
> This completes transforming arm_smmu_attach_dev() so that it does not
> disturb the HW if it fails.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 24 +++++++--------------
> 1 file changed, 8 insertions(+), 16 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 2dd6cb17112e98..39081d828a2132 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2676,6 +2676,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> struct arm_smmu_device *smmu;
> struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> struct arm_smmu_master *master;
> + struct arm_smmu_cd *cdptr;
>
> if (!fwspec)
> return -ENOENT;
> @@ -2704,6 +2705,12 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> if (ret)
> return ret;
>
> + if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
> + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> + if (!cdptr)
> + return -ENOMEM;
> + }
> +
> /*
> * Prevent arm_smmu_share_asid() from trying to change the ASID
> * of either the old or new domain while we are working on it.
> @@ -2723,13 +2730,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> switch (smmu_domain->stage) {
> case ARM_SMMU_DOMAIN_S1: {
> struct arm_smmu_cd target_cd;
> - struct arm_smmu_cd *cdptr;
> -
> - cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> - if (!cdptr) {
> - ret = -ENOMEM;
> - goto out_list_del;
> - }
>
> arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> @@ -2746,16 +2746,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> }
>
> arm_smmu_enable_ats(master, smmu_domain);
> - goto out_unlock;
> -
> -out_list_del:
> - spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - list_del_init(&master->domain_head);
> - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> -
> -out_unlock:
> mutex_unlock(&arm_smmu_asid_lock);
> - return ret;
> + return 0;
> }
>
> static int arm_smmu_attach_dev_ste(struct device *dev,
> --
> 2.43.2
>
Reviewed-by: Michael Shavit <mshavit@google.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 02/27] iommu/arm-smmu-v3: Do not ATC invalidate the entire domain
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-15 2:24 ` Nicolin Chen
-1 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-15 2:24 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:50PM -0400, Jason Gunthorpe wrote:
> At this point we know which master we are going to change the PCI config
> on, this is the only device we need to invalidate. Switch
> arm_smmu_atc_inv_domain() for arm_smmu_atc_inv_master().
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 02/27] iommu/arm-smmu-v3: Do not ATC invalidate the entire domain
@ 2024-03-15 2:24 ` Nicolin Chen
0 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-15 2:24 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:50PM -0400, Jason Gunthorpe wrote:
> At this point we know which master we are going to change the PCI config
> on, this is the only device we need to invalidate. Switch
> arm_smmu_atc_inv_domain() for arm_smmu_atc_inv_master().
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 03/27] iommu/arm-smmu-v3: Add a type for the CD entry
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-15 3:12 ` Nicolin Chen
-1 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-15 3:12 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:51PM -0400, Jason Gunthorpe wrote:
> Instead of passing a naked __le16 * around to represent a CD table entry
> wrap it in a "struct arm_smmu_cd" with an array of the correct size. This
> makes it much clearer which functions will comprise the "CD API".
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 03/27] iommu/arm-smmu-v3: Add a type for the CD entry
@ 2024-03-15 3:12 ` Nicolin Chen
0 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-15 3:12 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:51PM -0400, Jason Gunthorpe wrote:
> Instead of passing a naked __le16 * around to represent a CD table entry
> wrap it in a "struct arm_smmu_cd" with an array of the correct size. This
> makes it much clearer which functions will comprise the "CD API".
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 01/27] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-15 3:38 ` Nicolin Chen
-1 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-15 3:38 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:49PM -0400, Jason Gunthorpe wrote:
> The SVA code is wired to assume that the SVA is programmed onto the
> mm->pasid. The current core code always does this, so it is fine.
>
> Add a check for clarity.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 2610e82c0ecd0d..347c2fdd865c1a 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -581,6 +581,9 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> int ret = 0;
> struct mm_struct *mm = domain->mm;
>
> + if (mm_get_enqcmd_pasid(mm) != id)
> + return -EINVAL;
> +
This seems to get deleted entirely by a followup change. Is it
added here to ensure a 1:1 mapping for git-bisect?
Thanks
Nicolin
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 01/27] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID
@ 2024-03-15 3:38 ` Nicolin Chen
0 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-15 3:38 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:49PM -0400, Jason Gunthorpe wrote:
> The SVA code is wired to assume that the SVA is programmed onto the
> mm->pasid. The current core code always does this, so it is fine.
>
> Add a check for clarity.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 2610e82c0ecd0d..347c2fdd865c1a 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -581,6 +581,9 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> int ret = 0;
> struct mm_struct *mm = domain->mm;
>
> + if (mm_get_enqcmd_pasid(mm) != id)
> + return -EINVAL;
> +
This seems to get deleted entirely by a followup change. Is it
added here to ensure a 1:1 mapping for git-bisect?
Thanks
Nicolin
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-15 4:22 ` Nicolin Chen
-1 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-15 4:22 UTC (permalink / raw)
To: Michael Shavit, Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, patches,
Shameerali Kolothum Thodi
Hi Michael/Jason,
On Mon, Mar 04, 2024 at 07:43:52PM -0400, Jason Gunthorpe wrote:
> Prepare to put the CD code into the same mechanism. Add an ops indirection
> around all the STE specific code and make the worker functions independent
> of the entry content being processed.
>
> get_used and sync ops are provided to hook the correct code.
>
> Signed-off-by: Michael Shavit <mshavit@google.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 172 ++++++++++++--------
> 1 file changed, 104 insertions(+), 68 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index c60b067c1f553e..b7f947e36f596f 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -48,8 +48,20 @@ enum arm_smmu_msi_index {
> ARM_SMMU_MAX_MSIS,
> };
>
> -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu,
> - ioasid_t sid);
> +struct arm_smmu_entry_writer_ops;
> +struct arm_smmu_entry_writer {
> + const struct arm_smmu_entry_writer_ops *ops;
> + struct arm_smmu_master *master;
> +};
> +
> +struct arm_smmu_entry_writer_ops {
> + unsigned int num_entry_qwords;
I vaguely remember some related discussion, yet can't find it
out. So sorry for questioning this, if it's already discussed.
Aren't CD and STE having the same num_entry_qwords in terms of
their values? Feels like we can just use NUM_ENTRY_QWORDS?
> + __le64 v_bit;
> + void (*get_used)(const __le64 *entry, __le64 *used);
> + void (*sync)(struct arm_smmu_entry_writer *writer);
> +};
> +
> +#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
And this seems to be just a fixed "8"? Since both are defined
straightforwardly:
struct arm_smmu_ste {
__le64 data[8];
};
...
struct arm_smmu_cd {
__le64 data[8];
};
Might be a bit nitpicking, yet maybe the other way around?
#define NUM_ENTRY_QWORDS 8
...
struct arm_smmu_ste {
__le64 data[NUM_ENTRY_QWORDS];
};
...
struct arm_smmu_cd {
__le64 data[NUM_ENTRY_QWORDS];
};
Thanks
Nicolin
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code
@ 2024-03-15 4:22 ` Nicolin Chen
0 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-15 4:22 UTC (permalink / raw)
To: Michael Shavit, Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, patches,
Shameerali Kolothum Thodi
Hi Michael/Jason,
On Mon, Mar 04, 2024 at 07:43:52PM -0400, Jason Gunthorpe wrote:
> Prepare to put the CD code into the same mechanism. Add an ops indirection
> around all the STE specific code and make the worker functions independent
> of the entry content being processed.
>
> get_used and sync ops are provided to hook the correct code.
>
> Signed-off-by: Michael Shavit <mshavit@google.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 172 ++++++++++++--------
> 1 file changed, 104 insertions(+), 68 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index c60b067c1f553e..b7f947e36f596f 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -48,8 +48,20 @@ enum arm_smmu_msi_index {
> ARM_SMMU_MAX_MSIS,
> };
>
> -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu,
> - ioasid_t sid);
> +struct arm_smmu_entry_writer_ops;
> +struct arm_smmu_entry_writer {
> + const struct arm_smmu_entry_writer_ops *ops;
> + struct arm_smmu_master *master;
> +};
> +
> +struct arm_smmu_entry_writer_ops {
> + unsigned int num_entry_qwords;
I vaguely remember some related discussion, yet can't find it
out. So sorry for questioning this, if it's already discussed.
Aren't CD and STE having the same num_entry_qwords in terms of
their values? Feels like we can just use NUM_ENTRY_QWORDS?
> + __le64 v_bit;
> + void (*get_used)(const __le64 *entry, __le64 *used);
> + void (*sync)(struct arm_smmu_entry_writer *writer);
> +};
> +
> +#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
And this seems to be just a fixed "8"? Since both are defined
straightforwardly:
struct arm_smmu_ste {
__le64 data[8];
};
...
struct arm_smmu_cd {
__le64 data[8];
};
Might be a bit nitpicking, yet maybe the other way around?
#define NUM_ENTRY_QWORDS 8
...
struct arm_smmu_ste {
__le64 data[NUM_ENTRY_QWORDS];
};
...
struct arm_smmu_cd {
__le64 data[NUM_ENTRY_QWORDS];
};
Thanks
Nicolin
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code
2024-03-15 4:22 ` Nicolin Chen
@ 2024-03-15 5:20 ` Nicolin Chen
-1 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-15 5:20 UTC (permalink / raw)
To: Michael Shavit, Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, patches,
Shameerali Kolothum Thodi
On Thu, Mar 14, 2024 at 09:23:00PM -0700, Nicolin Chen wrote:
> Hi Michael/Jason,
>
> On Mon, Mar 04, 2024 at 07:43:52PM -0400, Jason Gunthorpe wrote:
> > Prepare to put the CD code into the same mechanism. Add an ops indirection
> > around all the STE specific code and make the worker functions independent
> > of the entry content being processed.
> >
> > get_used and sync ops are provided to hook the correct code.
> >
> > Signed-off-by: Michael Shavit <mshavit@google.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
With the following trivial comments being sent previously, I've
also retested this version with SVA cases covering two different
S1DSS configurations.
This seems to be the only patch not tagged with Tested-by. So,
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Thanks
Nicolin
> > ---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 172 ++++++++++++--------
> > 1 file changed, 104 insertions(+), 68 deletions(-)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index c60b067c1f553e..b7f947e36f596f 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -48,8 +48,20 @@ enum arm_smmu_msi_index {
> > ARM_SMMU_MAX_MSIS,
> > };
> >
> > -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu,
> > - ioasid_t sid);
> > +struct arm_smmu_entry_writer_ops;
> > +struct arm_smmu_entry_writer {
> > + const struct arm_smmu_entry_writer_ops *ops;
> > + struct arm_smmu_master *master;
> > +};
> > +
> > +struct arm_smmu_entry_writer_ops {
> > + unsigned int num_entry_qwords;
>
> I vaguely remember some related discussion, yet can't find it
> out. So sorry for questioning this, if it's already discussed.
> Aren't CD and STE having the same num_entry_qwords in terms of
> their values? Feels like we can just use NUM_ENTRY_QWORDS?
>
> > + __le64 v_bit;
> > + void (*get_used)(const __le64 *entry, __le64 *used);
> > + void (*sync)(struct arm_smmu_entry_writer *writer);
> > +};
> > +
> > +#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
>
> And this seems to be just a fixed "8"? Since both are defined
> straightforwardly:
>
> struct arm_smmu_ste {
> __le64 data[8];
> };
> ...
> struct arm_smmu_cd {
> __le64 data[8];
> };
>
> Might be a bit nitpicking, yet maybe the other way around?
>
> #define NUM_ENTRY_QWORDS 8
> ...
> struct arm_smmu_ste {
> __le64 data[NUM_ENTRY_QWORDS];
> };
> ...
> struct arm_smmu_cd {
> __le64 data[NUM_ENTRY_QWORDS];
> };
>
> Thanks
> Nicolin
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code
@ 2024-03-15 5:20 ` Nicolin Chen
0 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-15 5:20 UTC (permalink / raw)
To: Michael Shavit, Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, patches,
Shameerali Kolothum Thodi
On Thu, Mar 14, 2024 at 09:23:00PM -0700, Nicolin Chen wrote:
> Hi Michael/Jason,
>
> On Mon, Mar 04, 2024 at 07:43:52PM -0400, Jason Gunthorpe wrote:
> > Prepare to put the CD code into the same mechanism. Add an ops indirection
> > around all the STE specific code and make the worker functions independent
> > of the entry content being processed.
> >
> > get_used and sync ops are provided to hook the correct code.
> >
> > Signed-off-by: Michael Shavit <mshavit@google.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
With the following trivial comments being sent previously, I've
also retested this version with SVA cases covering two different
S1DSS configurations.
This seems to be the only patch not tagged with Tested-by. So,
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Thanks
Nicolin
> > ---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 172 ++++++++++++--------
> > 1 file changed, 104 insertions(+), 68 deletions(-)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index c60b067c1f553e..b7f947e36f596f 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -48,8 +48,20 @@ enum arm_smmu_msi_index {
> > ARM_SMMU_MAX_MSIS,
> > };
> >
> > -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu,
> > - ioasid_t sid);
> > +struct arm_smmu_entry_writer_ops;
> > +struct arm_smmu_entry_writer {
> > + const struct arm_smmu_entry_writer_ops *ops;
> > + struct arm_smmu_master *master;
> > +};
> > +
> > +struct arm_smmu_entry_writer_ops {
> > + unsigned int num_entry_qwords;
>
> I vaguely remember some related discussion, yet can't find it
> out. So sorry for questioning this, if it's already discussed.
> Aren't CD and STE having the same num_entry_qwords in terms of
> their values? Feels like we can just use NUM_ENTRY_QWORDS?
>
> > + __le64 v_bit;
> > + void (*get_used)(const __le64 *entry, __le64 *used);
> > + void (*sync)(struct arm_smmu_entry_writer *writer);
> > +};
> > +
> > +#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
>
> And this seems to be just a fixed "8"? Since both are defined
> straightforwardly:
>
> struct arm_smmu_ste {
> __le64 data[8];
> };
> ...
> struct arm_smmu_cd {
> __le64 data[8];
> };
>
> Might be a bit nitpicking, yet maybe the other way around?
>
> #define NUM_ENTRY_QWORDS 8
> ...
> struct arm_smmu_ste {
> __le64 data[NUM_ENTRY_QWORDS];
> };
> ...
> struct arm_smmu_cd {
> __le64 data[NUM_ENTRY_QWORDS];
> };
>
> Thanks
> Nicolin
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 06/27] iommu/arm-smmu-v3: Consolidate clearing a CD table entry
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-15 6:17 ` Nicolin Chen
-1 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-15 6:17 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:54PM -0400, Jason Gunthorpe wrote:
> A cleared entry is all 0's. Make arm_smmu_clear_cd() do this sequence.
>
> If we are clearing an entry and for some reason it is not already
> allocated in the CD table then something has gone wrong.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 06/27] iommu/arm-smmu-v3: Consolidate clearing a CD table entry
@ 2024-03-15 6:17 ` Nicolin Chen
0 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-15 6:17 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:54PM -0400, Jason Gunthorpe wrote:
> A cleared entry is all 0's. Make arm_smmu_clear_cd() do this sequence.
>
> If we are clearing an entry and for some reason it is not already
> allocated in the CD table then something has gone wrong.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-15 7:52 ` Nicolin Chen
-1 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-15 7:52 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:53PM -0400, Jason Gunthorpe wrote:
> CD table entries and STE's have the same essential programming sequence,
> just with different types and sizes.
Hmm.. I somehow remember that one of them was 4 qwords while
the other was 8? Yet now, in the final driver source code on
the smmuv3_newapi branch, they are both 8 qwords? Feels like
some of the statement and the change doesn't match with the
reality so well -- hopefully I am not missing something...
Thanks
Nicolin
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
@ 2024-03-15 7:52 ` Nicolin Chen
0 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-15 7:52 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:53PM -0400, Jason Gunthorpe wrote:
> CD table entries and STE's have the same essential programming sequence,
> just with different types and sizes.
Hmm.. I somehow remember that one of them was 4 qwords while
the other was 8? Yet now, in the final driver source code on
the smmuv3_newapi branch, they are both 8 qwords? Feels like
some of the statement and the change doesn't match with the
reality so well -- hopefully I am not missing something...
Thanks
Nicolin
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 11/27] iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd()
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-15 10:04 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-15 10:04 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Half the code was living in arm_smmu_domain_finalise_s1(), just move it
> here and take the values directly from the pgtbl_ops instead of storing
> copies.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 47 ++++++++-------------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 --
> 2 files changed, 18 insertions(+), 32 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 50d17e3ce0a956..dfdd48cf217c4e 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1301,15 +1301,25 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> struct arm_smmu_domain *smmu_domain)
> {
> struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
> + const struct io_pgtable_cfg *pgtbl_cfg =
> + &io_pgtable_ops_to_pgtable(smmu_domain->pgtbl_ops)->cfg;
> + typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr =
> + &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
>
> memset(target, 0, sizeof(*target));
>
> target->data[0] = cpu_to_le64(
> - cd->tcr |
> + FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) |
> + FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) |
> + FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) |
> + FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) |
> + FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) |
> + CTXDESC_CD_0_TCR_EPD1 |
> #ifdef __BIG_ENDIAN
> CTXDESC_CD_0_ENDI |
> #endif
> CTXDESC_CD_0_V |
> + FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) |
I guess you're trying to keep these ordered by bit position, in which
case EPD1 should go after ENDI.
> CTXDESC_CD_0_AA64 |
> (master->stall_enabled ? CTXDESC_CD_0_S : 0) |
> CTXDESC_CD_0_R |
> @@ -1317,9 +1327,9 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> CTXDESC_CD_0_ASET |
> FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid)
> );
> -
> - target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> - target->data[3] = cpu_to_le64(cd->mair);
> + target->data[1] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.ttbr &
> + CTXDESC_CD_1_TTB0_MASK);
> + target->data[3] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.mair);
> }
>
> void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> @@ -2305,13 +2315,11 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
> }
>
> static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
> - struct arm_smmu_domain *smmu_domain,
> - struct io_pgtable_cfg *pgtbl_cfg)
> + struct arm_smmu_domain *smmu_domain)
> {
> int ret;
> u32 asid;
> struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
> - typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
>
> refcount_set(&cd->refs, 1);
>
> @@ -2319,31 +2327,13 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
> mutex_lock(&arm_smmu_asid_lock);
> ret = xa_alloc(&arm_smmu_asid_xa, &asid, cd,
> XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> - if (ret)
> - goto out_unlock;
> -
> cd->asid = (u16)asid;
> - cd->ttbr = pgtbl_cfg->arm_lpae_s1_cfg.ttbr;
> - cd->tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) |
> - CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64;
> - cd->mair = pgtbl_cfg->arm_lpae_s1_cfg.mair;
> -
> - mutex_unlock(&arm_smmu_asid_lock);
> - return 0;
> -
> -out_unlock:
> mutex_unlock(&arm_smmu_asid_lock);
> return ret;
> }
>
> static int arm_smmu_domain_finalise_s2(struct arm_smmu_device *smmu,
> - struct arm_smmu_domain *smmu_domain,
> - struct io_pgtable_cfg *pgtbl_cfg)
> + struct arm_smmu_domain *smmu_domain)
> {
> int vmid;
> struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
> @@ -2367,8 +2357,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
> struct io_pgtable_cfg pgtbl_cfg;
> struct io_pgtable_ops *pgtbl_ops;
> int (*finalise_stage_fn)(struct arm_smmu_device *smmu,
> - struct arm_smmu_domain *smmu_domain,
> - struct io_pgtable_cfg *pgtbl_cfg);
> + struct arm_smmu_domain *smmu_domain);
>
> /* Restrict the stage to what we can actually support */
> if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S1))
> @@ -2411,7 +2400,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
> smmu_domain->domain.geometry.aperture_end = (1UL << pgtbl_cfg.ias) - 1;
> smmu_domain->domain.geometry.force_aperture = true;
>
> - ret = finalise_stage_fn(smmu, smmu_domain, &pgtbl_cfg);
> + ret = finalise_stage_fn(smmu, smmu_domain);
> if (ret < 0) {
> free_io_pgtable_ops(pgtbl_ops);
> return ret;
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 8eabcccb9420ba..468cd33b80ac35 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -587,9 +587,6 @@ struct arm_smmu_strtab_l1_desc {
>
> struct arm_smmu_ctx_desc {
> u16 asid;
> - u64 ttbr;
> - u64 tcr;
> - u64 mair;
>
> refcount_t refs;
> struct mm_struct *mm;
> --
> 2.43.2
>
Apart from the nit,
Reviewed-by: Michael Shavit <mshavit@google.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 11/27] iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd()
@ 2024-03-15 10:04 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-15 10:04 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Half the code was living in arm_smmu_domain_finalise_s1(), just move it
> here and take the values directly from the pgtbl_ops instead of storing
> copies.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 47 ++++++++-------------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 --
> 2 files changed, 18 insertions(+), 32 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 50d17e3ce0a956..dfdd48cf217c4e 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1301,15 +1301,25 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> struct arm_smmu_domain *smmu_domain)
> {
> struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
> + const struct io_pgtable_cfg *pgtbl_cfg =
> + &io_pgtable_ops_to_pgtable(smmu_domain->pgtbl_ops)->cfg;
> + typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr =
> + &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
>
> memset(target, 0, sizeof(*target));
>
> target->data[0] = cpu_to_le64(
> - cd->tcr |
> + FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) |
> + FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) |
> + FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) |
> + FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) |
> + FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) |
> + CTXDESC_CD_0_TCR_EPD1 |
> #ifdef __BIG_ENDIAN
> CTXDESC_CD_0_ENDI |
> #endif
> CTXDESC_CD_0_V |
> + FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) |
I guess you're trying to keep these ordered by bit position, in which
case EPD1 should go after ENDI.
> CTXDESC_CD_0_AA64 |
> (master->stall_enabled ? CTXDESC_CD_0_S : 0) |
> CTXDESC_CD_0_R |
> @@ -1317,9 +1327,9 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> CTXDESC_CD_0_ASET |
> FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid)
> );
> -
> - target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> - target->data[3] = cpu_to_le64(cd->mair);
> + target->data[1] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.ttbr &
> + CTXDESC_CD_1_TTB0_MASK);
> + target->data[3] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.mair);
> }
>
> void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> @@ -2305,13 +2315,11 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
> }
>
> static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
> - struct arm_smmu_domain *smmu_domain,
> - struct io_pgtable_cfg *pgtbl_cfg)
> + struct arm_smmu_domain *smmu_domain)
> {
> int ret;
> u32 asid;
> struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
> - typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
>
> refcount_set(&cd->refs, 1);
>
> @@ -2319,31 +2327,13 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
> mutex_lock(&arm_smmu_asid_lock);
> ret = xa_alloc(&arm_smmu_asid_xa, &asid, cd,
> XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> - if (ret)
> - goto out_unlock;
> -
> cd->asid = (u16)asid;
> - cd->ttbr = pgtbl_cfg->arm_lpae_s1_cfg.ttbr;
> - cd->tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) |
> - CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64;
> - cd->mair = pgtbl_cfg->arm_lpae_s1_cfg.mair;
> -
> - mutex_unlock(&arm_smmu_asid_lock);
> - return 0;
> -
> -out_unlock:
> mutex_unlock(&arm_smmu_asid_lock);
> return ret;
> }
>
> static int arm_smmu_domain_finalise_s2(struct arm_smmu_device *smmu,
> - struct arm_smmu_domain *smmu_domain,
> - struct io_pgtable_cfg *pgtbl_cfg)
> + struct arm_smmu_domain *smmu_domain)
> {
> int vmid;
> struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
> @@ -2367,8 +2357,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
> struct io_pgtable_cfg pgtbl_cfg;
> struct io_pgtable_ops *pgtbl_ops;
> int (*finalise_stage_fn)(struct arm_smmu_device *smmu,
> - struct arm_smmu_domain *smmu_domain,
> - struct io_pgtable_cfg *pgtbl_cfg);
> + struct arm_smmu_domain *smmu_domain);
>
> /* Restrict the stage to what we can actually support */
> if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S1))
> @@ -2411,7 +2400,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
> smmu_domain->domain.geometry.aperture_end = (1UL << pgtbl_cfg.ias) - 1;
> smmu_domain->domain.geometry.force_aperture = true;
>
> - ret = finalise_stage_fn(smmu, smmu_domain, &pgtbl_cfg);
> + ret = finalise_stage_fn(smmu, smmu_domain);
> if (ret < 0) {
> free_io_pgtable_ops(pgtbl_ops);
> return ret;
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 8eabcccb9420ba..468cd33b80ac35 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -587,9 +587,6 @@ struct arm_smmu_strtab_l1_desc {
>
> struct arm_smmu_ctx_desc {
> u16 asid;
> - u64 ttbr;
> - u64 tcr;
> - u64 mair;
>
> refcount_t refs;
> struct mm_struct *mm;
> --
> 2.43.2
>
Apart from the nit,
Reviewed-by: Michael Shavit <mshavit@google.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* RE: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-15 10:40 ` Shameerali Kolothum Thodi
-1 siblings, 0 replies; 232+ messages in thread
From: Shameerali Kolothum Thodi @ 2024-03-15 10:40 UTC (permalink / raw)
To: Jason Gunthorpe, iommu, Joerg Roedel, linux-arm-kernel,
Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches
> -----Original Message-----
> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Monday, March 4, 2024 11:44 PM
> To: iommu@lists.linux.dev; Joerg Roedel <joro@8bytes.org>; linux-arm-
> kernel@lists.infradead.org; Robin Murphy <robin.murphy@arm.com>; Will
> Deacon <will@kernel.org>
> Cc: Eric Auger <eric.auger@redhat.com>; Jean-Philippe Brucker <jean-
> philippe@linaro.org>; Moritz Fischer <mdf@kernel.org>; Michael Shavit
> <mshavit@google.com>; Nicolin Chen <nicolinc@nvidia.com>;
> patches@lists.linux.dev; Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>
> Subject: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
>
> Continuing the work of part 1 this focuses on the CD, PASID and SVA
> components:
>
> - attach_dev failure does not change the HW configuration.
>
> - Full PASID API support including:
> - S1/SVA domains attached to PASIDs
> - IDENTITY/BLOCKED/S1 attached to RID
> - Change of the RID domain while PASIDs are attached
>
> - Streamlined SVA support using the core infrastructure
>
> - Hitless, whenever possible, change between two domains
>
> Making the CD programming work like the new STE programming allows
> untangling some of the confusing SVA flows. From there the focus is on
> building out the core infrastructure for dealing with PASID and CD
> entries, then keeping track of unique SSID's for ATS invalidation.
>
> The ATS ordering is generalized so that the PASID flow can use it and put
> into a form where it is fully hitless, whenever possible. Care is taken to
> ensure that ATC flushes are present after any change in translation.
>
> Finally we simply kill the entire outdated SVA mmu_notifier implementation
> in one shot and switch it over to the newly created generic PASID & CD
> code. This avoids the messy and confusing approach of trying to
> incrementally untangle this in place. The new code is small and simple
> enough this is much better than trying to figure out smaller steps.
>
> Once SVA is resting on the right CD code it is straightforward to make the
> PASID interface functionally complete.
>
> It achieves the same goals as the several series from Michael and the S1DSS
> series from Nicolin that were trying to improve portions of the API.
>
> This is on github:
> https://github.com/jgunthorpe/linux/commits/smmuv3_newapi
Performed few tests with this series on a HiSilicon D06 board(SMMUv3).
-Host kernel: boot with translated and passthrough cases.
-Host kernel: ACC dev SVA test run with uadk/uadk_tool benchmark.
With Qemu branch:
https://github.com/nicolinc/qemu/commits/wip/iommufd_vsmmu-02292024/
-Guest with a n/w VF dev, legacy VFIO mode.
-Guest with a n/w VF dev, IOMMUFD mode.
-Hot plug(add/del) on both VFIO and IOMMUFD modes.
All works as expected.
FWIW:
Tested-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
Thanks,
Shameer
^ permalink raw reply [flat|nested] 232+ messages in thread
* RE: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
@ 2024-03-15 10:40 ` Shameerali Kolothum Thodi
0 siblings, 0 replies; 232+ messages in thread
From: Shameerali Kolothum Thodi @ 2024-03-15 10:40 UTC (permalink / raw)
To: Jason Gunthorpe, iommu, Joerg Roedel, linux-arm-kernel,
Robin Murphy, Will Deacon
Cc: Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches
> -----Original Message-----
> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Monday, March 4, 2024 11:44 PM
> To: iommu@lists.linux.dev; Joerg Roedel <joro@8bytes.org>; linux-arm-
> kernel@lists.infradead.org; Robin Murphy <robin.murphy@arm.com>; Will
> Deacon <will@kernel.org>
> Cc: Eric Auger <eric.auger@redhat.com>; Jean-Philippe Brucker <jean-
> philippe@linaro.org>; Moritz Fischer <mdf@kernel.org>; Michael Shavit
> <mshavit@google.com>; Nicolin Chen <nicolinc@nvidia.com>;
> patches@lists.linux.dev; Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>
> Subject: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
>
> Continuing the work of part 1 this focuses on the CD, PASID and SVA
> components:
>
> - attach_dev failure does not change the HW configuration.
>
> - Full PASID API support including:
> - S1/SVA domains attached to PASIDs
> - IDENTITY/BLOCKED/S1 attached to RID
> - Change of the RID domain while PASIDs are attached
>
> - Streamlined SVA support using the core infrastructure
>
> - Hitless, whenever possible, change between two domains
>
> Making the CD programming work like the new STE programming allows
> untangling some of the confusing SVA flows. From there the focus is on
> building out the core infrastructure for dealing with PASID and CD
> entries, then keeping track of unique SSID's for ATS invalidation.
>
> The ATS ordering is generalized so that the PASID flow can use it and put
> into a form where it is fully hitless, whenever possible. Care is taken to
> ensure that ATC flushes are present after any change in translation.
>
> Finally we simply kill the entire outdated SVA mmu_notifier implementation
> in one shot and switch it over to the newly created generic PASID & CD
> code. This avoids the messy and confusing approach of trying to
> incrementally untangle this in place. The new code is small and simple
> enough this is much better than trying to figure out smaller steps.
>
> Once SVA is resting on the right CD code it is straightforward to make the
> PASID interface functionally complete.
>
> It achieves the same goals as the several series from Michael and the S1DSS
> series from Nicolin that were trying to improve portions of the API.
>
> This is on github:
> https://github.com/jgunthorpe/linux/commits/smmuv3_newapi
Performed few tests with this series on a HiSilicon D06 board(SMMUv3).
-Host kernel: boot with translated and passthrough cases.
-Host kernel: ACC dev SVA test run with uadk/uadk_tool benchmark.
With Qemu branch:
https://github.com/nicolinc/qemu/commits/wip/iommufd_vsmmu-02292024/
-Guest with a n/w VF dev, legacy VFIO mode.
-Guest with a n/w VF dev, IOMMUFD mode.
-Hot plug(add/del) on both VFIO and IOMMUFD modes.
All works as expected.
FWIW:
Tested-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
Thanks,
Shameer
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 08/27] iommu/arm-smmu-v3: Move allocation of the cdtable into arm_smmu_get_cd_ptr()
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-16 3:31 ` Nicolin Chen
-1 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-16 3:31 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:56PM -0400, Jason Gunthorpe wrote:
> No reason to force callers to do two steps. Make arm_smmu_get_cd_ptr()
> able to return an entry in all cases except OOM.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 08/27] iommu/arm-smmu-v3: Move allocation of the cdtable into arm_smmu_get_cd_ptr()
@ 2024-03-16 3:31 ` Nicolin Chen
0 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-16 3:31 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:56PM -0400, Jason Gunthorpe wrote:
> No reason to force callers to do two steps. Make arm_smmu_get_cd_ptr()
> able to return an entry in all cases except OOM.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 09/27] iommu/arm-smmu-v3: Allocate the CD table entry in advance
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-16 4:16 ` Nicolin Chen
-1 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-16 4:16 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:57PM -0400, Jason Gunthorpe wrote:
> Avoid arm_smmu_attach_dev() having to undo the changes to the
> smmu_domain->devices list, acquire the cdptr earlier so we don't need to
> handle that error.
I should probably mention this in the other patch, yet PATCH-14
adding arm_smmu_attach_prepare() to this function doesn't have a
rollback for CD table allocation. I assume we're fine with that?
> Now there is a clear break in arm_smmu_attach_dev() where all the
> prep-work has been done non-disruptively and we commit to making the HW
> change, which cannot fail.
>
> This completes transforming arm_smmu_attach_dev() so that it does not
> disturb the HW if it fails.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 09/27] iommu/arm-smmu-v3: Allocate the CD table entry in advance
@ 2024-03-16 4:16 ` Nicolin Chen
0 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-16 4:16 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:57PM -0400, Jason Gunthorpe wrote:
> Avoid arm_smmu_attach_dev() having to undo the changes to the
> smmu_domain->devices list, acquire the cdptr earlier so we don't need to
> handle that error.
I should probably mention this in the other patch, yet PATCH-14
adding arm_smmu_attach_prepare() to this function doesn't have a
rollback for CD table allocation. I assume we're fine with that?
> Now there is a clear break in arm_smmu_attach_dev() where all the
> prep-work has been done non-disruptively and we commit to making the HW
> change, which cannot fail.
>
> This completes transforming arm_smmu_attach_dev() so that it does not
> disturb the HW if it fails.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 10/27] iommu/arm-smmu-v3: Move the CD generation for SVA into a function
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-16 5:19 ` Nicolin Chen
-1 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-16 5:19 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:58PM -0400, Jason Gunthorpe wrote:
> @@ -1070,8 +1062,7 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> * Each bit indicates that a used bit in a qword needs to be
> * changed after unused_update is applied.
> */
> - if ((unused_update[i] & target_used[i]) !=
> - (target[i] & target_used[i]))
> + if ((unused_update[i] & target_used[i]) != target[i])
> used_qword_diff |= 1 << i;
Would you mind elaborating why we drop this "& target_used[i]"
against the target[i] on the right?
Thanks
Nicolin
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 10/27] iommu/arm-smmu-v3: Move the CD generation for SVA into a function
@ 2024-03-16 5:19 ` Nicolin Chen
0 siblings, 0 replies; 232+ messages in thread
From: Nicolin Chen @ 2024-03-16 5:19 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:58PM -0400, Jason Gunthorpe wrote:
> @@ -1070,8 +1062,7 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> * Each bit indicates that a used bit in a qword needs to be
> * changed after unused_update is applied.
> */
> - if ((unused_update[i] & target_used[i]) !=
> - (target[i] & target_used[i]))
> + if ((unused_update[i] & target_used[i]) != target[i])
> used_qword_diff |= 1 << i;
Would you mind elaborating why we drop this "& target_used[i]"
against the target[i] on the right?
Thanks
Nicolin
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 02/27] iommu/arm-smmu-v3: Do not ATC invalidate the entire domain
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-16 18:09 ` Moritz Fischer
-1 siblings, 0 replies; 232+ messages in thread
From: Moritz Fischer @ 2024-03-16 18:09 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:50PM -0400, Jason Gunthorpe wrote:
> At this point we know which master we are going to change the PCI config
> on, this is the only device we need to invalidate. Switch
> arm_smmu_atc_inv_domain() for arm_smmu_atc_inv_master().
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Moritz Fischer <moritzf@google.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index d1bc151a5dff8c..9e9233331c4636 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2447,7 +2447,10 @@ static void arm_smmu_enable_ats(struct
> arm_smmu_master *master,
> pdev = to_pci_dev(master->dev);
> atomic_inc(&smmu_domain->nr_ats_masters);
> - arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, 0, 0);
> + /*
> + * ATC invalidation of PASID 0 causes the entire ATC to be flushed.
> + */
> + arm_smmu_atc_inv_master(master);
> if (pci_enable_ats(pdev, stu))
> dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu);
> }
> --
> 2.43.2
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 02/27] iommu/arm-smmu-v3: Do not ATC invalidate the entire domain
@ 2024-03-16 18:09 ` Moritz Fischer
0 siblings, 0 replies; 232+ messages in thread
From: Moritz Fischer @ 2024-03-16 18:09 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:50PM -0400, Jason Gunthorpe wrote:
> At this point we know which master we are going to change the PCI config
> on, this is the only device we need to invalidate. Switch
> arm_smmu_atc_inv_domain() for arm_smmu_atc_inv_master().
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Moritz Fischer <moritzf@google.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index d1bc151a5dff8c..9e9233331c4636 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2447,7 +2447,10 @@ static void arm_smmu_enable_ats(struct
> arm_smmu_master *master,
> pdev = to_pci_dev(master->dev);
> atomic_inc(&smmu_domain->nr_ats_masters);
> - arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, 0, 0);
> + /*
> + * ATC invalidation of PASID 0 causes the entire ATC to be flushed.
> + */
> + arm_smmu_atc_inv_master(master);
> if (pci_enable_ats(pdev, stu))
> dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu);
> }
> --
> 2.43.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 03/27] iommu/arm-smmu-v3: Add a type for the CD entry
2024-03-13 9:44 ` Michael Shavit
@ 2024-03-16 18:10 ` Moritz Fischer
-1 siblings, 0 replies; 232+ messages in thread
From: Moritz Fischer @ 2024-03-16 18:10 UTC (permalink / raw)
To: Michael Shavit
Cc: Jason Gunthorpe, iommu, Joerg Roedel, linux-arm-kernel,
Robin Murphy, Will Deacon, Eric Auger, Jean-Philippe Brucker,
Moritz Fischer, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Wed, Mar 13, 2024 at 05:44:35PM +0800, Michael Shavit wrote:
> On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > Instead of passing a naked __le16 * around to represent a CD table entry
> > wrap it in a "struct arm_smmu_cd" with an array of the correct size.
> This
> > makes it much clearer which functions will comprise the "CD API".
> >
> > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 +++++++++++---------
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 ++++++-
> > 2 files changed, 17 insertions(+), 10 deletions(-)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index 9e9233331c4636..c60b067c1f553e 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -1219,7 +1219,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
> > WRITE_ONCE(*dst, cpu_to_le64(val));
> > }
> >
> > -static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32
> ssid)
> > +static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master
> *master,
> > + u32 ssid)
> > {
> > __le64 *l1ptr;
> > unsigned int idx;
> > @@ -1228,7 +1229,8 @@ static __le64 *arm_smmu_get_cd_ptr(struct
> arm_smmu_master *master, u32 ssid)
> > struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
> >
> > if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
> > - return cd_table->cdtab + ssid * CTXDESC_CD_DWORDS;
> > + return (struct arm_smmu_cd *)(cd_table->cdtab +
> > + ssid * CTXDESC_CD_DWORDS);
> Can we define cd_table.cdtab as a union type to avoid this cast and
> make the struct definition more explicit?
+1, I think that'd make it more readable.
> >
> > idx = ssid >> CTXDESC_SPLIT;
> > l1_desc = &cd_table->l1_desc[idx];
> > @@ -1242,7 +1244,7 @@ static __le64 *arm_smmu_get_cd_ptr(struct
> arm_smmu_master *master, u32 ssid)
> > arm_smmu_sync_cd(master, ssid, false);
> > }
> > idx = ssid & (CTXDESC_L2_ENTRIES - 1);
> > - return l1_desc->l2ptr + idx * CTXDESC_CD_DWORDS;
> > + return &l1_desc->l2ptr[idx];
> > }
> >
> > int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> > @@ -1261,7 +1263,7 @@ int arm_smmu_write_ctx_desc(struct
> arm_smmu_master *master, int ssid,
> > */
> > u64 val;
> > bool cd_live;
> > - __le64 *cdptr;
> > + struct arm_smmu_cd *cdptr;
> > struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
> > struct arm_smmu_device *smmu = master->smmu;
> >
> > @@ -1272,7 +1274,7 @@ int arm_smmu_write_ctx_desc(struct
> arm_smmu_master *master, int ssid,
> > if (!cdptr)
> > return -ENOMEM;
> >
> > - val = le64_to_cpu(cdptr[0]);
> > + val = le64_to_cpu(cdptr->data[0]);
> > cd_live = !!(val & CTXDESC_CD_0_V);
> >
> > if (!cd) { /* (5) */
> > @@ -1289,9 +1291,9 @@ int arm_smmu_write_ctx_desc(struct
> arm_smmu_master *master, int ssid,
> > * this substream's traffic
> > */
> > } else { /* (1) and (2) */
> > - cdptr[1] = cpu_to_le64(cd->ttbr &
> CTXDESC_CD_1_TTB0_MASK);
> > - cdptr[2] = 0;
> > - cdptr[3] = cpu_to_le64(cd->mair);
> > + cdptr->data[1] = cpu_to_le64(cd->ttbr &
> CTXDESC_CD_1_TTB0_MASK);
> > + cdptr->data[2] = 0;
> > + cdptr->data[3] = cpu_to_le64(cd->mair);
> >
> > /*
> > * STE may be live, and the SMMU might read dwords of
> this CD in any
> > @@ -1323,7 +1325,7 @@ int arm_smmu_write_ctx_desc(struct
> arm_smmu_master *master, int ssid,
> > * field within an aligned 64-bit span of a structure can be
> altered
> > * without first making the structure invalid.
> > */
> > - WRITE_ONCE(cdptr[0], cpu_to_le64(val));
> > + WRITE_ONCE(cdptr->data[0], cpu_to_le64(val));
> > arm_smmu_sync_cd(master, ssid, true);
> > return 0;
> > }
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> > index 23baf117e7e4b5..7078ed569fd4d3 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> > @@ -282,6 +282,11 @@ struct arm_smmu_ste {
> > #define CTXDESC_L1_DESC_L2PTR_MASK GENMASK_ULL(51, 12)
> >
> > #define CTXDESC_CD_DWORDS 8
> > +
> > +struct arm_smmu_cd {
> > + __le64 data[CTXDESC_CD_DWORDS];
> > +};
> > +
> > #define CTXDESC_CD_0_TCR_T0SZ GENMASK_ULL(5, 0)
> > #define CTXDESC_CD_0_TCR_TG0 GENMASK_ULL(7, 6)
> > #define CTXDESC_CD_0_TCR_IRGN0 GENMASK_ULL(9, 8)
> > @@ -591,7 +596,7 @@ struct arm_smmu_ctx_desc {
> > };
> >
> > struct arm_smmu_l1_ctx_desc {
> > - __le64 *l2ptr;
> > + struct arm_smmu_cd *l2ptr;
> > dma_addr_t l2ptr_dma;
> > };
> >
> > --
> > 2.43.2
> >
> >
> Reviewed-by: Michael Shavit <mshavit@google.com>
Reviewed-by: Moritz Fischer <moritzf@google.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 03/27] iommu/arm-smmu-v3: Add a type for the CD entry
@ 2024-03-16 18:10 ` Moritz Fischer
0 siblings, 0 replies; 232+ messages in thread
From: Moritz Fischer @ 2024-03-16 18:10 UTC (permalink / raw)
To: Michael Shavit
Cc: Jason Gunthorpe, iommu, Joerg Roedel, linux-arm-kernel,
Robin Murphy, Will Deacon, Eric Auger, Jean-Philippe Brucker,
Moritz Fischer, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Wed, Mar 13, 2024 at 05:44:35PM +0800, Michael Shavit wrote:
> On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > Instead of passing a naked __le16 * around to represent a CD table entry
> > wrap it in a "struct arm_smmu_cd" with an array of the correct size.
> This
> > makes it much clearer which functions will comprise the "CD API".
> >
> > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 +++++++++++---------
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 ++++++-
> > 2 files changed, 17 insertions(+), 10 deletions(-)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index 9e9233331c4636..c60b067c1f553e 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -1219,7 +1219,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
> > WRITE_ONCE(*dst, cpu_to_le64(val));
> > }
> >
> > -static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32
> ssid)
> > +static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master
> *master,
> > + u32 ssid)
> > {
> > __le64 *l1ptr;
> > unsigned int idx;
> > @@ -1228,7 +1229,8 @@ static __le64 *arm_smmu_get_cd_ptr(struct
> arm_smmu_master *master, u32 ssid)
> > struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
> >
> > if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
> > - return cd_table->cdtab + ssid * CTXDESC_CD_DWORDS;
> > + return (struct arm_smmu_cd *)(cd_table->cdtab +
> > + ssid * CTXDESC_CD_DWORDS);
> Can we define cd_table.cdtab as a union type to avoid this cast and
> make the struct definition more explicit?
+1, I think that'd make it more readable.
> >
> > idx = ssid >> CTXDESC_SPLIT;
> > l1_desc = &cd_table->l1_desc[idx];
> > @@ -1242,7 +1244,7 @@ static __le64 *arm_smmu_get_cd_ptr(struct
> arm_smmu_master *master, u32 ssid)
> > arm_smmu_sync_cd(master, ssid, false);
> > }
> > idx = ssid & (CTXDESC_L2_ENTRIES - 1);
> > - return l1_desc->l2ptr + idx * CTXDESC_CD_DWORDS;
> > + return &l1_desc->l2ptr[idx];
> > }
> >
> > int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> > @@ -1261,7 +1263,7 @@ int arm_smmu_write_ctx_desc(struct
> arm_smmu_master *master, int ssid,
> > */
> > u64 val;
> > bool cd_live;
> > - __le64 *cdptr;
> > + struct arm_smmu_cd *cdptr;
> > struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
> > struct arm_smmu_device *smmu = master->smmu;
> >
> > @@ -1272,7 +1274,7 @@ int arm_smmu_write_ctx_desc(struct
> arm_smmu_master *master, int ssid,
> > if (!cdptr)
> > return -ENOMEM;
> >
> > - val = le64_to_cpu(cdptr[0]);
> > + val = le64_to_cpu(cdptr->data[0]);
> > cd_live = !!(val & CTXDESC_CD_0_V);
> >
> > if (!cd) { /* (5) */
> > @@ -1289,9 +1291,9 @@ int arm_smmu_write_ctx_desc(struct
> arm_smmu_master *master, int ssid,
> > * this substream's traffic
> > */
> > } else { /* (1) and (2) */
> > - cdptr[1] = cpu_to_le64(cd->ttbr &
> CTXDESC_CD_1_TTB0_MASK);
> > - cdptr[2] = 0;
> > - cdptr[3] = cpu_to_le64(cd->mair);
> > + cdptr->data[1] = cpu_to_le64(cd->ttbr &
> CTXDESC_CD_1_TTB0_MASK);
> > + cdptr->data[2] = 0;
> > + cdptr->data[3] = cpu_to_le64(cd->mair);
> >
> > /*
> > * STE may be live, and the SMMU might read dwords of
> this CD in any
> > @@ -1323,7 +1325,7 @@ int arm_smmu_write_ctx_desc(struct
> arm_smmu_master *master, int ssid,
> > * field within an aligned 64-bit span of a structure can be
> altered
> > * without first making the structure invalid.
> > */
> > - WRITE_ONCE(cdptr[0], cpu_to_le64(val));
> > + WRITE_ONCE(cdptr->data[0], cpu_to_le64(val));
> > arm_smmu_sync_cd(master, ssid, true);
> > return 0;
> > }
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> > index 23baf117e7e4b5..7078ed569fd4d3 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> > @@ -282,6 +282,11 @@ struct arm_smmu_ste {
> > #define CTXDESC_L1_DESC_L2PTR_MASK GENMASK_ULL(51, 12)
> >
> > #define CTXDESC_CD_DWORDS 8
> > +
> > +struct arm_smmu_cd {
> > + __le64 data[CTXDESC_CD_DWORDS];
> > +};
> > +
> > #define CTXDESC_CD_0_TCR_T0SZ GENMASK_ULL(5, 0)
> > #define CTXDESC_CD_0_TCR_TG0 GENMASK_ULL(7, 6)
> > #define CTXDESC_CD_0_TCR_IRGN0 GENMASK_ULL(9, 8)
> > @@ -591,7 +596,7 @@ struct arm_smmu_ctx_desc {
> > };
> >
> > struct arm_smmu_l1_ctx_desc {
> > - __le64 *l2ptr;
> > + struct arm_smmu_cd *l2ptr;
> > dma_addr_t l2ptr_dma;
> > };
> >
> > --
> > 2.43.2
> >
> >
> Reviewed-by: Michael Shavit <mshavit@google.com>
Reviewed-by: Moritz Fischer <moritzf@google.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-16 18:14 ` Moritz Fischer
-1 siblings, 0 replies; 232+ messages in thread
From: Moritz Fischer @ 2024-03-16 18:14 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:53PM -0400, Jason Gunthorpe wrote:
> CD table entries and STE's have the same essential programming sequence,
> just with different types and sizes.
> Have arm_smmu_write_ctx_desc() generate a target CD and call
> arm_smmu_write_entry() to do the programming. Due to the way the
> target CD is generated by modifying the existing CD this alone is not
> enough for the CD callers to be freed of the ordering requirements.
> The following patches will make the rest of the CD flow mirror the STE
> flow with precise CD contents generated in all cases.
> Currently the logic can't ensure that the CD always conforms to the used
> requirements until all the CD generation is moved to the new method. Add a
> temporary no_used_check to disable the assertions.
> Signed-off-by: Michael Shavit <mshavit@google.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 101 ++++++++++++++------
> 1 file changed, 74 insertions(+), 27 deletions(-)
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index b7f947e36f596f..237fd6d92c880b 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -57,11 +57,14 @@ struct arm_smmu_entry_writer {
> struct arm_smmu_entry_writer_ops {
> unsigned int num_entry_qwords;
> __le64 v_bit;
> + bool no_used_check;
> void (*get_used)(const __le64 *entry, __le64 *used);
> void (*sync)(struct arm_smmu_entry_writer *writer);
> };
> -#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
> +#define NUM_ENTRY_QWORDS \
> + (max(sizeof(struct arm_smmu_ste), sizeof(struct arm_smmu_cd)) / \
> + sizeof(u64))
> static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
> [EVTQ_MSI_INDEX] = {
> @@ -1056,7 +1059,8 @@ static u8 arm_smmu_entry_qword_diff(struct
> arm_smmu_entry_writer *writer,
> * allowed to set a bit to 1 if the used function doesn't say it
> * is used.
> */
> - WARN_ON_ONCE(target[i] & ~target_used[i]);
> + if (!writer->ops->no_used_check)
> + WARN_ON_ONCE(target[i] & ~target_used[i]);
> /* Bits can change because they are not currently being used */
> unused_update[i] = (entry[i] & cur_used[i]) |
> @@ -1065,7 +1069,8 @@ static u8 arm_smmu_entry_qword_diff(struct
> arm_smmu_entry_writer *writer,
> * Each bit indicates that a used bit in a qword needs to be
> * changed after unused_update is applied.
> */
> - if ((unused_update[i] & target_used[i]) != target[i])
> + if ((unused_update[i] & target_used[i]) !=
> + (target[i] & target_used[i]))
> used_qword_diff |= 1 << i;
> }
> return used_qword_diff;
> @@ -1161,8 +1166,11 @@ static void arm_smmu_write_entry(struct
> arm_smmu_entry_writer *writer,
> * in the entry. The target was already sanity checked by
> * compute_qword_diff().
> */
> - WARN_ON_ONCE(
> - entry_set(writer, entry, target, 0, num_entry_qwords));
> + if (writer->ops->no_used_check)
> + entry_set(writer, entry, target, 0, num_entry_qwords);
> + else
> + WARN_ON_ONCE(entry_set(writer, entry, target, 0,
> + num_entry_qwords));
> }
> }
> @@ -1242,6 +1250,59 @@ static struct arm_smmu_cd
> *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> return &l1_desc->l2ptr[idx];
> }
> +struct arm_smmu_cd_writer {
> + struct arm_smmu_entry_writer writer;
> + unsigned int ssid;
> +};
> +
> +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
> +{
> + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
> + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
> + return;
> + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
> +
> + /* EPD0 means T0SZ/TG0/IR0/OR0/SH0/TTB0 are IGNORED */
> + if (ent[0] & cpu_to_le64(CTXDESC_CD_0_TCR_EPD0)) {
> + used_bits[0] &= ~cpu_to_le64(
> + CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 |
> + CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 |
> + CTXDESC_CD_0_TCR_SH0);
> + used_bits[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK);
> + }
> +}
> +
> +static void arm_smmu_cd_writer_sync_entry(struct arm_smmu_entry_writer
> *writer)
> +{
> + struct arm_smmu_cd_writer *cd_writer =
> + container_of(writer, struct arm_smmu_cd_writer, writer);
> +
> + arm_smmu_sync_cd(writer->master, cd_writer->ssid, true);
> +}
> +
> +static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
> + .sync = arm_smmu_cd_writer_sync_entry,
> + .get_used = arm_smmu_get_cd_used,
> + .v_bit = cpu_to_le64(CTXDESC_CD_0_V),
> + .no_used_check = true,
> + .num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
> +};
> +
> +static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int
> ssid,
> + struct arm_smmu_cd *cdptr,
> + const struct arm_smmu_cd *target)
> +{
> + struct arm_smmu_cd_writer cd_writer = {
> + .writer = {
> + .ops = &arm_smmu_cd_writer_ops,
> + .master = master,
> + },
> + .ssid = ssid,
> + };
> +
> + arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
> +}
> +
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_ctx_desc *cd)
> {
> @@ -1258,17 +1319,20 @@ int arm_smmu_write_ctx_desc(struct
> arm_smmu_master *master, int ssid,
> */
> u64 val;
> bool cd_live;
> - struct arm_smmu_cd *cdptr;
> + struct arm_smmu_cd target;
> + struct arm_smmu_cd *cdptr = ⌖
> + struct arm_smmu_cd *cd_table_entry;
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
> struct arm_smmu_device *smmu = master->smmu;
> if (WARN_ON(ssid >= (1 << cd_table->s1cdmax)))
> return -E2BIG;
> - cdptr = arm_smmu_get_cd_ptr(master, ssid);
> - if (!cdptr)
> + cd_table_entry = arm_smmu_get_cd_ptr(master, ssid);
> + if (!cd_table_entry)
> return -ENOMEM;
> + target = *cd_table_entry;
> val = le64_to_cpu(cdptr->data[0]);
> cd_live = !!(val & CTXDESC_CD_0_V);
> @@ -1290,13 +1354,6 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master
> *master, int ssid,
> cdptr->data[2] = 0;
> cdptr->data[3] = cpu_to_le64(cd->mair);
> - /*
> - * STE may be live, and the SMMU might read dwords of this CD in any
> - * order. Ensure that it observes valid values before reading
> - * V=1.
> - */
> - arm_smmu_sync_cd(master, ssid, true);
> -
> val = cd->tcr |
> #ifdef __BIG_ENDIAN
> CTXDESC_CD_0_ENDI |
> @@ -1310,18 +1367,8 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master
> *master, int ssid,
> if (cd_table->stall_enabled)
> val |= CTXDESC_CD_0_S;
> }
> -
> - /*
> - * The SMMU accesses 64-bit values atomically. See IHI0070Ca 3.21.3
> - * "Configuration structures and configuration invalidation completion"
> - *
> - * The size of single-copy atomic reads made by the SMMU is
> - * IMPLEMENTATION DEFINED but must be at least 64 bits. Any single
> - * field within an aligned 64-bit span of a structure can be altered
> - * without first making the structure invalid.
> - */
> - WRITE_ONCE(cdptr->data[0], cpu_to_le64(val));
> - arm_smmu_sync_cd(master, ssid, true);
> + cdptr->data[0] = cpu_to_le64(val);
> + arm_smmu_write_cd_entry(master, ssid, cd_table_entry, &target);
> return 0;
> }
> --
> 2.43.2
Reviewed-by: Moritz Fischer <moritzf@google.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
@ 2024-03-16 18:14 ` Moritz Fischer
0 siblings, 0 replies; 232+ messages in thread
From: Moritz Fischer @ 2024-03-16 18:14 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:53PM -0400, Jason Gunthorpe wrote:
> CD table entries and STE's have the same essential programming sequence,
> just with different types and sizes.
> Have arm_smmu_write_ctx_desc() generate a target CD and call
> arm_smmu_write_entry() to do the programming. Due to the way the
> target CD is generated by modifying the existing CD this alone is not
> enough for the CD callers to be freed of the ordering requirements.
> The following patches will make the rest of the CD flow mirror the STE
> flow with precise CD contents generated in all cases.
> Currently the logic can't ensure that the CD always conforms to the used
> requirements until all the CD generation is moved to the new method. Add a
> temporary no_used_check to disable the assertions.
> Signed-off-by: Michael Shavit <mshavit@google.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 101 ++++++++++++++------
> 1 file changed, 74 insertions(+), 27 deletions(-)
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index b7f947e36f596f..237fd6d92c880b 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -57,11 +57,14 @@ struct arm_smmu_entry_writer {
> struct arm_smmu_entry_writer_ops {
> unsigned int num_entry_qwords;
> __le64 v_bit;
> + bool no_used_check;
> void (*get_used)(const __le64 *entry, __le64 *used);
> void (*sync)(struct arm_smmu_entry_writer *writer);
> };
> -#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
> +#define NUM_ENTRY_QWORDS \
> + (max(sizeof(struct arm_smmu_ste), sizeof(struct arm_smmu_cd)) / \
> + sizeof(u64))
> static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
> [EVTQ_MSI_INDEX] = {
> @@ -1056,7 +1059,8 @@ static u8 arm_smmu_entry_qword_diff(struct
> arm_smmu_entry_writer *writer,
> * allowed to set a bit to 1 if the used function doesn't say it
> * is used.
> */
> - WARN_ON_ONCE(target[i] & ~target_used[i]);
> + if (!writer->ops->no_used_check)
> + WARN_ON_ONCE(target[i] & ~target_used[i]);
> /* Bits can change because they are not currently being used */
> unused_update[i] = (entry[i] & cur_used[i]) |
> @@ -1065,7 +1069,8 @@ static u8 arm_smmu_entry_qword_diff(struct
> arm_smmu_entry_writer *writer,
> * Each bit indicates that a used bit in a qword needs to be
> * changed after unused_update is applied.
> */
> - if ((unused_update[i] & target_used[i]) != target[i])
> + if ((unused_update[i] & target_used[i]) !=
> + (target[i] & target_used[i]))
> used_qword_diff |= 1 << i;
> }
> return used_qword_diff;
> @@ -1161,8 +1166,11 @@ static void arm_smmu_write_entry(struct
> arm_smmu_entry_writer *writer,
> * in the entry. The target was already sanity checked by
> * compute_qword_diff().
> */
> - WARN_ON_ONCE(
> - entry_set(writer, entry, target, 0, num_entry_qwords));
> + if (writer->ops->no_used_check)
> + entry_set(writer, entry, target, 0, num_entry_qwords);
> + else
> + WARN_ON_ONCE(entry_set(writer, entry, target, 0,
> + num_entry_qwords));
> }
> }
> @@ -1242,6 +1250,59 @@ static struct arm_smmu_cd
> *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> return &l1_desc->l2ptr[idx];
> }
> +struct arm_smmu_cd_writer {
> + struct arm_smmu_entry_writer writer;
> + unsigned int ssid;
> +};
> +
> +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
> +{
> + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
> + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
> + return;
> + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
> +
> + /* EPD0 means T0SZ/TG0/IR0/OR0/SH0/TTB0 are IGNORED */
> + if (ent[0] & cpu_to_le64(CTXDESC_CD_0_TCR_EPD0)) {
> + used_bits[0] &= ~cpu_to_le64(
> + CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 |
> + CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 |
> + CTXDESC_CD_0_TCR_SH0);
> + used_bits[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK);
> + }
> +}
> +
> +static void arm_smmu_cd_writer_sync_entry(struct arm_smmu_entry_writer
> *writer)
> +{
> + struct arm_smmu_cd_writer *cd_writer =
> + container_of(writer, struct arm_smmu_cd_writer, writer);
> +
> + arm_smmu_sync_cd(writer->master, cd_writer->ssid, true);
> +}
> +
> +static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
> + .sync = arm_smmu_cd_writer_sync_entry,
> + .get_used = arm_smmu_get_cd_used,
> + .v_bit = cpu_to_le64(CTXDESC_CD_0_V),
> + .no_used_check = true,
> + .num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
> +};
> +
> +static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int
> ssid,
> + struct arm_smmu_cd *cdptr,
> + const struct arm_smmu_cd *target)
> +{
> + struct arm_smmu_cd_writer cd_writer = {
> + .writer = {
> + .ops = &arm_smmu_cd_writer_ops,
> + .master = master,
> + },
> + .ssid = ssid,
> + };
> +
> + arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
> +}
> +
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_ctx_desc *cd)
> {
> @@ -1258,17 +1319,20 @@ int arm_smmu_write_ctx_desc(struct
> arm_smmu_master *master, int ssid,
> */
> u64 val;
> bool cd_live;
> - struct arm_smmu_cd *cdptr;
> + struct arm_smmu_cd target;
> + struct arm_smmu_cd *cdptr = ⌖
> + struct arm_smmu_cd *cd_table_entry;
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
> struct arm_smmu_device *smmu = master->smmu;
> if (WARN_ON(ssid >= (1 << cd_table->s1cdmax)))
> return -E2BIG;
> - cdptr = arm_smmu_get_cd_ptr(master, ssid);
> - if (!cdptr)
> + cd_table_entry = arm_smmu_get_cd_ptr(master, ssid);
> + if (!cd_table_entry)
> return -ENOMEM;
> + target = *cd_table_entry;
> val = le64_to_cpu(cdptr->data[0]);
> cd_live = !!(val & CTXDESC_CD_0_V);
> @@ -1290,13 +1354,6 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master
> *master, int ssid,
> cdptr->data[2] = 0;
> cdptr->data[3] = cpu_to_le64(cd->mair);
> - /*
> - * STE may be live, and the SMMU might read dwords of this CD in any
> - * order. Ensure that it observes valid values before reading
> - * V=1.
> - */
> - arm_smmu_sync_cd(master, ssid, true);
> -
> val = cd->tcr |
> #ifdef __BIG_ENDIAN
> CTXDESC_CD_0_ENDI |
> @@ -1310,18 +1367,8 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master
> *master, int ssid,
> if (cd_table->stall_enabled)
> val |= CTXDESC_CD_0_S;
> }
> -
> - /*
> - * The SMMU accesses 64-bit values atomically. See IHI0070Ca 3.21.3
> - * "Configuration structures and configuration invalidation completion"
> - *
> - * The size of single-copy atomic reads made by the SMMU is
> - * IMPLEMENTATION DEFINED but must be at least 64 bits. Any single
> - * field within an aligned 64-bit span of a structure can be altered
> - * without first making the structure invalid.
> - */
> - WRITE_ONCE(cdptr->data[0], cpu_to_le64(val));
> - arm_smmu_sync_cd(master, ssid, true);
> + cdptr->data[0] = cpu_to_le64(val);
> + arm_smmu_write_cd_entry(master, ssid, cd_table_entry, &target);
> return 0;
> }
> --
> 2.43.2
Reviewed-by: Moritz Fischer <moritzf@google.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 06/27] iommu/arm-smmu-v3: Consolidate clearing a CD table entry
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-16 18:15 ` Moritz Fischer
-1 siblings, 0 replies; 232+ messages in thread
From: Moritz Fischer @ 2024-03-16 18:15 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:54PM -0400, Jason Gunthorpe wrote:
> A cleared entry is all 0's. Make arm_smmu_clear_cd() do this sequence.
> If we are clearing an entry and for some reason it is not already
> allocated in the CD table then something has gone wrong.
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 ++++++++++++++-----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 ++
> 3 files changed, 18 insertions(+), 6 deletions(-)
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 347c2fdd865c1a..bb9bb6fd7914ce 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -558,7 +558,7 @@ void arm_smmu_sva_remove_dev_pasid(struct
> iommu_domain *domain,
> mutex_lock(&sva_lock);
> - arm_smmu_write_ctx_desc(master, id, NULL);
> + arm_smmu_clear_cd(master, id);
> list_for_each_entry(t, &master->bonds, list) {
> if (t->mm == mm) {
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 237fd6d92c880b..3fb4a1523d1d3f 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1303,6 +1303,19 @@ static void arm_smmu_write_cd_entry(struct
> arm_smmu_master *master, int ssid,
> arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
> }
> +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> +{
> + struct arm_smmu_cd target = {};
> + struct arm_smmu_cd *cdptr;
> +
> + if (!master->cd_table.cdtab)
> + return;
> + cdptr = arm_smmu_get_cd_ptr(master, ssid);
> + if (WARN_ON(!cdptr))
> + return;
> + arm_smmu_write_cd_entry(master, ssid, cdptr, &target);
> +}
> +
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_ctx_desc *cd)
> {
> @@ -2702,9 +2715,7 @@ static int arm_smmu_attach_dev(struct iommu_domain
> *domain, struct device *dev)
> case ARM_SMMU_DOMAIN_S2:
> arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
> arm_smmu_install_ste_for_dev(master, &target);
> - if (master->cd_table.cdtab)
> - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID,
> - NULL);
> + arm_smmu_clear_cd(master, IOMMU_NO_PASID);
> break;
> }
> @@ -2752,8 +2763,7 @@ static int arm_smmu_attach_dev_ste(struct device
> *dev,
> * arm_smmu_domain->devices to avoid races updating the same context
> * descriptor from arm_smmu_share_asid().
> */
> - if (master->cd_table.cdtab)
> - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, NULL);
> + arm_smmu_clear_cd(master, IOMMU_NO_PASID);
> return 0;
> }
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 7078ed569fd4d3..87a7b57f566fbc 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -749,6 +749,8 @@ extern struct xarray arm_smmu_asid_xa;
> extern struct mutex arm_smmu_asid_lock;
> extern struct arm_smmu_ctx_desc quiet_cd;
> +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
> +
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int
> ssid,
> struct arm_smmu_ctx_desc *cd);
> void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
> --
> 2.43.2
Reviewed-by: Moritz Fischer <moritzf@google.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 06/27] iommu/arm-smmu-v3: Consolidate clearing a CD table entry
@ 2024-03-16 18:15 ` Moritz Fischer
0 siblings, 0 replies; 232+ messages in thread
From: Moritz Fischer @ 2024-03-16 18:15 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:54PM -0400, Jason Gunthorpe wrote:
> A cleared entry is all 0's. Make arm_smmu_clear_cd() do this sequence.
> If we are clearing an entry and for some reason it is not already
> allocated in the CD table then something has gone wrong.
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 ++++++++++++++-----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 ++
> 3 files changed, 18 insertions(+), 6 deletions(-)
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 347c2fdd865c1a..bb9bb6fd7914ce 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -558,7 +558,7 @@ void arm_smmu_sva_remove_dev_pasid(struct
> iommu_domain *domain,
> mutex_lock(&sva_lock);
> - arm_smmu_write_ctx_desc(master, id, NULL);
> + arm_smmu_clear_cd(master, id);
> list_for_each_entry(t, &master->bonds, list) {
> if (t->mm == mm) {
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 237fd6d92c880b..3fb4a1523d1d3f 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1303,6 +1303,19 @@ static void arm_smmu_write_cd_entry(struct
> arm_smmu_master *master, int ssid,
> arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
> }
> +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> +{
> + struct arm_smmu_cd target = {};
> + struct arm_smmu_cd *cdptr;
> +
> + if (!master->cd_table.cdtab)
> + return;
> + cdptr = arm_smmu_get_cd_ptr(master, ssid);
> + if (WARN_ON(!cdptr))
> + return;
> + arm_smmu_write_cd_entry(master, ssid, cdptr, &target);
> +}
> +
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_ctx_desc *cd)
> {
> @@ -2702,9 +2715,7 @@ static int arm_smmu_attach_dev(struct iommu_domain
> *domain, struct device *dev)
> case ARM_SMMU_DOMAIN_S2:
> arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
> arm_smmu_install_ste_for_dev(master, &target);
> - if (master->cd_table.cdtab)
> - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID,
> - NULL);
> + arm_smmu_clear_cd(master, IOMMU_NO_PASID);
> break;
> }
> @@ -2752,8 +2763,7 @@ static int arm_smmu_attach_dev_ste(struct device
> *dev,
> * arm_smmu_domain->devices to avoid races updating the same context
> * descriptor from arm_smmu_share_asid().
> */
> - if (master->cd_table.cdtab)
> - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, NULL);
> + arm_smmu_clear_cd(master, IOMMU_NO_PASID);
> return 0;
> }
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 7078ed569fd4d3..87a7b57f566fbc 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -749,6 +749,8 @@ extern struct xarray arm_smmu_asid_xa;
> extern struct mutex arm_smmu_asid_lock;
> extern struct arm_smmu_ctx_desc quiet_cd;
> +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
> +
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int
> ssid,
> struct arm_smmu_ctx_desc *cd);
> void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
> --
> 2.43.2
Reviewed-by: Moritz Fischer <moritzf@google.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 03/27] iommu/arm-smmu-v3: Add a type for the CD entry
2024-03-13 9:44 ` Michael Shavit
@ 2024-03-18 18:02 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-18 18:02 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Wed, Mar 13, 2024 at 05:44:35PM +0800, Michael Shavit wrote:
> On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > Instead of passing a naked __le16 * around to represent a CD table entry
> > wrap it in a "struct arm_smmu_cd" with an array of the correct size. This
> > makes it much clearer which functions will comprise the "CD API".
> >
> > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 +++++++++++---------
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 ++++++-
> > 2 files changed, 17 insertions(+), 10 deletions(-)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index 9e9233331c4636..c60b067c1f553e 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -1219,7 +1219,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
> > WRITE_ONCE(*dst, cpu_to_le64(val));
> > }
> >
> > -static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
> > +static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> > + u32 ssid)
> > {
> > __le64 *l1ptr;
> > unsigned int idx;
> > @@ -1228,7 +1229,8 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
> > struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
> >
> > if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
> > - return cd_table->cdtab + ssid * CTXDESC_CD_DWORDS;
> > + return (struct arm_smmu_cd *)(cd_table->cdtab +
> > + ssid * CTXDESC_CD_DWORDS);
>
> Can we define cd_table.cdtab as a union type to avoid this cast and
> make the struct definition more explicit?
Yes, those patches are in part 3 to fix the CD and STE to be like you
suggest. In the end the above looks like:
if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
return cd_table->cdtab.linear + ssid;
Thanks,
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 03/27] iommu/arm-smmu-v3: Add a type for the CD entry
@ 2024-03-18 18:02 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-18 18:02 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Wed, Mar 13, 2024 at 05:44:35PM +0800, Michael Shavit wrote:
> On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > Instead of passing a naked __le16 * around to represent a CD table entry
> > wrap it in a "struct arm_smmu_cd" with an array of the correct size. This
> > makes it much clearer which functions will comprise the "CD API".
> >
> > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 +++++++++++---------
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 ++++++-
> > 2 files changed, 17 insertions(+), 10 deletions(-)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index 9e9233331c4636..c60b067c1f553e 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -1219,7 +1219,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
> > WRITE_ONCE(*dst, cpu_to_le64(val));
> > }
> >
> > -static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
> > +static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> > + u32 ssid)
> > {
> > __le64 *l1ptr;
> > unsigned int idx;
> > @@ -1228,7 +1229,8 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
> > struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
> >
> > if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
> > - return cd_table->cdtab + ssid * CTXDESC_CD_DWORDS;
> > + return (struct arm_smmu_cd *)(cd_table->cdtab +
> > + ssid * CTXDESC_CD_DWORDS);
>
> Can we define cd_table.cdtab as a union type to avoid this cast and
> make the struct definition more explicit?
Yes, those patches are in part 3 to fix the CD and STE to be like you
suggest. In the end the above looks like:
if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
return cd_table->cdtab.linear + ssid;
Thanks,
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code
2024-03-15 4:22 ` Nicolin Chen
@ 2024-03-18 18:06 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-18 18:06 UTC (permalink / raw)
To: Nicolin Chen
Cc: Michael Shavit, iommu, Joerg Roedel, linux-arm-kernel,
Robin Murphy, Will Deacon, Eric Auger, Jean-Philippe Brucker,
Moritz Fischer, patches, Shameerali Kolothum Thodi
On Thu, Mar 14, 2024 at 09:22:48PM -0700, Nicolin Chen wrote:
> Hi Michael/Jason,
>
> On Mon, Mar 04, 2024 at 07:43:52PM -0400, Jason Gunthorpe wrote:
> > Prepare to put the CD code into the same mechanism. Add an ops indirection
> > around all the STE specific code and make the worker functions independent
> > of the entry content being processed.
> >
> > get_used and sync ops are provided to hook the correct code.
> >
> > Signed-off-by: Michael Shavit <mshavit@google.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 172 ++++++++++++--------
> > 1 file changed, 104 insertions(+), 68 deletions(-)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index c60b067c1f553e..b7f947e36f596f 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -48,8 +48,20 @@ enum arm_smmu_msi_index {
> > ARM_SMMU_MAX_MSIS,
> > };
> >
> > -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu,
> > - ioasid_t sid);
> > +struct arm_smmu_entry_writer_ops;
> > +struct arm_smmu_entry_writer {
> > + const struct arm_smmu_entry_writer_ops *ops;
> > + struct arm_smmu_master *master;
> > +};
> > +
> > +struct arm_smmu_entry_writer_ops {
> > + unsigned int num_entry_qwords;
>
> I vaguely remember some related discussion, yet can't find it
> out. So sorry for questioning this, if it's already discussed.
> Aren't CD and STE having the same num_entry_qwords in terms of
> their values? Feels like we can just use NUM_ENTRY_QWORDS?
They do, but the structs should reflect the HW and there is nothing in
the HW description that requires these to be the same.
It is true we go through some contortions to keep the sizes seperate,
and it is also true they are unlikely to ever change size.
> > +#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
> And this seems to be just a fixed "8"? Since both are defined
> straightforwardly:
Yes, again it should flow from the struct which should reflect the HW
layout. Making them the same size is a SW desire..
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code
@ 2024-03-18 18:06 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-18 18:06 UTC (permalink / raw)
To: Nicolin Chen
Cc: Michael Shavit, iommu, Joerg Roedel, linux-arm-kernel,
Robin Murphy, Will Deacon, Eric Auger, Jean-Philippe Brucker,
Moritz Fischer, patches, Shameerali Kolothum Thodi
On Thu, Mar 14, 2024 at 09:22:48PM -0700, Nicolin Chen wrote:
> Hi Michael/Jason,
>
> On Mon, Mar 04, 2024 at 07:43:52PM -0400, Jason Gunthorpe wrote:
> > Prepare to put the CD code into the same mechanism. Add an ops indirection
> > around all the STE specific code and make the worker functions independent
> > of the entry content being processed.
> >
> > get_used and sync ops are provided to hook the correct code.
> >
> > Signed-off-by: Michael Shavit <mshavit@google.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 172 ++++++++++++--------
> > 1 file changed, 104 insertions(+), 68 deletions(-)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index c60b067c1f553e..b7f947e36f596f 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -48,8 +48,20 @@ enum arm_smmu_msi_index {
> > ARM_SMMU_MAX_MSIS,
> > };
> >
> > -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu,
> > - ioasid_t sid);
> > +struct arm_smmu_entry_writer_ops;
> > +struct arm_smmu_entry_writer {
> > + const struct arm_smmu_entry_writer_ops *ops;
> > + struct arm_smmu_master *master;
> > +};
> > +
> > +struct arm_smmu_entry_writer_ops {
> > + unsigned int num_entry_qwords;
>
> I vaguely remember some related discussion, yet can't find it
> out. So sorry for questioning this, if it's already discussed.
> Aren't CD and STE having the same num_entry_qwords in terms of
> their values? Feels like we can just use NUM_ENTRY_QWORDS?
They do, but the structs should reflect the HW and there is nothing in
the HW description that requires these to be the same.
It is true we go through some contortions to keep the sizes seperate,
and it is also true they are unlikely to ever change size.
> > +#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
> And this seems to be just a fixed "8"? Since both are defined
> straightforwardly:
Yes, again it should flow from the struct which should reflect the HW
layout. Making them the same size is a SW desire..
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 07/27] iommu/arm-smmu-v3: Move the CD generation for S1 domains into a function
2024-03-13 12:13 ` Michael Shavit
@ 2024-03-18 18:11 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-18 18:11 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Wed, Mar 13, 2024 at 08:13:12PM +0800, Michael Shavit wrote:
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > index bb9bb6fd7914ce..6acc65f6d00a71 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > @@ -54,6 +54,29 @@ static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain
> > spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> > }
> >
> > +static void
> > +arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> > +{
> > + struct arm_smmu_master *master;
> > + struct arm_smmu_cd target_cd;
> > + unsigned long flags;
> > +
> > + spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> > + list_for_each_entry(master, &smmu_domain->devices, domain_head) {
> > + struct arm_smmu_cd *cdptr;
> > +
> > + /* S1 domains only support RID attachment right now */
> > + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> > + if (WARN_ON(!cdptr))
> > + continue;
> > +
> > + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> > + arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> > + &target_cd);
> > + }
> > + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> > +}
> > +
> > /*
> > * Check if the CPU ASID is available on the SMMU side. If a private context
> > * descriptor is using it, try to replace it.
> > @@ -97,7 +120,7 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid)
> > * be some overlap between use of both ASIDs, until we invalidate the
> > * TLB.
> > */
> > - arm_smmu_update_ctx_desc_devices(smmu_domain, IOMMU_NO_PASID, cd);
> > + arm_smmu_update_s1_domain_cd_entry(smmu_domain);
>
> Not to be too nitpicky, but is calling this
> arm_smmu_update_s1_domain_cd_entry the right choice here? Yes the RID
> domain has type "ARM_SMMU_DOMAIN_S1", but CDs are also stage 1
> translations.
A later patch will inline this function into
arm_smmu_realloc_s1_domain_asid(), so I wouldn't fuss too much over
the name..
But the name is trying to convey the two important details:
1) The domain is an ARM_SMMU_DOMAIN_S1
2) We are changing the CD Table Entry *not* the STE
One of the things that was really confusing about this code was
exactly what the smm_domain *was*, it is actually always a
ARM_SMMU_DOMAIN_S1.
Thanks,
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 07/27] iommu/arm-smmu-v3: Move the CD generation for S1 domains into a function
@ 2024-03-18 18:11 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-18 18:11 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Wed, Mar 13, 2024 at 08:13:12PM +0800, Michael Shavit wrote:
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > index bb9bb6fd7914ce..6acc65f6d00a71 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > @@ -54,6 +54,29 @@ static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain
> > spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> > }
> >
> > +static void
> > +arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> > +{
> > + struct arm_smmu_master *master;
> > + struct arm_smmu_cd target_cd;
> > + unsigned long flags;
> > +
> > + spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> > + list_for_each_entry(master, &smmu_domain->devices, domain_head) {
> > + struct arm_smmu_cd *cdptr;
> > +
> > + /* S1 domains only support RID attachment right now */
> > + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> > + if (WARN_ON(!cdptr))
> > + continue;
> > +
> > + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> > + arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> > + &target_cd);
> > + }
> > + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> > +}
> > +
> > /*
> > * Check if the CPU ASID is available on the SMMU side. If a private context
> > * descriptor is using it, try to replace it.
> > @@ -97,7 +120,7 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid)
> > * be some overlap between use of both ASIDs, until we invalidate the
> > * TLB.
> > */
> > - arm_smmu_update_ctx_desc_devices(smmu_domain, IOMMU_NO_PASID, cd);
> > + arm_smmu_update_s1_domain_cd_entry(smmu_domain);
>
> Not to be too nitpicky, but is calling this
> arm_smmu_update_s1_domain_cd_entry the right choice here? Yes the RID
> domain has type "ARM_SMMU_DOMAIN_S1", but CDs are also stage 1
> translations.
A later patch will inline this function into
arm_smmu_realloc_s1_domain_asid(), so I wouldn't fuss too much over
the name..
But the name is trying to convey the two important details:
1) The domain is an ARM_SMMU_DOMAIN_S1
2) We are changing the CD Table Entry *not* the STE
One of the things that was really confusing about this code was
exactly what the smm_domain *was*, it is actually always a
ARM_SMMU_DOMAIN_S1.
Thanks,
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 09/27] iommu/arm-smmu-v3: Allocate the CD table entry in advance
2024-03-16 4:16 ` Nicolin Chen
@ 2024-03-18 18:14 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-18 18:14 UTC (permalink / raw)
To: Nicolin Chen
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Fri, Mar 15, 2024 at 09:16:14PM -0700, Nicolin Chen wrote:
> On Mon, Mar 04, 2024 at 07:43:57PM -0400, Jason Gunthorpe wrote:
> > Avoid arm_smmu_attach_dev() having to undo the changes to the
> > smmu_domain->devices list, acquire the cdptr earlier so we don't need to
> > handle that error.
>
> I should probably mention this in the other patch, yet PATCH-14
> adding arm_smmu_attach_prepare() to this function doesn't have a
> rollback for CD table allocation. I assume we're fine with that?
Yeah, I think so. The CD table leafs are never freed by anything, so
even if you did succeed to attach a PASID and then detach the leaf
will still hang around.
If we care we could try to clean it up by consulting the PASID xarray
and checking if the entire leaf is unused.
Thanks,
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 09/27] iommu/arm-smmu-v3: Allocate the CD table entry in advance
@ 2024-03-18 18:14 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-18 18:14 UTC (permalink / raw)
To: Nicolin Chen
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Fri, Mar 15, 2024 at 09:16:14PM -0700, Nicolin Chen wrote:
> On Mon, Mar 04, 2024 at 07:43:57PM -0400, Jason Gunthorpe wrote:
> > Avoid arm_smmu_attach_dev() having to undo the changes to the
> > smmu_domain->devices list, acquire the cdptr earlier so we don't need to
> > handle that error.
>
> I should probably mention this in the other patch, yet PATCH-14
> adding arm_smmu_attach_prepare() to this function doesn't have a
> rollback for CD table allocation. I assume we're fine with that?
Yeah, I think so. The CD table leafs are never freed by anything, so
even if you did succeed to attach a PASID and then detach the leaf
will still hang around.
If we care we could try to clean it up by consulting the PASID xarray
and checking if the entire leaf is unused.
Thanks,
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 01/27] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID
2024-03-15 3:38 ` Nicolin Chen
@ 2024-03-18 18:16 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-18 18:16 UTC (permalink / raw)
To: Nicolin Chen
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Thu, Mar 14, 2024 at 08:38:30PM -0700, Nicolin Chen wrote:
> Hi Jason,
>
> On Mon, Mar 04, 2024 at 07:43:49PM -0400, Jason Gunthorpe wrote:
> > The SVA code is wired to assume that the SVA is programmed onto the
> > mm->pasid. The current core code always does this, so it is fine.
> >
> > Add a check for clarity.
> >
> > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++
> > 1 file changed, 3 insertions(+)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > index 2610e82c0ecd0d..347c2fdd865c1a 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > @@ -581,6 +581,9 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> > int ret = 0;
> > struct mm_struct *mm = domain->mm;
> >
> > + if (mm_get_enqcmd_pasid(mm) != id)
> > + return -EINVAL;
> > +
>
> This seems to get deleted entirely by a followup change. Is it
> added here to ensure a 1:1 mapping for git-bisect?
Right, it is here in the series primarily for clarity, but arguably it
should be backported too. I'll add a fixes line
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 01/27] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID
@ 2024-03-18 18:16 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-18 18:16 UTC (permalink / raw)
To: Nicolin Chen
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Thu, Mar 14, 2024 at 08:38:30PM -0700, Nicolin Chen wrote:
> Hi Jason,
>
> On Mon, Mar 04, 2024 at 07:43:49PM -0400, Jason Gunthorpe wrote:
> > The SVA code is wired to assume that the SVA is programmed onto the
> > mm->pasid. The current core code always does this, so it is fine.
> >
> > Add a check for clarity.
> >
> > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++
> > 1 file changed, 3 insertions(+)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > index 2610e82c0ecd0d..347c2fdd865c1a 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > @@ -581,6 +581,9 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> > int ret = 0;
> > struct mm_struct *mm = domain->mm;
> >
> > + if (mm_get_enqcmd_pasid(mm) != id)
> > + return -EINVAL;
> > +
>
> This seems to get deleted entirely by a followup change. Is it
> added here to ensure a 1:1 mapping for git-bisect?
Right, it is here in the series primarily for clarity, but arguably it
should be backported too. I'll add a fixes line
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 13/27] iommu/arm-smmu-v3: Make smmu_domain->devices into an allocated list
2024-03-04 23:44 ` Jason Gunthorpe
@ 2024-03-19 13:09 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 13:09 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> The next patch will need to store the same master twice (with different
> SSIDs), so allocate memory for each list element.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 11 ++++--
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 39 ++++++++++++++++---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 +++-
> 3 files changed, 47 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index e1fa2074c0b37b..9a285b828e8b12 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -38,12 +38,13 @@ static DEFINE_MUTEX(sva_lock);
> static void
> arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> {
> - struct arm_smmu_master *master;
> + struct arm_smmu_master_domain *master_domain;
> struct arm_smmu_cd target_cd;
> unsigned long flags;
>
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - list_for_each_entry(master, &smmu_domain->devices, domain_head) {
> + list_for_each_entry(master_domain, &smmu_domain->devices, devices_elm) {
> + struct arm_smmu_master *master = master_domain->master;
> struct arm_smmu_cd *cdptr;
>
> /* S1 domains only support RID attachment right now */
> @@ -300,7 +301,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
> {
> struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn);
> struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
> - struct arm_smmu_master *master;
> + struct arm_smmu_master_domain *master_domain;
> unsigned long flags;
>
> mutex_lock(&sva_lock);
> @@ -314,7 +315,9 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
> * but disable translation.
> */
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - list_for_each_entry(master, &smmu_domain->devices, domain_head) {
> + list_for_each_entry(master_domain, &smmu_domain->devices,
> + devices_elm) {
> + struct arm_smmu_master *master = master_domain->master;
> struct arm_smmu_cd target;
> struct arm_smmu_cd *cdptr;
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 7a0be7dd719793..51a1e7198fd1af 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2031,10 +2031,10 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master)
> int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
> unsigned long iova, size_t size)
> {
> + struct arm_smmu_master_domain *master_domain;
> int i;
> unsigned long flags;
> struct arm_smmu_cmdq_ent cmd;
> - struct arm_smmu_master *master;
> struct arm_smmu_cmdq_batch cmds;
>
> if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_ATS))
> @@ -2062,7 +2062,10 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
> cmds.num = 0;
>
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - list_for_each_entry(master, &smmu_domain->devices, domain_head) {
> + list_for_each_entry(master_domain, &smmu_domain->devices,
> + devices_elm) {
> + struct arm_smmu_master *master = master_domain->master;
> +
> if (!master->ats_enabled)
> continue;
>
> @@ -2554,9 +2557,26 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master)
> pci_disable_pasid(pdev);
> }
>
> +static struct arm_smmu_master_domain *
> +arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
> + struct arm_smmu_master *master)
> +{
> + struct arm_smmu_master_domain *master_domain;
> +
> + lockdep_assert_held(&smmu_domain->devices_lock);
> +
> + list_for_each_entry(master_domain, &smmu_domain->devices,
> + devices_elm) {
> + if (master_domain->master == master)
> + return master_domain;
> + }
> + return NULL;
> +}
> +
> static void arm_smmu_detach_dev(struct arm_smmu_master *master)
> {
> struct iommu_domain *domain = iommu_get_domain_for_dev(master->dev);
> + struct arm_smmu_master_domain *master_domain;
> struct arm_smmu_domain *smmu_domain;
> unsigned long flags;
>
> @@ -2567,7 +2587,11 @@ static void arm_smmu_detach_dev(struct arm_smmu_master *master)
> arm_smmu_disable_ats(master, smmu_domain);
>
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - list_del_init(&master->domain_head);
> + master_domain = arm_smmu_find_master_domain(smmu_domain, master);
> + if (master_domain) {
> + list_del(&master_domain->devices_elm);
> + kfree(master_domain);
> + }
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
>
> master->ats_enabled = false;
> @@ -2581,6 +2605,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
> struct arm_smmu_device *smmu;
> struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> + struct arm_smmu_master_domain *master_domain;
> struct arm_smmu_master *master;
> struct arm_smmu_cd *cdptr;
>
> @@ -2617,6 +2642,11 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> return -ENOMEM;
> }
>
> + master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL);
> + if (!master_domain)
> + return -ENOMEM;
> + master_domain->master = master;
> +
> /*
> * Prevent arm_smmu_share_asid() from trying to change the ASID
> * of either the old or new domain while we are working on it.
> @@ -2630,7 +2660,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> master->ats_enabled = arm_smmu_ats_supported(master);
>
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - list_add(&master->domain_head, &smmu_domain->devices);
> + list_add(&master_domain->devices_elm, &smmu_domain->devices);
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
>
> switch (smmu_domain->stage) {
> @@ -2949,7 +2979,6 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev)
> master->dev = dev;
> master->smmu = smmu;
> INIT_LIST_HEAD(&master->bonds);
> - INIT_LIST_HEAD(&master->domain_head);
> dev_iommu_priv_set(dev, master);
>
> ret = arm_smmu_insert_master(smmu, master);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index ab0a6587e07ec8..c4b79bc52991be 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -695,7 +695,6 @@ struct arm_smmu_stream {
> struct arm_smmu_master {
> struct arm_smmu_device *smmu;
> struct device *dev;
> - struct list_head domain_head;
> struct arm_smmu_stream *streams;
> /* Locked by the iommu core using the group mutex */
> struct arm_smmu_ctx_desc_cfg cd_table;
> @@ -729,12 +728,18 @@ struct arm_smmu_domain {
>
> struct iommu_domain domain;
>
> + /* List of struct arm_smmu_master_domain */
> struct list_head devices;
> spinlock_t devices_lock;
>
> struct list_head mmu_notifiers;
> };
>
> +struct arm_smmu_master_domain {
> + struct list_head devices_elm;
> + struct arm_smmu_master *master;
> +};
> +
> static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
> {
> return container_of(dom, struct arm_smmu_domain, domain);
> --
> 2.43.2
>
Reviewed-by: Michael Shavit <mshavit@google.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 13/27] iommu/arm-smmu-v3: Make smmu_domain->devices into an allocated list
@ 2024-03-19 13:09 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 13:09 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> The next patch will need to store the same master twice (with different
> SSIDs), so allocate memory for each list element.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 11 ++++--
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 39 ++++++++++++++++---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 +++-
> 3 files changed, 47 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index e1fa2074c0b37b..9a285b828e8b12 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -38,12 +38,13 @@ static DEFINE_MUTEX(sva_lock);
> static void
> arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> {
> - struct arm_smmu_master *master;
> + struct arm_smmu_master_domain *master_domain;
> struct arm_smmu_cd target_cd;
> unsigned long flags;
>
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - list_for_each_entry(master, &smmu_domain->devices, domain_head) {
> + list_for_each_entry(master_domain, &smmu_domain->devices, devices_elm) {
> + struct arm_smmu_master *master = master_domain->master;
> struct arm_smmu_cd *cdptr;
>
> /* S1 domains only support RID attachment right now */
> @@ -300,7 +301,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
> {
> struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn);
> struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
> - struct arm_smmu_master *master;
> + struct arm_smmu_master_domain *master_domain;
> unsigned long flags;
>
> mutex_lock(&sva_lock);
> @@ -314,7 +315,9 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
> * but disable translation.
> */
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - list_for_each_entry(master, &smmu_domain->devices, domain_head) {
> + list_for_each_entry(master_domain, &smmu_domain->devices,
> + devices_elm) {
> + struct arm_smmu_master *master = master_domain->master;
> struct arm_smmu_cd target;
> struct arm_smmu_cd *cdptr;
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 7a0be7dd719793..51a1e7198fd1af 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2031,10 +2031,10 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master)
> int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
> unsigned long iova, size_t size)
> {
> + struct arm_smmu_master_domain *master_domain;
> int i;
> unsigned long flags;
> struct arm_smmu_cmdq_ent cmd;
> - struct arm_smmu_master *master;
> struct arm_smmu_cmdq_batch cmds;
>
> if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_ATS))
> @@ -2062,7 +2062,10 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
> cmds.num = 0;
>
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - list_for_each_entry(master, &smmu_domain->devices, domain_head) {
> + list_for_each_entry(master_domain, &smmu_domain->devices,
> + devices_elm) {
> + struct arm_smmu_master *master = master_domain->master;
> +
> if (!master->ats_enabled)
> continue;
>
> @@ -2554,9 +2557,26 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master)
> pci_disable_pasid(pdev);
> }
>
> +static struct arm_smmu_master_domain *
> +arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
> + struct arm_smmu_master *master)
> +{
> + struct arm_smmu_master_domain *master_domain;
> +
> + lockdep_assert_held(&smmu_domain->devices_lock);
> +
> + list_for_each_entry(master_domain, &smmu_domain->devices,
> + devices_elm) {
> + if (master_domain->master == master)
> + return master_domain;
> + }
> + return NULL;
> +}
> +
> static void arm_smmu_detach_dev(struct arm_smmu_master *master)
> {
> struct iommu_domain *domain = iommu_get_domain_for_dev(master->dev);
> + struct arm_smmu_master_domain *master_domain;
> struct arm_smmu_domain *smmu_domain;
> unsigned long flags;
>
> @@ -2567,7 +2587,11 @@ static void arm_smmu_detach_dev(struct arm_smmu_master *master)
> arm_smmu_disable_ats(master, smmu_domain);
>
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - list_del_init(&master->domain_head);
> + master_domain = arm_smmu_find_master_domain(smmu_domain, master);
> + if (master_domain) {
> + list_del(&master_domain->devices_elm);
> + kfree(master_domain);
> + }
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
>
> master->ats_enabled = false;
> @@ -2581,6 +2605,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
> struct arm_smmu_device *smmu;
> struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> + struct arm_smmu_master_domain *master_domain;
> struct arm_smmu_master *master;
> struct arm_smmu_cd *cdptr;
>
> @@ -2617,6 +2642,11 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> return -ENOMEM;
> }
>
> + master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL);
> + if (!master_domain)
> + return -ENOMEM;
> + master_domain->master = master;
> +
> /*
> * Prevent arm_smmu_share_asid() from trying to change the ASID
> * of either the old or new domain while we are working on it.
> @@ -2630,7 +2660,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> master->ats_enabled = arm_smmu_ats_supported(master);
>
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - list_add(&master->domain_head, &smmu_domain->devices);
> + list_add(&master_domain->devices_elm, &smmu_domain->devices);
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
>
> switch (smmu_domain->stage) {
> @@ -2949,7 +2979,6 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev)
> master->dev = dev;
> master->smmu = smmu;
> INIT_LIST_HEAD(&master->bonds);
> - INIT_LIST_HEAD(&master->domain_head);
> dev_iommu_priv_set(dev, master);
>
> ret = arm_smmu_insert_master(smmu, master);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index ab0a6587e07ec8..c4b79bc52991be 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -695,7 +695,6 @@ struct arm_smmu_stream {
> struct arm_smmu_master {
> struct arm_smmu_device *smmu;
> struct device *dev;
> - struct list_head domain_head;
> struct arm_smmu_stream *streams;
> /* Locked by the iommu core using the group mutex */
> struct arm_smmu_ctx_desc_cfg cd_table;
> @@ -729,12 +728,18 @@ struct arm_smmu_domain {
>
> struct iommu_domain domain;
>
> + /* List of struct arm_smmu_master_domain */
> struct list_head devices;
> spinlock_t devices_lock;
>
> struct list_head mmu_notifiers;
> };
>
> +struct arm_smmu_master_domain {
> + struct list_head devices_elm;
> + struct arm_smmu_master *master;
> +};
> +
> static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
> {
> return container_of(dom, struct arm_smmu_domain, domain);
> --
> 2.43.2
>
Reviewed-by: Michael Shavit <mshavit@google.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 15/27] iommu/arm-smmu-v3: Add ssid to struct arm_smmu_master_domain
2024-03-04 23:44 ` Jason Gunthorpe
@ 2024-03-19 13:31 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 13:31 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Prepare to allow a S1 domain to be attached to a PASID as well. Keep track
> of the SSID the domain is using on each master in the
> arm_smmu_master_domain.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 15 ++++---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 42 +++++++++++++++----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 ++-
> 3 files changed, 43 insertions(+), 19 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 9a285b828e8b12..ab9de8e36c45f5 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -47,13 +47,12 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> struct arm_smmu_master *master = master_domain->master;
> struct arm_smmu_cd *cdptr;
>
> - /* S1 domains only support RID attachment right now */
> - cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> + cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid);
> if (WARN_ON(!cdptr))
> continue;
>
> arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> - arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> + arm_smmu_write_cd_entry(master, master_domain->ssid, cdptr,
> &target_cd);
> }
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> @@ -293,8 +292,8 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
> smmu_domain);
> }
>
> - arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), start,
> - size);
> + arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), start,
> + size);
> }
>
> static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
> @@ -331,7 +330,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
>
> arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid);
> - arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0);
> + arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0);
>
> smmu_mn->cleared = true;
> mutex_unlock(&sva_lock);
> @@ -410,8 +409,8 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn)
> */
> if (!smmu_mn->cleared) {
> arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid);
> - arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), 0,
> - 0);
> + arm_smmu_atc_inv_domain_sva(smmu_domain,
> + mm_get_enqcmd_pasid(mm), 0, 0);
> }
>
> /* Frees smmu_mn */
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 45f2190fc31786..26c6b9f6f34fd3 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2030,8 +2030,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master)
> return arm_smmu_cmdq_batch_submit(master->smmu, &cmds);
> }
>
> -int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
> - unsigned long iova, size_t size)
> +static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
> + ioasid_t ssid, unsigned long iova, size_t size)
> {
> struct arm_smmu_master_domain *master_domain;
> int i;
> @@ -2059,8 +2059,6 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
> if (!atomic_read(&smmu_domain->nr_ats_masters))
> return 0;
>
> - arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd);
> -
> cmds.num = 0;
>
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> @@ -2071,6 +2069,16 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
> if (!master->ats_enabled)
> continue;
>
> + /*
> + * Non-zero ssid means SVA is co-opting the S1 domain to issue
> + * invalidations for SVA PASIDs.
> + */
> + if (ssid != IOMMU_NO_PASID)
> + arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd);
> + else
> + arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size,
> + &cmd);
> +
> for (i = 0; i < master->num_streams; i++) {
> cmd.atc.sid = master->streams[i].id;
> arm_smmu_cmdq_batch_add(smmu_domain->smmu, &cmds, &cmd);
> @@ -2081,6 +2089,19 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
> return arm_smmu_cmdq_batch_submit(smmu_domain->smmu, &cmds);
> }
>
> +static int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
> + unsigned long iova, size_t size)
> +{
> + return __arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova,
> + size);
> +}
> +
> +int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain,
> + ioasid_t ssid, unsigned long iova, size_t size)
> +{
> + return __arm_smmu_atc_inv_domain(smmu_domain, ssid, iova, size);
> +}
> +
> /* IO_PGTABLE API */
> static void arm_smmu_tlb_inv_context(void *cookie)
> {
> @@ -2102,7 +2123,7 @@ static void arm_smmu_tlb_inv_context(void *cookie)
> cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid;
> arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
> }
> - arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, 0, 0);
> + arm_smmu_atc_inv_domain(smmu_domain, 0, 0);
> }
>
> static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd,
> @@ -2200,7 +2221,7 @@ static void arm_smmu_tlb_inv_range_domain(unsigned long iova, size_t size,
> * Unfortunately, this can't be leaf-only since we may have
> * zapped an entire table.
> */
> - arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova, size);
> + arm_smmu_atc_inv_domain(smmu_domain, iova, size);
> }
>
> void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
> @@ -2539,7 +2560,8 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master)
>
> static struct arm_smmu_master_domain *
> arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
> - struct arm_smmu_master *master)
> + struct arm_smmu_master *master,
> + ioasid_t ssid)
> {
> struct arm_smmu_master_domain *master_domain;
>
> @@ -2547,7 +2569,8 @@ arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
>
> list_for_each_entry(master_domain, &smmu_domain->devices,
> devices_elm) {
> - if (master_domain->master == master)
> + if (master_domain->master == master &&
> + master_domain->ssid == ssid)
> return master_domain;
> }
> return NULL;
> @@ -2580,7 +2603,8 @@ static void arm_smmu_remove_master_domain(struct arm_smmu_master *master,
> return;
>
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - master_domain = arm_smmu_find_master_domain(smmu_domain, master);
> + master_domain = arm_smmu_find_master_domain(smmu_domain, master,
> + IOMMU_NO_PASID);
> if (master_domain) {
> list_del(&master_domain->devices_elm);
> kfree(master_domain);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index c4b79bc52991be..98dc5885c48655 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -738,6 +738,7 @@ struct arm_smmu_domain {
> struct arm_smmu_master_domain {
> struct list_head devices_elm;
> struct arm_smmu_master *master;
> + u16 ssid;
> };
SMMU_IDR1.SSIDSIZE may be up to 20 according to the spec. The value
read from this register is used to populate iommu.max_pasids as well
as to compute the smmu/master ssid_bits fields so u16 is insufficient
here.
>
> static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
> @@ -769,8 +770,8 @@ void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
> size_t granule, bool leaf,
> struct arm_smmu_domain *smmu_domain);
> bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd);
> -int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
> - unsigned long iova, size_t size);
> +int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain,
> + ioasid_t ssid, unsigned long iova, size_t size);
>
> #ifdef CONFIG_ARM_SMMU_V3_SVA
> bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
> --
> 2.43.2
>
Apart from ssid size issue
Reviewed-by: Michael Shavit <mshavit@google.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 15/27] iommu/arm-smmu-v3: Add ssid to struct arm_smmu_master_domain
@ 2024-03-19 13:31 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 13:31 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Prepare to allow a S1 domain to be attached to a PASID as well. Keep track
> of the SSID the domain is using on each master in the
> arm_smmu_master_domain.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 15 ++++---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 42 +++++++++++++++----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 ++-
> 3 files changed, 43 insertions(+), 19 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 9a285b828e8b12..ab9de8e36c45f5 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -47,13 +47,12 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> struct arm_smmu_master *master = master_domain->master;
> struct arm_smmu_cd *cdptr;
>
> - /* S1 domains only support RID attachment right now */
> - cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> + cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid);
> if (WARN_ON(!cdptr))
> continue;
>
> arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> - arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> + arm_smmu_write_cd_entry(master, master_domain->ssid, cdptr,
> &target_cd);
> }
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> @@ -293,8 +292,8 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
> smmu_domain);
> }
>
> - arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), start,
> - size);
> + arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), start,
> + size);
> }
>
> static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
> @@ -331,7 +330,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
>
> arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid);
> - arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0);
> + arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0);
>
> smmu_mn->cleared = true;
> mutex_unlock(&sva_lock);
> @@ -410,8 +409,8 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn)
> */
> if (!smmu_mn->cleared) {
> arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid);
> - arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), 0,
> - 0);
> + arm_smmu_atc_inv_domain_sva(smmu_domain,
> + mm_get_enqcmd_pasid(mm), 0, 0);
> }
>
> /* Frees smmu_mn */
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 45f2190fc31786..26c6b9f6f34fd3 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2030,8 +2030,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master)
> return arm_smmu_cmdq_batch_submit(master->smmu, &cmds);
> }
>
> -int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
> - unsigned long iova, size_t size)
> +static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
> + ioasid_t ssid, unsigned long iova, size_t size)
> {
> struct arm_smmu_master_domain *master_domain;
> int i;
> @@ -2059,8 +2059,6 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
> if (!atomic_read(&smmu_domain->nr_ats_masters))
> return 0;
>
> - arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd);
> -
> cmds.num = 0;
>
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> @@ -2071,6 +2069,16 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
> if (!master->ats_enabled)
> continue;
>
> + /*
> + * Non-zero ssid means SVA is co-opting the S1 domain to issue
> + * invalidations for SVA PASIDs.
> + */
> + if (ssid != IOMMU_NO_PASID)
> + arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd);
> + else
> + arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size,
> + &cmd);
> +
> for (i = 0; i < master->num_streams; i++) {
> cmd.atc.sid = master->streams[i].id;
> arm_smmu_cmdq_batch_add(smmu_domain->smmu, &cmds, &cmd);
> @@ -2081,6 +2089,19 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
> return arm_smmu_cmdq_batch_submit(smmu_domain->smmu, &cmds);
> }
>
> +static int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
> + unsigned long iova, size_t size)
> +{
> + return __arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova,
> + size);
> +}
> +
> +int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain,
> + ioasid_t ssid, unsigned long iova, size_t size)
> +{
> + return __arm_smmu_atc_inv_domain(smmu_domain, ssid, iova, size);
> +}
> +
> /* IO_PGTABLE API */
> static void arm_smmu_tlb_inv_context(void *cookie)
> {
> @@ -2102,7 +2123,7 @@ static void arm_smmu_tlb_inv_context(void *cookie)
> cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid;
> arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
> }
> - arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, 0, 0);
> + arm_smmu_atc_inv_domain(smmu_domain, 0, 0);
> }
>
> static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd,
> @@ -2200,7 +2221,7 @@ static void arm_smmu_tlb_inv_range_domain(unsigned long iova, size_t size,
> * Unfortunately, this can't be leaf-only since we may have
> * zapped an entire table.
> */
> - arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova, size);
> + arm_smmu_atc_inv_domain(smmu_domain, iova, size);
> }
>
> void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
> @@ -2539,7 +2560,8 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master)
>
> static struct arm_smmu_master_domain *
> arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
> - struct arm_smmu_master *master)
> + struct arm_smmu_master *master,
> + ioasid_t ssid)
> {
> struct arm_smmu_master_domain *master_domain;
>
> @@ -2547,7 +2569,8 @@ arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
>
> list_for_each_entry(master_domain, &smmu_domain->devices,
> devices_elm) {
> - if (master_domain->master == master)
> + if (master_domain->master == master &&
> + master_domain->ssid == ssid)
> return master_domain;
> }
> return NULL;
> @@ -2580,7 +2603,8 @@ static void arm_smmu_remove_master_domain(struct arm_smmu_master *master,
> return;
>
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - master_domain = arm_smmu_find_master_domain(smmu_domain, master);
> + master_domain = arm_smmu_find_master_domain(smmu_domain, master,
> + IOMMU_NO_PASID);
> if (master_domain) {
> list_del(&master_domain->devices_elm);
> kfree(master_domain);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index c4b79bc52991be..98dc5885c48655 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -738,6 +738,7 @@ struct arm_smmu_domain {
> struct arm_smmu_master_domain {
> struct list_head devices_elm;
> struct arm_smmu_master *master;
> + u16 ssid;
> };
SMMU_IDR1.SSIDSIZE may be up to 20 according to the spec. The value
read from this register is used to populate iommu.max_pasids as well
as to compute the smmu/master ssid_bits fields so u16 is insufficient
here.
>
> static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
> @@ -769,8 +770,8 @@ void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
> size_t granule, bool leaf,
> struct arm_smmu_domain *smmu_domain);
> bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd);
> -int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
> - unsigned long iova, size_t size);
> +int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain,
> + ioasid_t ssid, unsigned long iova, size_t size);
>
> #ifdef CONFIG_ARM_SMMU_V3_SVA
> bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
> --
> 2.43.2
>
Apart from ssid size issue
Reviewed-by: Michael Shavit <mshavit@google.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 16/27] iommu/arm-smmu-v3: Keep track of valid CD entries in the cd_table
2024-03-04 23:44 ` Jason Gunthorpe
@ 2024-03-19 13:55 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 13:55 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> We no longer need a master->sva_enable to control what attaches are
> allowed.
>
> Instead keep track inside the cd_table how many valid CD entries exist,
> and if the RID has a valid entry.
>
> Replace all the attach focused master->sva_enabled tests with a check if
> the CD has valid entries (or not). If there are any valid entries then the
> CD table must be currently programmed to the STE.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 5 +---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 ++++++++++---------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 10 +++++++
> 3 files changed, 25 insertions(+), 16 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index ab9de8e36c45f5..82b9c4d4061c3d 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -433,9 +433,6 @@ static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm,
> if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1)
> return -ENODEV;
>
I assume this doesn't matter because of subsequent patches, but the
check above could also be removed since used_sid precisely means that
the attached domain is an ARM_SMMU_DOMAIN_S1 domain.
> - if (!master || !master->sva_enabled)
> - return -ENODEV;
> -
> bond = kzalloc(sizeof(*bond), GFP_KERNEL);
> if (!bond)
> return -ENOMEM;
> @@ -638,7 +635,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> struct mm_struct *mm = domain->mm;
> struct arm_smmu_cd target;
>
> - if (mm_get_enqcmd_pasid(mm) != id)
> + if (mm_get_enqcmd_pasid(mm) != id || !master->cd_table.used_sid)
> return -EINVAL;
>
> if (!arm_smmu_get_cd_ptr(master, id))
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 26c6b9f6f34fd3..fc19585bf192c6 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1285,6 +1285,8 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_cd *cdptr,
> const struct arm_smmu_cd *target)
> {
> + bool target_valid = target->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
> + bool cur_valid = cdptr->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
> struct arm_smmu_cd_writer cd_writer = {
> .writer = {
> .ops = &arm_smmu_cd_writer_ops,
> @@ -1293,6 +1295,15 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> .ssid = ssid,
> };
>
> + if (cur_valid != target_valid) {
> + if (cur_valid)
> + master->cd_table.used_ssids--;
> + else
> + master->cd_table.used_ssids++;
> + }
> + if (ssid == IOMMU_NO_PASID)
> + master->cd_table.used_sid = target_valid;
> +
> arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
> }
>
> @@ -2725,16 +2736,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> master = dev_iommu_priv_get(dev);
> smmu = master->smmu;
>
> - /*
> - * Checking that SVA is disabled ensures that this device isn't bound to
> - * any mm, and can be safely detached from its old domain. Bonds cannot
> - * be removed concurrently since we're holding the group mutex.
> - */
> - if (arm_smmu_master_sva_enabled(master)) {
> - dev_err(dev, "cannot attach - SVA enabled\n");
> - return -EBUSY;
> - }
> -
> mutex_lock(&smmu_domain->init_mutex);
>
> if (!smmu_domain->smmu) {
> @@ -2750,7 +2751,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> if (!cdptr)
> return -ENOMEM;
> - }
> + } else if (arm_smmu_ssids_in_use(&master->cd_table))
> + return -EBUSY;
>
> /*
> * Prevent arm_smmu_share_asid() from trying to change the ASID
> @@ -2825,7 +2827,7 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
> struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> struct attach_state state = {};
>
> - if (arm_smmu_master_sva_enabled(master))
> + if (arm_smmu_ssids_in_use(&master->cd_table))
> return -EBUSY;
>
> /*
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 98dc5885c48655..7e1f6af4ce4e79 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -602,11 +602,21 @@ struct arm_smmu_ctx_desc_cfg {
> dma_addr_t cdtab_dma;
> struct arm_smmu_l1_ctx_desc *l1_desc;
> unsigned int num_l1_ents;
> + unsigned int used_ssids;
> + bool used_sid;
This probably deserves a comment. There's plenty of places where the
"rid" domain is handled as the CD with ssid 0; but we don't count it
as a used_ssid here.
I also don't find the meaning of used_sid obvious, especially if I
didn't have the context from the commit description.
> u8 s1fmt;
> /* log2 of the maximum number of CDs supported by this table */
> u8 s1cdmax;
> };
>
> +/* True if the cd table has SSIDS > 0 in use. */
> +static inline bool arm_smmu_ssids_in_use(struct arm_smmu_ctx_desc_cfg *cd_table)
> +{
> + if (cd_table->used_sid)
> + return cd_table->used_ssids > 1;
> + return cd_table->used_ssids;
> +}
> +
> struct arm_smmu_s2_cfg {
> u16 vmid;
> };
> --
> 2.43.2
>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 16/27] iommu/arm-smmu-v3: Keep track of valid CD entries in the cd_table
@ 2024-03-19 13:55 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 13:55 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> We no longer need a master->sva_enable to control what attaches are
> allowed.
>
> Instead keep track inside the cd_table how many valid CD entries exist,
> and if the RID has a valid entry.
>
> Replace all the attach focused master->sva_enabled tests with a check if
> the CD has valid entries (or not). If there are any valid entries then the
> CD table must be currently programmed to the STE.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 5 +---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 ++++++++++---------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 10 +++++++
> 3 files changed, 25 insertions(+), 16 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index ab9de8e36c45f5..82b9c4d4061c3d 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -433,9 +433,6 @@ static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm,
> if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1)
> return -ENODEV;
>
I assume this doesn't matter because of subsequent patches, but the
check above could also be removed since used_sid precisely means that
the attached domain is an ARM_SMMU_DOMAIN_S1 domain.
> - if (!master || !master->sva_enabled)
> - return -ENODEV;
> -
> bond = kzalloc(sizeof(*bond), GFP_KERNEL);
> if (!bond)
> return -ENOMEM;
> @@ -638,7 +635,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> struct mm_struct *mm = domain->mm;
> struct arm_smmu_cd target;
>
> - if (mm_get_enqcmd_pasid(mm) != id)
> + if (mm_get_enqcmd_pasid(mm) != id || !master->cd_table.used_sid)
> return -EINVAL;
>
> if (!arm_smmu_get_cd_ptr(master, id))
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 26c6b9f6f34fd3..fc19585bf192c6 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1285,6 +1285,8 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_cd *cdptr,
> const struct arm_smmu_cd *target)
> {
> + bool target_valid = target->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
> + bool cur_valid = cdptr->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
> struct arm_smmu_cd_writer cd_writer = {
> .writer = {
> .ops = &arm_smmu_cd_writer_ops,
> @@ -1293,6 +1295,15 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> .ssid = ssid,
> };
>
> + if (cur_valid != target_valid) {
> + if (cur_valid)
> + master->cd_table.used_ssids--;
> + else
> + master->cd_table.used_ssids++;
> + }
> + if (ssid == IOMMU_NO_PASID)
> + master->cd_table.used_sid = target_valid;
> +
> arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
> }
>
> @@ -2725,16 +2736,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> master = dev_iommu_priv_get(dev);
> smmu = master->smmu;
>
> - /*
> - * Checking that SVA is disabled ensures that this device isn't bound to
> - * any mm, and can be safely detached from its old domain. Bonds cannot
> - * be removed concurrently since we're holding the group mutex.
> - */
> - if (arm_smmu_master_sva_enabled(master)) {
> - dev_err(dev, "cannot attach - SVA enabled\n");
> - return -EBUSY;
> - }
> -
> mutex_lock(&smmu_domain->init_mutex);
>
> if (!smmu_domain->smmu) {
> @@ -2750,7 +2751,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> if (!cdptr)
> return -ENOMEM;
> - }
> + } else if (arm_smmu_ssids_in_use(&master->cd_table))
> + return -EBUSY;
>
> /*
> * Prevent arm_smmu_share_asid() from trying to change the ASID
> @@ -2825,7 +2827,7 @@ static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
> struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> struct attach_state state = {};
>
> - if (arm_smmu_master_sva_enabled(master))
> + if (arm_smmu_ssids_in_use(&master->cd_table))
> return -EBUSY;
>
> /*
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 98dc5885c48655..7e1f6af4ce4e79 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -602,11 +602,21 @@ struct arm_smmu_ctx_desc_cfg {
> dma_addr_t cdtab_dma;
> struct arm_smmu_l1_ctx_desc *l1_desc;
> unsigned int num_l1_ents;
> + unsigned int used_ssids;
> + bool used_sid;
This probably deserves a comment. There's plenty of places where the
"rid" domain is handled as the CD with ssid 0; but we don't count it
as a used_ssid here.
I also don't find the meaning of used_sid obvious, especially if I
didn't have the context from the commit description.
> u8 s1fmt;
> /* log2 of the maximum number of CDs supported by this table */
> u8 s1cdmax;
> };
>
> +/* True if the cd table has SSIDS > 0 in use. */
> +static inline bool arm_smmu_ssids_in_use(struct arm_smmu_ctx_desc_cfg *cd_table)
> +{
> + if (cd_table->used_sid)
> + return cd_table->used_ssids > 1;
> + return cd_table->used_ssids;
> +}
> +
> struct arm_smmu_s2_cfg {
> u16 vmid;
> };
> --
> 2.43.2
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 18/27] iommu/arm-smmu-v3: Make SVA allocate a normal arm_smmu_domain
2024-03-04 23:44 ` Jason Gunthorpe
@ 2024-03-19 14:52 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 14:52 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Currently the SVA domain is a naked struct iommu_domain, allocate a struct
> arm_smmu_domain instead.
>
> This is necessary to be able to use the struct arm_master_domain
> mechanism.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 19 ++++++-----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 34 +++++++++++--------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++-
> 3 files changed, 33 insertions(+), 24 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 82b9c4d4061c3d..d633316f2e45bc 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -654,7 +654,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
>
> static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
> {
> - kfree(domain);
> + kfree(to_smmu_domain(domain));
> }
>
> static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
> @@ -662,14 +662,17 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
> .free = arm_smmu_sva_domain_free
> };
>
> -struct iommu_domain *arm_smmu_sva_domain_alloc(void)
> +struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned type)
> {
> - struct iommu_domain *domain;
> + struct arm_smmu_domain *smmu_domain;
>
> - domain = kzalloc(sizeof(*domain), GFP_KERNEL);
> - if (!domain)
> - return NULL;
> - domain->ops = &arm_smmu_sva_domain_ops;
> + if (type != IOMMU_DOMAIN_SVA)
> + return ERR_PTR(-EOPNOTSUPP);
>
> - return domain;
> + smmu_domain = arm_smmu_domain_alloc();
> + if (IS_ERR(smmu_domain))
> + return ERR_CAST(smmu_domain);
> + smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
> +
> + return &smmu_domain->domain;
> }
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index dd7f841cd19b3c..2db2b822292a87 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2291,23 +2291,10 @@ static bool arm_smmu_capable(struct device *dev, enum iommu_cap cap)
> }
> }
>
> -static struct iommu_domain *arm_smmu_domain_alloc(unsigned type)
> -{
> -
> - if (type == IOMMU_DOMAIN_SVA)
> - return arm_smmu_sva_domain_alloc();
> - return ERR_PTR(-EOPNOTSUPP);
> -}
> -
> -static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
> +struct arm_smmu_domain *arm_smmu_domain_alloc(void)
Consider renaming arm_smmu_domain_free as well since there's asymmetry
between arm_smmu_domain_alloc and arm_smmu_domain_free that could be a
little confusing:
1. arm_smmu_domain_alloc is shared between arm_smmu_sva_domain_alloc
and arm_smmu_domain_alloc_paging
2. arm_smmu_domain_free is only used by paging domains, with SVA
domains freed through arm_smmu_sva_domain_free.
> {
> struct arm_smmu_domain *smmu_domain;
>
> - /*
> - * Allocate the domain and initialise some of its data structures.
> - * We can't really do anything meaningful until we've added a
> - * master.
> - */
> smmu_domain = kzalloc(sizeof(*smmu_domain), GFP_KERNEL);
> if (!smmu_domain)
> return ERR_PTR(-ENOMEM);
> @@ -2317,6 +2304,23 @@ static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
> spin_lock_init(&smmu_domain->devices_lock);
> INIT_LIST_HEAD(&smmu_domain->mmu_notifiers);
>
> + return smmu_domain;
> +}
> +
> +static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
> +{
> + struct arm_smmu_domain *smmu_domain;
> +
> + smmu_domain = arm_smmu_domain_alloc();
> + if (IS_ERR(smmu_domain))
> + return ERR_CAST(smmu_domain);
> +
> + /*
> + * Allocate the domain and initialise some of its data structures.
> + * We can't really do anything meaningful until we've added a
> + * master.
> + */
> +
> if (dev) {
> struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> int ret;
> @@ -3288,7 +3292,7 @@ static struct iommu_ops arm_smmu_ops = {
> .identity_domain = &arm_smmu_identity_domain,
> .blocked_domain = &arm_smmu_blocked_domain,
> .capable = arm_smmu_capable,
> - .domain_alloc = arm_smmu_domain_alloc,
> + .domain_alloc = arm_smmu_sva_domain_alloc,
> .domain_alloc_paging = arm_smmu_domain_alloc_paging,
> .probe_device = arm_smmu_probe_device,
> .release_device = arm_smmu_release_device,
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 7e1f6af4ce4e79..c47e07d695bef2 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -759,6 +759,8 @@ static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
> extern struct xarray arm_smmu_asid_xa;
> extern struct mutex arm_smmu_asid_lock;
>
> +struct arm_smmu_domain *arm_smmu_domain_alloc(void);
> +
> void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
> struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> u32 ssid);
> @@ -791,7 +793,7 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master);
> int arm_smmu_master_disable_sva(struct arm_smmu_master *master);
> bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master);
> void arm_smmu_sva_notifier_synchronize(void);
> -struct iommu_domain *arm_smmu_sva_domain_alloc(void);
> +struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned int type);
> void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
> struct device *dev, ioasid_t id);
> #else /* CONFIG_ARM_SMMU_V3_SVA */
> --
> 2.43.2
>
>
Reviewed-by: Michael Shavit <mshavit@google.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 18/27] iommu/arm-smmu-v3: Make SVA allocate a normal arm_smmu_domain
@ 2024-03-19 14:52 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 14:52 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Currently the SVA domain is a naked struct iommu_domain, allocate a struct
> arm_smmu_domain instead.
>
> This is necessary to be able to use the struct arm_master_domain
> mechanism.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 19 ++++++-----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 34 +++++++++++--------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++-
> 3 files changed, 33 insertions(+), 24 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 82b9c4d4061c3d..d633316f2e45bc 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -654,7 +654,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
>
> static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
> {
> - kfree(domain);
> + kfree(to_smmu_domain(domain));
> }
>
> static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
> @@ -662,14 +662,17 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
> .free = arm_smmu_sva_domain_free
> };
>
> -struct iommu_domain *arm_smmu_sva_domain_alloc(void)
> +struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned type)
> {
> - struct iommu_domain *domain;
> + struct arm_smmu_domain *smmu_domain;
>
> - domain = kzalloc(sizeof(*domain), GFP_KERNEL);
> - if (!domain)
> - return NULL;
> - domain->ops = &arm_smmu_sva_domain_ops;
> + if (type != IOMMU_DOMAIN_SVA)
> + return ERR_PTR(-EOPNOTSUPP);
>
> - return domain;
> + smmu_domain = arm_smmu_domain_alloc();
> + if (IS_ERR(smmu_domain))
> + return ERR_CAST(smmu_domain);
> + smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
> +
> + return &smmu_domain->domain;
> }
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index dd7f841cd19b3c..2db2b822292a87 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2291,23 +2291,10 @@ static bool arm_smmu_capable(struct device *dev, enum iommu_cap cap)
> }
> }
>
> -static struct iommu_domain *arm_smmu_domain_alloc(unsigned type)
> -{
> -
> - if (type == IOMMU_DOMAIN_SVA)
> - return arm_smmu_sva_domain_alloc();
> - return ERR_PTR(-EOPNOTSUPP);
> -}
> -
> -static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
> +struct arm_smmu_domain *arm_smmu_domain_alloc(void)
Consider renaming arm_smmu_domain_free as well since there's asymmetry
between arm_smmu_domain_alloc and arm_smmu_domain_free that could be a
little confusing:
1. arm_smmu_domain_alloc is shared between arm_smmu_sva_domain_alloc
and arm_smmu_domain_alloc_paging
2. arm_smmu_domain_free is only used by paging domains, with SVA
domains freed through arm_smmu_sva_domain_free.
> {
> struct arm_smmu_domain *smmu_domain;
>
> - /*
> - * Allocate the domain and initialise some of its data structures.
> - * We can't really do anything meaningful until we've added a
> - * master.
> - */
> smmu_domain = kzalloc(sizeof(*smmu_domain), GFP_KERNEL);
> if (!smmu_domain)
> return ERR_PTR(-ENOMEM);
> @@ -2317,6 +2304,23 @@ static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
> spin_lock_init(&smmu_domain->devices_lock);
> INIT_LIST_HEAD(&smmu_domain->mmu_notifiers);
>
> + return smmu_domain;
> +}
> +
> +static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
> +{
> + struct arm_smmu_domain *smmu_domain;
> +
> + smmu_domain = arm_smmu_domain_alloc();
> + if (IS_ERR(smmu_domain))
> + return ERR_CAST(smmu_domain);
> +
> + /*
> + * Allocate the domain and initialise some of its data structures.
> + * We can't really do anything meaningful until we've added a
> + * master.
> + */
> +
> if (dev) {
> struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> int ret;
> @@ -3288,7 +3292,7 @@ static struct iommu_ops arm_smmu_ops = {
> .identity_domain = &arm_smmu_identity_domain,
> .blocked_domain = &arm_smmu_blocked_domain,
> .capable = arm_smmu_capable,
> - .domain_alloc = arm_smmu_domain_alloc,
> + .domain_alloc = arm_smmu_sva_domain_alloc,
> .domain_alloc_paging = arm_smmu_domain_alloc_paging,
> .probe_device = arm_smmu_probe_device,
> .release_device = arm_smmu_release_device,
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 7e1f6af4ce4e79..c47e07d695bef2 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -759,6 +759,8 @@ static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
> extern struct xarray arm_smmu_asid_xa;
> extern struct mutex arm_smmu_asid_lock;
>
> +struct arm_smmu_domain *arm_smmu_domain_alloc(void);
> +
> void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
> struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> u32 ssid);
> @@ -791,7 +793,7 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master);
> int arm_smmu_master_disable_sva(struct arm_smmu_master *master);
> bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master);
> void arm_smmu_sva_notifier_synchronize(void);
> -struct iommu_domain *arm_smmu_sva_domain_alloc(void);
> +struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned int type);
> void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
> struct device *dev, ioasid_t id);
> #else /* CONFIG_ARM_SMMU_V3_SVA */
> --
> 2.43.2
>
>
Reviewed-by: Michael Shavit <mshavit@google.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 20/27] iommu: Add ops->domain_alloc_sva()
2024-03-04 23:44 ` Jason Gunthorpe
@ 2024-03-19 15:09 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 15:09 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Make a new op that receives the device and the mm_struct that the SVA
> domain should be created for. Unlike domain_alloc_paging() the dev
> argument is never NULL here.
>
> This allows drivers to fully initialize the SVA domain and allocate the
> mmu_notifier during allocation. It allows the notifier lifetime to follow
> the lifetime of the iommu_domain.
>
> Since we have only one call site, upgrade the new op to return ERR_PTR
> instead of NULL.
>
> Change SMMUv3 to use the new op.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 11 +++++++----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 2 +-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 +++++-
> drivers/iommu/iommu-sva.c | 4 ++--
> drivers/iommu/iommu.c | 12 +++++++++---
> include/linux/iommu.h | 3 +++
> 6 files changed, 27 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index d633316f2e45bc..30b1cf587efffb 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -662,17 +662,20 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
> .free = arm_smmu_sva_domain_free
> };
>
> -struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned type)
> +struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> + struct mm_struct *mm)
> {
> + struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> + struct arm_smmu_device *smmu = master->smmu;
> struct arm_smmu_domain *smmu_domain;
>
> - if (type != IOMMU_DOMAIN_SVA)
> - return ERR_PTR(-EOPNOTSUPP);
> -
> smmu_domain = arm_smmu_domain_alloc();
> if (IS_ERR(smmu_domain))
> return ERR_CAST(smmu_domain);
> +
> + smmu_domain->domain.type = IOMMU_DOMAIN_SVA;
> smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
> + smmu_domain->smmu = smmu;
>
> return &smmu_domain->domain;
> }
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 6d15fe3a160acf..a255a02a5fc8a9 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -3311,8 +3311,8 @@ static struct iommu_ops arm_smmu_ops = {
> .identity_domain = &arm_smmu_identity_domain,
> .blocked_domain = &arm_smmu_blocked_domain,
> .capable = arm_smmu_capable,
> - .domain_alloc = arm_smmu_sva_domain_alloc,
> .domain_alloc_paging = arm_smmu_domain_alloc_paging,
> + .domain_alloc_sva = arm_smmu_sva_domain_alloc,
> .probe_device = arm_smmu_probe_device,
> .release_device = arm_smmu_release_device,
> .device_group = arm_smmu_device_group,
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index c47e07d695bef2..f62bd2ef36a18d 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -793,7 +793,8 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master);
> int arm_smmu_master_disable_sva(struct arm_smmu_master *master);
> bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master);
> void arm_smmu_sva_notifier_synchronize(void);
> -struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned int type);
> +struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> + struct mm_struct *mm);
> void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
> struct device *dev, ioasid_t id);
> #else /* CONFIG_ARM_SMMU_V3_SVA */
> @@ -839,5 +840,8 @@ static inline void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
> ioasid_t id)
> {
> }
> +
> +#define arm_smmu_sva_domain_alloc NULL
> +
> #endif /* CONFIG_ARM_SMMU_V3_SVA */
> #endif /* _ARM_SMMU_V3_H */
> diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c
> index 65814cbc840200..aa20ee6a1d8ae7 100644
> --- a/drivers/iommu/iommu-sva.c
> +++ b/drivers/iommu/iommu-sva.c
> @@ -108,8 +108,8 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm
>
> /* Allocate a new domain and set it on device pasid. */
> domain = iommu_sva_domain_alloc(dev, mm);
> - if (!domain) {
> - ret = -ENOMEM;
> + if (IS_ERR(domain)) {
> + ret = PTR_ERR(domain);
> goto out_free_handle;
> }
>
> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
> index d14413916f93a0..f9a2b8363ad941 100644
> --- a/drivers/iommu/iommu.c
> +++ b/drivers/iommu/iommu.c
> @@ -3629,9 +3629,15 @@ struct iommu_domain *iommu_sva_domain_alloc(struct device *dev,
> const struct iommu_ops *ops = dev_iommu_ops(dev);
> struct iommu_domain *domain;
>
> - domain = ops->domain_alloc(IOMMU_DOMAIN_SVA);
> - if (!domain)
> - return NULL;
> + if (ops->domain_alloc_sva) {
> + domain = ops->domain_alloc_sva(dev, mm);
> + if (IS_ERR(domain))
> + return domain;
> + } else {
> + domain = ops->domain_alloc(IOMMU_DOMAIN_SVA);
> + if (!domain)
> + return ERR_PTR(-ENOMEM);
> + }
>
> domain->type = IOMMU_DOMAIN_SVA;
> mmgrab(mm);
> diff --git a/include/linux/iommu.h b/include/linux/iommu.h
> index 5e27cb3a3be99b..0274044f1fb084 100644
> --- a/include/linux/iommu.h
> +++ b/include/linux/iommu.h
> @@ -419,6 +419,7 @@ static inline int __iommu_copy_struct_from_user_array(
> * Upon failure, ERR_PTR must be returned.
> * @domain_alloc_paging: Allocate an iommu_domain that can be used for
> * UNMANAGED, DMA, and DMA_FQ domain types.
> + * @domain_alloc_sva: Allocate an iommu_domain for Shared Virtual Addressing.
> * @probe_device: Add device to iommu driver handling
> * @release_device: Remove device from iommu driver handling
> * @probe_finalize: Do final setup work after the device is added to an IOMMU
> @@ -459,6 +460,8 @@ struct iommu_ops {
> struct device *dev, u32 flags, struct iommu_domain *parent,
> const struct iommu_user_data *user_data);
> struct iommu_domain *(*domain_alloc_paging)(struct device *dev);
> + struct iommu_domain *(*domain_alloc_sva)(struct device *dev,
> + struct mm_struct *mm);
>
> struct iommu_device *(*probe_device)(struct device *dev);
> void (*release_device)(struct device *dev);
> --
> 2.43.2
>
Reviewed-by: Michael Shavit <mshavit@google.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 20/27] iommu: Add ops->domain_alloc_sva()
@ 2024-03-19 15:09 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 15:09 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Make a new op that receives the device and the mm_struct that the SVA
> domain should be created for. Unlike domain_alloc_paging() the dev
> argument is never NULL here.
>
> This allows drivers to fully initialize the SVA domain and allocate the
> mmu_notifier during allocation. It allows the notifier lifetime to follow
> the lifetime of the iommu_domain.
>
> Since we have only one call site, upgrade the new op to return ERR_PTR
> instead of NULL.
>
> Change SMMUv3 to use the new op.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 11 +++++++----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 2 +-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 +++++-
> drivers/iommu/iommu-sva.c | 4 ++--
> drivers/iommu/iommu.c | 12 +++++++++---
> include/linux/iommu.h | 3 +++
> 6 files changed, 27 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index d633316f2e45bc..30b1cf587efffb 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -662,17 +662,20 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
> .free = arm_smmu_sva_domain_free
> };
>
> -struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned type)
> +struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> + struct mm_struct *mm)
> {
> + struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> + struct arm_smmu_device *smmu = master->smmu;
> struct arm_smmu_domain *smmu_domain;
>
> - if (type != IOMMU_DOMAIN_SVA)
> - return ERR_PTR(-EOPNOTSUPP);
> -
> smmu_domain = arm_smmu_domain_alloc();
> if (IS_ERR(smmu_domain))
> return ERR_CAST(smmu_domain);
> +
> + smmu_domain->domain.type = IOMMU_DOMAIN_SVA;
> smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
> + smmu_domain->smmu = smmu;
>
> return &smmu_domain->domain;
> }
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 6d15fe3a160acf..a255a02a5fc8a9 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -3311,8 +3311,8 @@ static struct iommu_ops arm_smmu_ops = {
> .identity_domain = &arm_smmu_identity_domain,
> .blocked_domain = &arm_smmu_blocked_domain,
> .capable = arm_smmu_capable,
> - .domain_alloc = arm_smmu_sva_domain_alloc,
> .domain_alloc_paging = arm_smmu_domain_alloc_paging,
> + .domain_alloc_sva = arm_smmu_sva_domain_alloc,
> .probe_device = arm_smmu_probe_device,
> .release_device = arm_smmu_release_device,
> .device_group = arm_smmu_device_group,
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index c47e07d695bef2..f62bd2ef36a18d 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -793,7 +793,8 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master);
> int arm_smmu_master_disable_sva(struct arm_smmu_master *master);
> bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master);
> void arm_smmu_sva_notifier_synchronize(void);
> -struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned int type);
> +struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> + struct mm_struct *mm);
> void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
> struct device *dev, ioasid_t id);
> #else /* CONFIG_ARM_SMMU_V3_SVA */
> @@ -839,5 +840,8 @@ static inline void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
> ioasid_t id)
> {
> }
> +
> +#define arm_smmu_sva_domain_alloc NULL
> +
> #endif /* CONFIG_ARM_SMMU_V3_SVA */
> #endif /* _ARM_SMMU_V3_H */
> diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c
> index 65814cbc840200..aa20ee6a1d8ae7 100644
> --- a/drivers/iommu/iommu-sva.c
> +++ b/drivers/iommu/iommu-sva.c
> @@ -108,8 +108,8 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm
>
> /* Allocate a new domain and set it on device pasid. */
> domain = iommu_sva_domain_alloc(dev, mm);
> - if (!domain) {
> - ret = -ENOMEM;
> + if (IS_ERR(domain)) {
> + ret = PTR_ERR(domain);
> goto out_free_handle;
> }
>
> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
> index d14413916f93a0..f9a2b8363ad941 100644
> --- a/drivers/iommu/iommu.c
> +++ b/drivers/iommu/iommu.c
> @@ -3629,9 +3629,15 @@ struct iommu_domain *iommu_sva_domain_alloc(struct device *dev,
> const struct iommu_ops *ops = dev_iommu_ops(dev);
> struct iommu_domain *domain;
>
> - domain = ops->domain_alloc(IOMMU_DOMAIN_SVA);
> - if (!domain)
> - return NULL;
> + if (ops->domain_alloc_sva) {
> + domain = ops->domain_alloc_sva(dev, mm);
> + if (IS_ERR(domain))
> + return domain;
> + } else {
> + domain = ops->domain_alloc(IOMMU_DOMAIN_SVA);
> + if (!domain)
> + return ERR_PTR(-ENOMEM);
> + }
>
> domain->type = IOMMU_DOMAIN_SVA;
> mmgrab(mm);
> diff --git a/include/linux/iommu.h b/include/linux/iommu.h
> index 5e27cb3a3be99b..0274044f1fb084 100644
> --- a/include/linux/iommu.h
> +++ b/include/linux/iommu.h
> @@ -419,6 +419,7 @@ static inline int __iommu_copy_struct_from_user_array(
> * Upon failure, ERR_PTR must be returned.
> * @domain_alloc_paging: Allocate an iommu_domain that can be used for
> * UNMANAGED, DMA, and DMA_FQ domain types.
> + * @domain_alloc_sva: Allocate an iommu_domain for Shared Virtual Addressing.
> * @probe_device: Add device to iommu driver handling
> * @release_device: Remove device from iommu driver handling
> * @probe_finalize: Do final setup work after the device is added to an IOMMU
> @@ -459,6 +460,8 @@ struct iommu_ops {
> struct device *dev, u32 flags, struct iommu_domain *parent,
> const struct iommu_user_data *user_data);
> struct iommu_domain *(*domain_alloc_paging)(struct device *dev);
> + struct iommu_domain *(*domain_alloc_sva)(struct device *dev,
> + struct mm_struct *mm);
>
> struct iommu_device *(*probe_device)(struct device *dev);
> void (*release_device)(struct device *dev);
> --
> 2.43.2
>
Reviewed-by: Michael Shavit <mshavit@google.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 12/27] iommu/arm-smmu-v3: Start building a generic PASID layer
2024-03-04 23:44 ` Jason Gunthorpe
@ 2024-03-19 16:11 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 16:11 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Add arm_smmu_set_pasid()/arm_smmu_remove_pasid() which are to be used by
> callers that already constructed the arm_smmu_cd they wish to program.
>
> These functions will encapsulate the shared logic to setup a CD entry that
> will be shared by SVA and S1 domain cases.
>
> Prior fixes had already moved this logic most up into
> __arm_smmu_sva_bind(), move it to it's final home.
>
> This does not yet relieve the SVA restrictions:
> - The RID domain is a S1 domain
> - The programmed PASID is the mm_get_enqcmd_pasid()
> - Nothing changes while SVA is running (sva_enable)
>
> Also, SVA invalidation will still iterate over the S1 domain's master
> list.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 39 +++++++++----------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 29 ++++++++++++++
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 +++
> 3 files changed, 53 insertions(+), 21 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 159350a78d4cf9..e1fa2074c0b37b 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -416,12 +416,10 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn)
> arm_smmu_free_shared_cd(cd);
> }
>
> -static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid,
> - struct mm_struct *mm)
> +static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm,
> + struct arm_smmu_cd *target)
> {
> int ret;
> - struct arm_smmu_cd target;
> - struct arm_smmu_cd *cdptr;
> struct arm_smmu_bond *bond;
> struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
> @@ -448,19 +446,10 @@ static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid,
> goto err_free_bond;
> }
>
> - cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm));
> - if (!cdptr) {
> - ret = -ENOMEM;
> - goto err_put_notifier;
> - }
> - arm_smmu_make_sva_cd(&target, master, mm, bond->smmu_mn->cd->asid);
> - arm_smmu_write_cd_entry(master, pasid, cdptr, &target);
> -
> + arm_smmu_make_sva_cd(target, master, mm, bond->smmu_mn->cd->asid);
This can probably already move out to arm_smmu_sva_set_dev_pasid in
this commit. Removing the arm_smmu_get_cd_ptr pre-alloc might also be
possible if we're careful with failure of arm_smmu_set_pasid.
This is eventually addressed in "iommu/arm-smmu-v3: Put the SVA mmu
notifier in the smmu_domain", but that patch is already big enough as
it is and the change fits nicely here.
> list_add(&bond->list, &master->bonds);
> return 0;
>
> -err_put_notifier:
> - arm_smmu_mmu_notifier_put(bond->smmu_mn);
> err_free_bond:
> kfree(bond);
> return ret;
> @@ -621,10 +610,9 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
> struct arm_smmu_bond *bond = NULL, *t;
> struct arm_smmu_master *master = dev_iommu_priv_get(dev);
>
> + arm_smmu_remove_pasid(master, to_smmu_domain(domain), id);
> +
> mutex_lock(&sva_lock);
> -
> - arm_smmu_clear_cd(master, id);
> -
> list_for_each_entry(t, &master->bonds, list) {
> if (t->mm == mm) {
> bond = t;
> @@ -643,17 +631,26 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
> static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> struct device *dev, ioasid_t id)
> {
> + struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> int ret = 0;
> struct mm_struct *mm = domain->mm;
> + struct arm_smmu_cd target;
>
> if (mm_get_enqcmd_pasid(mm) != id)
> return -EINVAL;
>
> - mutex_lock(&sva_lock);
> - ret = __arm_smmu_sva_bind(dev, id, mm);
> - mutex_unlock(&sva_lock);
> + if (!arm_smmu_get_cd_ptr(master, id))
> + return -ENOMEM;
>
> - return ret;
> + mutex_lock(&sva_lock);
> + ret = __arm_smmu_sva_bind(dev, mm, &target);
> + mutex_unlock(&sva_lock);
> + if (ret)
> + return ret;
> +
> + /* This cannot fail since we preallocated the cdptr */
> + arm_smmu_set_pasid(master, to_smmu_domain(domain), id, &target);
> + return 0;
> }
>
> static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index dfdd48cf217c4e..7a0be7dd719793 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2656,6 +2656,35 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> return 0;
> }
>
> +static bool arm_smmu_is_s1_domain(struct iommu_domain *domain)
> +{
> + if (!domain || !(domain->type & __IOMMU_DOMAIN_PAGING))
> + return false;
> + return to_smmu_domain(domain)->stage == ARM_SMMU_DOMAIN_S1;
> +}
> +
> +int arm_smmu_set_pasid(struct arm_smmu_master *master,
> + struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
> + const struct arm_smmu_cd *cd)
> +{
> + struct arm_smmu_cd *cdptr;
> +
> + if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)))
> + return -ENODEV;
> +
> + cdptr = arm_smmu_get_cd_ptr(master, pasid);
> + if (!cdptr)
> + return -ENOMEM;
> + arm_smmu_write_cd_entry(master, pasid, cdptr, cd);
> + return 0;
> +}
> +
> +void arm_smmu_remove_pasid(struct arm_smmu_master *master,
> + struct arm_smmu_domain *smmu_domain, ioasid_t pasid)
> +{
> + arm_smmu_clear_cd(master, pasid);
> +}
> +
> static int arm_smmu_attach_dev_ste(struct device *dev,
> struct arm_smmu_ste *ste)
> {
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 468cd33b80ac35..ab0a6587e07ec8 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -753,6 +753,12 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_cd *cdptr,
> const struct arm_smmu_cd *target);
>
> +int arm_smmu_set_pasid(struct arm_smmu_master *master,
> + struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
> + const struct arm_smmu_cd *cd);
> +void arm_smmu_remove_pasid(struct arm_smmu_master *master,
> + struct arm_smmu_domain *smmu_domain, ioasid_t pasid);
> +
> void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
> void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
> size_t granule, bool leaf,
> --
> 2.43.2
>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 12/27] iommu/arm-smmu-v3: Start building a generic PASID layer
@ 2024-03-19 16:11 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 16:11 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Add arm_smmu_set_pasid()/arm_smmu_remove_pasid() which are to be used by
> callers that already constructed the arm_smmu_cd they wish to program.
>
> These functions will encapsulate the shared logic to setup a CD entry that
> will be shared by SVA and S1 domain cases.
>
> Prior fixes had already moved this logic most up into
> __arm_smmu_sva_bind(), move it to it's final home.
>
> This does not yet relieve the SVA restrictions:
> - The RID domain is a S1 domain
> - The programmed PASID is the mm_get_enqcmd_pasid()
> - Nothing changes while SVA is running (sva_enable)
>
> Also, SVA invalidation will still iterate over the S1 domain's master
> list.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 39 +++++++++----------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 29 ++++++++++++++
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 +++
> 3 files changed, 53 insertions(+), 21 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 159350a78d4cf9..e1fa2074c0b37b 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -416,12 +416,10 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn)
> arm_smmu_free_shared_cd(cd);
> }
>
> -static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid,
> - struct mm_struct *mm)
> +static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm,
> + struct arm_smmu_cd *target)
> {
> int ret;
> - struct arm_smmu_cd target;
> - struct arm_smmu_cd *cdptr;
> struct arm_smmu_bond *bond;
> struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
> @@ -448,19 +446,10 @@ static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid,
> goto err_free_bond;
> }
>
> - cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm));
> - if (!cdptr) {
> - ret = -ENOMEM;
> - goto err_put_notifier;
> - }
> - arm_smmu_make_sva_cd(&target, master, mm, bond->smmu_mn->cd->asid);
> - arm_smmu_write_cd_entry(master, pasid, cdptr, &target);
> -
> + arm_smmu_make_sva_cd(target, master, mm, bond->smmu_mn->cd->asid);
This can probably already move out to arm_smmu_sva_set_dev_pasid in
this commit. Removing the arm_smmu_get_cd_ptr pre-alloc might also be
possible if we're careful with failure of arm_smmu_set_pasid.
This is eventually addressed in "iommu/arm-smmu-v3: Put the SVA mmu
notifier in the smmu_domain", but that patch is already big enough as
it is and the change fits nicely here.
> list_add(&bond->list, &master->bonds);
> return 0;
>
> -err_put_notifier:
> - arm_smmu_mmu_notifier_put(bond->smmu_mn);
> err_free_bond:
> kfree(bond);
> return ret;
> @@ -621,10 +610,9 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
> struct arm_smmu_bond *bond = NULL, *t;
> struct arm_smmu_master *master = dev_iommu_priv_get(dev);
>
> + arm_smmu_remove_pasid(master, to_smmu_domain(domain), id);
> +
> mutex_lock(&sva_lock);
> -
> - arm_smmu_clear_cd(master, id);
> -
> list_for_each_entry(t, &master->bonds, list) {
> if (t->mm == mm) {
> bond = t;
> @@ -643,17 +631,26 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
> static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> struct device *dev, ioasid_t id)
> {
> + struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> int ret = 0;
> struct mm_struct *mm = domain->mm;
> + struct arm_smmu_cd target;
>
> if (mm_get_enqcmd_pasid(mm) != id)
> return -EINVAL;
>
> - mutex_lock(&sva_lock);
> - ret = __arm_smmu_sva_bind(dev, id, mm);
> - mutex_unlock(&sva_lock);
> + if (!arm_smmu_get_cd_ptr(master, id))
> + return -ENOMEM;
>
> - return ret;
> + mutex_lock(&sva_lock);
> + ret = __arm_smmu_sva_bind(dev, mm, &target);
> + mutex_unlock(&sva_lock);
> + if (ret)
> + return ret;
> +
> + /* This cannot fail since we preallocated the cdptr */
> + arm_smmu_set_pasid(master, to_smmu_domain(domain), id, &target);
> + return 0;
> }
>
> static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index dfdd48cf217c4e..7a0be7dd719793 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2656,6 +2656,35 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> return 0;
> }
>
> +static bool arm_smmu_is_s1_domain(struct iommu_domain *domain)
> +{
> + if (!domain || !(domain->type & __IOMMU_DOMAIN_PAGING))
> + return false;
> + return to_smmu_domain(domain)->stage == ARM_SMMU_DOMAIN_S1;
> +}
> +
> +int arm_smmu_set_pasid(struct arm_smmu_master *master,
> + struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
> + const struct arm_smmu_cd *cd)
> +{
> + struct arm_smmu_cd *cdptr;
> +
> + if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)))
> + return -ENODEV;
> +
> + cdptr = arm_smmu_get_cd_ptr(master, pasid);
> + if (!cdptr)
> + return -ENOMEM;
> + arm_smmu_write_cd_entry(master, pasid, cdptr, cd);
> + return 0;
> +}
> +
> +void arm_smmu_remove_pasid(struct arm_smmu_master *master,
> + struct arm_smmu_domain *smmu_domain, ioasid_t pasid)
> +{
> + arm_smmu_clear_cd(master, pasid);
> +}
> +
> static int arm_smmu_attach_dev_ste(struct device *dev,
> struct arm_smmu_ste *ste)
> {
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 468cd33b80ac35..ab0a6587e07ec8 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -753,6 +753,12 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_cd *cdptr,
> const struct arm_smmu_cd *target);
>
> +int arm_smmu_set_pasid(struct arm_smmu_master *master,
> + struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
> + const struct arm_smmu_cd *cd);
> +void arm_smmu_remove_pasid(struct arm_smmu_master *master,
> + struct arm_smmu_domain *smmu_domain, ioasid_t pasid);
> +
> void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
> void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
> size_t granule, bool leaf,
> --
> 2.43.2
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 21/27] iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain
2024-03-04 23:44 ` Jason Gunthorpe
@ 2024-03-19 16:23 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 16:23 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> This removes all the notifier de-duplication logic in the driver and
> relies on the core code to de-duplicate and allocate only one SVA domain
> per mm per smmu instance. This naturally gives a 1:1 relationship between
> SVA domain and mmu notifier.
>
> Remove all of the previous mmu_notifier, bond, shared cd, and cd refcount
> logic entirely.
>
> For the purpose of organizing patches lightly remove BTM support. The next
> patches will add it back in. BTM is a performance optimization so this is
> bisection friendly functionally invisible change. Note that the current
> code never even enables ARM_SMMU_FEAT_BTM so none of this code is ever
> even used.
>
> The bond/shared_cd/btm/asid allocator are tightly wound together and
> changing them all at once would make this patch too big. The core issue is
> that having a single SVA domain per-smmu instance conflicts with the
> design of having a global ASID table that BTM currently needs, as we would
> end up having to assign multiple SVA domains to the same ASID.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 389 ++++--------------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 81 +---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 14 +-
> 3 files changed, 101 insertions(+), 383 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 30b1cf587efffb..cb0da4e5a5517a 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -13,29 +13,9 @@
> #include "../../iommu-sva.h"
> #include "../../io-pgtable-arm.h"
>
> -struct arm_smmu_mmu_notifier {
> - struct mmu_notifier mn;
> - struct arm_smmu_ctx_desc *cd;
> - bool cleared;
> - refcount_t refs;
> - struct list_head list;
> - struct arm_smmu_domain *domain;
> -};
> -
> -#define mn_to_smmu(mn) container_of(mn, struct arm_smmu_mmu_notifier, mn)
> -
> -struct arm_smmu_bond {
> - struct mm_struct *mm;
> - struct arm_smmu_mmu_notifier *smmu_mn;
> - struct list_head list;
> -};
> -
> -#define sva_to_bond(handle) \
> - container_of(handle, struct arm_smmu_bond, sva)
> -
> static DEFINE_MUTEX(sva_lock);
>
> -static void
> +static void __maybe_unused
> arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> {
> struct arm_smmu_master_domain *master_domain;
> @@ -58,58 +38,6 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> }
>
> -/*
> - * Check if the CPU ASID is available on the SMMU side. If a private context
> - * descriptor is using it, try to replace it.
> - */
> -static struct arm_smmu_ctx_desc *
> -arm_smmu_share_asid(struct mm_struct *mm, u16 asid)
> -{
> - int ret;
> - u32 new_asid;
> - struct arm_smmu_ctx_desc *cd;
> - struct arm_smmu_device *smmu;
> - struct arm_smmu_domain *smmu_domain;
> -
> - cd = xa_load(&arm_smmu_asid_xa, asid);
> - if (!cd)
> - return NULL;
> -
> - if (cd->mm) {
> - if (WARN_ON(cd->mm != mm))
> - return ERR_PTR(-EINVAL);
> - /* All devices bound to this mm use the same cd struct. */
> - refcount_inc(&cd->refs);
> - return cd;
> - }
> -
> - smmu_domain = container_of(cd, struct arm_smmu_domain, cd);
> - smmu = smmu_domain->smmu;
> -
> - ret = xa_alloc(&arm_smmu_asid_xa, &new_asid, cd,
> - XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> - if (ret)
> - return ERR_PTR(-ENOSPC);
> - /*
> - * Race with unmap: TLB invalidations will start targeting the new ASID,
> - * which isn't assigned yet. We'll do an invalidate-all on the old ASID
> - * later, so it doesn't matter.
> - */
> - cd->asid = new_asid;
> - /*
> - * Update ASID and invalidate CD in all associated masters. There will
> - * be some overlap between use of both ASIDs, until we invalidate the
> - * TLB.
> - */
> - arm_smmu_update_s1_domain_cd_entry(smmu_domain);
> -
> - /* Invalidate TLB entries previously associated with that context */
> - arm_smmu_tlb_inv_asid(smmu, asid);
> -
> - xa_erase(&arm_smmu_asid_xa, asid);
> - return NULL;
> -}
> -
> static u64 page_size_to_cd(void)
> {
> static_assert(PAGE_SIZE == SZ_4K || PAGE_SIZE == SZ_16K ||
> @@ -123,7 +51,8 @@ static u64 page_size_to_cd(void)
>
> static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target,
> struct arm_smmu_master *master,
> - struct mm_struct *mm, u16 asid)
> + struct mm_struct *mm, u16 asid,
> + bool btm_invalidation)
> {
> u64 par;
>
> @@ -144,7 +73,7 @@ static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target,
> (master->stall_enabled ? CTXDESC_CD_0_S : 0) |
> CTXDESC_CD_0_R |
> CTXDESC_CD_0_A |
> - CTXDESC_CD_0_ASET |
> + (btm_invalidation ? 0 : CTXDESC_CD_0_ASET) |
> FIELD_PREP(CTXDESC_CD_0_ASID, asid));
>
> /*
> @@ -186,69 +115,6 @@ static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target,
> target->data[3] = cpu_to_le64(read_sysreg(mair_el1));
> }
>
> -static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm)
> -{
> - u16 asid;
> - int err = 0;
> - struct arm_smmu_ctx_desc *cd;
> - struct arm_smmu_ctx_desc *ret = NULL;
> -
> - /* Don't free the mm until we release the ASID */
> - mmgrab(mm);
> -
> - asid = arm64_mm_context_get(mm);
> - if (!asid) {
> - err = -ESRCH;
> - goto out_drop_mm;
> - }
> -
> - cd = kzalloc(sizeof(*cd), GFP_KERNEL);
> - if (!cd) {
> - err = -ENOMEM;
> - goto out_put_context;
> - }
> -
> - refcount_set(&cd->refs, 1);
> -
> - mutex_lock(&arm_smmu_asid_lock);
> - ret = arm_smmu_share_asid(mm, asid);
> - if (ret) {
> - mutex_unlock(&arm_smmu_asid_lock);
> - goto out_free_cd;
> - }
> -
> - err = xa_insert(&arm_smmu_asid_xa, asid, cd, GFP_KERNEL);
> - mutex_unlock(&arm_smmu_asid_lock);
> -
> - if (err)
> - goto out_free_asid;
> -
> - cd->asid = asid;
> - cd->mm = mm;
> -
> - return cd;
> -
> -out_free_asid:
> - arm_smmu_free_asid(cd);
> -out_free_cd:
> - kfree(cd);
> -out_put_context:
> - arm64_mm_context_put(mm);
> -out_drop_mm:
> - mmdrop(mm);
> - return err < 0 ? ERR_PTR(err) : ret;
> -}
> -
> -static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd)
> -{
> - if (arm_smmu_free_asid(cd)) {
> - /* Unpin ASID */
> - arm64_mm_context_put(cd->mm);
> - mmdrop(cd->mm);
> - kfree(cd);
> - }
> -}
> -
> /*
> * Cloned from the MAX_TLBI_OPS in arch/arm64/include/asm/tlbflush.h, this
> * is used as a threshold to replace per-page TLBI commands to issue in the
> @@ -263,8 +129,8 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
> unsigned long start,
> unsigned long end)
> {
> - struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn);
> - struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
> + struct arm_smmu_domain *smmu_domain =
> + container_of(mn, struct arm_smmu_domain, mmu_notifier);
> size_t size;
>
> /*
> @@ -281,34 +147,27 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
> size = 0;
> }
>
> - if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM)) {
> + if (!smmu_domain->btm_invalidation) {
> if (!size)
> arm_smmu_tlb_inv_asid(smmu_domain->smmu,
> - smmu_mn->cd->asid);
> + smmu_domain->cd.asid);
> else
> arm_smmu_tlb_inv_range_asid(start, size,
> - smmu_mn->cd->asid,
> + smmu_domain->cd.asid,
> PAGE_SIZE, false,
> smmu_domain);
> }
>
> - arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), start,
> - size);
> + arm_smmu_atc_inv_domain(smmu_domain, start, size);
> }
>
> static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
> {
> - struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn);
> - struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
> + struct arm_smmu_domain *smmu_domain =
> + container_of(mn, struct arm_smmu_domain, mmu_notifier);
> struct arm_smmu_master_domain *master_domain;
> unsigned long flags;
>
> - mutex_lock(&sva_lock);
> - if (smmu_mn->cleared) {
> - mutex_unlock(&sva_lock);
> - return;
> - }
> -
> /*
> * DMA may still be running. Keep the cd valid to avoid C_BAD_CD events,
> * but disable translation.
> @@ -320,25 +179,27 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
> struct arm_smmu_cd target;
> struct arm_smmu_cd *cdptr;
>
> - cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm));
> + cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid);
> if (WARN_ON(!cdptr))
> continue;
> - arm_smmu_make_sva_cd(&target, master, NULL, smmu_mn->cd->asid);
> - arm_smmu_write_cd_entry(master, mm_get_enqcmd_pasid(mm), cdptr,
> + arm_smmu_make_sva_cd(&target, master, NULL,
> + smmu_domain->cd.asid,
> + smmu_domain->btm_invalidation);
> + arm_smmu_write_cd_entry(master, master_domain->ssid, cdptr,
> &target);
> }
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
>
> - arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid);
> - arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0);
> -
> - smmu_mn->cleared = true;
> - mutex_unlock(&sva_lock);
> + arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
> + arm_smmu_atc_inv_domain(smmu_domain, 0, 0);
> }
>
> static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn)
> {
> - kfree(mn_to_smmu(mn));
> + struct arm_smmu_domain *smmu_domain =
> + container_of(mn, struct arm_smmu_domain, mmu_notifier);
> +
> + kfree(smmu_domain);
> }
>
> static const struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = {
> @@ -347,113 +208,6 @@ static const struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = {
> .free_notifier = arm_smmu_mmu_notifier_free,
> };
>
> -/* Allocate or get existing MMU notifier for this {domain, mm} pair */
> -static struct arm_smmu_mmu_notifier *
> -arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain,
> - struct mm_struct *mm)
> -{
> - int ret;
> - struct arm_smmu_ctx_desc *cd;
> - struct arm_smmu_mmu_notifier *smmu_mn;
> -
> - list_for_each_entry(smmu_mn, &smmu_domain->mmu_notifiers, list) {
> - if (smmu_mn->mn.mm == mm) {
> - refcount_inc(&smmu_mn->refs);
> - return smmu_mn;
> - }
> - }
> -
> - cd = arm_smmu_alloc_shared_cd(mm);
> - if (IS_ERR(cd))
> - return ERR_CAST(cd);
> -
> - smmu_mn = kzalloc(sizeof(*smmu_mn), GFP_KERNEL);
> - if (!smmu_mn) {
> - ret = -ENOMEM;
> - goto err_free_cd;
> - }
> -
> - refcount_set(&smmu_mn->refs, 1);
> - smmu_mn->cd = cd;
> - smmu_mn->domain = smmu_domain;
> - smmu_mn->mn.ops = &arm_smmu_mmu_notifier_ops;
> -
> - ret = mmu_notifier_register(&smmu_mn->mn, mm);
> - if (ret) {
> - kfree(smmu_mn);
> - goto err_free_cd;
> - }
> -
> - list_add(&smmu_mn->list, &smmu_domain->mmu_notifiers);
> - return smmu_mn;
> -
> -err_free_cd:
> - arm_smmu_free_shared_cd(cd);
> - return ERR_PTR(ret);
> -}
> -
> -static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn)
> -{
> - struct mm_struct *mm = smmu_mn->mn.mm;
> - struct arm_smmu_ctx_desc *cd = smmu_mn->cd;
> - struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
> -
> - if (!refcount_dec_and_test(&smmu_mn->refs))
> - return;
> -
> - list_del(&smmu_mn->list);
> -
> - /*
> - * If we went through clear(), we've already invalidated, and no
> - * new TLB entry can have been formed.
> - */
> - if (!smmu_mn->cleared) {
> - arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid);
> - arm_smmu_atc_inv_domain_sva(smmu_domain,
> - mm_get_enqcmd_pasid(mm), 0, 0);
> - }
> -
> - /* Frees smmu_mn */
> - mmu_notifier_put(&smmu_mn->mn);
> - arm_smmu_free_shared_cd(cd);
> -}
> -
> -static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm,
> - struct arm_smmu_cd *target)
> -{
> - int ret;
> - struct arm_smmu_bond *bond;
> - struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> - struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
> - struct arm_smmu_domain *smmu_domain;
> -
> - if (!(domain->type & __IOMMU_DOMAIN_PAGING))
> - return -ENODEV;
> - smmu_domain = to_smmu_domain(domain);
> - if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1)
> - return -ENODEV;
> -
> - bond = kzalloc(sizeof(*bond), GFP_KERNEL);
> - if (!bond)
> - return -ENOMEM;
> -
> - bond->mm = mm;
> -
> - bond->smmu_mn = arm_smmu_mmu_notifier_get(smmu_domain, mm);
> - if (IS_ERR(bond->smmu_mn)) {
> - ret = PTR_ERR(bond->smmu_mn);
> - goto err_free_bond;
> - }
> -
> - arm_smmu_make_sva_cd(target, master, mm, bond->smmu_mn->cd->asid);
> - list_add(&bond->list, &master->bonds);
> - return 0;
> -
> -err_free_bond:
> - kfree(bond);
> - return ret;
> -}
> -
> bool arm_smmu_sva_supported(struct arm_smmu_device *smmu)
> {
> unsigned long reg, fld;
> @@ -581,11 +335,6 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master)
> int arm_smmu_master_disable_sva(struct arm_smmu_master *master)
> {
> mutex_lock(&sva_lock);
> - if (!list_empty(&master->bonds)) {
> - dev_err(master->dev, "cannot disable SVA, device is bound\n");
> - mutex_unlock(&sva_lock);
> - return -EBUSY;
> - }
> arm_smmu_master_sva_disable_iopf(master);
> master->sva_enabled = false;
> mutex_unlock(&sva_lock);
> @@ -602,59 +351,54 @@ void arm_smmu_sva_notifier_synchronize(void)
> mmu_notifier_synchronize();
> }
>
> -void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
> - struct device *dev, ioasid_t id)
> -{
> - struct mm_struct *mm = domain->mm;
> - struct arm_smmu_bond *bond = NULL, *t;
> - struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> -
> - arm_smmu_remove_pasid(master, to_smmu_domain(domain), id);
> -
> - mutex_lock(&sva_lock);
> - list_for_each_entry(t, &master->bonds, list) {
> - if (t->mm == mm) {
> - bond = t;
> - break;
> - }
> - }
> -
> - if (!WARN_ON(!bond)) {
> - list_del(&bond->list);
> - arm_smmu_mmu_notifier_put(bond->smmu_mn);
> - kfree(bond);
> - }
> - mutex_unlock(&sva_lock);
> -}
> -
> static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> struct device *dev, ioasid_t id)
> {
> + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> - int ret = 0;
> - struct mm_struct *mm = domain->mm;
> struct arm_smmu_cd target;
> + int ret;
>
> - if (mm_get_enqcmd_pasid(mm) != id || !master->cd_table.used_sid)
> + /* Prevent arm_smmu_mm_release from being called while we are attaching */
> + if (!mmget_not_zero(domain->mm))
> return -EINVAL;
>
> - if (!arm_smmu_get_cd_ptr(master, id))
> - return -ENOMEM;
> + /*
> + * This does not need the arm_smmu_asid_lock because SVA domains never
> + * get reassigned
> + */
> + arm_smmu_make_sva_cd(&target, master, smmu_domain->domain.mm,
> + smmu_domain->cd.asid,
> + smmu_domain->btm_invalidation);
>
> - mutex_lock(&sva_lock);
> - ret = __arm_smmu_sva_bind(dev, mm, &target);
> - mutex_unlock(&sva_lock);
> - if (ret)
> - return ret;
> + ret = arm_smmu_set_pasid(master, smmu_domain, id, &target);
>
> - /* This cannot fail since we preallocated the cdptr */
> - arm_smmu_set_pasid(master, to_smmu_domain(domain), id, &target);
> - return 0;
> + mmput(domain->mm);
> + return ret;
> }
>
> static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
> {
> - kfree(to_smmu_domain(domain));
> + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> +
> + /*
> + * Ensure the ASID is empty in the iommu cache before allowing reuse.
> + */
> + arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
> +
> + /*
> + * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can
> + * still be called/running at this point. We allow the ASID to be
> + * reused, and if there is a race then it just suffers harmless
> + * unnecessary invalidation.
> + */
> + xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
> +
> + /*
> + * Actual free is defered to the SRCU callback
> + * arm_smmu_mmu_notifier_free()
> + */
> + mmu_notifier_put(&smmu_domain->mmu_notifier);
> }
I'm far from familiar with mmu_notifier and how the sva layers manage
its reference count....but is calling mmu_notifier_unregister instead
of mmu_notifier_put possible here? Such that invalidating the TL,
releasing the allocated ASID, and freeing the smmu_domain can take
place here after the mmu_notifier is gone.
>
> static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
> @@ -668,6 +412,8 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> struct arm_smmu_device *smmu = master->smmu;
> struct arm_smmu_domain *smmu_domain;
> + u32 asid;
> + int ret;
>
> smmu_domain = arm_smmu_domain_alloc();
> if (IS_ERR(smmu_domain))
> @@ -677,5 +423,22 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
> smmu_domain->smmu = smmu;
>
> + ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
> + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> + if (ret)
> + goto err_free;
> +
> + smmu_domain->cd.asid = asid;
> + smmu_domain->mmu_notifier.ops = &arm_smmu_mmu_notifier_ops;
> + ret = mmu_notifier_register(&smmu_domain->mmu_notifier, mm);
> + if (ret)
> + goto err_asid;
> +
> return &smmu_domain->domain;
> +
> +err_asid:
> + xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
> +err_free:
> + kfree(smmu_domain);
> + return ERR_PTR(ret);
> }
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index a255a02a5fc8a9..5642321b2124d9 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1436,22 +1436,6 @@ static void arm_smmu_free_cd_tables(struct arm_smmu_master *master)
> cd_table->cdtab = NULL;
> }
>
> -bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd)
> -{
> - bool free;
> - struct arm_smmu_ctx_desc *old_cd;
> -
> - if (!cd->asid)
> - return false;
> -
> - free = refcount_dec_and_test(&cd->refs);
> - if (free) {
> - old_cd = xa_erase(&arm_smmu_asid_xa, cd->asid);
> - WARN_ON(old_cd != cd);
> - }
> - return free;
> -}
> -
> /* Stream table manipulation functions */
> static void
> arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc)
> @@ -2042,8 +2026,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master,
> return arm_smmu_cmdq_batch_submit(master->smmu, &cmds);
> }
>
> -static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
> - ioasid_t ssid, unsigned long iova, size_t size)
> +int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
> + unsigned long iova, size_t size)
> {
> struct arm_smmu_master_domain *master_domain;
> int i;
> @@ -2081,15 +2065,7 @@ static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
> if (!master->ats_enabled)
> continue;
>
> - /*
> - * Non-zero ssid means SVA is co-opting the S1 domain to issue
> - * invalidations for SVA PASIDs.
> - */
> - if (ssid != IOMMU_NO_PASID)
> - arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd);
> - else
> - arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size,
> - &cmd);
> + arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size, &cmd);
>
> for (i = 0; i < master->num_streams; i++) {
> cmd.atc.sid = master->streams[i].id;
> @@ -2101,19 +2077,6 @@ static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
> return arm_smmu_cmdq_batch_submit(smmu_domain->smmu, &cmds);
> }
>
> -static int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
> - unsigned long iova, size_t size)
> -{
> - return __arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova,
> - size);
> -}
> -
> -int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain,
> - ioasid_t ssid, unsigned long iova, size_t size)
> -{
> - return __arm_smmu_atc_inv_domain(smmu_domain, ssid, iova, size);
> -}
> -
> /* IO_PGTABLE API */
> static void arm_smmu_tlb_inv_context(void *cookie)
> {
> @@ -2302,7 +2265,6 @@ struct arm_smmu_domain *arm_smmu_domain_alloc(void)
> mutex_init(&smmu_domain->init_mutex);
> INIT_LIST_HEAD(&smmu_domain->devices);
> spin_lock_init(&smmu_domain->devices_lock);
> - INIT_LIST_HEAD(&smmu_domain->mmu_notifiers);
>
> return smmu_domain;
> }
> @@ -2345,7 +2307,7 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
> if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
> /* Prevent SVA from touching the CD while we're freeing it */
> mutex_lock(&arm_smmu_asid_lock);
> - arm_smmu_free_asid(&smmu_domain->cd);
> + xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
> mutex_unlock(&arm_smmu_asid_lock);
> } else {
> struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
> @@ -2363,11 +2325,9 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
> u32 asid;
> struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
>
> - refcount_set(&cd->refs, 1);
> -
> /* Prevent SVA from modifying the ASID until it is written to the CD */
> mutex_lock(&arm_smmu_asid_lock);
> - ret = xa_alloc(&arm_smmu_asid_xa, &asid, cd,
> + ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
> XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> cd->asid = (u16)asid;
> mutex_unlock(&arm_smmu_asid_lock);
> @@ -2818,7 +2778,11 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
> struct arm_smmu_cd *cdptr;
> int ret;
>
> - if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)))
> + if (smmu_domain->smmu != master->smmu)
> + return -EINVAL;
> +
> + if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)) ||
> + !master->cd_table.used_sid)
> return -ENODEV;
>
> cdptr = arm_smmu_get_cd_ptr(master, pasid);
> @@ -2839,9 +2803,18 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
> return ret;
> }
>
> -void arm_smmu_remove_pasid(struct arm_smmu_master *master,
> - struct arm_smmu_domain *smmu_domain, ioasid_t pasid)
> +static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
> {
> + struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> + struct arm_smmu_domain *smmu_domain;
> + struct iommu_domain *domain;
> +
> + domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA);
> + if (WARN_ON(IS_ERR(domain)) || !domain)
> + return;
> +
> + smmu_domain = to_smmu_domain(domain);
> +
> mutex_lock(&arm_smmu_asid_lock);
> arm_smmu_clear_cd(master, pasid);
> if (master->ats_enabled)
> @@ -3115,7 +3088,6 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev)
>
> master->dev = dev;
> master->smmu = smmu;
> - INIT_LIST_HEAD(&master->bonds);
> dev_iommu_priv_set(dev, master);
>
> ret = arm_smmu_insert_master(smmu, master);
> @@ -3296,17 +3268,6 @@ static int arm_smmu_def_domain_type(struct device *dev)
> return 0;
> }
>
> -static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
> -{
> - struct iommu_domain *domain;
> -
> - domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA);
> - if (WARN_ON(IS_ERR(domain)) || !domain)
> - return;
> -
> - arm_smmu_sva_remove_dev_pasid(domain, dev, pasid);
> -}
> -
> static struct iommu_ops arm_smmu_ops = {
> .identity_domain = &arm_smmu_identity_domain,
> .blocked_domain = &arm_smmu_blocked_domain,
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index f62bd2ef36a18d..cfae4d69cd810c 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -587,9 +587,6 @@ struct arm_smmu_strtab_l1_desc {
>
> struct arm_smmu_ctx_desc {
> u16 asid;
> -
> - refcount_t refs;
> - struct mm_struct *mm;
> };
>
> struct arm_smmu_l1_ctx_desc {
> @@ -713,7 +710,6 @@ struct arm_smmu_master {
> bool stall_enabled;
> bool sva_enabled;
> bool iopf_enabled;
> - struct list_head bonds;
> unsigned int ssid_bits;
> };
>
> @@ -742,7 +738,8 @@ struct arm_smmu_domain {
> struct list_head devices;
> spinlock_t devices_lock;
>
> - struct list_head mmu_notifiers;
> + struct mmu_notifier mmu_notifier;
> + bool btm_invalidation;
> };
>
> struct arm_smmu_master_domain {
> @@ -781,9 +778,8 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
> void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
> size_t granule, bool leaf,
> struct arm_smmu_domain *smmu_domain);
> -bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd);
> -int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain,
> - ioasid_t ssid, unsigned long iova, size_t size);
> +int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
> + unsigned long iova, size_t size);
>
> #ifdef CONFIG_ARM_SMMU_V3_SVA
> bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
> @@ -795,8 +791,6 @@ bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master);
> void arm_smmu_sva_notifier_synchronize(void);
> struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> struct mm_struct *mm);
> -void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
> - struct device *dev, ioasid_t id);
> #else /* CONFIG_ARM_SMMU_V3_SVA */
> static inline bool arm_smmu_sva_supported(struct arm_smmu_device *smmu)
> {
> --
> 2.43.2
>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 21/27] iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain
@ 2024-03-19 16:23 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 16:23 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> This removes all the notifier de-duplication logic in the driver and
> relies on the core code to de-duplicate and allocate only one SVA domain
> per mm per smmu instance. This naturally gives a 1:1 relationship between
> SVA domain and mmu notifier.
>
> Remove all of the previous mmu_notifier, bond, shared cd, and cd refcount
> logic entirely.
>
> For the purpose of organizing patches lightly remove BTM support. The next
> patches will add it back in. BTM is a performance optimization so this is
> bisection friendly functionally invisible change. Note that the current
> code never even enables ARM_SMMU_FEAT_BTM so none of this code is ever
> even used.
>
> The bond/shared_cd/btm/asid allocator are tightly wound together and
> changing them all at once would make this patch too big. The core issue is
> that having a single SVA domain per-smmu instance conflicts with the
> design of having a global ASID table that BTM currently needs, as we would
> end up having to assign multiple SVA domains to the same ASID.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 389 ++++--------------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 81 +---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 14 +-
> 3 files changed, 101 insertions(+), 383 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 30b1cf587efffb..cb0da4e5a5517a 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -13,29 +13,9 @@
> #include "../../iommu-sva.h"
> #include "../../io-pgtable-arm.h"
>
> -struct arm_smmu_mmu_notifier {
> - struct mmu_notifier mn;
> - struct arm_smmu_ctx_desc *cd;
> - bool cleared;
> - refcount_t refs;
> - struct list_head list;
> - struct arm_smmu_domain *domain;
> -};
> -
> -#define mn_to_smmu(mn) container_of(mn, struct arm_smmu_mmu_notifier, mn)
> -
> -struct arm_smmu_bond {
> - struct mm_struct *mm;
> - struct arm_smmu_mmu_notifier *smmu_mn;
> - struct list_head list;
> -};
> -
> -#define sva_to_bond(handle) \
> - container_of(handle, struct arm_smmu_bond, sva)
> -
> static DEFINE_MUTEX(sva_lock);
>
> -static void
> +static void __maybe_unused
> arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> {
> struct arm_smmu_master_domain *master_domain;
> @@ -58,58 +38,6 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> }
>
> -/*
> - * Check if the CPU ASID is available on the SMMU side. If a private context
> - * descriptor is using it, try to replace it.
> - */
> -static struct arm_smmu_ctx_desc *
> -arm_smmu_share_asid(struct mm_struct *mm, u16 asid)
> -{
> - int ret;
> - u32 new_asid;
> - struct arm_smmu_ctx_desc *cd;
> - struct arm_smmu_device *smmu;
> - struct arm_smmu_domain *smmu_domain;
> -
> - cd = xa_load(&arm_smmu_asid_xa, asid);
> - if (!cd)
> - return NULL;
> -
> - if (cd->mm) {
> - if (WARN_ON(cd->mm != mm))
> - return ERR_PTR(-EINVAL);
> - /* All devices bound to this mm use the same cd struct. */
> - refcount_inc(&cd->refs);
> - return cd;
> - }
> -
> - smmu_domain = container_of(cd, struct arm_smmu_domain, cd);
> - smmu = smmu_domain->smmu;
> -
> - ret = xa_alloc(&arm_smmu_asid_xa, &new_asid, cd,
> - XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> - if (ret)
> - return ERR_PTR(-ENOSPC);
> - /*
> - * Race with unmap: TLB invalidations will start targeting the new ASID,
> - * which isn't assigned yet. We'll do an invalidate-all on the old ASID
> - * later, so it doesn't matter.
> - */
> - cd->asid = new_asid;
> - /*
> - * Update ASID and invalidate CD in all associated masters. There will
> - * be some overlap between use of both ASIDs, until we invalidate the
> - * TLB.
> - */
> - arm_smmu_update_s1_domain_cd_entry(smmu_domain);
> -
> - /* Invalidate TLB entries previously associated with that context */
> - arm_smmu_tlb_inv_asid(smmu, asid);
> -
> - xa_erase(&arm_smmu_asid_xa, asid);
> - return NULL;
> -}
> -
> static u64 page_size_to_cd(void)
> {
> static_assert(PAGE_SIZE == SZ_4K || PAGE_SIZE == SZ_16K ||
> @@ -123,7 +51,8 @@ static u64 page_size_to_cd(void)
>
> static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target,
> struct arm_smmu_master *master,
> - struct mm_struct *mm, u16 asid)
> + struct mm_struct *mm, u16 asid,
> + bool btm_invalidation)
> {
> u64 par;
>
> @@ -144,7 +73,7 @@ static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target,
> (master->stall_enabled ? CTXDESC_CD_0_S : 0) |
> CTXDESC_CD_0_R |
> CTXDESC_CD_0_A |
> - CTXDESC_CD_0_ASET |
> + (btm_invalidation ? 0 : CTXDESC_CD_0_ASET) |
> FIELD_PREP(CTXDESC_CD_0_ASID, asid));
>
> /*
> @@ -186,69 +115,6 @@ static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target,
> target->data[3] = cpu_to_le64(read_sysreg(mair_el1));
> }
>
> -static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm)
> -{
> - u16 asid;
> - int err = 0;
> - struct arm_smmu_ctx_desc *cd;
> - struct arm_smmu_ctx_desc *ret = NULL;
> -
> - /* Don't free the mm until we release the ASID */
> - mmgrab(mm);
> -
> - asid = arm64_mm_context_get(mm);
> - if (!asid) {
> - err = -ESRCH;
> - goto out_drop_mm;
> - }
> -
> - cd = kzalloc(sizeof(*cd), GFP_KERNEL);
> - if (!cd) {
> - err = -ENOMEM;
> - goto out_put_context;
> - }
> -
> - refcount_set(&cd->refs, 1);
> -
> - mutex_lock(&arm_smmu_asid_lock);
> - ret = arm_smmu_share_asid(mm, asid);
> - if (ret) {
> - mutex_unlock(&arm_smmu_asid_lock);
> - goto out_free_cd;
> - }
> -
> - err = xa_insert(&arm_smmu_asid_xa, asid, cd, GFP_KERNEL);
> - mutex_unlock(&arm_smmu_asid_lock);
> -
> - if (err)
> - goto out_free_asid;
> -
> - cd->asid = asid;
> - cd->mm = mm;
> -
> - return cd;
> -
> -out_free_asid:
> - arm_smmu_free_asid(cd);
> -out_free_cd:
> - kfree(cd);
> -out_put_context:
> - arm64_mm_context_put(mm);
> -out_drop_mm:
> - mmdrop(mm);
> - return err < 0 ? ERR_PTR(err) : ret;
> -}
> -
> -static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd)
> -{
> - if (arm_smmu_free_asid(cd)) {
> - /* Unpin ASID */
> - arm64_mm_context_put(cd->mm);
> - mmdrop(cd->mm);
> - kfree(cd);
> - }
> -}
> -
> /*
> * Cloned from the MAX_TLBI_OPS in arch/arm64/include/asm/tlbflush.h, this
> * is used as a threshold to replace per-page TLBI commands to issue in the
> @@ -263,8 +129,8 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
> unsigned long start,
> unsigned long end)
> {
> - struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn);
> - struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
> + struct arm_smmu_domain *smmu_domain =
> + container_of(mn, struct arm_smmu_domain, mmu_notifier);
> size_t size;
>
> /*
> @@ -281,34 +147,27 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
> size = 0;
> }
>
> - if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM)) {
> + if (!smmu_domain->btm_invalidation) {
> if (!size)
> arm_smmu_tlb_inv_asid(smmu_domain->smmu,
> - smmu_mn->cd->asid);
> + smmu_domain->cd.asid);
> else
> arm_smmu_tlb_inv_range_asid(start, size,
> - smmu_mn->cd->asid,
> + smmu_domain->cd.asid,
> PAGE_SIZE, false,
> smmu_domain);
> }
>
> - arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), start,
> - size);
> + arm_smmu_atc_inv_domain(smmu_domain, start, size);
> }
>
> static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
> {
> - struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn);
> - struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
> + struct arm_smmu_domain *smmu_domain =
> + container_of(mn, struct arm_smmu_domain, mmu_notifier);
> struct arm_smmu_master_domain *master_domain;
> unsigned long flags;
>
> - mutex_lock(&sva_lock);
> - if (smmu_mn->cleared) {
> - mutex_unlock(&sva_lock);
> - return;
> - }
> -
> /*
> * DMA may still be running. Keep the cd valid to avoid C_BAD_CD events,
> * but disable translation.
> @@ -320,25 +179,27 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
> struct arm_smmu_cd target;
> struct arm_smmu_cd *cdptr;
>
> - cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm));
> + cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid);
> if (WARN_ON(!cdptr))
> continue;
> - arm_smmu_make_sva_cd(&target, master, NULL, smmu_mn->cd->asid);
> - arm_smmu_write_cd_entry(master, mm_get_enqcmd_pasid(mm), cdptr,
> + arm_smmu_make_sva_cd(&target, master, NULL,
> + smmu_domain->cd.asid,
> + smmu_domain->btm_invalidation);
> + arm_smmu_write_cd_entry(master, master_domain->ssid, cdptr,
> &target);
> }
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
>
> - arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid);
> - arm_smmu_atc_inv_domain_sva(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0);
> -
> - smmu_mn->cleared = true;
> - mutex_unlock(&sva_lock);
> + arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
> + arm_smmu_atc_inv_domain(smmu_domain, 0, 0);
> }
>
> static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn)
> {
> - kfree(mn_to_smmu(mn));
> + struct arm_smmu_domain *smmu_domain =
> + container_of(mn, struct arm_smmu_domain, mmu_notifier);
> +
> + kfree(smmu_domain);
> }
>
> static const struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = {
> @@ -347,113 +208,6 @@ static const struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = {
> .free_notifier = arm_smmu_mmu_notifier_free,
> };
>
> -/* Allocate or get existing MMU notifier for this {domain, mm} pair */
> -static struct arm_smmu_mmu_notifier *
> -arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain,
> - struct mm_struct *mm)
> -{
> - int ret;
> - struct arm_smmu_ctx_desc *cd;
> - struct arm_smmu_mmu_notifier *smmu_mn;
> -
> - list_for_each_entry(smmu_mn, &smmu_domain->mmu_notifiers, list) {
> - if (smmu_mn->mn.mm == mm) {
> - refcount_inc(&smmu_mn->refs);
> - return smmu_mn;
> - }
> - }
> -
> - cd = arm_smmu_alloc_shared_cd(mm);
> - if (IS_ERR(cd))
> - return ERR_CAST(cd);
> -
> - smmu_mn = kzalloc(sizeof(*smmu_mn), GFP_KERNEL);
> - if (!smmu_mn) {
> - ret = -ENOMEM;
> - goto err_free_cd;
> - }
> -
> - refcount_set(&smmu_mn->refs, 1);
> - smmu_mn->cd = cd;
> - smmu_mn->domain = smmu_domain;
> - smmu_mn->mn.ops = &arm_smmu_mmu_notifier_ops;
> -
> - ret = mmu_notifier_register(&smmu_mn->mn, mm);
> - if (ret) {
> - kfree(smmu_mn);
> - goto err_free_cd;
> - }
> -
> - list_add(&smmu_mn->list, &smmu_domain->mmu_notifiers);
> - return smmu_mn;
> -
> -err_free_cd:
> - arm_smmu_free_shared_cd(cd);
> - return ERR_PTR(ret);
> -}
> -
> -static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn)
> -{
> - struct mm_struct *mm = smmu_mn->mn.mm;
> - struct arm_smmu_ctx_desc *cd = smmu_mn->cd;
> - struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
> -
> - if (!refcount_dec_and_test(&smmu_mn->refs))
> - return;
> -
> - list_del(&smmu_mn->list);
> -
> - /*
> - * If we went through clear(), we've already invalidated, and no
> - * new TLB entry can have been formed.
> - */
> - if (!smmu_mn->cleared) {
> - arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid);
> - arm_smmu_atc_inv_domain_sva(smmu_domain,
> - mm_get_enqcmd_pasid(mm), 0, 0);
> - }
> -
> - /* Frees smmu_mn */
> - mmu_notifier_put(&smmu_mn->mn);
> - arm_smmu_free_shared_cd(cd);
> -}
> -
> -static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm,
> - struct arm_smmu_cd *target)
> -{
> - int ret;
> - struct arm_smmu_bond *bond;
> - struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> - struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
> - struct arm_smmu_domain *smmu_domain;
> -
> - if (!(domain->type & __IOMMU_DOMAIN_PAGING))
> - return -ENODEV;
> - smmu_domain = to_smmu_domain(domain);
> - if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1)
> - return -ENODEV;
> -
> - bond = kzalloc(sizeof(*bond), GFP_KERNEL);
> - if (!bond)
> - return -ENOMEM;
> -
> - bond->mm = mm;
> -
> - bond->smmu_mn = arm_smmu_mmu_notifier_get(smmu_domain, mm);
> - if (IS_ERR(bond->smmu_mn)) {
> - ret = PTR_ERR(bond->smmu_mn);
> - goto err_free_bond;
> - }
> -
> - arm_smmu_make_sva_cd(target, master, mm, bond->smmu_mn->cd->asid);
> - list_add(&bond->list, &master->bonds);
> - return 0;
> -
> -err_free_bond:
> - kfree(bond);
> - return ret;
> -}
> -
> bool arm_smmu_sva_supported(struct arm_smmu_device *smmu)
> {
> unsigned long reg, fld;
> @@ -581,11 +335,6 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master)
> int arm_smmu_master_disable_sva(struct arm_smmu_master *master)
> {
> mutex_lock(&sva_lock);
> - if (!list_empty(&master->bonds)) {
> - dev_err(master->dev, "cannot disable SVA, device is bound\n");
> - mutex_unlock(&sva_lock);
> - return -EBUSY;
> - }
> arm_smmu_master_sva_disable_iopf(master);
> master->sva_enabled = false;
> mutex_unlock(&sva_lock);
> @@ -602,59 +351,54 @@ void arm_smmu_sva_notifier_synchronize(void)
> mmu_notifier_synchronize();
> }
>
> -void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
> - struct device *dev, ioasid_t id)
> -{
> - struct mm_struct *mm = domain->mm;
> - struct arm_smmu_bond *bond = NULL, *t;
> - struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> -
> - arm_smmu_remove_pasid(master, to_smmu_domain(domain), id);
> -
> - mutex_lock(&sva_lock);
> - list_for_each_entry(t, &master->bonds, list) {
> - if (t->mm == mm) {
> - bond = t;
> - break;
> - }
> - }
> -
> - if (!WARN_ON(!bond)) {
> - list_del(&bond->list);
> - arm_smmu_mmu_notifier_put(bond->smmu_mn);
> - kfree(bond);
> - }
> - mutex_unlock(&sva_lock);
> -}
> -
> static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> struct device *dev, ioasid_t id)
> {
> + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> - int ret = 0;
> - struct mm_struct *mm = domain->mm;
> struct arm_smmu_cd target;
> + int ret;
>
> - if (mm_get_enqcmd_pasid(mm) != id || !master->cd_table.used_sid)
> + /* Prevent arm_smmu_mm_release from being called while we are attaching */
> + if (!mmget_not_zero(domain->mm))
> return -EINVAL;
>
> - if (!arm_smmu_get_cd_ptr(master, id))
> - return -ENOMEM;
> + /*
> + * This does not need the arm_smmu_asid_lock because SVA domains never
> + * get reassigned
> + */
> + arm_smmu_make_sva_cd(&target, master, smmu_domain->domain.mm,
> + smmu_domain->cd.asid,
> + smmu_domain->btm_invalidation);
>
> - mutex_lock(&sva_lock);
> - ret = __arm_smmu_sva_bind(dev, mm, &target);
> - mutex_unlock(&sva_lock);
> - if (ret)
> - return ret;
> + ret = arm_smmu_set_pasid(master, smmu_domain, id, &target);
>
> - /* This cannot fail since we preallocated the cdptr */
> - arm_smmu_set_pasid(master, to_smmu_domain(domain), id, &target);
> - return 0;
> + mmput(domain->mm);
> + return ret;
> }
>
> static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
> {
> - kfree(to_smmu_domain(domain));
> + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> +
> + /*
> + * Ensure the ASID is empty in the iommu cache before allowing reuse.
> + */
> + arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
> +
> + /*
> + * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can
> + * still be called/running at this point. We allow the ASID to be
> + * reused, and if there is a race then it just suffers harmless
> + * unnecessary invalidation.
> + */
> + xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
> +
> + /*
> + * Actual free is defered to the SRCU callback
> + * arm_smmu_mmu_notifier_free()
> + */
> + mmu_notifier_put(&smmu_domain->mmu_notifier);
> }
I'm far from familiar with mmu_notifier and how the sva layers manage
its reference count....but is calling mmu_notifier_unregister instead
of mmu_notifier_put possible here? Such that invalidating the TL,
releasing the allocated ASID, and freeing the smmu_domain can take
place here after the mmu_notifier is gone.
>
> static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
> @@ -668,6 +412,8 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> struct arm_smmu_device *smmu = master->smmu;
> struct arm_smmu_domain *smmu_domain;
> + u32 asid;
> + int ret;
>
> smmu_domain = arm_smmu_domain_alloc();
> if (IS_ERR(smmu_domain))
> @@ -677,5 +423,22 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
> smmu_domain->smmu = smmu;
>
> + ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
> + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> + if (ret)
> + goto err_free;
> +
> + smmu_domain->cd.asid = asid;
> + smmu_domain->mmu_notifier.ops = &arm_smmu_mmu_notifier_ops;
> + ret = mmu_notifier_register(&smmu_domain->mmu_notifier, mm);
> + if (ret)
> + goto err_asid;
> +
> return &smmu_domain->domain;
> +
> +err_asid:
> + xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
> +err_free:
> + kfree(smmu_domain);
> + return ERR_PTR(ret);
> }
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index a255a02a5fc8a9..5642321b2124d9 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1436,22 +1436,6 @@ static void arm_smmu_free_cd_tables(struct arm_smmu_master *master)
> cd_table->cdtab = NULL;
> }
>
> -bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd)
> -{
> - bool free;
> - struct arm_smmu_ctx_desc *old_cd;
> -
> - if (!cd->asid)
> - return false;
> -
> - free = refcount_dec_and_test(&cd->refs);
> - if (free) {
> - old_cd = xa_erase(&arm_smmu_asid_xa, cd->asid);
> - WARN_ON(old_cd != cd);
> - }
> - return free;
> -}
> -
> /* Stream table manipulation functions */
> static void
> arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc)
> @@ -2042,8 +2026,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master,
> return arm_smmu_cmdq_batch_submit(master->smmu, &cmds);
> }
>
> -static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
> - ioasid_t ssid, unsigned long iova, size_t size)
> +int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
> + unsigned long iova, size_t size)
> {
> struct arm_smmu_master_domain *master_domain;
> int i;
> @@ -2081,15 +2065,7 @@ static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
> if (!master->ats_enabled)
> continue;
>
> - /*
> - * Non-zero ssid means SVA is co-opting the S1 domain to issue
> - * invalidations for SVA PASIDs.
> - */
> - if (ssid != IOMMU_NO_PASID)
> - arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd);
> - else
> - arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size,
> - &cmd);
> + arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size, &cmd);
>
> for (i = 0; i < master->num_streams; i++) {
> cmd.atc.sid = master->streams[i].id;
> @@ -2101,19 +2077,6 @@ static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
> return arm_smmu_cmdq_batch_submit(smmu_domain->smmu, &cmds);
> }
>
> -static int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
> - unsigned long iova, size_t size)
> -{
> - return __arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova,
> - size);
> -}
> -
> -int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain,
> - ioasid_t ssid, unsigned long iova, size_t size)
> -{
> - return __arm_smmu_atc_inv_domain(smmu_domain, ssid, iova, size);
> -}
> -
> /* IO_PGTABLE API */
> static void arm_smmu_tlb_inv_context(void *cookie)
> {
> @@ -2302,7 +2265,6 @@ struct arm_smmu_domain *arm_smmu_domain_alloc(void)
> mutex_init(&smmu_domain->init_mutex);
> INIT_LIST_HEAD(&smmu_domain->devices);
> spin_lock_init(&smmu_domain->devices_lock);
> - INIT_LIST_HEAD(&smmu_domain->mmu_notifiers);
>
> return smmu_domain;
> }
> @@ -2345,7 +2307,7 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
> if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
> /* Prevent SVA from touching the CD while we're freeing it */
> mutex_lock(&arm_smmu_asid_lock);
> - arm_smmu_free_asid(&smmu_domain->cd);
> + xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
> mutex_unlock(&arm_smmu_asid_lock);
> } else {
> struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
> @@ -2363,11 +2325,9 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
> u32 asid;
> struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
>
> - refcount_set(&cd->refs, 1);
> -
> /* Prevent SVA from modifying the ASID until it is written to the CD */
> mutex_lock(&arm_smmu_asid_lock);
> - ret = xa_alloc(&arm_smmu_asid_xa, &asid, cd,
> + ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
> XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> cd->asid = (u16)asid;
> mutex_unlock(&arm_smmu_asid_lock);
> @@ -2818,7 +2778,11 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
> struct arm_smmu_cd *cdptr;
> int ret;
>
> - if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)))
> + if (smmu_domain->smmu != master->smmu)
> + return -EINVAL;
> +
> + if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)) ||
> + !master->cd_table.used_sid)
> return -ENODEV;
>
> cdptr = arm_smmu_get_cd_ptr(master, pasid);
> @@ -2839,9 +2803,18 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
> return ret;
> }
>
> -void arm_smmu_remove_pasid(struct arm_smmu_master *master,
> - struct arm_smmu_domain *smmu_domain, ioasid_t pasid)
> +static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
> {
> + struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> + struct arm_smmu_domain *smmu_domain;
> + struct iommu_domain *domain;
> +
> + domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA);
> + if (WARN_ON(IS_ERR(domain)) || !domain)
> + return;
> +
> + smmu_domain = to_smmu_domain(domain);
> +
> mutex_lock(&arm_smmu_asid_lock);
> arm_smmu_clear_cd(master, pasid);
> if (master->ats_enabled)
> @@ -3115,7 +3088,6 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev)
>
> master->dev = dev;
> master->smmu = smmu;
> - INIT_LIST_HEAD(&master->bonds);
> dev_iommu_priv_set(dev, master);
>
> ret = arm_smmu_insert_master(smmu, master);
> @@ -3296,17 +3268,6 @@ static int arm_smmu_def_domain_type(struct device *dev)
> return 0;
> }
>
> -static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
> -{
> - struct iommu_domain *domain;
> -
> - domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA);
> - if (WARN_ON(IS_ERR(domain)) || !domain)
> - return;
> -
> - arm_smmu_sva_remove_dev_pasid(domain, dev, pasid);
> -}
> -
> static struct iommu_ops arm_smmu_ops = {
> .identity_domain = &arm_smmu_identity_domain,
> .blocked_domain = &arm_smmu_blocked_domain,
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index f62bd2ef36a18d..cfae4d69cd810c 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -587,9 +587,6 @@ struct arm_smmu_strtab_l1_desc {
>
> struct arm_smmu_ctx_desc {
> u16 asid;
> -
> - refcount_t refs;
> - struct mm_struct *mm;
> };
>
> struct arm_smmu_l1_ctx_desc {
> @@ -713,7 +710,6 @@ struct arm_smmu_master {
> bool stall_enabled;
> bool sva_enabled;
> bool iopf_enabled;
> - struct list_head bonds;
> unsigned int ssid_bits;
> };
>
> @@ -742,7 +738,8 @@ struct arm_smmu_domain {
> struct list_head devices;
> spinlock_t devices_lock;
>
> - struct list_head mmu_notifiers;
> + struct mmu_notifier mmu_notifier;
> + bool btm_invalidation;
> };
>
> struct arm_smmu_master_domain {
> @@ -781,9 +778,8 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
> void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
> size_t granule, bool leaf,
> struct arm_smmu_domain *smmu_domain);
> -bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd);
> -int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain,
> - ioasid_t ssid, unsigned long iova, size_t size);
> +int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain,
> + unsigned long iova, size_t size);
>
> #ifdef CONFIG_ARM_SMMU_V3_SVA
> bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
> @@ -795,8 +791,6 @@ bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master);
> void arm_smmu_sva_notifier_synchronize(void);
> struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> struct mm_struct *mm);
> -void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
> - struct device *dev, ioasid_t id);
> #else /* CONFIG_ARM_SMMU_V3_SVA */
> static inline bool arm_smmu_sva_supported(struct arm_smmu_device *smmu)
> {
> --
> 2.43.2
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 22/27] iommu/arm-smmu-v3: Consolidate freeing the ASID/VMID
2024-03-04 23:44 ` Jason Gunthorpe
@ 2024-03-19 16:44 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 16:44 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> The SMMUv3 IOTLB is tagged with a VMID/ASID cache tag. Any time the
> underlying translation is changed these need to be invalidated. At boot
> time the IOTLB starts out empty and all cache tags are available for
> allocation.
>
> When a tag is taken out of the allocator the code assumes the IOTLB
> doesn't reference it, and immediately programs it into a STE/CD. If the
> cache is referencing the tag then it will have stale data and IOMMU will
> become incoherent.
>
> Thus, whenever an ASID/VMID is freed back to the allocator we need to know
> that the IOTLB doesn't have any references to it. The SVA code correctly
> had an invalidation here, but the paging code does not.
Isn't that....bad?
>
> Consolidate freeing the VMID/ASID to one place and consistently flush both
> ID types before returning to their allocators.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 9 ++---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 36 +++++++++++++------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 +
> 3 files changed, 29 insertions(+), 17 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index cb0da4e5a5517a..3a9f4ef47c6b6f 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -381,18 +381,13 @@ static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
> {
> struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
>
> - /*
> - * Ensure the ASID is empty in the iommu cache before allowing reuse.
> - */
> - arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
> -
> /*
> * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can
> * still be called/running at this point. We allow the ASID to be
> * reused, and if there is a race then it just suffers harmless
> * unnecessary invalidation.
> */
> - xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
> + arm_smmu_domain_free_id(smmu_domain);
>
> /*
> * Actual free is defered to the SRCU callback
> @@ -437,7 +432,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> return &smmu_domain->domain;
>
> err_asid:
> - xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
> + arm_smmu_domain_free_id(smmu_domain);
> err_free:
> kfree(smmu_domain);
> return ERR_PTR(ret);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 5642321b2124d9..4f22eb810c8dbd 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2296,25 +2296,41 @@ static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
> return &smmu_domain->domain;
> }
>
> -static void arm_smmu_domain_free(struct iommu_domain *domain)
> +/*
> + * Return the domain's ASID or VMID back to the allocator. All IDs in the
> + * allocator do not have an IOTLB entries referencing them.
> + */
> +void arm_smmu_domain_free_id(struct arm_smmu_domain *smmu_domain)
> {
> - struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> struct arm_smmu_device *smmu = smmu_domain->smmu;
>
> - free_io_pgtable_ops(smmu_domain->pgtbl_ops);
> + if ((smmu_domain->stage == ARM_SMMU_DOMAIN_S1 ||
> + smmu_domain->domain.type == IOMMU_DOMAIN_SVA) &&
> + smmu_domain->cd.asid) {
> + arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid);
>
> - /* Free the ASID or VMID */
> - if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
> /* Prevent SVA from touching the CD while we're freeing it */
> mutex_lock(&arm_smmu_asid_lock);
> xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
> mutex_unlock(&arm_smmu_asid_lock);
> - } else {
> - struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
> - if (cfg->vmid)
> - ida_free(&smmu->vmid_map, cfg->vmid);
> - }
> + } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S2 &&
> + smmu_domain->s2_cfg.vmid) {
> + struct arm_smmu_cmdq_ent cmd = {
> + .opcode = CMDQ_OP_TLBI_S12_VMALL,
> + .tlbi.vmid = smmu_domain->s2_cfg.vmid
> + };
>
> + arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
> + ida_free(&smmu->vmid_map, smmu_domain->s2_cfg.vmid);
> + }
> +}
> +
There's room to refactor and share the tlb id invalidation logic with
arm_smmu_tlb_inv_context by the way, but this works too.
> +static void arm_smmu_domain_free(struct iommu_domain *domain)
> +{
> + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> +
> + free_io_pgtable_ops(smmu_domain->pgtbl_ops);
> + arm_smmu_domain_free_id(smmu_domain);
> kfree(smmu_domain);
> }
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index cfae4d69cd810c..4631f0ac396dc3 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -774,6 +774,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
> void arm_smmu_remove_pasid(struct arm_smmu_master *master,
> struct arm_smmu_domain *smmu_domain, ioasid_t pasid);
>
> +void arm_smmu_domain_free_id(struct arm_smmu_domain *smmu_domain);
> void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
> void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
> size_t granule, bool leaf,
> --
> 2.43.2
>
>
Reviewed-by: Michael Shavit <mshavit@google.com>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 22/27] iommu/arm-smmu-v3: Consolidate freeing the ASID/VMID
@ 2024-03-19 16:44 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 16:44 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> The SMMUv3 IOTLB is tagged with a VMID/ASID cache tag. Any time the
> underlying translation is changed these need to be invalidated. At boot
> time the IOTLB starts out empty and all cache tags are available for
> allocation.
>
> When a tag is taken out of the allocator the code assumes the IOTLB
> doesn't reference it, and immediately programs it into a STE/CD. If the
> cache is referencing the tag then it will have stale data and IOMMU will
> become incoherent.
>
> Thus, whenever an ASID/VMID is freed back to the allocator we need to know
> that the IOTLB doesn't have any references to it. The SVA code correctly
> had an invalidation here, but the paging code does not.
Isn't that....bad?
>
> Consolidate freeing the VMID/ASID to one place and consistently flush both
> ID types before returning to their allocators.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 9 ++---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 36 +++++++++++++------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 +
> 3 files changed, 29 insertions(+), 17 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index cb0da4e5a5517a..3a9f4ef47c6b6f 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -381,18 +381,13 @@ static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
> {
> struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
>
> - /*
> - * Ensure the ASID is empty in the iommu cache before allowing reuse.
> - */
> - arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
> -
> /*
> * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can
> * still be called/running at this point. We allow the ASID to be
> * reused, and if there is a race then it just suffers harmless
> * unnecessary invalidation.
> */
> - xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
> + arm_smmu_domain_free_id(smmu_domain);
>
> /*
> * Actual free is defered to the SRCU callback
> @@ -437,7 +432,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> return &smmu_domain->domain;
>
> err_asid:
> - xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
> + arm_smmu_domain_free_id(smmu_domain);
> err_free:
> kfree(smmu_domain);
> return ERR_PTR(ret);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 5642321b2124d9..4f22eb810c8dbd 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2296,25 +2296,41 @@ static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
> return &smmu_domain->domain;
> }
>
> -static void arm_smmu_domain_free(struct iommu_domain *domain)
> +/*
> + * Return the domain's ASID or VMID back to the allocator. All IDs in the
> + * allocator do not have an IOTLB entries referencing them.
> + */
> +void arm_smmu_domain_free_id(struct arm_smmu_domain *smmu_domain)
> {
> - struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> struct arm_smmu_device *smmu = smmu_domain->smmu;
>
> - free_io_pgtable_ops(smmu_domain->pgtbl_ops);
> + if ((smmu_domain->stage == ARM_SMMU_DOMAIN_S1 ||
> + smmu_domain->domain.type == IOMMU_DOMAIN_SVA) &&
> + smmu_domain->cd.asid) {
> + arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid);
>
> - /* Free the ASID or VMID */
> - if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
> /* Prevent SVA from touching the CD while we're freeing it */
> mutex_lock(&arm_smmu_asid_lock);
> xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
> mutex_unlock(&arm_smmu_asid_lock);
> - } else {
> - struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
> - if (cfg->vmid)
> - ida_free(&smmu->vmid_map, cfg->vmid);
> - }
> + } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S2 &&
> + smmu_domain->s2_cfg.vmid) {
> + struct arm_smmu_cmdq_ent cmd = {
> + .opcode = CMDQ_OP_TLBI_S12_VMALL,
> + .tlbi.vmid = smmu_domain->s2_cfg.vmid
> + };
>
> + arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
> + ida_free(&smmu->vmid_map, smmu_domain->s2_cfg.vmid);
> + }
> +}
> +
There's room to refactor and share the tlb id invalidation logic with
arm_smmu_tlb_inv_context by the way, but this works too.
> +static void arm_smmu_domain_free(struct iommu_domain *domain)
> +{
> + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> +
> + free_io_pgtable_ops(smmu_domain->pgtbl_ops);
> + arm_smmu_domain_free_id(smmu_domain);
> kfree(smmu_domain);
> }
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index cfae4d69cd810c..4631f0ac396dc3 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -774,6 +774,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
> void arm_smmu_remove_pasid(struct arm_smmu_master *master,
> struct arm_smmu_domain *smmu_domain, ioasid_t pasid);
>
> +void arm_smmu_domain_free_id(struct arm_smmu_domain *smmu_domain);
> void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
> void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
> size_t granule, bool leaf,
> --
> 2.43.2
>
>
Reviewed-by: Michael Shavit <mshavit@google.com>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 24/27] iommu/arm-smmu-v3: Bring back SVA BTM support
2024-03-04 23:44 ` Jason Gunthorpe
@ 2024-03-19 17:07 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 17:07 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> BTM support is a feature where the CPU TLB invalidation can be forwarded
> to the IOMMU and also invalidate the IOTLB. For this to work the CPU and
> IOMMU ASID must be the same.
>
> Retain the prior SVA design here of keeping the ASID allocator for the
> IOMMU private to SMMU and force SVA domains to set an ASID that matches
> the CPU ASID.
>
> This requires changing the ASID assigned to a S1 domain if it happens to
> be overlapping with the required CPU ASID. We hold on to the CPU ASID so
> long as the SVA iommu_domain exists, so SVA domain conflict is not
> possible.
>
> With the asid per-smmu we no longer have a problem that two per-smmu
> iommu_domain's would need to share a CPU ASID entry in the IOMMU's xarray.
>
> Use the same ASID move algorithm for the S1 domains as before with some
> streamlining around how the xarray is being used. Do not synchronize the
> ASID's if BTM mode is not supported. Just leave BTM features off
> everywhere.
>
> Audit all the places that touch cd->asid and think carefully about how the
> locking works with the change to the cd->asid by the move algorithm. Use
> xarray internal locking during xa_alloc() instead of double locking. Add a
> note that concurrent S1 invalidation doesn't fully work.
>
> Note that this is all still dead code, ARM_SMMU_FEAT_BTM is never set.
Clearly a lot of work/thought has gone into this patch, but I have to
wonder if this complexity is worth carrying forward for a feature that
was never enabled... Is moving this to part 3 or dropping it all
together an option?
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 133 ++++++++++++++++--
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 15 +-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 +-
> 3 files changed, 129 insertions(+), 21 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 3acd699433b7d8..0b5aeaa3a85575 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -15,12 +15,33 @@
>
> static DEFINE_MUTEX(sva_lock);
>
> -static void __maybe_unused
> -arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> +static int arm_smmu_realloc_s1_domain_asid(struct arm_smmu_device *smmu,
> + struct arm_smmu_domain *smmu_domain)
> {
> struct arm_smmu_master_domain *master_domain;
> + u32 old_asid = smmu_domain->cd.asid;
> struct arm_smmu_cd target_cd;
> unsigned long flags;
> + int ret;
> +
> + lockdep_assert_held(&smmu->asid_lock);
> +
> + /*
> + * FIXME: The unmap and invalidation path doesn't take any locks but
> + * this is not fully safe. Since updating the CD tables is not atomic
> + * there is always a hole where invalidating only one ASID of two active
> + * ASIDs during unmap will cause the IOTLB to become stale.
> + *
> + * This approach is to hopefully shift the racing CPUs to the new ASID
> + * before we start programming the HW. This increases the chance that
> + * racing IOPTE changes will pick up an invalidation for the new ASID
> + * and we achieve eventual consistency. For the brief period where the
> + * old ASID is still in the CD entries it will become incoherent.
> + */
> + ret = xa_alloc(&smmu->asid_map, &smmu_domain->cd.asid, smmu_domain,
> + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> + if (ret)
> + return ret;
>
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> list_for_each_entry(master_domain, &smmu_domain->devices, devices_elm) {
> @@ -36,6 +57,10 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> &target_cd);
> }
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> +
> + /* Clean the ASID we are about to assign to a new translation */
> + arm_smmu_tlb_inv_asid(smmu, old_asid);
> + return 0;
> }
>
> static u64 page_size_to_cd(void)
> @@ -148,12 +173,12 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
> }
>
> if (!smmu_domain->btm_invalidation) {
> + ioasid_t asid = READ_ONCE(smmu_domain->cd.asid);
> +
> if (!size)
> - arm_smmu_tlb_inv_asid(smmu_domain->smmu,
> - smmu_domain->cd.asid);
> + arm_smmu_tlb_inv_asid(smmu_domain->smmu, asid);
> else
> - arm_smmu_tlb_inv_range_asid(start, size,
> - smmu_domain->cd.asid,
> + arm_smmu_tlb_inv_range_asid(start, size, asid,
> PAGE_SIZE, false,
> smmu_domain);
> }
> @@ -182,6 +207,8 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
> cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid);
> if (WARN_ON(!cdptr))
> continue;
> +
> + /* An SVA ASID never changes, no asid_lock required */
> arm_smmu_make_sva_cd(&target, master, NULL,
> smmu_domain->cd.asid,
> smmu_domain->btm_invalidation);
> @@ -388,6 +415,8 @@ static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
> * unnecessary invalidation.
> */
> arm_smmu_domain_free_id(smmu_domain);
> + if (smmu_domain->btm_invalidation)
> + arm64_mm_context_put(domain->mm);
>
> /*
> * Actual free is defered to the SRCU callback
> @@ -401,13 +430,97 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
> .free = arm_smmu_sva_domain_free
> };
>
> +static int arm_smmu_share_asid(struct arm_smmu_device *smmu,
> + struct arm_smmu_domain *smmu_domain,
> + struct mm_struct *mm)
> +{
> + struct arm_smmu_domain *old_s1_domain;
> + int ret;
> +
> + /*
> + * Notice that BTM is never currently enabled, this is all dead code.
> + * The specification cautions:
> + *
> + * Note: Arm expects that SMMU stage 2 address spaces are generally
> + * shared with their respective PE virtual machine stage 2
> + * configuration. If broadcast invalidation is required to be avoided
> + * for a particular SMMU stage 2 address space, Arm recommends that a
> + * hypervisor configures the STE with a VMID that is not allocated for
> + * virtual machine use on the PEs
> + *
> + * However, in Linux, both KVM and SMMU think they own the VMID pool.
> + * Unfortunately the ARM design is problematic for Linux as we do not
> + * currently share the S2 table with KVM. This creates a situation where
> + * the S2 needs to have the same VMID as KVM in order to allow the guest
> + * to use BTM, however we must still invalidate the S2 directly since it
> + * is a different radix tree. What Linux would like is something like
> + * ASET for the STE to disable BTM only for the S2.
> + *
> + * Arguably in a system with BTM the driver should prefer to use a S1
> + * table in all cases execpt when explicitly asked to create a nesting
> + * parent. Then it should use the VMID of KVM to enable BTM in the
> + * guest. We cannot optimize away the resulting double invalidation of
> + * the S2 :( Or we simply ignore BTM entirely as we are doing now.
> + */
> + if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM))
> + return xa_alloc(&smmu->asid_map, &smmu_domain->cd.asid,
> + smmu_domain,
> + XA_LIMIT(1, (1 << smmu->asid_bits) - 1),
> + GFP_KERNEL);
> +
> + /* At this point the caller ensures we have a mmget() */
> + smmu_domain->cd.asid = arm64_mm_context_get(mm);
> +
> + mutex_lock(&smmu->asid_lock);
> + old_s1_domain = xa_store(&smmu->asid_map, smmu_domain->cd.asid,
> + smmu_domain, GFP_KERNEL);
> + if (xa_err(old_s1_domain)) {
> + ret = xa_err(old_s1_domain);
> + goto out_put_asid;
> + }
> +
> + /*
> + * In BTM mode the CPU ASID and the IOMMU ASID have to be the same.
> + * Unfortunately we run separate allocators for this and the IOMMU
> + * ASID can already have been assigned to a S1 domain. SVA domains
> + * always align to their CPU ASIDs. In this case we change
> + * the S1 domain's ASID, update the CD entry and flush the caches.
> + *
> + * This is a bit tricky, all the places writing to a S1 CD, reading the
> + * S1 ASID, or doing xa_erase must hold the asid_lock or xa_lock to
> + * avoid IOTLB incoherence.
> + */
> + if (old_s1_domain) {
> + if (WARN_ON(old_s1_domain->domain.type == IOMMU_DOMAIN_SVA)) {
> + ret = -EINVAL;
> + goto out_restore_s1;
> + }
> + ret = arm_smmu_realloc_s1_domain_asid(smmu, old_s1_domain);
> + if (ret)
> + goto out_restore_s1;
> + }
> +
> + smmu_domain->btm_invalidation = true;
> +
> + ret = 0;
> + goto out_unlock;
> +
> +out_restore_s1:
> + xa_store(&smmu->asid_map, smmu_domain->cd.asid, old_s1_domain,
> + GFP_KERNEL);
> +out_put_asid:
> + arm64_mm_context_put(mm);
> +out_unlock:
> + mutex_unlock(&smmu->asid_lock);
> + return ret;
> +}
> +
> struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> struct mm_struct *mm)
> {
> struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> struct arm_smmu_device *smmu = master->smmu;
> struct arm_smmu_domain *smmu_domain;
> - u32 asid;
> int ret;
>
> smmu_domain = arm_smmu_domain_alloc();
> @@ -418,12 +531,10 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
> smmu_domain->smmu = smmu;
>
> - ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain,
> - XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> + ret = arm_smmu_share_asid(smmu, smmu_domain, mm);
> if (ret)
> goto err_free;
>
> - smmu_domain->cd.asid = asid;
> smmu_domain->mmu_notifier.ops = &arm_smmu_mmu_notifier_ops;
> ret = mmu_notifier_register(&smmu_domain->mmu_notifier, mm);
> if (ret)
> @@ -433,6 +544,8 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
>
> err_asid:
> arm_smmu_domain_free_id(smmu_domain);
> + if (smmu_domain->btm_invalidation)
> + arm64_mm_context_put(mm);
> err_free:
> kfree(smmu_domain);
> return ERR_PTR(ret);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 1a72dd63e0ca14..01737a0fc82828 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1314,6 +1314,8 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr =
> &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
>
> + lockdep_assert_held(&master->smmu->asid_lock);
> +
> memset(target, 0, sizeof(*target));
>
> target->data[0] = cpu_to_le64(
> @@ -2089,7 +2091,7 @@ static void arm_smmu_tlb_inv_context(void *cookie)
> * careful, 007.
> */
> if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
> - arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid);
> + arm_smmu_tlb_inv_asid(smmu, READ_ONCE(smmu_domain->cd.asid));
> } else {
> cmd.opcode = CMDQ_OP_TLBI_S12_VMALL;
> cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid;
> @@ -2334,17 +2336,10 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
> static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
> struct arm_smmu_domain *smmu_domain)
> {
> - int ret;
> - u32 asid;
> struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
>
> - /* Prevent SVA from modifying the ASID until it is written to the CD */
> - mutex_lock(&smmu->asid_lock);
> - ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain,
> - XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> - cd->asid = (u16)asid;
> - mutex_unlock(&smmu->asid_lock);
> - return ret;
> + return xa_alloc(&smmu->asid_map, &cd->asid, smmu_domain,
> + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> }
>
> static int arm_smmu_domain_finalise_s2(struct arm_smmu_device *smmu,
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 6becdbae905598..a23593f1830106 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -586,7 +586,7 @@ struct arm_smmu_strtab_l1_desc {
> };
>
> struct arm_smmu_ctx_desc {
> - u16 asid;
> + u32 asid;
> };
>
> struct arm_smmu_l1_ctx_desc {
> --
> 2.43.2
>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 24/27] iommu/arm-smmu-v3: Bring back SVA BTM support
@ 2024-03-19 17:07 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-19 17:07 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> BTM support is a feature where the CPU TLB invalidation can be forwarded
> to the IOMMU and also invalidate the IOTLB. For this to work the CPU and
> IOMMU ASID must be the same.
>
> Retain the prior SVA design here of keeping the ASID allocator for the
> IOMMU private to SMMU and force SVA domains to set an ASID that matches
> the CPU ASID.
>
> This requires changing the ASID assigned to a S1 domain if it happens to
> be overlapping with the required CPU ASID. We hold on to the CPU ASID so
> long as the SVA iommu_domain exists, so SVA domain conflict is not
> possible.
>
> With the asid per-smmu we no longer have a problem that two per-smmu
> iommu_domain's would need to share a CPU ASID entry in the IOMMU's xarray.
>
> Use the same ASID move algorithm for the S1 domains as before with some
> streamlining around how the xarray is being used. Do not synchronize the
> ASID's if BTM mode is not supported. Just leave BTM features off
> everywhere.
>
> Audit all the places that touch cd->asid and think carefully about how the
> locking works with the change to the cd->asid by the move algorithm. Use
> xarray internal locking during xa_alloc() instead of double locking. Add a
> note that concurrent S1 invalidation doesn't fully work.
>
> Note that this is all still dead code, ARM_SMMU_FEAT_BTM is never set.
Clearly a lot of work/thought has gone into this patch, but I have to
wonder if this complexity is worth carrying forward for a feature that
was never enabled... Is moving this to part 3 or dropping it all
together an option?
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 133 ++++++++++++++++--
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 15 +-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 +-
> 3 files changed, 129 insertions(+), 21 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 3acd699433b7d8..0b5aeaa3a85575 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -15,12 +15,33 @@
>
> static DEFINE_MUTEX(sva_lock);
>
> -static void __maybe_unused
> -arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> +static int arm_smmu_realloc_s1_domain_asid(struct arm_smmu_device *smmu,
> + struct arm_smmu_domain *smmu_domain)
> {
> struct arm_smmu_master_domain *master_domain;
> + u32 old_asid = smmu_domain->cd.asid;
> struct arm_smmu_cd target_cd;
> unsigned long flags;
> + int ret;
> +
> + lockdep_assert_held(&smmu->asid_lock);
> +
> + /*
> + * FIXME: The unmap and invalidation path doesn't take any locks but
> + * this is not fully safe. Since updating the CD tables is not atomic
> + * there is always a hole where invalidating only one ASID of two active
> + * ASIDs during unmap will cause the IOTLB to become stale.
> + *
> + * This approach is to hopefully shift the racing CPUs to the new ASID
> + * before we start programming the HW. This increases the chance that
> + * racing IOPTE changes will pick up an invalidation for the new ASID
> + * and we achieve eventual consistency. For the brief period where the
> + * old ASID is still in the CD entries it will become incoherent.
> + */
> + ret = xa_alloc(&smmu->asid_map, &smmu_domain->cd.asid, smmu_domain,
> + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> + if (ret)
> + return ret;
>
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> list_for_each_entry(master_domain, &smmu_domain->devices, devices_elm) {
> @@ -36,6 +57,10 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> &target_cd);
> }
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> +
> + /* Clean the ASID we are about to assign to a new translation */
> + arm_smmu_tlb_inv_asid(smmu, old_asid);
> + return 0;
> }
>
> static u64 page_size_to_cd(void)
> @@ -148,12 +173,12 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
> }
>
> if (!smmu_domain->btm_invalidation) {
> + ioasid_t asid = READ_ONCE(smmu_domain->cd.asid);
> +
> if (!size)
> - arm_smmu_tlb_inv_asid(smmu_domain->smmu,
> - smmu_domain->cd.asid);
> + arm_smmu_tlb_inv_asid(smmu_domain->smmu, asid);
> else
> - arm_smmu_tlb_inv_range_asid(start, size,
> - smmu_domain->cd.asid,
> + arm_smmu_tlb_inv_range_asid(start, size, asid,
> PAGE_SIZE, false,
> smmu_domain);
> }
> @@ -182,6 +207,8 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
> cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid);
> if (WARN_ON(!cdptr))
> continue;
> +
> + /* An SVA ASID never changes, no asid_lock required */
> arm_smmu_make_sva_cd(&target, master, NULL,
> smmu_domain->cd.asid,
> smmu_domain->btm_invalidation);
> @@ -388,6 +415,8 @@ static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
> * unnecessary invalidation.
> */
> arm_smmu_domain_free_id(smmu_domain);
> + if (smmu_domain->btm_invalidation)
> + arm64_mm_context_put(domain->mm);
>
> /*
> * Actual free is defered to the SRCU callback
> @@ -401,13 +430,97 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = {
> .free = arm_smmu_sva_domain_free
> };
>
> +static int arm_smmu_share_asid(struct arm_smmu_device *smmu,
> + struct arm_smmu_domain *smmu_domain,
> + struct mm_struct *mm)
> +{
> + struct arm_smmu_domain *old_s1_domain;
> + int ret;
> +
> + /*
> + * Notice that BTM is never currently enabled, this is all dead code.
> + * The specification cautions:
> + *
> + * Note: Arm expects that SMMU stage 2 address spaces are generally
> + * shared with their respective PE virtual machine stage 2
> + * configuration. If broadcast invalidation is required to be avoided
> + * for a particular SMMU stage 2 address space, Arm recommends that a
> + * hypervisor configures the STE with a VMID that is not allocated for
> + * virtual machine use on the PEs
> + *
> + * However, in Linux, both KVM and SMMU think they own the VMID pool.
> + * Unfortunately the ARM design is problematic for Linux as we do not
> + * currently share the S2 table with KVM. This creates a situation where
> + * the S2 needs to have the same VMID as KVM in order to allow the guest
> + * to use BTM, however we must still invalidate the S2 directly since it
> + * is a different radix tree. What Linux would like is something like
> + * ASET for the STE to disable BTM only for the S2.
> + *
> + * Arguably in a system with BTM the driver should prefer to use a S1
> + * table in all cases execpt when explicitly asked to create a nesting
> + * parent. Then it should use the VMID of KVM to enable BTM in the
> + * guest. We cannot optimize away the resulting double invalidation of
> + * the S2 :( Or we simply ignore BTM entirely as we are doing now.
> + */
> + if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM))
> + return xa_alloc(&smmu->asid_map, &smmu_domain->cd.asid,
> + smmu_domain,
> + XA_LIMIT(1, (1 << smmu->asid_bits) - 1),
> + GFP_KERNEL);
> +
> + /* At this point the caller ensures we have a mmget() */
> + smmu_domain->cd.asid = arm64_mm_context_get(mm);
> +
> + mutex_lock(&smmu->asid_lock);
> + old_s1_domain = xa_store(&smmu->asid_map, smmu_domain->cd.asid,
> + smmu_domain, GFP_KERNEL);
> + if (xa_err(old_s1_domain)) {
> + ret = xa_err(old_s1_domain);
> + goto out_put_asid;
> + }
> +
> + /*
> + * In BTM mode the CPU ASID and the IOMMU ASID have to be the same.
> + * Unfortunately we run separate allocators for this and the IOMMU
> + * ASID can already have been assigned to a S1 domain. SVA domains
> + * always align to their CPU ASIDs. In this case we change
> + * the S1 domain's ASID, update the CD entry and flush the caches.
> + *
> + * This is a bit tricky, all the places writing to a S1 CD, reading the
> + * S1 ASID, or doing xa_erase must hold the asid_lock or xa_lock to
> + * avoid IOTLB incoherence.
> + */
> + if (old_s1_domain) {
> + if (WARN_ON(old_s1_domain->domain.type == IOMMU_DOMAIN_SVA)) {
> + ret = -EINVAL;
> + goto out_restore_s1;
> + }
> + ret = arm_smmu_realloc_s1_domain_asid(smmu, old_s1_domain);
> + if (ret)
> + goto out_restore_s1;
> + }
> +
> + smmu_domain->btm_invalidation = true;
> +
> + ret = 0;
> + goto out_unlock;
> +
> +out_restore_s1:
> + xa_store(&smmu->asid_map, smmu_domain->cd.asid, old_s1_domain,
> + GFP_KERNEL);
> +out_put_asid:
> + arm64_mm_context_put(mm);
> +out_unlock:
> + mutex_unlock(&smmu->asid_lock);
> + return ret;
> +}
> +
> struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> struct mm_struct *mm)
> {
> struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> struct arm_smmu_device *smmu = master->smmu;
> struct arm_smmu_domain *smmu_domain;
> - u32 asid;
> int ret;
>
> smmu_domain = arm_smmu_domain_alloc();
> @@ -418,12 +531,10 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> smmu_domain->domain.ops = &arm_smmu_sva_domain_ops;
> smmu_domain->smmu = smmu;
>
> - ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain,
> - XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> + ret = arm_smmu_share_asid(smmu, smmu_domain, mm);
> if (ret)
> goto err_free;
>
> - smmu_domain->cd.asid = asid;
> smmu_domain->mmu_notifier.ops = &arm_smmu_mmu_notifier_ops;
> ret = mmu_notifier_register(&smmu_domain->mmu_notifier, mm);
> if (ret)
> @@ -433,6 +544,8 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
>
> err_asid:
> arm_smmu_domain_free_id(smmu_domain);
> + if (smmu_domain->btm_invalidation)
> + arm64_mm_context_put(mm);
> err_free:
> kfree(smmu_domain);
> return ERR_PTR(ret);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 1a72dd63e0ca14..01737a0fc82828 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1314,6 +1314,8 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr =
> &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
>
> + lockdep_assert_held(&master->smmu->asid_lock);
> +
> memset(target, 0, sizeof(*target));
>
> target->data[0] = cpu_to_le64(
> @@ -2089,7 +2091,7 @@ static void arm_smmu_tlb_inv_context(void *cookie)
> * careful, 007.
> */
> if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
> - arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid);
> + arm_smmu_tlb_inv_asid(smmu, READ_ONCE(smmu_domain->cd.asid));
> } else {
> cmd.opcode = CMDQ_OP_TLBI_S12_VMALL;
> cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid;
> @@ -2334,17 +2336,10 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
> static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
> struct arm_smmu_domain *smmu_domain)
> {
> - int ret;
> - u32 asid;
> struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
>
> - /* Prevent SVA from modifying the ASID until it is written to the CD */
> - mutex_lock(&smmu->asid_lock);
> - ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain,
> - XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> - cd->asid = (u16)asid;
> - mutex_unlock(&smmu->asid_lock);
> - return ret;
> + return xa_alloc(&smmu->asid_map, &cd->asid, smmu_domain,
> + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> }
>
> static int arm_smmu_domain_finalise_s2(struct arm_smmu_device *smmu,
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 6becdbae905598..a23593f1830106 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -586,7 +586,7 @@ struct arm_smmu_strtab_l1_desc {
> };
>
> struct arm_smmu_ctx_desc {
> - u16 asid;
> + u32 asid;
> };
>
> struct arm_smmu_l1_ctx_desc {
> --
> 2.43.2
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 22/27] iommu/arm-smmu-v3: Consolidate freeing the ASID/VMID
2024-03-19 16:44 ` Michael Shavit
@ 2024-03-19 18:37 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-19 18:37 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Wed, Mar 20, 2024 at 12:44:42AM +0800, Michael Shavit wrote:
> On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > The SMMUv3 IOTLB is tagged with a VMID/ASID cache tag. Any time the
> > underlying translation is changed these need to be invalidated. At boot
> > time the IOTLB starts out empty and all cache tags are available for
> > allocation.
> >
> > When a tag is taken out of the allocator the code assumes the IOTLB
> > doesn't reference it, and immediately programs it into a STE/CD. If the
> > cache is referencing the tag then it will have stale data and IOMMU will
> > become incoherent.
> >
> > Thus, whenever an ASID/VMID is freed back to the allocator we need to know
> > that the IOTLB doesn't have any references to it. The SVA code correctly
> > had an invalidation here, but the paging code does not.
>
> Isn't that....bad?
Ah, that needed some more work I forgot to do.
The paging code hides the invalidation in the free_io_pgtable_ops()
called during domain free which calls back to
arm_smmu_tlb_inv_context().
free_io_pgtable_ops() is the only thing that calls
io_pgtable_tlb_flush_all() and thus arm_smmu_tlb_inv_context().
So we should also delete arm_smmu_tlb_inv_context() and NULL
ops.tlb_flush_all.
Then arm_smmu_domain_free_id() consistently handles the final
invalidation on all code paths instead of having it split up.
Thanks,
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 22/27] iommu/arm-smmu-v3: Consolidate freeing the ASID/VMID
@ 2024-03-19 18:37 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-19 18:37 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Wed, Mar 20, 2024 at 12:44:42AM +0800, Michael Shavit wrote:
> On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > The SMMUv3 IOTLB is tagged with a VMID/ASID cache tag. Any time the
> > underlying translation is changed these need to be invalidated. At boot
> > time the IOTLB starts out empty and all cache tags are available for
> > allocation.
> >
> > When a tag is taken out of the allocator the code assumes the IOTLB
> > doesn't reference it, and immediately programs it into a STE/CD. If the
> > cache is referencing the tag then it will have stale data and IOMMU will
> > become incoherent.
> >
> > Thus, whenever an ASID/VMID is freed back to the allocator we need to know
> > that the IOTLB doesn't have any references to it. The SVA code correctly
> > had an invalidation here, but the paging code does not.
>
> Isn't that....bad?
Ah, that needed some more work I forgot to do.
The paging code hides the invalidation in the free_io_pgtable_ops()
called during domain free which calls back to
arm_smmu_tlb_inv_context().
free_io_pgtable_ops() is the only thing that calls
io_pgtable_tlb_flush_all() and thus arm_smmu_tlb_inv_context().
So we should also delete arm_smmu_tlb_inv_context() and NULL
ops.tlb_flush_all.
Then arm_smmu_domain_free_id() consistently handles the final
invalidation on all code paths instead of having it split up.
Thanks,
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
2024-03-15 7:52 ` Nicolin Chen
@ 2024-03-20 12:46 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 12:46 UTC (permalink / raw)
To: Nicolin Chen
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Fri, Mar 15, 2024 at 12:52:43AM -0700, Nicolin Chen wrote:
> On Mon, Mar 04, 2024 at 07:43:53PM -0400, Jason Gunthorpe wrote:
> > CD table entries and STE's have the same essential programming sequence,
> > just with different types and sizes.
>
> Hmm.. I somehow remember that one of them was 4 qwords while
> the other was 8? Yet now, in the final driver source code on
> the smmuv3_newapi branch, they are both 8 qwords?
Yeah they are both 8, it is a mistake in the commit message. I fixed
it.
Thanks,
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
@ 2024-03-20 12:46 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 12:46 UTC (permalink / raw)
To: Nicolin Chen
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Fri, Mar 15, 2024 at 12:52:43AM -0700, Nicolin Chen wrote:
> On Mon, Mar 04, 2024 at 07:43:53PM -0400, Jason Gunthorpe wrote:
> > CD table entries and STE's have the same essential programming sequence,
> > just with different types and sizes.
>
> Hmm.. I somehow remember that one of them was 4 qwords while
> the other was 8? Yet now, in the final driver source code on
> the smmuv3_newapi branch, they are both 8 qwords?
Yeah they are both 8, it is a mistake in the commit message. I fixed
it.
Thanks,
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 11/27] iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd()
2024-03-15 10:04 ` Michael Shavit
@ 2024-03-20 12:50 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 12:50 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Fri, Mar 15, 2024 at 06:04:19PM +0800, Michael Shavit wrote:
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index 50d17e3ce0a956..dfdd48cf217c4e 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -1301,15 +1301,25 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> > struct arm_smmu_domain *smmu_domain)
> > {
> > struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
> > + const struct io_pgtable_cfg *pgtbl_cfg =
> > + &io_pgtable_ops_to_pgtable(smmu_domain->pgtbl_ops)->cfg;
> > + typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr =
> > + &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
> >
> > memset(target, 0, sizeof(*target));
> >
> > target->data[0] = cpu_to_le64(
> > - cd->tcr |
> > + FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) |
> > + FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) |
> > + FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) |
> > + FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) |
> > + FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) |
> > + CTXDESC_CD_0_TCR_EPD1 |
> > #ifdef __BIG_ENDIAN
> > CTXDESC_CD_0_ENDI |
> > #endif
> > CTXDESC_CD_0_V |
> > + FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) |
>
> I guess you're trying to keep these ordered by bit position, in which
> case EPD1 should go after ENDI.
Right, I missed the #defines are not in bit order :\
Thanks,
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 11/27] iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd()
@ 2024-03-20 12:50 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 12:50 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Fri, Mar 15, 2024 at 06:04:19PM +0800, Michael Shavit wrote:
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index 50d17e3ce0a956..dfdd48cf217c4e 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -1301,15 +1301,25 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> > struct arm_smmu_domain *smmu_domain)
> > {
> > struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
> > + const struct io_pgtable_cfg *pgtbl_cfg =
> > + &io_pgtable_ops_to_pgtable(smmu_domain->pgtbl_ops)->cfg;
> > + typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr =
> > + &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
> >
> > memset(target, 0, sizeof(*target));
> >
> > target->data[0] = cpu_to_le64(
> > - cd->tcr |
> > + FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) |
> > + FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) |
> > + FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) |
> > + FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) |
> > + FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) |
> > + CTXDESC_CD_0_TCR_EPD1 |
> > #ifdef __BIG_ENDIAN
> > CTXDESC_CD_0_ENDI |
> > #endif
> > CTXDESC_CD_0_V |
> > + FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) |
>
> I guess you're trying to keep these ordered by bit position, in which
> case EPD1 should go after ENDI.
Right, I missed the #defines are not in bit order :\
Thanks,
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 15/27] iommu/arm-smmu-v3: Add ssid to struct arm_smmu_master_domain
2024-03-19 13:31 ` Michael Shavit
@ 2024-03-20 12:53 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 12:53 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 19, 2024 at 09:31:17PM +0800, Michael Shavit wrote:
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> > @@ -738,6 +738,7 @@ struct arm_smmu_domain {
> > struct arm_smmu_master_domain {
> > struct list_head devices_elm;
> > struct arm_smmu_master *master;
> > + u16 ssid;
> > };
>
> SMMU_IDR1.SSIDSIZE may be up to 20 according to the spec. The value
> read from this register is used to populate iommu.max_pasids as well
> as to compute the smmu/master ssid_bits fields so u16 is insufficient
> here.
Right, that should be ioasid_t
Thanks,
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 15/27] iommu/arm-smmu-v3: Add ssid to struct arm_smmu_master_domain
@ 2024-03-20 12:53 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 12:53 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 19, 2024 at 09:31:17PM +0800, Michael Shavit wrote:
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> > @@ -738,6 +738,7 @@ struct arm_smmu_domain {
> > struct arm_smmu_master_domain {
> > struct list_head devices_elm;
> > struct arm_smmu_master *master;
> > + u16 ssid;
> > };
>
> SMMU_IDR1.SSIDSIZE may be up to 20 according to the spec. The value
> read from this register is used to populate iommu.max_pasids as well
> as to compute the smmu/master ssid_bits fields so u16 is insufficient
> here.
Right, that should be ioasid_t
Thanks,
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 24/27] iommu/arm-smmu-v3: Bring back SVA BTM support
2024-03-19 17:07 ` Michael Shavit
@ 2024-03-20 13:05 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 13:05 UTC (permalink / raw)
To: Michael Shavit, Will Deacon
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Eric Auger,
Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen, patches,
Shameerali Kolothum Thodi
On Wed, Mar 20, 2024 at 01:07:01AM +0800, Michael Shavit wrote:
> On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > BTM support is a feature where the CPU TLB invalidation can be forwarded
> > to the IOMMU and also invalidate the IOTLB. For this to work the CPU and
> > IOMMU ASID must be the same.
> >
> > Retain the prior SVA design here of keeping the ASID allocator for the
> > IOMMU private to SMMU and force SVA domains to set an ASID that matches
> > the CPU ASID.
> >
> > This requires changing the ASID assigned to a S1 domain if it happens to
> > be overlapping with the required CPU ASID. We hold on to the CPU ASID so
> > long as the SVA iommu_domain exists, so SVA domain conflict is not
> > possible.
> >
> > With the asid per-smmu we no longer have a problem that two per-smmu
> > iommu_domain's would need to share a CPU ASID entry in the IOMMU's xarray.
> >
> > Use the same ASID move algorithm for the S1 domains as before with some
> > streamlining around how the xarray is being used. Do not synchronize the
> > ASID's if BTM mode is not supported. Just leave BTM features off
> > everywhere.
> >
> > Audit all the places that touch cd->asid and think carefully about how the
> > locking works with the change to the cd->asid by the move algorithm. Use
> > xarray internal locking during xa_alloc() instead of double locking. Add a
> > note that concurrent S1 invalidation doesn't fully work.
> >
> > Note that this is all still dead code, ARM_SMMU_FEAT_BTM is never set.
>
> Clearly a lot of work/thought has gone into this patch, but I have to
> wonder if this complexity is worth carrying forward for a feature that
> was never enabled... Is moving this to part 3 or dropping it all
> together an option?
Yes, we can do any of those. If we drop it here then it will be moved
to this series:
https://lore.kernel.org/linux-iommu/20240208151837.35068-1-shameerali.kolothum.thodi@huawei.com/
Which is the final bit of work to actually enable the feature. We are
getting quite close, it needs some of Nicolin's iommufd enablement
work to cross the finish line, I think.
I agree dead code like this should not be in the kernel..
Will? Please advise
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 24/27] iommu/arm-smmu-v3: Bring back SVA BTM support
@ 2024-03-20 13:05 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 13:05 UTC (permalink / raw)
To: Michael Shavit, Will Deacon
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Eric Auger,
Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen, patches,
Shameerali Kolothum Thodi
On Wed, Mar 20, 2024 at 01:07:01AM +0800, Michael Shavit wrote:
> On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > BTM support is a feature where the CPU TLB invalidation can be forwarded
> > to the IOMMU and also invalidate the IOTLB. For this to work the CPU and
> > IOMMU ASID must be the same.
> >
> > Retain the prior SVA design here of keeping the ASID allocator for the
> > IOMMU private to SMMU and force SVA domains to set an ASID that matches
> > the CPU ASID.
> >
> > This requires changing the ASID assigned to a S1 domain if it happens to
> > be overlapping with the required CPU ASID. We hold on to the CPU ASID so
> > long as the SVA iommu_domain exists, so SVA domain conflict is not
> > possible.
> >
> > With the asid per-smmu we no longer have a problem that two per-smmu
> > iommu_domain's would need to share a CPU ASID entry in the IOMMU's xarray.
> >
> > Use the same ASID move algorithm for the S1 domains as before with some
> > streamlining around how the xarray is being used. Do not synchronize the
> > ASID's if BTM mode is not supported. Just leave BTM features off
> > everywhere.
> >
> > Audit all the places that touch cd->asid and think carefully about how the
> > locking works with the change to the cd->asid by the move algorithm. Use
> > xarray internal locking during xa_alloc() instead of double locking. Add a
> > note that concurrent S1 invalidation doesn't fully work.
> >
> > Note that this is all still dead code, ARM_SMMU_FEAT_BTM is never set.
>
> Clearly a lot of work/thought has gone into this patch, but I have to
> wonder if this complexity is worth carrying forward for a feature that
> was never enabled... Is moving this to part 3 or dropping it all
> together an option?
Yes, we can do any of those. If we drop it here then it will be moved
to this series:
https://lore.kernel.org/linux-iommu/20240208151837.35068-1-shameerali.kolothum.thodi@huawei.com/
Which is the final bit of work to actually enable the feature. We are
getting quite close, it needs some of Nicolin's iommufd enablement
work to cross the finish line, I think.
I agree dead code like this should not be in the kernel..
Will? Please advise
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 10/27] iommu/arm-smmu-v3: Move the CD generation for SVA into a function
2024-03-16 5:19 ` Nicolin Chen
@ 2024-03-20 13:09 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 13:09 UTC (permalink / raw)
To: Nicolin Chen
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Fri, Mar 15, 2024 at 10:19:45PM -0700, Nicolin Chen wrote:
> On Mon, Mar 04, 2024 at 07:43:58PM -0400, Jason Gunthorpe wrote:
> > @@ -1070,8 +1062,7 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> > * Each bit indicates that a used bit in a qword needs to be
> > * changed after unused_update is applied.
> > */
> > - if ((unused_update[i] & target_used[i]) !=
> > - (target[i] & target_used[i]))
> > + if ((unused_update[i] & target_used[i]) != target[i])
> > used_qword_diff |= 1 << i;
>
> Would you mind elaborating why we drop this "& target_used[i]"
> against the target[i] on the right?
It is undoning the temporary code a few patches before did and
bringing it back to the way it was in part 1.
It is ugly, the programmer requires that the target be properly
constructed but the way I migrated the CD code over doesn't guarentee
proper construction until we reach this patch because it edits the
current CD..
So there is this inbetween step where the programming code has to fix
an improperly constructed thing.
I didn't try too hard to simplify this, let me look again.
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 10/27] iommu/arm-smmu-v3: Move the CD generation for SVA into a function
@ 2024-03-20 13:09 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 13:09 UTC (permalink / raw)
To: Nicolin Chen
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, patches, Shameerali Kolothum Thodi
On Fri, Mar 15, 2024 at 10:19:45PM -0700, Nicolin Chen wrote:
> On Mon, Mar 04, 2024 at 07:43:58PM -0400, Jason Gunthorpe wrote:
> > @@ -1070,8 +1062,7 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> > * Each bit indicates that a used bit in a qword needs to be
> > * changed after unused_update is applied.
> > */
> > - if ((unused_update[i] & target_used[i]) !=
> > - (target[i] & target_used[i]))
> > + if ((unused_update[i] & target_used[i]) != target[i])
> > used_qword_diff |= 1 << i;
>
> Would you mind elaborating why we drop this "& target_used[i]"
> against the target[i] on the right?
It is undoning the temporary code a few patches before did and
bringing it back to the way it was in part 1.
It is ugly, the programmer requires that the target be properly
constructed but the way I migrated the CD code over doesn't guarentee
proper construction until we reach this patch because it edits the
current CD..
So there is this inbetween step where the programming code has to fix
an improperly constructed thing.
I didn't try too hard to simplify this, let me look again.
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 16/27] iommu/arm-smmu-v3: Keep track of valid CD entries in the cd_table
2024-03-19 13:55 ` Michael Shavit
@ 2024-03-20 18:21 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 18:21 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 19, 2024 at 09:55:17PM +0800, Michael Shavit wrote:
> On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > We no longer need a master->sva_enable to control what attaches are
> > allowed.
> >
> > Instead keep track inside the cd_table how many valid CD entries exist,
> > and if the RID has a valid entry.
> >
> > Replace all the attach focused master->sva_enabled tests with a check if
> > the CD has valid entries (or not). If there are any valid entries then the
> > CD table must be currently programmed to the STE.
> >
> > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> > .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 5 +---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 ++++++++++---------
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 10 +++++++
> > 3 files changed, 25 insertions(+), 16 deletions(-)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > index ab9de8e36c45f5..82b9c4d4061c3d 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > @@ -433,9 +433,6 @@ static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm,
> > if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1)
> > return -ENODEV;
> >
> I assume this doesn't matter because of subsequent patches, but the
> check above could also be removed since used_sid precisely means that
> the attached domain is an ARM_SMMU_DOMAIN_S1 domain.
Right, but lets move the delete here for clarity.
The same comment applies to some later patches too that do:
if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)) ||
!master->cd_table.used_sid)
return -ENODEV;
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> > index 98dc5885c48655..7e1f6af4ce4e79 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> > @@ -602,11 +602,21 @@ struct arm_smmu_ctx_desc_cfg {
> > dma_addr_t cdtab_dma;
> > struct arm_smmu_l1_ctx_desc *l1_desc;
> > unsigned int num_l1_ents;
> > + unsigned int used_ssids;
> > + bool used_sid;
>
> This probably deserves a comment. There's plenty of places where the
> "rid" domain is handled as the CD with ssid 0; but we don't count it
> as a used_ssid here.
As a page table? I didn't think so, the only way to get a CD page table
installed is through arm_smmu_write_cd_entry() which will capture
this..
Non paging domains don't get captured here, they are translating the
RID but they are not using the CD table.
> I also don't find the meaning of used_sid obvious, especially if I
> didn't have the context from the commit description.
Hum, okay, so looking over all of this again I think we can
simplify. At the end there was only one place using used_sid and it
can instead be calling arm_smmu_ssids_in_use() directly:
@@ -2987,11 +2986,13 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
* When the last user of the CD table goes away downgrade the STE back
* to a non-cd_table one.
*/
- if (last_ssid && !master->cd_table.used_sid) {
+ if (!arm_smmu_ssids_in_use(&master->cd_table)) {
struct iommu_domain *sid_domain =
iommu_get_domain_for_dev(master->dev);
- sid_domain->ops->attach_dev(sid_domain, master->dev);
+ if (domain->type == IOMMU_DOMAIN_IDENTITY ||
+ domain->type == IOMMU_DOMAIN_BLOCKED)
+ sid_domain->ops->attach_dev(sid_domain, dev);
}
Then we can get rid of used_sid and just have used_ssids count the
!0 ssids directly.
I reorganized a bunch of things in the in between patches so we go
more directly to this final outcome.
Thanks,
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 16/27] iommu/arm-smmu-v3: Keep track of valid CD entries in the cd_table
@ 2024-03-20 18:21 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 18:21 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 19, 2024 at 09:55:17PM +0800, Michael Shavit wrote:
> On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > We no longer need a master->sva_enable to control what attaches are
> > allowed.
> >
> > Instead keep track inside the cd_table how many valid CD entries exist,
> > and if the RID has a valid entry.
> >
> > Replace all the attach focused master->sva_enabled tests with a check if
> > the CD has valid entries (or not). If there are any valid entries then the
> > CD table must be currently programmed to the STE.
> >
> > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> > .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 5 +---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 ++++++++++---------
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 10 +++++++
> > 3 files changed, 25 insertions(+), 16 deletions(-)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > index ab9de8e36c45f5..82b9c4d4061c3d 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > @@ -433,9 +433,6 @@ static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm,
> > if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1)
> > return -ENODEV;
> >
> I assume this doesn't matter because of subsequent patches, but the
> check above could also be removed since used_sid precisely means that
> the attached domain is an ARM_SMMU_DOMAIN_S1 domain.
Right, but lets move the delete here for clarity.
The same comment applies to some later patches too that do:
if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)) ||
!master->cd_table.used_sid)
return -ENODEV;
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> > index 98dc5885c48655..7e1f6af4ce4e79 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> > @@ -602,11 +602,21 @@ struct arm_smmu_ctx_desc_cfg {
> > dma_addr_t cdtab_dma;
> > struct arm_smmu_l1_ctx_desc *l1_desc;
> > unsigned int num_l1_ents;
> > + unsigned int used_ssids;
> > + bool used_sid;
>
> This probably deserves a comment. There's plenty of places where the
> "rid" domain is handled as the CD with ssid 0; but we don't count it
> as a used_ssid here.
As a page table? I didn't think so, the only way to get a CD page table
installed is through arm_smmu_write_cd_entry() which will capture
this..
Non paging domains don't get captured here, they are translating the
RID but they are not using the CD table.
> I also don't find the meaning of used_sid obvious, especially if I
> didn't have the context from the commit description.
Hum, okay, so looking over all of this again I think we can
simplify. At the end there was only one place using used_sid and it
can instead be calling arm_smmu_ssids_in_use() directly:
@@ -2987,11 +2986,13 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
* When the last user of the CD table goes away downgrade the STE back
* to a non-cd_table one.
*/
- if (last_ssid && !master->cd_table.used_sid) {
+ if (!arm_smmu_ssids_in_use(&master->cd_table)) {
struct iommu_domain *sid_domain =
iommu_get_domain_for_dev(master->dev);
- sid_domain->ops->attach_dev(sid_domain, master->dev);
+ if (domain->type == IOMMU_DOMAIN_IDENTITY ||
+ domain->type == IOMMU_DOMAIN_BLOCKED)
+ sid_domain->ops->attach_dev(sid_domain, dev);
}
Then we can get rid of used_sid and just have used_ssids count the
!0 ssids directly.
I reorganized a bunch of things in the in between patches so we go
more directly to this final outcome.
Thanks,
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 12/27] iommu/arm-smmu-v3: Start building a generic PASID layer
2024-03-19 16:11 ` Michael Shavit
@ 2024-03-20 18:32 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 18:32 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Wed, Mar 20, 2024 at 12:11:36AM +0800, Michael Shavit wrote:
> > @@ -448,19 +446,10 @@ static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid,
> > goto err_free_bond;
> > }
> >
> > - cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm));
> > - if (!cdptr) {
> > - ret = -ENOMEM;
> > - goto err_put_notifier;
> > - }
> > - arm_smmu_make_sva_cd(&target, master, mm, bond->smmu_mn->cd->asid);
> > - arm_smmu_write_cd_entry(master, pasid, cdptr, &target);
> > -
> > + arm_smmu_make_sva_cd(target, master, mm, bond->smmu_mn->cd->asid);
> This can probably already move out to arm_smmu_sva_set_dev_pasid in
> this commit. Removing the arm_smmu_get_cd_ptr pre-alloc might also be
> possible if we're careful with failure of arm_smmu_set_pasid.
> This is eventually addressed in "iommu/arm-smmu-v3: Put the SVA mmu
> notifier in the smmu_domain", but that patch is already big enough as
> it is and the change fits nicely here.
Since all this bond related code is going to be deleted I didn't want
to mess with it too much, but sure it looks like this:
static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
struct device *dev, ioasid_t id)
{
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct mm_struct *mm = domain->mm;
struct arm_smmu_bond *bond;
struct arm_smmu_cd target;
int ret;
if (mm_get_enqcmd_pasid(mm) != id)
return -EINVAL;
mutex_lock(&sva_lock);
bond = __arm_smmu_sva_bind(dev, mm);
if (IS_ERR(bond)) {
mutex_unlock(&sva_lock);
return PTR_ERR(bond);
}
arm_smmu_make_sva_cd(&target, master, mm, bond->smmu_mn->cd->asid);
ret = arm_smmu_set_pasid(master, to_smmu_domain(domain), id, &target);
if (ret) {
list_del(&bond->list);
arm_smmu_mmu_notifier_put(bond->smmu_mn);
kfree(bond);
mutex_unlock(&sva_lock);
return ret;
}
mutex_unlock(&sva_lock);
return 0;
}
Which is much closer to the final arrangment in later patches.
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 12/27] iommu/arm-smmu-v3: Start building a generic PASID layer
@ 2024-03-20 18:32 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 18:32 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Wed, Mar 20, 2024 at 12:11:36AM +0800, Michael Shavit wrote:
> > @@ -448,19 +446,10 @@ static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid,
> > goto err_free_bond;
> > }
> >
> > - cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm));
> > - if (!cdptr) {
> > - ret = -ENOMEM;
> > - goto err_put_notifier;
> > - }
> > - arm_smmu_make_sva_cd(&target, master, mm, bond->smmu_mn->cd->asid);
> > - arm_smmu_write_cd_entry(master, pasid, cdptr, &target);
> > -
> > + arm_smmu_make_sva_cd(target, master, mm, bond->smmu_mn->cd->asid);
> This can probably already move out to arm_smmu_sva_set_dev_pasid in
> this commit. Removing the arm_smmu_get_cd_ptr pre-alloc might also be
> possible if we're careful with failure of arm_smmu_set_pasid.
> This is eventually addressed in "iommu/arm-smmu-v3: Put the SVA mmu
> notifier in the smmu_domain", but that patch is already big enough as
> it is and the change fits nicely here.
Since all this bond related code is going to be deleted I didn't want
to mess with it too much, but sure it looks like this:
static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
struct device *dev, ioasid_t id)
{
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct mm_struct *mm = domain->mm;
struct arm_smmu_bond *bond;
struct arm_smmu_cd target;
int ret;
if (mm_get_enqcmd_pasid(mm) != id)
return -EINVAL;
mutex_lock(&sva_lock);
bond = __arm_smmu_sva_bind(dev, mm);
if (IS_ERR(bond)) {
mutex_unlock(&sva_lock);
return PTR_ERR(bond);
}
arm_smmu_make_sva_cd(&target, master, mm, bond->smmu_mn->cd->asid);
ret = arm_smmu_set_pasid(master, to_smmu_domain(domain), id, &target);
if (ret) {
list_del(&bond->list);
arm_smmu_mmu_notifier_put(bond->smmu_mn);
kfree(bond);
mutex_unlock(&sva_lock);
return ret;
}
mutex_unlock(&sva_lock);
return 0;
}
Which is much closer to the final arrangment in later patches.
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 21/27] iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain
2024-03-19 16:23 ` Michael Shavit
@ 2024-03-20 18:35 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 18:35 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Wed, Mar 20, 2024 at 12:23:02AM +0800, Michael Shavit wrote:
> > static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
> > {
> > - kfree(to_smmu_domain(domain));
> > + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> > +
> > + /*
> > + * Ensure the ASID is empty in the iommu cache before allowing reuse.
> > + */
> > + arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
> > +
> > + /*
> > + * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can
> > + * still be called/running at this point. We allow the ASID to be
> > + * reused, and if there is a race then it just suffers harmless
> > + * unnecessary invalidation.
> > + */
> > + xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
> > +
> > + /*
> > + * Actual free is defered to the SRCU callback
> > + * arm_smmu_mmu_notifier_free()
> > + */
> > + mmu_notifier_put(&smmu_domain->mmu_notifier);
> > }
>
> I'm far from familiar with mmu_notifier and how the sva layers manage
> its reference count....but is calling mmu_notifier_unregister instead
> of mmu_notifier_put possible here?
Possible yes, but it is not good for performance..
The reason I created the mmnut_notifier_put() scheme is to avoid the
synchronize_srcu() penalty in places like this. We don't want to wait
around for a grace period just to destroy an iommu_domain.
So the put scheme piggybacks on the call_srcu that the does the rest
of the cleanup and we get the last bits of freeing done asynchronously.
All of this is because the notifier ops callers are protected by srcu
locks for performance and we can't guarentee no ops are running
without a grace period.
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 21/27] iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain
@ 2024-03-20 18:35 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 18:35 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Wed, Mar 20, 2024 at 12:23:02AM +0800, Michael Shavit wrote:
> > static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
> > {
> > - kfree(to_smmu_domain(domain));
> > + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> > +
> > + /*
> > + * Ensure the ASID is empty in the iommu cache before allowing reuse.
> > + */
> > + arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid);
> > +
> > + /*
> > + * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can
> > + * still be called/running at this point. We allow the ASID to be
> > + * reused, and if there is a race then it just suffers harmless
> > + * unnecessary invalidation.
> > + */
> > + xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
> > +
> > + /*
> > + * Actual free is defered to the SRCU callback
> > + * arm_smmu_mmu_notifier_free()
> > + */
> > + mmu_notifier_put(&smmu_domain->mmu_notifier);
> > }
>
> I'm far from familiar with mmu_notifier and how the sva layers manage
> its reference count....but is calling mmu_notifier_unregister instead
> of mmu_notifier_put possible here?
Possible yes, but it is not good for performance..
The reason I created the mmnut_notifier_put() scheme is to avoid the
synchronize_srcu() penalty in places like this. We don't want to wait
around for a grace period just to destroy an iommu_domain.
So the put scheme piggybacks on the call_srcu that the does the rest
of the cleanup and we get the last bits of freeing done asynchronously.
All of this is because the notifier ops callers are protected by srcu
locks for performance and we can't guarentee no ops are running
without a grace period.
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 18/27] iommu/arm-smmu-v3: Make SVA allocate a normal arm_smmu_domain
2024-03-19 14:52 ` Michael Shavit
@ 2024-03-20 23:20 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 23:20 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 19, 2024 at 10:52:11PM +0800, Michael Shavit wrote:
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index dd7f841cd19b3c..2db2b822292a87 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -2291,23 +2291,10 @@ static bool arm_smmu_capable(struct device *dev, enum iommu_cap cap)
> > }
> > }
> >
> > -static struct iommu_domain *arm_smmu_domain_alloc(unsigned type)
> > -{
> > -
> > - if (type == IOMMU_DOMAIN_SVA)
> > - return arm_smmu_sva_domain_alloc();
> > - return ERR_PTR(-EOPNOTSUPP);
> > -}
> > -
> > -static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
> > +struct arm_smmu_domain *arm_smmu_domain_alloc(void)
>
> Consider renaming arm_smmu_domain_free as well since there's asymmetry
> between arm_smmu_domain_alloc and arm_smmu_domain_free that could be a
> little confusing:
> 1. arm_smmu_domain_alloc is shared between arm_smmu_sva_domain_alloc
> and arm_smmu_domain_alloc_paging
> 2. arm_smmu_domain_free is only used by paging domains, with SVA
> domains freed through arm_smmu_sva_domain_free.
Yeah, that is a good idea, done
Thanks,
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 18/27] iommu/arm-smmu-v3: Make SVA allocate a normal arm_smmu_domain
@ 2024-03-20 23:20 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-20 23:20 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 19, 2024 at 10:52:11PM +0800, Michael Shavit wrote:
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index dd7f841cd19b3c..2db2b822292a87 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -2291,23 +2291,10 @@ static bool arm_smmu_capable(struct device *dev, enum iommu_cap cap)
> > }
> > }
> >
> > -static struct iommu_domain *arm_smmu_domain_alloc(unsigned type)
> > -{
> > -
> > - if (type == IOMMU_DOMAIN_SVA)
> > - return arm_smmu_sva_domain_alloc();
> > - return ERR_PTR(-EOPNOTSUPP);
> > -}
> > -
> > -static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev)
> > +struct arm_smmu_domain *arm_smmu_domain_alloc(void)
>
> Consider renaming arm_smmu_domain_free as well since there's asymmetry
> between arm_smmu_domain_alloc and arm_smmu_domain_free that could be a
> little confusing:
> 1. arm_smmu_domain_alloc is shared between arm_smmu_sva_domain_alloc
> and arm_smmu_domain_alloc_paging
> 2. arm_smmu_domain_free is only used by paging domains, with SVA
> domains freed through arm_smmu_sva_domain_free.
Yeah, that is a good idea, done
Thanks,
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 19/27] iommu/arm-smmu-v3: Keep track of arm_smmu_master_domain for SVA
2024-03-04 23:44 ` Jason Gunthorpe
@ 2024-03-21 10:47 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-21 10:47 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi, Daniel Mentz
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Currently the smmu_domain->devices list is unused for SVA domains.
> Fill it in with the SSID and master of every arm_smmu_set_pasid()
> using the same logic as the RID attach.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 23 +++++++++++++++++++--
> 1 file changed, 21 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 2db2b822292a87..6d15fe3a160acf 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2603,7 +2603,8 @@ to_smmu_domain_devices(struct iommu_domain *domain)
> /* The domain can be NULL only when processing the first attach */
> if (!domain)
> return NULL;
> - if (domain->type & __IOMMU_DOMAIN_PAGING)
> + if ((domain->type & __IOMMU_DOMAIN_PAGING) ||
> + domain->type == IOMMU_DOMAIN_SVA)
> return to_smmu_domain(domain);
> return NULL;
> }
> @@ -2813,7 +2814,9 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
> struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
> const struct arm_smmu_cd *cd)
> {
> + struct attach_state state = {.ssid = pasid};
> struct arm_smmu_cd *cdptr;
> + int ret;
>
> if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)))
> return -ENODEV;
> @@ -2821,14 +2824,30 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
> cdptr = arm_smmu_get_cd_ptr(master, pasid);
> if (!cdptr)
> return -ENOMEM;
> +
> + mutex_lock(&arm_smmu_asid_lock);
> + ret = arm_smmu_attach_prepare(master, &smmu_domain->domain, &state);
> + if (ret)
> + goto out_unlock;
> +
> arm_smmu_write_cd_entry(master, pasid, cdptr, cd);
> - return 0;
> +
> + arm_smmu_attach_commit(master, &state);
> +
> +out_unlock:
> + mutex_unlock(&arm_smmu_asid_lock);
> + return ret;
> }
arm_smmu_attach_commit tries to remove the master_domain entry from
the previous domain that this master was attached to. It gets a
pointer to the previous domain from the iommu framework with
iommu_get_domain_for_dev().
But in this path, arm_smmu_attach_prepare is creating a master_domain
entry for the pasid domain, which may be different from the one
returned by iommu_get_domain_for_dev() on the next attach.
I think this ended up being safe in the end because afaict the iommu
framework requires detaching the previous pasid domain before
attaching a new one. But nonetheless this is pretty fragile and
doesn't look intentional.
>
> void arm_smmu_remove_pasid(struct arm_smmu_master *master,
> struct arm_smmu_domain *smmu_domain, ioasid_t pasid)
> {
> + mutex_lock(&arm_smmu_asid_lock);
> arm_smmu_clear_cd(master, pasid);
> + if (master->ats_enabled)
> + arm_smmu_atc_inv_master(master, pasid);
> + arm_smmu_remove_master_domain(master, &smmu_domain->domain, pasid);
> + mutex_unlock(&arm_smmu_asid_lock);
> }
>
> static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
> --
> 2.43.2
>
>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 19/27] iommu/arm-smmu-v3: Keep track of arm_smmu_master_domain for SVA
@ 2024-03-21 10:47 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-21 10:47 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi, Daniel Mentz
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Currently the smmu_domain->devices list is unused for SVA domains.
> Fill it in with the SSID and master of every arm_smmu_set_pasid()
> using the same logic as the RID attach.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 23 +++++++++++++++++++--
> 1 file changed, 21 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 2db2b822292a87..6d15fe3a160acf 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2603,7 +2603,8 @@ to_smmu_domain_devices(struct iommu_domain *domain)
> /* The domain can be NULL only when processing the first attach */
> if (!domain)
> return NULL;
> - if (domain->type & __IOMMU_DOMAIN_PAGING)
> + if ((domain->type & __IOMMU_DOMAIN_PAGING) ||
> + domain->type == IOMMU_DOMAIN_SVA)
> return to_smmu_domain(domain);
> return NULL;
> }
> @@ -2813,7 +2814,9 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
> struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
> const struct arm_smmu_cd *cd)
> {
> + struct attach_state state = {.ssid = pasid};
> struct arm_smmu_cd *cdptr;
> + int ret;
>
> if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)))
> return -ENODEV;
> @@ -2821,14 +2824,30 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
> cdptr = arm_smmu_get_cd_ptr(master, pasid);
> if (!cdptr)
> return -ENOMEM;
> +
> + mutex_lock(&arm_smmu_asid_lock);
> + ret = arm_smmu_attach_prepare(master, &smmu_domain->domain, &state);
> + if (ret)
> + goto out_unlock;
> +
> arm_smmu_write_cd_entry(master, pasid, cdptr, cd);
> - return 0;
> +
> + arm_smmu_attach_commit(master, &state);
> +
> +out_unlock:
> + mutex_unlock(&arm_smmu_asid_lock);
> + return ret;
> }
arm_smmu_attach_commit tries to remove the master_domain entry from
the previous domain that this master was attached to. It gets a
pointer to the previous domain from the iommu framework with
iommu_get_domain_for_dev().
But in this path, arm_smmu_attach_prepare is creating a master_domain
entry for the pasid domain, which may be different from the one
returned by iommu_get_domain_for_dev() on the next attach.
I think this ended up being safe in the end because afaict the iommu
framework requires detaching the previous pasid domain before
attaching a new one. But nonetheless this is pretty fragile and
doesn't look intentional.
>
> void arm_smmu_remove_pasid(struct arm_smmu_master *master,
> struct arm_smmu_domain *smmu_domain, ioasid_t pasid)
> {
> + mutex_lock(&arm_smmu_asid_lock);
> arm_smmu_clear_cd(master, pasid);
> + if (master->ats_enabled)
> + arm_smmu_atc_inv_master(master, pasid);
> + arm_smmu_remove_master_domain(master, &smmu_domain->domain, pasid);
> + mutex_unlock(&arm_smmu_asid_lock);
> }
>
> static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
> --
> 2.43.2
>
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS
2024-03-04 23:44 ` Jason Gunthorpe
@ 2024-03-21 12:26 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-21 12:26 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> The core code allows the domain to be changed on the fly without a forced
> stop in BLOCKED/IDENTITY. In this flow the driver should just continually
> maintain the ATS with no change while the STE is updated.
>
> ATS relies on a linked list smmu_domain->devices to keep track of which
> masters have the domain programmed, but this list is also used by
> arm_smmu_share_asid(), unrelated to ats.
>
> Create two new functions to encapsulate this combined logic:
> arm_smmu_attach_prepare()
> <caller generates and sets the STE>
> arm_smmu_attach_commit()
>
> The two functions can sequence both enabling ATS and disabling across
> the STE store. Have every update of the STE use this sequence.
>
> Installing a S1/S2 domain always enables the ATS if the PCIe device
> supports it.
>
> The enable flow is now ordered differently to allow it to be hitless:
>
> 1) Add the master to the new smmu_domain->devices list
> 2) Program the STE
> 3) Enable ATS at PCIe
> 4) Remove the master from the old smmu_domain
>
> This flow ensures that invalidations to either domain will generate an ATC
> invalidation to the device while the STE is being switched. Thus we don't
> need to turn off the ATS anymore for correctness.
>
> The disable flow is the reverse:
> 1) Disable ATS at PCIe
> 2) Program the STE
> 3) Invalidate the ATC
> 4) Remove the master from the old smmu_domain
>
> Move the nr_ats_masters adjustments to be close to the list
> manipulations. It is a count of the number of ATS enabled masters
> currently in the list. This is stricly before and after the STE/CD are
> revised, and done under the list's spin_lock.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 198 ++++++++++++++------
> 1 file changed, 140 insertions(+), 58 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 51a1e7198fd1af..45f2190fc31786 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1527,7 +1527,8 @@ static void arm_smmu_make_bypass_ste(struct arm_smmu_ste *target)
> }
>
> static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
> - struct arm_smmu_master *master)
> + struct arm_smmu_master *master,
> + bool ats_enabled)
> {
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
> struct arm_smmu_device *smmu = master->smmu;
> @@ -1550,7 +1551,7 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
> STRTAB_STE_1_S1STALLD :
> 0) |
> FIELD_PREP(STRTAB_STE_1_EATS,
> - master->ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0));
> + ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0));
>
> if (smmu->features & ARM_SMMU_FEAT_E2H) {
> /*
> @@ -1578,7 +1579,8 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
>
> static void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target,
> struct arm_smmu_master *master,
> - struct arm_smmu_domain *smmu_domain)
> + struct arm_smmu_domain *smmu_domain,
> + bool ats_enabled)
> {
> struct arm_smmu_s2_cfg *s2_cfg = &smmu_domain->s2_cfg;
> const struct io_pgtable_cfg *pgtbl_cfg =
> @@ -1594,7 +1596,7 @@ static void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target,
>
> target->data[1] = cpu_to_le64(
> FIELD_PREP(STRTAB_STE_1_EATS,
> - master->ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0) |
> + ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0) |
> FIELD_PREP(STRTAB_STE_1_SHCFG,
> STRTAB_STE_1_SHCFG_INCOMING));
>
> @@ -2470,22 +2472,16 @@ static bool arm_smmu_ats_supported(struct arm_smmu_master *master)
> return dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev));
> }
>
> -static void arm_smmu_enable_ats(struct arm_smmu_master *master,
> - struct arm_smmu_domain *smmu_domain)
> +static void arm_smmu_enable_ats(struct arm_smmu_master *master)
> {
> size_t stu;
> struct pci_dev *pdev;
> struct arm_smmu_device *smmu = master->smmu;
>
> - /* Don't enable ATS at the endpoint if it's not enabled in the STE */
> - if (!master->ats_enabled)
> - return;
> -
> /* Smallest Translation Unit: log2 of the smallest supported granule */
> stu = __ffs(smmu->pgsize_bitmap);
> pdev = to_pci_dev(master->dev);
>
> - atomic_inc(&smmu_domain->nr_ats_masters);
> /*
> * ATC invalidation of PASID 0 causes the entire ATC to be flushed.
> */
> @@ -2494,22 +2490,6 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master,
> dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu);
> }
>
> -static void arm_smmu_disable_ats(struct arm_smmu_master *master,
> - struct arm_smmu_domain *smmu_domain)
> -{
> - if (!master->ats_enabled)
> - return;
> -
> - pci_disable_ats(to_pci_dev(master->dev));
> - /*
> - * Ensure ATS is disabled at the endpoint before we issue the
> - * ATC invalidation via the SMMU.
> - */
> - wmb();
> - arm_smmu_atc_inv_master(master);
> - atomic_dec(&smmu_domain->nr_ats_masters);
> -}
> -
> static int arm_smmu_enable_pasid(struct arm_smmu_master *master)
> {
> int ret;
> @@ -2573,39 +2553,145 @@ arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
> return NULL;
> }
>
> -static void arm_smmu_detach_dev(struct arm_smmu_master *master)
> +/*
> + * If the domain uses the smmu_domain->devices list return the arm_smmu_domain
> + * structure, otherwise NULL. These domains track attached devices so they can
> + * issue invalidations.
> + */
> +static struct arm_smmu_domain *
> +to_smmu_domain_devices(struct iommu_domain *domain)
> {
> - struct iommu_domain *domain = iommu_get_domain_for_dev(master->dev);
> + /* The domain can be NULL only when processing the first attach */
> + if (!domain)
> + return NULL;
> + if (domain->type & __IOMMU_DOMAIN_PAGING)
> + return to_smmu_domain(domain);
> + return NULL;
> +}
> +
> +static void arm_smmu_remove_master_domain(struct arm_smmu_master *master,
> + struct iommu_domain *domain)
> +{
> + struct arm_smmu_domain *smmu_domain = to_smmu_domain_devices(domain);
> struct arm_smmu_master_domain *master_domain;
> - struct arm_smmu_domain *smmu_domain;
> unsigned long flags;
>
> - if (!domain || !(domain->type & __IOMMU_DOMAIN_PAGING))
> + if (!smmu_domain)
> return;
>
> - smmu_domain = to_smmu_domain(domain);
> - arm_smmu_disable_ats(master, smmu_domain);
> -
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> master_domain = arm_smmu_find_master_domain(smmu_domain, master);
> if (master_domain) {
> list_del(&master_domain->devices_elm);
> kfree(master_domain);
> + if (master->ats_enabled)
> + atomic_dec(&smmu_domain->nr_ats_masters);
> }
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> +}
>
> - master->ats_enabled = false;
> +struct attach_state {
> + bool want_ats;
> + bool disable_ats;
> +};
> +
> +/*
> + * Prepare to attach a domain to a master. If disable_ats is not set this will
> + * turn on ATS if supported. smmu_domain can be NULL if the domain being
> + * attached does not have a page table and does not require invalidation
> + * tracking.
> + */
> +static int arm_smmu_attach_prepare(struct arm_smmu_master *master,
> + struct iommu_domain *domain,
> + struct attach_state *state)
> +{
> + struct arm_smmu_domain *smmu_domain =
> + to_smmu_domain_devices(domain);
> + struct arm_smmu_master_domain *master_domain;
> + unsigned long flags;
> +
> + /*
> + * arm_smmu_share_asid() must not see two domains pointing to the same
> + * arm_smmu_master_domain contents otherwise it could randomly write one
> + * or the other to the CD.
> + */
> + lockdep_assert_held(&arm_smmu_asid_lock);
> +
> + state->want_ats = !state->disable_ats && arm_smmu_ats_supported(master);
> +
> + if (smmu_domain) {
> + master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL);
> + if (!master_domain)
> + return -ENOMEM;
> + master_domain->master = master;
> +
> + /*
> + * During prepare we want the current smmu_domain and new
> + * smmu_domain to be in the devices list before we change any
> + * HW. This ensures that both domains will send ATS
> + * invalidations to the master until we are done.
> + *
> + * It is tempting to make this list only track masters that are
> + * using ATS, but arm_smmu_share_asid() also uses this to change
> + * the ASID of a domain, unrelated to ATS.
> + *
> + * Notice if we are re-attaching the same domain then the list
> + * will have two identical entries and commit will remove only
> + * one of them.
> + */
> + spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> + if (state->want_ats)
> + atomic_inc(&smmu_domain->nr_ats_masters);
> + list_add(&master_domain->devices_elm, &smmu_domain->devices);
> + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> + }
> +
> + if (!state->want_ats && master->ats_enabled) {
> + pci_disable_ats(to_pci_dev(master->dev));
> + /*
> + * This is probably overkill, but the config write for disabling
> + * ATS should complete before the STE is configured to generate
> + * UR to avoid AER noise.
> + */
> + wmb();
> + }
> + return 0;
> +}
> +
> +/*
> + * Commit is done after the STE/CD are configured with the EATS setting. It
> + * completes synchronizing the PCI device's ATC and finishes manipulating the
> + * smmu_domain->devices list.
> + */
> +static void arm_smmu_attach_commit(struct arm_smmu_master *master,
> + struct attach_state *state)
> +{
> + lockdep_assert_held(&arm_smmu_asid_lock);
> +
> + if (state->want_ats && !master->ats_enabled) {
> + arm_smmu_enable_ats(master);
> + } else if (master->ats_enabled) {
> + /*
> + * The translation has changed, flush the ATC. At this point the
> + * SMMU is translating for the new domain and both the old&new
> + * domain will issue invalidations.
> + */
> + arm_smmu_atc_inv_master(master);
> + }
> + master->ats_enabled = state->want_ats;
> +
> + arm_smmu_remove_master_domain(master,
> + iommu_get_domain_for_dev(master->dev));
> }
>
> static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> {
> int ret = 0;
> - unsigned long flags;
> struct arm_smmu_ste target;
> struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
> struct arm_smmu_device *smmu;
> struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> - struct arm_smmu_master_domain *master_domain;
> + struct attach_state state = {};
> struct arm_smmu_master *master;
> struct arm_smmu_cd *cdptr;
>
> @@ -2642,11 +2728,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> return -ENOMEM;
> }
>
> - master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL);
> - if (!master_domain)
> - return -ENOMEM;
> - master_domain->master = master;
> -
> /*
> * Prevent arm_smmu_share_asid() from trying to change the ASID
> * of either the old or new domain while we are working on it.
> @@ -2655,13 +2736,11 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> */
> mutex_lock(&arm_smmu_asid_lock);
>
> - arm_smmu_detach_dev(master);
> -
> - master->ats_enabled = arm_smmu_ats_supported(master);
> -
> - spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - list_add(&master_domain->devices_elm, &smmu_domain->devices);
> - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> + ret = arm_smmu_attach_prepare(master, domain, &state);
> + if (ret) {
> + mutex_unlock(&arm_smmu_asid_lock);
> + return ret;
> + }
>
> switch (smmu_domain->stage) {
> case ARM_SMMU_DOMAIN_S1: {
> @@ -2670,18 +2749,19 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> &target_cd);
> - arm_smmu_make_cdtable_ste(&target, master);
> + arm_smmu_make_cdtable_ste(&target, master, state.want_ats);
> arm_smmu_install_ste_for_dev(master, &target);
> break;
> }
> case ARM_SMMU_DOMAIN_S2:
> - arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
> + arm_smmu_make_s2_domain_ste(&target, master, smmu_domain,
> + state.want_ats);
> arm_smmu_install_ste_for_dev(master, &target);
> arm_smmu_clear_cd(master, IOMMU_NO_PASID);
> break;
> }
>
> - arm_smmu_enable_ats(master, smmu_domain);
> + arm_smmu_attach_commit(master, &state);
> mutex_unlock(&arm_smmu_asid_lock);
> return 0;
> }
> @@ -2715,10 +2795,11 @@ void arm_smmu_remove_pasid(struct arm_smmu_master *master,
> arm_smmu_clear_cd(master, pasid);
> }
>
> -static int arm_smmu_attach_dev_ste(struct device *dev,
> - struct arm_smmu_ste *ste)
> +static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
> + struct device *dev, struct arm_smmu_ste *ste)
> {
> struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> + struct attach_state state = {};
>
> if (arm_smmu_master_sva_enabled(master))
> return -EBUSY;
> @@ -2736,9 +2817,10 @@ static int arm_smmu_attach_dev_ste(struct device *dev,
> * the stream (STE.EATS == 0b00), causing F_BAD_ATS_TREQ and
> * F_TRANSL_FORBIDDEN events (IHI0070Ea 5.2 Stream Table Entry).
> */
> - arm_smmu_detach_dev(master);
> -
> + state.disable_ats = true;
> + arm_smmu_attach_prepare(master, domain, &state);
> arm_smmu_install_ste_for_dev(master, ste);
> + arm_smmu_attach_commit(master, &state);
> mutex_unlock(&arm_smmu_asid_lock);
>
> /*
> @@ -2756,7 +2838,7 @@ static int arm_smmu_attach_dev_identity(struct iommu_domain *domain,
> struct arm_smmu_ste ste;
>
> arm_smmu_make_bypass_ste(&ste);
> - return arm_smmu_attach_dev_ste(dev, &ste);
> + return arm_smmu_attach_dev_ste(domain, dev, &ste);
> }
>
> static const struct iommu_domain_ops arm_smmu_identity_ops = {
> @@ -2774,7 +2856,7 @@ static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain,
> struct arm_smmu_ste ste;
>
> arm_smmu_make_abort_ste(&ste);
> - return arm_smmu_attach_dev_ste(dev, &ste);
> + return arm_smmu_attach_dev_ste(domain, dev, &ste);
> }
>
> static const struct iommu_domain_ops arm_smmu_blocked_ops = {
> --
> 2.43.2
>
Overall I think the patch works, but it took me a while to really
digest the big picture. Just to make sure I fully understand:
We're trying to satisfy the following invariants for correctness:
1. Devices cannot get translations from a domain that was detached
after arm_smmu_attach_dev() returns.
2. Devices cannot get cached translations from a domain after
arm_smmu_atc_inv_domain() returns, regardless of whether the domain is
simultaneously being attached/detached from a device.
Apart from note point 2. above, the behaviour of translations while in
the middle of an arm_smmu_attach_dev doesn't have well defined
requirements:
1. Before this patch, devices may get translations belonging to the
old domain, then aborts or identity translations, and then the new
domain while arm_smmu_attach_dev() is in progress.
2. Ater this patch, devices may get an arbitrary mix of translations
belonging to the old domain and the new domain while
arm_smmu_attach_dev() is in progress.
While disabling and re-enabling ATS inside arm_smmu_attach_dev() would
meet the same requirements as this patch, it's not optimal since the
device may still have traffic on other pasids than the one being
manipulated.
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS
@ 2024-03-21 12:26 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-21 12:26 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> The core code allows the domain to be changed on the fly without a forced
> stop in BLOCKED/IDENTITY. In this flow the driver should just continually
> maintain the ATS with no change while the STE is updated.
>
> ATS relies on a linked list smmu_domain->devices to keep track of which
> masters have the domain programmed, but this list is also used by
> arm_smmu_share_asid(), unrelated to ats.
>
> Create two new functions to encapsulate this combined logic:
> arm_smmu_attach_prepare()
> <caller generates and sets the STE>
> arm_smmu_attach_commit()
>
> The two functions can sequence both enabling ATS and disabling across
> the STE store. Have every update of the STE use this sequence.
>
> Installing a S1/S2 domain always enables the ATS if the PCIe device
> supports it.
>
> The enable flow is now ordered differently to allow it to be hitless:
>
> 1) Add the master to the new smmu_domain->devices list
> 2) Program the STE
> 3) Enable ATS at PCIe
> 4) Remove the master from the old smmu_domain
>
> This flow ensures that invalidations to either domain will generate an ATC
> invalidation to the device while the STE is being switched. Thus we don't
> need to turn off the ATS anymore for correctness.
>
> The disable flow is the reverse:
> 1) Disable ATS at PCIe
> 2) Program the STE
> 3) Invalidate the ATC
> 4) Remove the master from the old smmu_domain
>
> Move the nr_ats_masters adjustments to be close to the list
> manipulations. It is a count of the number of ATS enabled masters
> currently in the list. This is stricly before and after the STE/CD are
> revised, and done under the list's spin_lock.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 198 ++++++++++++++------
> 1 file changed, 140 insertions(+), 58 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 51a1e7198fd1af..45f2190fc31786 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1527,7 +1527,8 @@ static void arm_smmu_make_bypass_ste(struct arm_smmu_ste *target)
> }
>
> static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
> - struct arm_smmu_master *master)
> + struct arm_smmu_master *master,
> + bool ats_enabled)
> {
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
> struct arm_smmu_device *smmu = master->smmu;
> @@ -1550,7 +1551,7 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
> STRTAB_STE_1_S1STALLD :
> 0) |
> FIELD_PREP(STRTAB_STE_1_EATS,
> - master->ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0));
> + ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0));
>
> if (smmu->features & ARM_SMMU_FEAT_E2H) {
> /*
> @@ -1578,7 +1579,8 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target,
>
> static void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target,
> struct arm_smmu_master *master,
> - struct arm_smmu_domain *smmu_domain)
> + struct arm_smmu_domain *smmu_domain,
> + bool ats_enabled)
> {
> struct arm_smmu_s2_cfg *s2_cfg = &smmu_domain->s2_cfg;
> const struct io_pgtable_cfg *pgtbl_cfg =
> @@ -1594,7 +1596,7 @@ static void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target,
>
> target->data[1] = cpu_to_le64(
> FIELD_PREP(STRTAB_STE_1_EATS,
> - master->ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0) |
> + ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0) |
> FIELD_PREP(STRTAB_STE_1_SHCFG,
> STRTAB_STE_1_SHCFG_INCOMING));
>
> @@ -2470,22 +2472,16 @@ static bool arm_smmu_ats_supported(struct arm_smmu_master *master)
> return dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev));
> }
>
> -static void arm_smmu_enable_ats(struct arm_smmu_master *master,
> - struct arm_smmu_domain *smmu_domain)
> +static void arm_smmu_enable_ats(struct arm_smmu_master *master)
> {
> size_t stu;
> struct pci_dev *pdev;
> struct arm_smmu_device *smmu = master->smmu;
>
> - /* Don't enable ATS at the endpoint if it's not enabled in the STE */
> - if (!master->ats_enabled)
> - return;
> -
> /* Smallest Translation Unit: log2 of the smallest supported granule */
> stu = __ffs(smmu->pgsize_bitmap);
> pdev = to_pci_dev(master->dev);
>
> - atomic_inc(&smmu_domain->nr_ats_masters);
> /*
> * ATC invalidation of PASID 0 causes the entire ATC to be flushed.
> */
> @@ -2494,22 +2490,6 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master,
> dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu);
> }
>
> -static void arm_smmu_disable_ats(struct arm_smmu_master *master,
> - struct arm_smmu_domain *smmu_domain)
> -{
> - if (!master->ats_enabled)
> - return;
> -
> - pci_disable_ats(to_pci_dev(master->dev));
> - /*
> - * Ensure ATS is disabled at the endpoint before we issue the
> - * ATC invalidation via the SMMU.
> - */
> - wmb();
> - arm_smmu_atc_inv_master(master);
> - atomic_dec(&smmu_domain->nr_ats_masters);
> -}
> -
> static int arm_smmu_enable_pasid(struct arm_smmu_master *master)
> {
> int ret;
> @@ -2573,39 +2553,145 @@ arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
> return NULL;
> }
>
> -static void arm_smmu_detach_dev(struct arm_smmu_master *master)
> +/*
> + * If the domain uses the smmu_domain->devices list return the arm_smmu_domain
> + * structure, otherwise NULL. These domains track attached devices so they can
> + * issue invalidations.
> + */
> +static struct arm_smmu_domain *
> +to_smmu_domain_devices(struct iommu_domain *domain)
> {
> - struct iommu_domain *domain = iommu_get_domain_for_dev(master->dev);
> + /* The domain can be NULL only when processing the first attach */
> + if (!domain)
> + return NULL;
> + if (domain->type & __IOMMU_DOMAIN_PAGING)
> + return to_smmu_domain(domain);
> + return NULL;
> +}
> +
> +static void arm_smmu_remove_master_domain(struct arm_smmu_master *master,
> + struct iommu_domain *domain)
> +{
> + struct arm_smmu_domain *smmu_domain = to_smmu_domain_devices(domain);
> struct arm_smmu_master_domain *master_domain;
> - struct arm_smmu_domain *smmu_domain;
> unsigned long flags;
>
> - if (!domain || !(domain->type & __IOMMU_DOMAIN_PAGING))
> + if (!smmu_domain)
> return;
>
> - smmu_domain = to_smmu_domain(domain);
> - arm_smmu_disable_ats(master, smmu_domain);
> -
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> master_domain = arm_smmu_find_master_domain(smmu_domain, master);
> if (master_domain) {
> list_del(&master_domain->devices_elm);
> kfree(master_domain);
> + if (master->ats_enabled)
> + atomic_dec(&smmu_domain->nr_ats_masters);
> }
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> +}
>
> - master->ats_enabled = false;
> +struct attach_state {
> + bool want_ats;
> + bool disable_ats;
> +};
> +
> +/*
> + * Prepare to attach a domain to a master. If disable_ats is not set this will
> + * turn on ATS if supported. smmu_domain can be NULL if the domain being
> + * attached does not have a page table and does not require invalidation
> + * tracking.
> + */
> +static int arm_smmu_attach_prepare(struct arm_smmu_master *master,
> + struct iommu_domain *domain,
> + struct attach_state *state)
> +{
> + struct arm_smmu_domain *smmu_domain =
> + to_smmu_domain_devices(domain);
> + struct arm_smmu_master_domain *master_domain;
> + unsigned long flags;
> +
> + /*
> + * arm_smmu_share_asid() must not see two domains pointing to the same
> + * arm_smmu_master_domain contents otherwise it could randomly write one
> + * or the other to the CD.
> + */
> + lockdep_assert_held(&arm_smmu_asid_lock);
> +
> + state->want_ats = !state->disable_ats && arm_smmu_ats_supported(master);
> +
> + if (smmu_domain) {
> + master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL);
> + if (!master_domain)
> + return -ENOMEM;
> + master_domain->master = master;
> +
> + /*
> + * During prepare we want the current smmu_domain and new
> + * smmu_domain to be in the devices list before we change any
> + * HW. This ensures that both domains will send ATS
> + * invalidations to the master until we are done.
> + *
> + * It is tempting to make this list only track masters that are
> + * using ATS, but arm_smmu_share_asid() also uses this to change
> + * the ASID of a domain, unrelated to ATS.
> + *
> + * Notice if we are re-attaching the same domain then the list
> + * will have two identical entries and commit will remove only
> + * one of them.
> + */
> + spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> + if (state->want_ats)
> + atomic_inc(&smmu_domain->nr_ats_masters);
> + list_add(&master_domain->devices_elm, &smmu_domain->devices);
> + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> + }
> +
> + if (!state->want_ats && master->ats_enabled) {
> + pci_disable_ats(to_pci_dev(master->dev));
> + /*
> + * This is probably overkill, but the config write for disabling
> + * ATS should complete before the STE is configured to generate
> + * UR to avoid AER noise.
> + */
> + wmb();
> + }
> + return 0;
> +}
> +
> +/*
> + * Commit is done after the STE/CD are configured with the EATS setting. It
> + * completes synchronizing the PCI device's ATC and finishes manipulating the
> + * smmu_domain->devices list.
> + */
> +static void arm_smmu_attach_commit(struct arm_smmu_master *master,
> + struct attach_state *state)
> +{
> + lockdep_assert_held(&arm_smmu_asid_lock);
> +
> + if (state->want_ats && !master->ats_enabled) {
> + arm_smmu_enable_ats(master);
> + } else if (master->ats_enabled) {
> + /*
> + * The translation has changed, flush the ATC. At this point the
> + * SMMU is translating for the new domain and both the old&new
> + * domain will issue invalidations.
> + */
> + arm_smmu_atc_inv_master(master);
> + }
> + master->ats_enabled = state->want_ats;
> +
> + arm_smmu_remove_master_domain(master,
> + iommu_get_domain_for_dev(master->dev));
> }
>
> static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> {
> int ret = 0;
> - unsigned long flags;
> struct arm_smmu_ste target;
> struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
> struct arm_smmu_device *smmu;
> struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> - struct arm_smmu_master_domain *master_domain;
> + struct attach_state state = {};
> struct arm_smmu_master *master;
> struct arm_smmu_cd *cdptr;
>
> @@ -2642,11 +2728,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> return -ENOMEM;
> }
>
> - master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL);
> - if (!master_domain)
> - return -ENOMEM;
> - master_domain->master = master;
> -
> /*
> * Prevent arm_smmu_share_asid() from trying to change the ASID
> * of either the old or new domain while we are working on it.
> @@ -2655,13 +2736,11 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> */
> mutex_lock(&arm_smmu_asid_lock);
>
> - arm_smmu_detach_dev(master);
> -
> - master->ats_enabled = arm_smmu_ats_supported(master);
> -
> - spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - list_add(&master_domain->devices_elm, &smmu_domain->devices);
> - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> + ret = arm_smmu_attach_prepare(master, domain, &state);
> + if (ret) {
> + mutex_unlock(&arm_smmu_asid_lock);
> + return ret;
> + }
>
> switch (smmu_domain->stage) {
> case ARM_SMMU_DOMAIN_S1: {
> @@ -2670,18 +2749,19 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> &target_cd);
> - arm_smmu_make_cdtable_ste(&target, master);
> + arm_smmu_make_cdtable_ste(&target, master, state.want_ats);
> arm_smmu_install_ste_for_dev(master, &target);
> break;
> }
> case ARM_SMMU_DOMAIN_S2:
> - arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
> + arm_smmu_make_s2_domain_ste(&target, master, smmu_domain,
> + state.want_ats);
> arm_smmu_install_ste_for_dev(master, &target);
> arm_smmu_clear_cd(master, IOMMU_NO_PASID);
> break;
> }
>
> - arm_smmu_enable_ats(master, smmu_domain);
> + arm_smmu_attach_commit(master, &state);
> mutex_unlock(&arm_smmu_asid_lock);
> return 0;
> }
> @@ -2715,10 +2795,11 @@ void arm_smmu_remove_pasid(struct arm_smmu_master *master,
> arm_smmu_clear_cd(master, pasid);
> }
>
> -static int arm_smmu_attach_dev_ste(struct device *dev,
> - struct arm_smmu_ste *ste)
> +static int arm_smmu_attach_dev_ste(struct iommu_domain *domain,
> + struct device *dev, struct arm_smmu_ste *ste)
> {
> struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> + struct attach_state state = {};
>
> if (arm_smmu_master_sva_enabled(master))
> return -EBUSY;
> @@ -2736,9 +2817,10 @@ static int arm_smmu_attach_dev_ste(struct device *dev,
> * the stream (STE.EATS == 0b00), causing F_BAD_ATS_TREQ and
> * F_TRANSL_FORBIDDEN events (IHI0070Ea 5.2 Stream Table Entry).
> */
> - arm_smmu_detach_dev(master);
> -
> + state.disable_ats = true;
> + arm_smmu_attach_prepare(master, domain, &state);
> arm_smmu_install_ste_for_dev(master, ste);
> + arm_smmu_attach_commit(master, &state);
> mutex_unlock(&arm_smmu_asid_lock);
>
> /*
> @@ -2756,7 +2838,7 @@ static int arm_smmu_attach_dev_identity(struct iommu_domain *domain,
> struct arm_smmu_ste ste;
>
> arm_smmu_make_bypass_ste(&ste);
> - return arm_smmu_attach_dev_ste(dev, &ste);
> + return arm_smmu_attach_dev_ste(domain, dev, &ste);
> }
>
> static const struct iommu_domain_ops arm_smmu_identity_ops = {
> @@ -2774,7 +2856,7 @@ static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain,
> struct arm_smmu_ste ste;
>
> arm_smmu_make_abort_ste(&ste);
> - return arm_smmu_attach_dev_ste(dev, &ste);
> + return arm_smmu_attach_dev_ste(domain, dev, &ste);
> }
>
> static const struct iommu_domain_ops arm_smmu_blocked_ops = {
> --
> 2.43.2
>
Overall I think the patch works, but it took me a while to really
digest the big picture. Just to make sure I fully understand:
We're trying to satisfy the following invariants for correctness:
1. Devices cannot get translations from a domain that was detached
after arm_smmu_attach_dev() returns.
2. Devices cannot get cached translations from a domain after
arm_smmu_atc_inv_domain() returns, regardless of whether the domain is
simultaneously being attached/detached from a device.
Apart from note point 2. above, the behaviour of translations while in
the middle of an arm_smmu_attach_dev doesn't have well defined
requirements:
1. Before this patch, devices may get translations belonging to the
old domain, then aborts or identity translations, and then the new
domain while arm_smmu_attach_dev() is in progress.
2. Ater this patch, devices may get an arbitrary mix of translations
belonging to the old domain and the new domain while
arm_smmu_attach_dev() is in progress.
While disabling and re-enabling ATS inside arm_smmu_attach_dev() would
meet the same requirements as this patch, it's not optimal since the
device may still have traffic on other pasids than the one being
manipulated.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS
2024-03-21 12:26 ` Michael Shavit
@ 2024-03-21 13:28 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-21 13:28 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Thu, Mar 21, 2024 at 08:26:43PM +0800, Michael Shavit wrote:
> Overall I think the patch works, but it took me a while to really
> digest the big picture. Just to make sure I fully understand:
>
> We're trying to satisfy the following invariants for correctness:
> 1. Devices cannot get translations from a domain that was detached
> after arm_smmu_attach_dev() returns.
Yes, regardless of ATS this is the API requirement.
> 2. Devices cannot get cached translations from a domain after
> arm_smmu_atc_inv_domain() returns, regardless of whether the domain is
> simultaneously being attached/detached from a device.
There is no simultaneously here since the group lock is held by the
core code.
> Apart from note point 2. above, the behaviour of translations while in
> the middle of an arm_smmu_attach_dev doesn't have well defined
> requirements:
> 1. Before this patch, devices may get translations belonging to the
> old domain, then aborts or identity translations, and then the new
> domain while arm_smmu_attach_dev() is in progress.
Yes
> 2. Ater this patch, devices may get an arbitrary mix of translations
> belonging to the old domain and the new domain while
> arm_smmu_attach_dev() is in progress.
Yes, but no aborts.
> While disabling and re-enabling ATS inside arm_smmu_attach_dev() would
> meet the same requirements as this patch, it's not optimal since the
> device may still have traffic on other pasids than the one being
> manipulated.
Yes, we can't just disable ATS due to the PASIDs. A SVA PASID might be
present and disabling ATS to change the RID domain would completely
wreck it. Today this scenario is prevented by sva_enable, which is
removed in following patches.
I added this note for context to the commit message:
This is part of the bigger picture to allow changing the RID domain while
a PASID is in use. If a SVA PASID is relying on ATS to function then
changing the RID domain cannot just temporarily toggle ATS off without
also wrecking the SVA PASID. The new infrastructure here is organized so
that the PASID attach/detach flows will make use of it as well in
following patches.
Thanks,
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS
@ 2024-03-21 13:28 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-21 13:28 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Thu, Mar 21, 2024 at 08:26:43PM +0800, Michael Shavit wrote:
> Overall I think the patch works, but it took me a while to really
> digest the big picture. Just to make sure I fully understand:
>
> We're trying to satisfy the following invariants for correctness:
> 1. Devices cannot get translations from a domain that was detached
> after arm_smmu_attach_dev() returns.
Yes, regardless of ATS this is the API requirement.
> 2. Devices cannot get cached translations from a domain after
> arm_smmu_atc_inv_domain() returns, regardless of whether the domain is
> simultaneously being attached/detached from a device.
There is no simultaneously here since the group lock is held by the
core code.
> Apart from note point 2. above, the behaviour of translations while in
> the middle of an arm_smmu_attach_dev doesn't have well defined
> requirements:
> 1. Before this patch, devices may get translations belonging to the
> old domain, then aborts or identity translations, and then the new
> domain while arm_smmu_attach_dev() is in progress.
Yes
> 2. Ater this patch, devices may get an arbitrary mix of translations
> belonging to the old domain and the new domain while
> arm_smmu_attach_dev() is in progress.
Yes, but no aborts.
> While disabling and re-enabling ATS inside arm_smmu_attach_dev() would
> meet the same requirements as this patch, it's not optimal since the
> device may still have traffic on other pasids than the one being
> manipulated.
Yes, we can't just disable ATS due to the PASIDs. A SVA PASID might be
present and disabling ATS to change the RID domain would completely
wreck it. Today this scenario is prevented by sva_enable, which is
removed in following patches.
I added this note for context to the commit message:
This is part of the bigger picture to allow changing the RID domain while
a PASID is in use. If a SVA PASID is relying on ATS to function then
changing the RID domain cannot just temporarily toggle ATS off without
also wrecking the SVA PASID. The new infrastructure here is organized so
that the PASID attach/detach flows will make use of it as well in
following patches.
Thanks,
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 19/27] iommu/arm-smmu-v3: Keep track of arm_smmu_master_domain for SVA
2024-03-21 10:47 ` Michael Shavit
@ 2024-03-21 13:55 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-21 13:55 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi, Daniel Mentz
On Thu, Mar 21, 2024 at 06:47:41PM +0800, Michael Shavit wrote:
> On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > Currently the smmu_domain->devices list is unused for SVA domains.
> > Fill it in with the SSID and master of every arm_smmu_set_pasid()
> > using the same logic as the RID attach.
> >
> > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 23 +++++++++++++++++++--
> > 1 file changed, 21 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index 2db2b822292a87..6d15fe3a160acf 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -2603,7 +2603,8 @@ to_smmu_domain_devices(struct iommu_domain *domain)
> > /* The domain can be NULL only when processing the first attach */
> > if (!domain)
> > return NULL;
> > - if (domain->type & __IOMMU_DOMAIN_PAGING)
> > + if ((domain->type & __IOMMU_DOMAIN_PAGING) ||
> > + domain->type == IOMMU_DOMAIN_SVA)
> > return to_smmu_domain(domain);
> > return NULL;
> > }
> > @@ -2813,7 +2814,9 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
> > struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
> > const struct arm_smmu_cd *cd)
> > {
> > + struct attach_state state = {.ssid = pasid};
> > struct arm_smmu_cd *cdptr;
> > + int ret;
> >
> > if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)))
> > return -ENODEV;
> > @@ -2821,14 +2824,30 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
> > cdptr = arm_smmu_get_cd_ptr(master, pasid);
> > if (!cdptr)
> > return -ENOMEM;
> > +
> > + mutex_lock(&arm_smmu_asid_lock);
> > + ret = arm_smmu_attach_prepare(master, &smmu_domain->domain, &state);
> > + if (ret)
> > + goto out_unlock;
> > +
> > arm_smmu_write_cd_entry(master, pasid, cdptr, cd);
> > - return 0;
> > +
> > + arm_smmu_attach_commit(master, &state);
> > +
> > +out_unlock:
> > + mutex_unlock(&arm_smmu_asid_lock);
> > + return ret;
> > }
>
> arm_smmu_attach_commit tries to remove the master_domain entry from
> the previous domain that this master was attached to. It gets a
> pointer to the previous domain from the iommu framework with
> iommu_get_domain_for_dev().
> But in this path, arm_smmu_attach_prepare is creating a master_domain
> entry for the pasid domain, which may be different from the one
> returned by iommu_get_domain_for_dev() on the next attach.
>
> I think this ended up being safe in the end because afaict the iommu
> framework requires detaching the previous pasid domain before
> attaching a new one. But nonetheless this is pretty fragile and
> doesn't look intentional.
Oh yeah, this is a mistake!
Commit needs to take in the old domain from the caller which knows
what it actually is.
Thanks,
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 19/27] iommu/arm-smmu-v3: Keep track of arm_smmu_master_domain for SVA
@ 2024-03-21 13:55 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-21 13:55 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi, Daniel Mentz
On Thu, Mar 21, 2024 at 06:47:41PM +0800, Michael Shavit wrote:
> On Tue, Mar 5, 2024 at 7:44 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > Currently the smmu_domain->devices list is unused for SVA domains.
> > Fill it in with the SSID and master of every arm_smmu_set_pasid()
> > using the same logic as the RID attach.
> >
> > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 23 +++++++++++++++++++--
> > 1 file changed, 21 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index 2db2b822292a87..6d15fe3a160acf 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -2603,7 +2603,8 @@ to_smmu_domain_devices(struct iommu_domain *domain)
> > /* The domain can be NULL only when processing the first attach */
> > if (!domain)
> > return NULL;
> > - if (domain->type & __IOMMU_DOMAIN_PAGING)
> > + if ((domain->type & __IOMMU_DOMAIN_PAGING) ||
> > + domain->type == IOMMU_DOMAIN_SVA)
> > return to_smmu_domain(domain);
> > return NULL;
> > }
> > @@ -2813,7 +2814,9 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
> > struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
> > const struct arm_smmu_cd *cd)
> > {
> > + struct attach_state state = {.ssid = pasid};
> > struct arm_smmu_cd *cdptr;
> > + int ret;
> >
> > if (!arm_smmu_is_s1_domain(iommu_get_domain_for_dev(master->dev)))
> > return -ENODEV;
> > @@ -2821,14 +2824,30 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
> > cdptr = arm_smmu_get_cd_ptr(master, pasid);
> > if (!cdptr)
> > return -ENOMEM;
> > +
> > + mutex_lock(&arm_smmu_asid_lock);
> > + ret = arm_smmu_attach_prepare(master, &smmu_domain->domain, &state);
> > + if (ret)
> > + goto out_unlock;
> > +
> > arm_smmu_write_cd_entry(master, pasid, cdptr, cd);
> > - return 0;
> > +
> > + arm_smmu_attach_commit(master, &state);
> > +
> > +out_unlock:
> > + mutex_unlock(&arm_smmu_asid_lock);
> > + return ret;
> > }
>
> arm_smmu_attach_commit tries to remove the master_domain entry from
> the previous domain that this master was attached to. It gets a
> pointer to the previous domain from the iommu framework with
> iommu_get_domain_for_dev().
> But in this path, arm_smmu_attach_prepare is creating a master_domain
> entry for the pasid domain, which may be different from the one
> returned by iommu_get_domain_for_dev() on the next attach.
>
> I think this ended up being safe in the end because afaict the iommu
> framework requires detaching the previous pasid domain before
> attaching a new one. But nonetheless this is pretty fragile and
> doesn't look intentional.
Oh yeah, this is a mistake!
Commit needs to take in the old domain from the caller which knows
what it actually is.
Thanks,
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS
2024-03-21 13:28 ` Jason Gunthorpe
@ 2024-03-21 14:53 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-21 14:53 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Thu, Mar 21, 2024 at 9:28 PM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> On Thu, Mar 21, 2024 at 08:26:43PM +0800, Michael Shavit wrote:
> > Overall I think the patch works, but it took me a while to really
> > digest the big picture. Just to make sure I fully understand:
> >
> > We're trying to satisfy the following invariants for correctness:
> > 1. Devices cannot get translations from a domain that was detached
> > after arm_smmu_attach_dev() returns.
>
> Yes, regardless of ATS this is the API requirement.
>
> > 2. Devices cannot get cached translations from a domain after
> > arm_smmu_atc_inv_domain() returns, regardless of whether the domain is
> > simultaneously being attached/detached from a device.
>
> There is no simultaneously here since the group lock is held by the
> core code.
I meant a call to arm_smmu_attach_dev concurrent with
arm_smmu_atc_inv_domain (through tlb_flush_all).
>
> > Apart from note point 2. above, the behaviour of translations while in
> > the middle of an arm_smmu_attach_dev doesn't have well defined
> > requirements:
> > 1. Before this patch, devices may get translations belonging to the
> > old domain, then aborts or identity translations, and then the new
> > domain while arm_smmu_attach_dev() is in progress.
>
> Yes
>
> > 2. Ater this patch, devices may get an arbitrary mix of translations
> > belonging to the old domain and the new domain while
> > arm_smmu_attach_dev() is in progress.
>
> Yes, but no aborts.
>
> > While disabling and re-enabling ATS inside arm_smmu_attach_dev() would
> > meet the same requirements as this patch, it's not optimal since the
> > device may still have traffic on other pasids than the one being
> > manipulated.
>
> Yes, we can't just disable ATS due to the PASIDs. A SVA PASID might be
> present and disabling ATS to change the RID domain would completely
> wreck it. Today this scenario is prevented by sva_enable, which is
> removed in following patches.
What do you mean by wrecking? We might slow it down to a crawl but we
wouldn't be corrupting or destroying anything by disabling ATS while
SVA is running would we?
Oh does disabling ATS abort all transactions rather than making it transparent?
>
> I added this note for context to the commit message:
>
> This is part of the bigger picture to allow changing the RID domain while
> a PASID is in use. If a SVA PASID is relying on ATS to function then
> changing the RID domain cannot just temporarily toggle ATS off without
> also wrecking the SVA PASID. The new infrastructure here is organized so
> that the PASID attach/detach flows will make use of it as well in
> following patches.
>
> Thanks,
> Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS
@ 2024-03-21 14:53 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-21 14:53 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Thu, Mar 21, 2024 at 9:28 PM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> On Thu, Mar 21, 2024 at 08:26:43PM +0800, Michael Shavit wrote:
> > Overall I think the patch works, but it took me a while to really
> > digest the big picture. Just to make sure I fully understand:
> >
> > We're trying to satisfy the following invariants for correctness:
> > 1. Devices cannot get translations from a domain that was detached
> > after arm_smmu_attach_dev() returns.
>
> Yes, regardless of ATS this is the API requirement.
>
> > 2. Devices cannot get cached translations from a domain after
> > arm_smmu_atc_inv_domain() returns, regardless of whether the domain is
> > simultaneously being attached/detached from a device.
>
> There is no simultaneously here since the group lock is held by the
> core code.
I meant a call to arm_smmu_attach_dev concurrent with
arm_smmu_atc_inv_domain (through tlb_flush_all).
>
> > Apart from note point 2. above, the behaviour of translations while in
> > the middle of an arm_smmu_attach_dev doesn't have well defined
> > requirements:
> > 1. Before this patch, devices may get translations belonging to the
> > old domain, then aborts or identity translations, and then the new
> > domain while arm_smmu_attach_dev() is in progress.
>
> Yes
>
> > 2. Ater this patch, devices may get an arbitrary mix of translations
> > belonging to the old domain and the new domain while
> > arm_smmu_attach_dev() is in progress.
>
> Yes, but no aborts.
>
> > While disabling and re-enabling ATS inside arm_smmu_attach_dev() would
> > meet the same requirements as this patch, it's not optimal since the
> > device may still have traffic on other pasids than the one being
> > manipulated.
>
> Yes, we can't just disable ATS due to the PASIDs. A SVA PASID might be
> present and disabling ATS to change the RID domain would completely
> wreck it. Today this scenario is prevented by sva_enable, which is
> removed in following patches.
What do you mean by wrecking? We might slow it down to a crawl but we
wouldn't be corrupting or destroying anything by disabling ATS while
SVA is running would we?
Oh does disabling ATS abort all transactions rather than making it transparent?
>
> I added this note for context to the commit message:
>
> This is part of the bigger picture to allow changing the RID domain while
> a PASID is in use. If a SVA PASID is relying on ATS to function then
> changing the RID domain cannot just temporarily toggle ATS off without
> also wrecking the SVA PASID. The new infrastructure here is organized so
> that the PASID attach/detach flows will make use of it as well in
> following patches.
>
> Thanks,
> Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS
2024-03-21 14:53 ` Michael Shavit
@ 2024-03-21 14:57 ` Michael Shavit
-1 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-21 14:57 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Thu, Mar 21, 2024 at 10:53 PM Michael Shavit <mshavit@google.com> wrote:
>
> On Thu, Mar 21, 2024 at 9:28 PM Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > On Thu, Mar 21, 2024 at 08:26:43PM +0800, Michael Shavit wrote:
> > > Overall I think the patch works, but it took me a while to really
> > > digest the big picture. Just to make sure I fully understand:
> > >
> > > We're trying to satisfy the following invariants for correctness:
> > > 1. Devices cannot get translations from a domain that was detached
> > > after arm_smmu_attach_dev() returns.
> >
> > Yes, regardless of ATS this is the API requirement.
> >
> > > 2. Devices cannot get cached translations from a domain after
> > > arm_smmu_atc_inv_domain() returns, regardless of whether the domain is
> > > simultaneously being attached/detached from a device.
> >
> > There is no simultaneously here since the group lock is held by the
> > core code.
>
> I meant a call to arm_smmu_attach_dev concurrent with
> arm_smmu_atc_inv_domain (through tlb_flush_all).
>
> >
> > > Apart from note point 2. above, the behaviour of translations while in
> > > the middle of an arm_smmu_attach_dev doesn't have well defined
> > > requirements:
> > > 1. Before this patch, devices may get translations belonging to the
> > > old domain, then aborts or identity translations, and then the new
> > > domain while arm_smmu_attach_dev() is in progress.
> >
> > Yes
> >
> > > 2. Ater this patch, devices may get an arbitrary mix of translations
> > > belonging to the old domain and the new domain while
> > > arm_smmu_attach_dev() is in progress.
> >
> > Yes, but no aborts.
> >
> > > While disabling and re-enabling ATS inside arm_smmu_attach_dev() would
> > > meet the same requirements as this patch, it's not optimal since the
> > > device may still have traffic on other pasids than the one being
> > > manipulated.
> >
> > Yes, we can't just disable ATS due to the PASIDs. A SVA PASID might be
> > present and disabling ATS to change the RID domain would completely
> > wreck it. Today this scenario is prevented by sva_enable, which is
> > removed in following patches.
>
> What do you mean by wrecking? We might slow it down to a crawl but we
> wouldn't be corrupting or destroying anything by disabling ATS while
> SVA is running would we?
> Oh does disabling ATS abort all transactions rather than making it transparent?
>
> >
> > I added this note for context to the commit message:
> >
> > This is part of the bigger picture to allow changing the RID domain while
> > a PASID is in use. If a SVA PASID is relying on ATS to function then
> > changing the RID domain cannot just temporarily toggle ATS off without
> > also wrecking the SVA PASID. The new infrastructure here is organized so
> > that the PASID attach/detach flows will make use of it as well in
> > following patches.
Huh ok yeah, I thought this was just enabling/disabling caching of
translations in the ATC but looks like it's disabling translation for
those PCIe devices.
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS
@ 2024-03-21 14:57 ` Michael Shavit
0 siblings, 0 replies; 232+ messages in thread
From: Michael Shavit @ 2024-03-21 14:57 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Thu, Mar 21, 2024 at 10:53 PM Michael Shavit <mshavit@google.com> wrote:
>
> On Thu, Mar 21, 2024 at 9:28 PM Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > On Thu, Mar 21, 2024 at 08:26:43PM +0800, Michael Shavit wrote:
> > > Overall I think the patch works, but it took me a while to really
> > > digest the big picture. Just to make sure I fully understand:
> > >
> > > We're trying to satisfy the following invariants for correctness:
> > > 1. Devices cannot get translations from a domain that was detached
> > > after arm_smmu_attach_dev() returns.
> >
> > Yes, regardless of ATS this is the API requirement.
> >
> > > 2. Devices cannot get cached translations from a domain after
> > > arm_smmu_atc_inv_domain() returns, regardless of whether the domain is
> > > simultaneously being attached/detached from a device.
> >
> > There is no simultaneously here since the group lock is held by the
> > core code.
>
> I meant a call to arm_smmu_attach_dev concurrent with
> arm_smmu_atc_inv_domain (through tlb_flush_all).
>
> >
> > > Apart from note point 2. above, the behaviour of translations while in
> > > the middle of an arm_smmu_attach_dev doesn't have well defined
> > > requirements:
> > > 1. Before this patch, devices may get translations belonging to the
> > > old domain, then aborts or identity translations, and then the new
> > > domain while arm_smmu_attach_dev() is in progress.
> >
> > Yes
> >
> > > 2. Ater this patch, devices may get an arbitrary mix of translations
> > > belonging to the old domain and the new domain while
> > > arm_smmu_attach_dev() is in progress.
> >
> > Yes, but no aborts.
> >
> > > While disabling and re-enabling ATS inside arm_smmu_attach_dev() would
> > > meet the same requirements as this patch, it's not optimal since the
> > > device may still have traffic on other pasids than the one being
> > > manipulated.
> >
> > Yes, we can't just disable ATS due to the PASIDs. A SVA PASID might be
> > present and disabling ATS to change the RID domain would completely
> > wreck it. Today this scenario is prevented by sva_enable, which is
> > removed in following patches.
>
> What do you mean by wrecking? We might slow it down to a crawl but we
> wouldn't be corrupting or destroying anything by disabling ATS while
> SVA is running would we?
> Oh does disabling ATS abort all transactions rather than making it transparent?
>
> >
> > I added this note for context to the commit message:
> >
> > This is part of the bigger picture to allow changing the RID domain while
> > a PASID is in use. If a SVA PASID is relying on ATS to function then
> > changing the RID domain cannot just temporarily toggle ATS off without
> > also wrecking the SVA PASID. The new infrastructure here is organized so
> > that the PASID attach/detach flows will make use of it as well in
> > following patches.
Huh ok yeah, I thought this was just enabling/disabling caching of
translations in the ATC but looks like it's disabling translation for
those PCIe devices.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS
2024-03-21 14:53 ` Michael Shavit
@ 2024-03-21 17:32 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-21 17:32 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Thu, Mar 21, 2024 at 10:53:20PM +0800, Michael Shavit wrote:
> On Thu, Mar 21, 2024 at 9:28 PM Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > On Thu, Mar 21, 2024 at 08:26:43PM +0800, Michael Shavit wrote:
> > > Overall I think the patch works, but it took me a while to really
> > > digest the big picture. Just to make sure I fully understand:
> > >
> > > We're trying to satisfy the following invariants for correctness:
> > > 1. Devices cannot get translations from a domain that was detached
> > > after arm_smmu_attach_dev() returns.
> >
> > Yes, regardless of ATS this is the API requirement.
> >
> > > 2. Devices cannot get cached translations from a domain after
> > > arm_smmu_atc_inv_domain() returns, regardless of whether the domain is
> > > simultaneously being attached/detached from a device.
> >
> > There is no simultaneously here since the group lock is held by the
> > core code.
>
> I meant a call to arm_smmu_attach_dev concurrent with
> arm_smmu_atc_inv_domain (through tlb_flush_all).
Oh, yes that is the concurrency that drives alot of this organization
> > Yes, we can't just disable ATS due to the PASIDs. A SVA PASID might be
> > present and disabling ATS to change the RID domain would completely
> > wreck it. Today this scenario is prevented by sva_enable, which is
> > removed in following patches.
>
> What do you mean by wrecking? We might slow it down to a crawl but we
> wouldn't be corrupting or destroying anything by disabling ATS while
> SVA is running would we?
> Oh does disabling ATS abort all transactions rather than making it transparent?
As you deduced, without ATS a non-present page will return an abort
back to the device instead of a non-present ATS reply which should
then trigger PRI.
IOW the SVA domain can no longer handle page faults and is effectively
non-functioning if ATS is disabled.
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS
@ 2024-03-21 17:32 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-21 17:32 UTC (permalink / raw)
To: Michael Shavit
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer, Nicolin Chen,
patches, Shameerali Kolothum Thodi
On Thu, Mar 21, 2024 at 10:53:20PM +0800, Michael Shavit wrote:
> On Thu, Mar 21, 2024 at 9:28 PM Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > On Thu, Mar 21, 2024 at 08:26:43PM +0800, Michael Shavit wrote:
> > > Overall I think the patch works, but it took me a while to really
> > > digest the big picture. Just to make sure I fully understand:
> > >
> > > We're trying to satisfy the following invariants for correctness:
> > > 1. Devices cannot get translations from a domain that was detached
> > > after arm_smmu_attach_dev() returns.
> >
> > Yes, regardless of ATS this is the API requirement.
> >
> > > 2. Devices cannot get cached translations from a domain after
> > > arm_smmu_atc_inv_domain() returns, regardless of whether the domain is
> > > simultaneously being attached/detached from a device.
> >
> > There is no simultaneously here since the group lock is held by the
> > core code.
>
> I meant a call to arm_smmu_attach_dev concurrent with
> arm_smmu_atc_inv_domain (through tlb_flush_all).
Oh, yes that is the concurrency that drives alot of this organization
> > Yes, we can't just disable ATS due to the PASIDs. A SVA PASID might be
> > present and disabling ATS to change the RID domain would completely
> > wreck it. Today this scenario is prevented by sva_enable, which is
> > removed in following patches.
>
> What do you mean by wrecking? We might slow it down to a crawl but we
> wouldn't be corrupting or destroying anything by disabling ATS while
> SVA is running would we?
> Oh does disabling ATS abort all transactions rather than making it transparent?
As you deduced, without ATS a non-present page will return an abort
back to the device instead of a non-present ATS reply which should
then trigger PRI.
IOW the SVA domain can no longer handle page faults and is effectively
non-functioning if ATS is disabled.
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 01/27] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-22 17:48 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-22 17:48 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:49PM -0400, Jason Gunthorpe wrote:
> The SVA code is wired to assume that the SVA is programmed onto the
> mm->pasid. The current core code always does this, so it is fine.
>
> Add a check for clarity.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 2610e82c0ecd0d..347c2fdd865c1a 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -581,6 +581,9 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> int ret = 0;
> struct mm_struct *mm = domain->mm;
>
> + if (mm_get_enqcmd_pasid(mm) != id)
> + return -EINVAL;
> +
I am not sure if that is needed, the only caller in the tree is the IOMMU code
and it does the right thing, as that check is removed later anyway, I don’t
think this patch adds much.
> mutex_lock(&sva_lock);
> ret = __arm_smmu_sva_bind(dev, id, mm);
> mutex_unlock(&sva_lock);
> --
> 2.43.2
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 01/27] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID
@ 2024-03-22 17:48 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-22 17:48 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:49PM -0400, Jason Gunthorpe wrote:
> The SVA code is wired to assume that the SVA is programmed onto the
> mm->pasid. The current core code always does this, so it is fine.
>
> Add a check for clarity.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 2610e82c0ecd0d..347c2fdd865c1a 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -581,6 +581,9 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> int ret = 0;
> struct mm_struct *mm = domain->mm;
>
> + if (mm_get_enqcmd_pasid(mm) != id)
> + return -EINVAL;
> +
I am not sure if that is needed, the only caller in the tree is the IOMMU code
and it does the right thing, as that check is removed later anyway, I don’t
think this patch adds much.
> mutex_lock(&sva_lock);
> ret = __arm_smmu_sva_bind(dev, id, mm);
> mutex_unlock(&sva_lock);
> --
> 2.43.2
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 02/27] iommu/arm-smmu-v3: Do not ATC invalidate the entire domain
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-22 17:51 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-22 17:51 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:50PM -0400, Jason Gunthorpe wrote:
> At this point we know which master we are going to change the PCI config
> on, this is the only device we need to invalidate. Switch
> arm_smmu_atc_inv_domain() for arm_smmu_atc_inv_master().
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index d1bc151a5dff8c..9e9233331c4636 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2447,7 +2447,10 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master,
> pdev = to_pci_dev(master->dev);
>
> atomic_inc(&smmu_domain->nr_ats_masters);
> - arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, 0, 0);
> + /*
> + * ATC invalidation of PASID 0 causes the entire ATC to be flushed.
> + */
> + arm_smmu_atc_inv_master(master);
> if (pci_enable_ats(pdev, stu))
> dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu);
> }
> --
> 2.43.2
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 02/27] iommu/arm-smmu-v3: Do not ATC invalidate the entire domain
@ 2024-03-22 17:51 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-22 17:51 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:50PM -0400, Jason Gunthorpe wrote:
> At this point we know which master we are going to change the PCI config
> on, this is the only device we need to invalidate. Switch
> arm_smmu_atc_inv_domain() for arm_smmu_atc_inv_master().
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index d1bc151a5dff8c..9e9233331c4636 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2447,7 +2447,10 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master,
> pdev = to_pci_dev(master->dev);
>
> atomic_inc(&smmu_domain->nr_ats_masters);
> - arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, 0, 0);
> + /*
> + * ATC invalidation of PASID 0 causes the entire ATC to be flushed.
> + */
> + arm_smmu_atc_inv_master(master);
> if (pci_enable_ats(pdev, stu))
> dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu);
> }
> --
> 2.43.2
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 03/27] iommu/arm-smmu-v3: Add a type for the CD entry
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-22 17:52 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-22 17:52 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:51PM -0400, Jason Gunthorpe wrote:
> Instead of passing a naked __le16 * around to represent a CD table entry
> wrap it in a "struct arm_smmu_cd" with an array of the correct size. This
> makes it much clearer which functions will comprise the "CD API".
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 +++++++++++---------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 ++++++-
> 2 files changed, 17 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 9e9233331c4636..c60b067c1f553e 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1219,7 +1219,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
> WRITE_ONCE(*dst, cpu_to_le64(val));
> }
>
> -static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
> +static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> + u32 ssid)
> {
> __le64 *l1ptr;
> unsigned int idx;
> @@ -1228,7 +1229,8 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
>
> if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
> - return cd_table->cdtab + ssid * CTXDESC_CD_DWORDS;
> + return (struct arm_smmu_cd *)(cd_table->cdtab +
> + ssid * CTXDESC_CD_DWORDS);
>
> idx = ssid >> CTXDESC_SPLIT;
> l1_desc = &cd_table->l1_desc[idx];
> @@ -1242,7 +1244,7 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
> arm_smmu_sync_cd(master, ssid, false);
> }
> idx = ssid & (CTXDESC_L2_ENTRIES - 1);
> - return l1_desc->l2ptr + idx * CTXDESC_CD_DWORDS;
> + return &l1_desc->l2ptr[idx];
> }
>
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> @@ -1261,7 +1263,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> */
> u64 val;
> bool cd_live;
> - __le64 *cdptr;
> + struct arm_smmu_cd *cdptr;
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
> struct arm_smmu_device *smmu = master->smmu;
>
> @@ -1272,7 +1274,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> if (!cdptr)
> return -ENOMEM;
>
> - val = le64_to_cpu(cdptr[0]);
> + val = le64_to_cpu(cdptr->data[0]);
> cd_live = !!(val & CTXDESC_CD_0_V);
>
> if (!cd) { /* (5) */
> @@ -1289,9 +1291,9 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> * this substream's traffic
> */
> } else { /* (1) and (2) */
> - cdptr[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> - cdptr[2] = 0;
> - cdptr[3] = cpu_to_le64(cd->mair);
> + cdptr->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> + cdptr->data[2] = 0;
> + cdptr->data[3] = cpu_to_le64(cd->mair);
>
> /*
> * STE may be live, and the SMMU might read dwords of this CD in any
> @@ -1323,7 +1325,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> * field within an aligned 64-bit span of a structure can be altered
> * without first making the structure invalid.
> */
> - WRITE_ONCE(cdptr[0], cpu_to_le64(val));
> + WRITE_ONCE(cdptr->data[0], cpu_to_le64(val));
> arm_smmu_sync_cd(master, ssid, true);
> return 0;
> }
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 23baf117e7e4b5..7078ed569fd4d3 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -282,6 +282,11 @@ struct arm_smmu_ste {
> #define CTXDESC_L1_DESC_L2PTR_MASK GENMASK_ULL(51, 12)
>
> #define CTXDESC_CD_DWORDS 8
> +
> +struct arm_smmu_cd {
> + __le64 data[CTXDESC_CD_DWORDS];
> +};
> +
> #define CTXDESC_CD_0_TCR_T0SZ GENMASK_ULL(5, 0)
> #define CTXDESC_CD_0_TCR_TG0 GENMASK_ULL(7, 6)
> #define CTXDESC_CD_0_TCR_IRGN0 GENMASK_ULL(9, 8)
> @@ -591,7 +596,7 @@ struct arm_smmu_ctx_desc {
> };
>
> struct arm_smmu_l1_ctx_desc {
> - __le64 *l2ptr;
> + struct arm_smmu_cd *l2ptr;
> dma_addr_t l2ptr_dma;
> };
>
> --
> 2.43.2
>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 03/27] iommu/arm-smmu-v3: Add a type for the CD entry
@ 2024-03-22 17:52 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-22 17:52 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:51PM -0400, Jason Gunthorpe wrote:
> Instead of passing a naked __le16 * around to represent a CD table entry
> wrap it in a "struct arm_smmu_cd" with an array of the correct size. This
> makes it much clearer which functions will comprise the "CD API".
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 +++++++++++---------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 ++++++-
> 2 files changed, 17 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 9e9233331c4636..c60b067c1f553e 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1219,7 +1219,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
> WRITE_ONCE(*dst, cpu_to_le64(val));
> }
>
> -static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
> +static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> + u32 ssid)
> {
> __le64 *l1ptr;
> unsigned int idx;
> @@ -1228,7 +1229,8 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
>
> if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
> - return cd_table->cdtab + ssid * CTXDESC_CD_DWORDS;
> + return (struct arm_smmu_cd *)(cd_table->cdtab +
> + ssid * CTXDESC_CD_DWORDS);
>
> idx = ssid >> CTXDESC_SPLIT;
> l1_desc = &cd_table->l1_desc[idx];
> @@ -1242,7 +1244,7 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid)
> arm_smmu_sync_cd(master, ssid, false);
> }
> idx = ssid & (CTXDESC_L2_ENTRIES - 1);
> - return l1_desc->l2ptr + idx * CTXDESC_CD_DWORDS;
> + return &l1_desc->l2ptr[idx];
> }
>
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> @@ -1261,7 +1263,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> */
> u64 val;
> bool cd_live;
> - __le64 *cdptr;
> + struct arm_smmu_cd *cdptr;
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
> struct arm_smmu_device *smmu = master->smmu;
>
> @@ -1272,7 +1274,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> if (!cdptr)
> return -ENOMEM;
>
> - val = le64_to_cpu(cdptr[0]);
> + val = le64_to_cpu(cdptr->data[0]);
> cd_live = !!(val & CTXDESC_CD_0_V);
>
> if (!cd) { /* (5) */
> @@ -1289,9 +1291,9 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> * this substream's traffic
> */
> } else { /* (1) and (2) */
> - cdptr[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> - cdptr[2] = 0;
> - cdptr[3] = cpu_to_le64(cd->mair);
> + cdptr->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> + cdptr->data[2] = 0;
> + cdptr->data[3] = cpu_to_le64(cd->mair);
>
> /*
> * STE may be live, and the SMMU might read dwords of this CD in any
> @@ -1323,7 +1325,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> * field within an aligned 64-bit span of a structure can be altered
> * without first making the structure invalid.
> */
> - WRITE_ONCE(cdptr[0], cpu_to_le64(val));
> + WRITE_ONCE(cdptr->data[0], cpu_to_le64(val));
> arm_smmu_sync_cd(master, ssid, true);
> return 0;
> }
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 23baf117e7e4b5..7078ed569fd4d3 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -282,6 +282,11 @@ struct arm_smmu_ste {
> #define CTXDESC_L1_DESC_L2PTR_MASK GENMASK_ULL(51, 12)
>
> #define CTXDESC_CD_DWORDS 8
> +
> +struct arm_smmu_cd {
> + __le64 data[CTXDESC_CD_DWORDS];
> +};
> +
> #define CTXDESC_CD_0_TCR_T0SZ GENMASK_ULL(5, 0)
> #define CTXDESC_CD_0_TCR_TG0 GENMASK_ULL(7, 6)
> #define CTXDESC_CD_0_TCR_IRGN0 GENMASK_ULL(9, 8)
> @@ -591,7 +596,7 @@ struct arm_smmu_ctx_desc {
> };
>
> struct arm_smmu_l1_ctx_desc {
> - __le64 *l2ptr;
> + struct arm_smmu_cd *l2ptr;
> dma_addr_t l2ptr_dma;
> };
>
> --
> 2.43.2
>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-22 18:14 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-22 18:14 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:52PM -0400, Jason Gunthorpe wrote:
> Prepare to put the CD code into the same mechanism. Add an ops indirection
> around all the STE specific code and make the worker functions independent
> of the entry content being processed.
>
> get_used and sync ops are provided to hook the correct code.
>
> Signed-off-by: Michael Shavit <mshavit@google.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 172 ++++++++++++--------
> 1 file changed, 104 insertions(+), 68 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index c60b067c1f553e..b7f947e36f596f 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -48,8 +48,20 @@ enum arm_smmu_msi_index {
> ARM_SMMU_MAX_MSIS,
> };
>
> -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu,
> - ioasid_t sid);
> +struct arm_smmu_entry_writer_ops;
> +struct arm_smmu_entry_writer {
> + const struct arm_smmu_entry_writer_ops *ops;
> + struct arm_smmu_master *master;
> +};
> +
> +struct arm_smmu_entry_writer_ops {
> + unsigned int num_entry_qwords;
> + __le64 v_bit;
> + void (*get_used)(const __le64 *entry, __le64 *used);
> + void (*sync)(struct arm_smmu_entry_writer *writer);
> +};
> +
> +#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
>
> static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
> [EVTQ_MSI_INDEX] = {
> @@ -982,43 +994,42 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid)
> * would be nice if this was complete according to the spec, but minimally it
> * has to capture the bits this driver uses.
> */
> -static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent,
> - struct arm_smmu_ste *used_bits)
> +static void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits)
> {
> - unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent->data[0]));
> + unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent[0]));
>
> - used_bits->data[0] = cpu_to_le64(STRTAB_STE_0_V);
> - if (!(ent->data[0] & cpu_to_le64(STRTAB_STE_0_V)))
> + used_bits[0] = cpu_to_le64(STRTAB_STE_0_V);
> + if (!(ent[0] & cpu_to_le64(STRTAB_STE_0_V)))
> return;
>
> - used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_CFG);
> + used_bits[0] |= cpu_to_le64(STRTAB_STE_0_CFG);
>
> /* S1 translates */
> if (cfg & BIT(0)) {
> - used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT |
> - STRTAB_STE_0_S1CTXPTR_MASK |
> - STRTAB_STE_0_S1CDMAX);
> - used_bits->data[1] |=
> + used_bits[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT |
> + STRTAB_STE_0_S1CTXPTR_MASK |
> + STRTAB_STE_0_S1CDMAX);
> + used_bits[1] |=
> cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR |
> STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH |
> STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW |
> STRTAB_STE_1_EATS);
> - used_bits->data[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);
> + used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);
> }
>
> /* S2 translates */
> if (cfg & BIT(1)) {
> - used_bits->data[1] |=
> + used_bits[1] |=
> cpu_to_le64(STRTAB_STE_1_EATS | STRTAB_STE_1_SHCFG);
> - used_bits->data[2] |=
> + used_bits[2] |=
> cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR |
> STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI |
> STRTAB_STE_2_S2PTW | STRTAB_STE_2_S2R);
> - used_bits->data[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK);
> + used_bits[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK);
> }
>
> if (cfg == STRTAB_STE_0_CFG_BYPASS)
> - used_bits->data[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG);
> + used_bits[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG);
> }
>
> /*
> @@ -1027,57 +1038,55 @@ static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent,
> * unused_update is an intermediate value of entry that has unused bits set to
> * their new values.
> */
> -static u8 arm_smmu_entry_qword_diff(const struct arm_smmu_ste *entry,
> - const struct arm_smmu_ste *target,
> - struct arm_smmu_ste *unused_update)
> +static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> + const __le64 *entry, const __le64 *target,
> + __le64 *unused_update)
> {
> - struct arm_smmu_ste target_used = {};
> - struct arm_smmu_ste cur_used = {};
> + __le64 target_used[NUM_ENTRY_QWORDS] = {};
> + __le64 cur_used[NUM_ENTRY_QWORDS] = {};
This is confusing to me, the function was modified to be generic, so its has
args are __le64 * instead of struct arm_smmu_ste *.
But NUM_ENTRY_QWORDS is defined as “(sizeof(struct arm_smmu_ste) / sizeof(u64))”
and in the same function writer->ops->num_entry_qwords is used nterchangeably,
I understand that this not a constant and the compiler would complain.
But since for any other num_entry_qwords larger than NUM_ENTRY_QWORDS it fails,
and we know STEs and CDs both have the same size, we simplify the code and make
it a constant everywhere.
I see in the next patch, that this is redefined to be the max between STE and
CD, but again, this hardware and it never changes, so my opinion is to simplify
the code, as there is no need to generalize this part.
> u8 used_qword_diff = 0;
> unsigned int i;
>
> - arm_smmu_get_ste_used(entry, &cur_used);
> - arm_smmu_get_ste_used(target, &target_used);
> + writer->ops->get_used(entry, cur_used);
> + writer->ops->get_used(target, target_used);
>
> - for (i = 0; i != ARRAY_SIZE(target_used.data); i++) {
> + for (i = 0; i != writer->ops->num_entry_qwords; i++) {
> /*
> * Check that masks are up to date, the make functions are not
> * allowed to set a bit to 1 if the used function doesn't say it
> * is used.
> */
> - WARN_ON_ONCE(target->data[i] & ~target_used.data[i]);
> + WARN_ON_ONCE(target[i] & ~target_used[i]);
>
> /* Bits can change because they are not currently being used */
> - unused_update->data[i] = (entry->data[i] & cur_used.data[i]) |
> - (target->data[i] & ~cur_used.data[i]);
> + unused_update[i] = (entry[i] & cur_used[i]) |
> + (target[i] & ~cur_used[i]);
> /*
> * Each bit indicates that a used bit in a qword needs to be
> * changed after unused_update is applied.
> */
> - if ((unused_update->data[i] & target_used.data[i]) !=
> - target->data[i])
> + if ((unused_update[i] & target_used[i]) != target[i])
> used_qword_diff |= 1 << i;
> }
> return used_qword_diff;
> }
>
> -static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid,
> - struct arm_smmu_ste *entry,
> - const struct arm_smmu_ste *target, unsigned int start,
> +static bool entry_set(struct arm_smmu_entry_writer *writer, __le64 *entry,
> + const __le64 *target, unsigned int start,
> unsigned int len)
> {
> bool changed = false;
> unsigned int i;
>
> for (i = start; len != 0; len--, i++) {
> - if (entry->data[i] != target->data[i]) {
> - WRITE_ONCE(entry->data[i], target->data[i]);
> + if (entry[i] != target[i]) {
> + WRITE_ONCE(entry[i], target[i]);
> changed = true;
> }
> }
>
> if (changed)
> - arm_smmu_sync_ste_for_sid(smmu, sid);
> + writer->ops->sync(writer);
> return changed;
> }
>
> @@ -1107,17 +1116,15 @@ static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid,
> * V=0 process. This relies on the IGNORED behavior described in the
> * specification.
> */
> -static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> - struct arm_smmu_ste *entry,
> - const struct arm_smmu_ste *target)
> +static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer,
> + __le64 *entry, const __le64 *target)
> {
> - unsigned int num_entry_qwords = ARRAY_SIZE(target->data);
> - struct arm_smmu_device *smmu = master->smmu;
> - struct arm_smmu_ste unused_update;
> + unsigned int num_entry_qwords = writer->ops->num_entry_qwords;
> + __le64 unused_update[NUM_ENTRY_QWORDS];
> u8 used_qword_diff;
>
> used_qword_diff =
> - arm_smmu_entry_qword_diff(entry, target, &unused_update);
> + arm_smmu_entry_qword_diff(writer, entry, target, unused_update);
> if (hweight8(used_qword_diff) == 1) {
> /*
> * Only one qword needs its used bits to be changed. This is a
> @@ -1133,22 +1140,21 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> * writing it in the next step anyways. This can save a sync
> * when the only change is in that qword.
> */
> - unused_update.data[critical_qword_index] =
> - entry->data[critical_qword_index];
> - entry_set(smmu, sid, entry, &unused_update, 0, num_entry_qwords);
> - entry_set(smmu, sid, entry, target, critical_qword_index, 1);
> - entry_set(smmu, sid, entry, target, 0, num_entry_qwords);
> + unused_update[critical_qword_index] =
> + entry[critical_qword_index];
> + entry_set(writer, entry, unused_update, 0, num_entry_qwords);
> + entry_set(writer, entry, target, critical_qword_index, 1);
> + entry_set(writer, entry, target, 0, num_entry_qwords);
> } else if (used_qword_diff) {
> /*
> * At least two qwords need their inuse bits to be changed. This
> * requires a breaking update, zero the V bit, write all qwords
> * but 0, then set qword 0
> */
> - unused_update.data[0] = entry->data[0] &
> - cpu_to_le64(~STRTAB_STE_0_V);
> - entry_set(smmu, sid, entry, &unused_update, 0, 1);
> - entry_set(smmu, sid, entry, target, 1, num_entry_qwords - 1);
> - entry_set(smmu, sid, entry, target, 0, 1);
> + unused_update[0] = entry[0] & (~writer->ops->v_bit);
> + entry_set(writer, entry, unused_update, 0, 1);
> + entry_set(writer, entry, target, 1, num_entry_qwords - 1);
> + entry_set(writer, entry, target, 0, 1);
> } else {
> /*
> * No inuse bit changed. Sanity check that all unused bits are 0
> @@ -1156,18 +1162,7 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> * compute_qword_diff().
> */
> WARN_ON_ONCE(
> - entry_set(smmu, sid, entry, target, 0, num_entry_qwords));
> - }
> -
> - /* It's likely that we'll want to use the new STE soon */
> - if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) {
> - struct arm_smmu_cmdq_ent
> - prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG,
> - .prefetch = {
> - .sid = sid,
> - } };
> -
> - arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
> + entry_set(writer, entry, target, 0, num_entry_qwords));
> }
> }
>
> @@ -1440,17 +1435,58 @@ arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc)
> WRITE_ONCE(*dst, cpu_to_le64(val));
> }
>
> -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu, u32 sid)
> +struct arm_smmu_ste_writer {
> + struct arm_smmu_entry_writer writer;
> + u32 sid;
> +};
> +
> +static void arm_smmu_ste_writer_sync_entry(struct arm_smmu_entry_writer *writer)
> {
> + struct arm_smmu_ste_writer *ste_writer =
> + container_of(writer, struct arm_smmu_ste_writer, writer);
> struct arm_smmu_cmdq_ent cmd = {
> .opcode = CMDQ_OP_CFGI_STE,
> .cfgi = {
> - .sid = sid,
> + .sid = ste_writer->sid,
> .leaf = true,
> },
> };
>
> - arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
> + arm_smmu_cmdq_issue_cmd_with_sync(writer->master->smmu, &cmd);
> +}
> +
> +static const struct arm_smmu_entry_writer_ops arm_smmu_ste_writer_ops = {
> + .sync = arm_smmu_ste_writer_sync_entry,
> + .get_used = arm_smmu_get_ste_used,
> + .v_bit = cpu_to_le64(STRTAB_STE_0_V),
> + .num_entry_qwords = sizeof(struct arm_smmu_ste) / sizeof(u64),
> +};
> +
> +static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> + struct arm_smmu_ste *ste,
> + const struct arm_smmu_ste *target)
> +{
> + struct arm_smmu_device *smmu = master->smmu;
> + struct arm_smmu_ste_writer ste_writer = {
> + .writer = {
> + .ops = &arm_smmu_ste_writer_ops,
> + .master = master,
> + },
> + .sid = sid,
> + };
> +
> + arm_smmu_write_entry(&ste_writer.writer, ste->data, target->data);
> +
> + /* It's likely that we'll want to use the new STE soon */
> + if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) {
> + struct arm_smmu_cmdq_ent
> + prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG,
> + .prefetch = {
> + .sid = sid,
> + } };
> +
> + arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
> + }
> }
>
> static void arm_smmu_make_abort_ste(struct arm_smmu_ste *target)
> --
> 2.43.2
>
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code
@ 2024-03-22 18:14 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-22 18:14 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:52PM -0400, Jason Gunthorpe wrote:
> Prepare to put the CD code into the same mechanism. Add an ops indirection
> around all the STE specific code and make the worker functions independent
> of the entry content being processed.
>
> get_used and sync ops are provided to hook the correct code.
>
> Signed-off-by: Michael Shavit <mshavit@google.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 172 ++++++++++++--------
> 1 file changed, 104 insertions(+), 68 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index c60b067c1f553e..b7f947e36f596f 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -48,8 +48,20 @@ enum arm_smmu_msi_index {
> ARM_SMMU_MAX_MSIS,
> };
>
> -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu,
> - ioasid_t sid);
> +struct arm_smmu_entry_writer_ops;
> +struct arm_smmu_entry_writer {
> + const struct arm_smmu_entry_writer_ops *ops;
> + struct arm_smmu_master *master;
> +};
> +
> +struct arm_smmu_entry_writer_ops {
> + unsigned int num_entry_qwords;
> + __le64 v_bit;
> + void (*get_used)(const __le64 *entry, __le64 *used);
> + void (*sync)(struct arm_smmu_entry_writer *writer);
> +};
> +
> +#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
>
> static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
> [EVTQ_MSI_INDEX] = {
> @@ -982,43 +994,42 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid)
> * would be nice if this was complete according to the spec, but minimally it
> * has to capture the bits this driver uses.
> */
> -static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent,
> - struct arm_smmu_ste *used_bits)
> +static void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits)
> {
> - unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent->data[0]));
> + unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent[0]));
>
> - used_bits->data[0] = cpu_to_le64(STRTAB_STE_0_V);
> - if (!(ent->data[0] & cpu_to_le64(STRTAB_STE_0_V)))
> + used_bits[0] = cpu_to_le64(STRTAB_STE_0_V);
> + if (!(ent[0] & cpu_to_le64(STRTAB_STE_0_V)))
> return;
>
> - used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_CFG);
> + used_bits[0] |= cpu_to_le64(STRTAB_STE_0_CFG);
>
> /* S1 translates */
> if (cfg & BIT(0)) {
> - used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT |
> - STRTAB_STE_0_S1CTXPTR_MASK |
> - STRTAB_STE_0_S1CDMAX);
> - used_bits->data[1] |=
> + used_bits[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT |
> + STRTAB_STE_0_S1CTXPTR_MASK |
> + STRTAB_STE_0_S1CDMAX);
> + used_bits[1] |=
> cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR |
> STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH |
> STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW |
> STRTAB_STE_1_EATS);
> - used_bits->data[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);
> + used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);
> }
>
> /* S2 translates */
> if (cfg & BIT(1)) {
> - used_bits->data[1] |=
> + used_bits[1] |=
> cpu_to_le64(STRTAB_STE_1_EATS | STRTAB_STE_1_SHCFG);
> - used_bits->data[2] |=
> + used_bits[2] |=
> cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR |
> STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI |
> STRTAB_STE_2_S2PTW | STRTAB_STE_2_S2R);
> - used_bits->data[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK);
> + used_bits[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK);
> }
>
> if (cfg == STRTAB_STE_0_CFG_BYPASS)
> - used_bits->data[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG);
> + used_bits[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG);
> }
>
> /*
> @@ -1027,57 +1038,55 @@ static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent,
> * unused_update is an intermediate value of entry that has unused bits set to
> * their new values.
> */
> -static u8 arm_smmu_entry_qword_diff(const struct arm_smmu_ste *entry,
> - const struct arm_smmu_ste *target,
> - struct arm_smmu_ste *unused_update)
> +static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> + const __le64 *entry, const __le64 *target,
> + __le64 *unused_update)
> {
> - struct arm_smmu_ste target_used = {};
> - struct arm_smmu_ste cur_used = {};
> + __le64 target_used[NUM_ENTRY_QWORDS] = {};
> + __le64 cur_used[NUM_ENTRY_QWORDS] = {};
This is confusing to me, the function was modified to be generic, so its has
args are __le64 * instead of struct arm_smmu_ste *.
But NUM_ENTRY_QWORDS is defined as “(sizeof(struct arm_smmu_ste) / sizeof(u64))”
and in the same function writer->ops->num_entry_qwords is used nterchangeably,
I understand that this not a constant and the compiler would complain.
But since for any other num_entry_qwords larger than NUM_ENTRY_QWORDS it fails,
and we know STEs and CDs both have the same size, we simplify the code and make
it a constant everywhere.
I see in the next patch, that this is redefined to be the max between STE and
CD, but again, this hardware and it never changes, so my opinion is to simplify
the code, as there is no need to generalize this part.
> u8 used_qword_diff = 0;
> unsigned int i;
>
> - arm_smmu_get_ste_used(entry, &cur_used);
> - arm_smmu_get_ste_used(target, &target_used);
> + writer->ops->get_used(entry, cur_used);
> + writer->ops->get_used(target, target_used);
>
> - for (i = 0; i != ARRAY_SIZE(target_used.data); i++) {
> + for (i = 0; i != writer->ops->num_entry_qwords; i++) {
> /*
> * Check that masks are up to date, the make functions are not
> * allowed to set a bit to 1 if the used function doesn't say it
> * is used.
> */
> - WARN_ON_ONCE(target->data[i] & ~target_used.data[i]);
> + WARN_ON_ONCE(target[i] & ~target_used[i]);
>
> /* Bits can change because they are not currently being used */
> - unused_update->data[i] = (entry->data[i] & cur_used.data[i]) |
> - (target->data[i] & ~cur_used.data[i]);
> + unused_update[i] = (entry[i] & cur_used[i]) |
> + (target[i] & ~cur_used[i]);
> /*
> * Each bit indicates that a used bit in a qword needs to be
> * changed after unused_update is applied.
> */
> - if ((unused_update->data[i] & target_used.data[i]) !=
> - target->data[i])
> + if ((unused_update[i] & target_used[i]) != target[i])
> used_qword_diff |= 1 << i;
> }
> return used_qword_diff;
> }
>
> -static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid,
> - struct arm_smmu_ste *entry,
> - const struct arm_smmu_ste *target, unsigned int start,
> +static bool entry_set(struct arm_smmu_entry_writer *writer, __le64 *entry,
> + const __le64 *target, unsigned int start,
> unsigned int len)
> {
> bool changed = false;
> unsigned int i;
>
> for (i = start; len != 0; len--, i++) {
> - if (entry->data[i] != target->data[i]) {
> - WRITE_ONCE(entry->data[i], target->data[i]);
> + if (entry[i] != target[i]) {
> + WRITE_ONCE(entry[i], target[i]);
> changed = true;
> }
> }
>
> if (changed)
> - arm_smmu_sync_ste_for_sid(smmu, sid);
> + writer->ops->sync(writer);
> return changed;
> }
>
> @@ -1107,17 +1116,15 @@ static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid,
> * V=0 process. This relies on the IGNORED behavior described in the
> * specification.
> */
> -static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> - struct arm_smmu_ste *entry,
> - const struct arm_smmu_ste *target)
> +static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer,
> + __le64 *entry, const __le64 *target)
> {
> - unsigned int num_entry_qwords = ARRAY_SIZE(target->data);
> - struct arm_smmu_device *smmu = master->smmu;
> - struct arm_smmu_ste unused_update;
> + unsigned int num_entry_qwords = writer->ops->num_entry_qwords;
> + __le64 unused_update[NUM_ENTRY_QWORDS];
> u8 used_qword_diff;
>
> used_qword_diff =
> - arm_smmu_entry_qword_diff(entry, target, &unused_update);
> + arm_smmu_entry_qword_diff(writer, entry, target, unused_update);
> if (hweight8(used_qword_diff) == 1) {
> /*
> * Only one qword needs its used bits to be changed. This is a
> @@ -1133,22 +1140,21 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> * writing it in the next step anyways. This can save a sync
> * when the only change is in that qword.
> */
> - unused_update.data[critical_qword_index] =
> - entry->data[critical_qword_index];
> - entry_set(smmu, sid, entry, &unused_update, 0, num_entry_qwords);
> - entry_set(smmu, sid, entry, target, critical_qword_index, 1);
> - entry_set(smmu, sid, entry, target, 0, num_entry_qwords);
> + unused_update[critical_qword_index] =
> + entry[critical_qword_index];
> + entry_set(writer, entry, unused_update, 0, num_entry_qwords);
> + entry_set(writer, entry, target, critical_qword_index, 1);
> + entry_set(writer, entry, target, 0, num_entry_qwords);
> } else if (used_qword_diff) {
> /*
> * At least two qwords need their inuse bits to be changed. This
> * requires a breaking update, zero the V bit, write all qwords
> * but 0, then set qword 0
> */
> - unused_update.data[0] = entry->data[0] &
> - cpu_to_le64(~STRTAB_STE_0_V);
> - entry_set(smmu, sid, entry, &unused_update, 0, 1);
> - entry_set(smmu, sid, entry, target, 1, num_entry_qwords - 1);
> - entry_set(smmu, sid, entry, target, 0, 1);
> + unused_update[0] = entry[0] & (~writer->ops->v_bit);
> + entry_set(writer, entry, unused_update, 0, 1);
> + entry_set(writer, entry, target, 1, num_entry_qwords - 1);
> + entry_set(writer, entry, target, 0, 1);
> } else {
> /*
> * No inuse bit changed. Sanity check that all unused bits are 0
> @@ -1156,18 +1162,7 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> * compute_qword_diff().
> */
> WARN_ON_ONCE(
> - entry_set(smmu, sid, entry, target, 0, num_entry_qwords));
> - }
> -
> - /* It's likely that we'll want to use the new STE soon */
> - if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) {
> - struct arm_smmu_cmdq_ent
> - prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG,
> - .prefetch = {
> - .sid = sid,
> - } };
> -
> - arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
> + entry_set(writer, entry, target, 0, num_entry_qwords));
> }
> }
>
> @@ -1440,17 +1435,58 @@ arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc)
> WRITE_ONCE(*dst, cpu_to_le64(val));
> }
>
> -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu, u32 sid)
> +struct arm_smmu_ste_writer {
> + struct arm_smmu_entry_writer writer;
> + u32 sid;
> +};
> +
> +static void arm_smmu_ste_writer_sync_entry(struct arm_smmu_entry_writer *writer)
> {
> + struct arm_smmu_ste_writer *ste_writer =
> + container_of(writer, struct arm_smmu_ste_writer, writer);
> struct arm_smmu_cmdq_ent cmd = {
> .opcode = CMDQ_OP_CFGI_STE,
> .cfgi = {
> - .sid = sid,
> + .sid = ste_writer->sid,
> .leaf = true,
> },
> };
>
> - arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
> + arm_smmu_cmdq_issue_cmd_with_sync(writer->master->smmu, &cmd);
> +}
> +
> +static const struct arm_smmu_entry_writer_ops arm_smmu_ste_writer_ops = {
> + .sync = arm_smmu_ste_writer_sync_entry,
> + .get_used = arm_smmu_get_ste_used,
> + .v_bit = cpu_to_le64(STRTAB_STE_0_V),
> + .num_entry_qwords = sizeof(struct arm_smmu_ste) / sizeof(u64),
> +};
> +
> +static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> + struct arm_smmu_ste *ste,
> + const struct arm_smmu_ste *target)
> +{
> + struct arm_smmu_device *smmu = master->smmu;
> + struct arm_smmu_ste_writer ste_writer = {
> + .writer = {
> + .ops = &arm_smmu_ste_writer_ops,
> + .master = master,
> + },
> + .sid = sid,
> + };
> +
> + arm_smmu_write_entry(&ste_writer.writer, ste->data, target->data);
> +
> + /* It's likely that we'll want to use the new STE soon */
> + if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) {
> + struct arm_smmu_cmdq_ent
> + prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG,
> + .prefetch = {
> + .sid = sid,
> + } };
> +
> + arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
> + }
> }
>
> static void arm_smmu_make_abort_ste(struct arm_smmu_ste *target)
> --
> 2.43.2
>
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 06/27] iommu/arm-smmu-v3: Consolidate clearing a CD table entry
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-22 18:36 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-22 18:36 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:54PM -0400, Jason Gunthorpe wrote:
> A cleared entry is all 0's. Make arm_smmu_clear_cd() do this sequence.
>
> If we are clearing an entry and for some reason it is not already
> allocated in the CD table then something has gone wrong.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 ++++++++++++++-----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 ++
> 3 files changed, 18 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 347c2fdd865c1a..bb9bb6fd7914ce 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -558,7 +558,7 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
>
> mutex_lock(&sva_lock);
>
> - arm_smmu_write_ctx_desc(master, id, NULL);
> + arm_smmu_clear_cd(master, id);
>
> list_for_each_entry(t, &master->bonds, list) {
> if (t->mm == mm) {
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 237fd6d92c880b..3fb4a1523d1d3f 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1303,6 +1303,19 @@ static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
> }
>
> +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> +{
> + struct arm_smmu_cd target = {};
> + struct arm_smmu_cd *cdptr;
> +
> + if (!master->cd_table.cdtab)
> + return;
> + cdptr = arm_smmu_get_cd_ptr(master, ssid);
> + if (WARN_ON(!cdptr))
> + return;
I don’t understand the SVA code enough, but AFAICT, arm_smmu_sva_set_dev_pasid
can allocate the L2 CD table through arm_smmu_write_ctx_desc. And if it failed
before allocating the CD table, then remove_dev_pasid would be called, which
warns here, the previous code would tolerate that, but that might regress on
systems with panic_on_warn, so I am not sure if that is necessary.
Otherwise, Reviewed-by: Mostafa Saleh <smostafa@google.com>
> + arm_smmu_write_cd_entry(master, ssid, cdptr, &target);
> +}
> +
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_ctx_desc *cd)
> {
> @@ -2702,9 +2715,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> case ARM_SMMU_DOMAIN_S2:
> arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
> arm_smmu_install_ste_for_dev(master, &target);
> - if (master->cd_table.cdtab)
> - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID,
> - NULL);
> + arm_smmu_clear_cd(master, IOMMU_NO_PASID);
> break;
> }
>
> @@ -2752,8 +2763,7 @@ static int arm_smmu_attach_dev_ste(struct device *dev,
> * arm_smmu_domain->devices to avoid races updating the same context
> * descriptor from arm_smmu_share_asid().
> */
> - if (master->cd_table.cdtab)
> - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, NULL);
> + arm_smmu_clear_cd(master, IOMMU_NO_PASID);
> return 0;
> }
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 7078ed569fd4d3..87a7b57f566fbc 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -749,6 +749,8 @@ extern struct xarray arm_smmu_asid_xa;
> extern struct mutex arm_smmu_asid_lock;
> extern struct arm_smmu_ctx_desc quiet_cd;
>
> +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
> +
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid,
> struct arm_smmu_ctx_desc *cd);
> void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
> --
> 2.43.2
>
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 06/27] iommu/arm-smmu-v3: Consolidate clearing a CD table entry
@ 2024-03-22 18:36 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-22 18:36 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:54PM -0400, Jason Gunthorpe wrote:
> A cleared entry is all 0's. Make arm_smmu_clear_cd() do this sequence.
>
> If we are clearing an entry and for some reason it is not already
> allocated in the CD table then something has gone wrong.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 ++++++++++++++-----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 ++
> 3 files changed, 18 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 347c2fdd865c1a..bb9bb6fd7914ce 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -558,7 +558,7 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain,
>
> mutex_lock(&sva_lock);
>
> - arm_smmu_write_ctx_desc(master, id, NULL);
> + arm_smmu_clear_cd(master, id);
>
> list_for_each_entry(t, &master->bonds, list) {
> if (t->mm == mm) {
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 237fd6d92c880b..3fb4a1523d1d3f 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1303,6 +1303,19 @@ static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
> }
>
> +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> +{
> + struct arm_smmu_cd target = {};
> + struct arm_smmu_cd *cdptr;
> +
> + if (!master->cd_table.cdtab)
> + return;
> + cdptr = arm_smmu_get_cd_ptr(master, ssid);
> + if (WARN_ON(!cdptr))
> + return;
I don’t understand the SVA code enough, but AFAICT, arm_smmu_sva_set_dev_pasid
can allocate the L2 CD table through arm_smmu_write_ctx_desc. And if it failed
before allocating the CD table, then remove_dev_pasid would be called, which
warns here, the previous code would tolerate that, but that might regress on
systems with panic_on_warn, so I am not sure if that is necessary.
Otherwise, Reviewed-by: Mostafa Saleh <smostafa@google.com>
> + arm_smmu_write_cd_entry(master, ssid, cdptr, &target);
> +}
> +
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_ctx_desc *cd)
> {
> @@ -2702,9 +2715,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> case ARM_SMMU_DOMAIN_S2:
> arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
> arm_smmu_install_ste_for_dev(master, &target);
> - if (master->cd_table.cdtab)
> - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID,
> - NULL);
> + arm_smmu_clear_cd(master, IOMMU_NO_PASID);
> break;
> }
>
> @@ -2752,8 +2763,7 @@ static int arm_smmu_attach_dev_ste(struct device *dev,
> * arm_smmu_domain->devices to avoid races updating the same context
> * descriptor from arm_smmu_share_asid().
> */
> - if (master->cd_table.cdtab)
> - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, NULL);
> + arm_smmu_clear_cd(master, IOMMU_NO_PASID);
> return 0;
> }
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 7078ed569fd4d3..87a7b57f566fbc 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -749,6 +749,8 @@ extern struct xarray arm_smmu_asid_xa;
> extern struct mutex arm_smmu_asid_lock;
> extern struct arm_smmu_ctx_desc quiet_cd;
>
> +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
> +
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid,
> struct arm_smmu_ctx_desc *cd);
> void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
> --
> 2.43.2
>
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 08/27] iommu/arm-smmu-v3: Move allocation of the cdtable into arm_smmu_get_cd_ptr()
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-22 19:07 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-22 19:07 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:56PM -0400, Jason Gunthorpe wrote:
> No reason to force callers to do two steps. Make arm_smmu_get_cd_ptr()
> able to return an entry in all cases except OOM
I believe the current code is more clear, as it is explicit about which path
is expected to allocate.
As there are many callers for arm_smmu_get_cd_ptr() directly and indirectly,
and it read-modify-writes the cdtable, it would be a pain to debug not
knowing which one could allocate, and this patch only abstracts one
allocating call, so it is not much code less.
For example, (again I don’t know much about SVA) I think there might be a
race condition as follows:
arm_smmu_attach_dev
arm_smmu_domain_finalise() => set domain stage
[....]
arm_smmu_get_cd_ptr() => RMW master->cd_table
arm_smmu_sva_set_dev_pasid
__arm_smmu_sva_bind
Check stage is valid
[...]
arm_smmu_write_ctx_desc
arm_smmu_get_cd_ptr => RMW master->cd_table
If this path is true though, I guess the in current code, we would need some
barriers in arm_smmu_get_cd_ptr(), arm_smmu_get_cd_ptr()
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 12 ++++++------
> 1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index e25dbb982feeee..2dd6cb17112e98 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -106,6 +106,7 @@ static struct arm_smmu_option_prop arm_smmu_options[] = {
>
> static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
> struct arm_smmu_device *smmu);
> +static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master);
>
> static void parse_driver_options(struct arm_smmu_device *smmu)
> {
> @@ -1231,6 +1232,11 @@ struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> struct arm_smmu_device *smmu = master->smmu;
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
>
> + if (!master->cd_table.cdtab) {
> + if (arm_smmu_alloc_cd_tables(master))
> + return NULL;
> + }
> +
> if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
> return (struct arm_smmu_cd *)(cd_table->cdtab +
> ssid * CTXDESC_CD_DWORDS);
> @@ -2719,12 +2725,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> struct arm_smmu_cd target_cd;
> struct arm_smmu_cd *cdptr;
>
> - if (!master->cd_table.cdtab) {
> - ret = arm_smmu_alloc_cd_tables(master);
> - if (ret)
> - goto out_list_del;
> - }
> -
> cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> if (!cdptr) {
> ret = -ENOMEM;
> --
> 2.43.2
>
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 08/27] iommu/arm-smmu-v3: Move allocation of the cdtable into arm_smmu_get_cd_ptr()
@ 2024-03-22 19:07 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-22 19:07 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:56PM -0400, Jason Gunthorpe wrote:
> No reason to force callers to do two steps. Make arm_smmu_get_cd_ptr()
> able to return an entry in all cases except OOM
I believe the current code is more clear, as it is explicit about which path
is expected to allocate.
As there are many callers for arm_smmu_get_cd_ptr() directly and indirectly,
and it read-modify-writes the cdtable, it would be a pain to debug not
knowing which one could allocate, and this patch only abstracts one
allocating call, so it is not much code less.
For example, (again I don’t know much about SVA) I think there might be a
race condition as follows:
arm_smmu_attach_dev
arm_smmu_domain_finalise() => set domain stage
[....]
arm_smmu_get_cd_ptr() => RMW master->cd_table
arm_smmu_sva_set_dev_pasid
__arm_smmu_sva_bind
Check stage is valid
[...]
arm_smmu_write_ctx_desc
arm_smmu_get_cd_ptr => RMW master->cd_table
If this path is true though, I guess the in current code, we would need some
barriers in arm_smmu_get_cd_ptr(), arm_smmu_get_cd_ptr()
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 12 ++++++------
> 1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index e25dbb982feeee..2dd6cb17112e98 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -106,6 +106,7 @@ static struct arm_smmu_option_prop arm_smmu_options[] = {
>
> static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
> struct arm_smmu_device *smmu);
> +static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master);
>
> static void parse_driver_options(struct arm_smmu_device *smmu)
> {
> @@ -1231,6 +1232,11 @@ struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> struct arm_smmu_device *smmu = master->smmu;
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
>
> + if (!master->cd_table.cdtab) {
> + if (arm_smmu_alloc_cd_tables(master))
> + return NULL;
> + }
> +
> if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR)
> return (struct arm_smmu_cd *)(cd_table->cdtab +
> ssid * CTXDESC_CD_DWORDS);
> @@ -2719,12 +2725,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> struct arm_smmu_cd target_cd;
> struct arm_smmu_cd *cdptr;
>
> - if (!master->cd_table.cdtab) {
> - ret = arm_smmu_alloc_cd_tables(master);
> - if (ret)
> - goto out_list_del;
> - }
> -
> cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> if (!cdptr) {
> ret = -ENOMEM;
> --
> 2.43.2
>
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 09/27] iommu/arm-smmu-v3: Allocate the CD table entry in advance
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-22 19:15 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-22 19:15 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:57PM -0400, Jason Gunthorpe wrote:
> Avoid arm_smmu_attach_dev() having to undo the changes to the
> smmu_domain->devices list, acquire the cdptr earlier so we don't need to
> handle that error.
>
> Now there is a clear break in arm_smmu_attach_dev() where all the
> prep-work has been done non-disruptively and we commit to making the HW
> change, which cannot fail.
>
> This completes transforming arm_smmu_attach_dev() so that it does not
> disturb the HW if it fails.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 24 +++++++--------------
> 1 file changed, 8 insertions(+), 16 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 2dd6cb17112e98..39081d828a2132 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2676,6 +2676,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> struct arm_smmu_device *smmu;
> struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> struct arm_smmu_master *master;
> + struct arm_smmu_cd *cdptr;
>
> if (!fwspec)
> return -ENOENT;
> @@ -2704,6 +2705,12 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> if (ret)
> return ret;
>
> + if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
> + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> + if (!cdptr)
> + return -ENOMEM;
> + }
> +
> /*
> * Prevent arm_smmu_share_asid() from trying to change the ASID
> * of either the old or new domain while we are working on it.
> @@ -2723,13 +2730,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> switch (smmu_domain->stage) {
> case ARM_SMMU_DOMAIN_S1: {
> struct arm_smmu_cd target_cd;
> - struct arm_smmu_cd *cdptr;
> -
> - cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> - if (!cdptr) {
> - ret = -ENOMEM;
> - goto out_list_del;
> - }
>
> arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> @@ -2746,16 +2746,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> }
>
> arm_smmu_enable_ats(master, smmu_domain);
> - goto out_unlock;
> -
> -out_list_del:
> - spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - list_del_init(&master->domain_head);
> - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> -
> -out_unlock:
> mutex_unlock(&arm_smmu_asid_lock);
> - return ret;
> + return 0;
> }
>
> static int arm_smmu_attach_dev_ste(struct device *dev,
> --
> 2.43.2
>
I believe this is fine, I couldn’t break it. With the comment on the previous
patch, where we explicitly allocate the CD here and not inside
arm_smmu_get_cd_ptr().
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 09/27] iommu/arm-smmu-v3: Allocate the CD table entry in advance
@ 2024-03-22 19:15 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-22 19:15 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:57PM -0400, Jason Gunthorpe wrote:
> Avoid arm_smmu_attach_dev() having to undo the changes to the
> smmu_domain->devices list, acquire the cdptr earlier so we don't need to
> handle that error.
>
> Now there is a clear break in arm_smmu_attach_dev() where all the
> prep-work has been done non-disruptively and we commit to making the HW
> change, which cannot fail.
>
> This completes transforming arm_smmu_attach_dev() so that it does not
> disturb the HW if it fails.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 24 +++++++--------------
> 1 file changed, 8 insertions(+), 16 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 2dd6cb17112e98..39081d828a2132 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2676,6 +2676,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> struct arm_smmu_device *smmu;
> struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> struct arm_smmu_master *master;
> + struct arm_smmu_cd *cdptr;
>
> if (!fwspec)
> return -ENOENT;
> @@ -2704,6 +2705,12 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> if (ret)
> return ret;
>
> + if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
> + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> + if (!cdptr)
> + return -ENOMEM;
> + }
> +
> /*
> * Prevent arm_smmu_share_asid() from trying to change the ASID
> * of either the old or new domain while we are working on it.
> @@ -2723,13 +2730,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> switch (smmu_domain->stage) {
> case ARM_SMMU_DOMAIN_S1: {
> struct arm_smmu_cd target_cd;
> - struct arm_smmu_cd *cdptr;
> -
> - cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> - if (!cdptr) {
> - ret = -ENOMEM;
> - goto out_list_del;
> - }
>
> arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> @@ -2746,16 +2746,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> }
>
> arm_smmu_enable_ats(master, smmu_domain);
> - goto out_unlock;
> -
> -out_list_del:
> - spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> - list_del_init(&master->domain_head);
> - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> -
> -out_unlock:
> mutex_unlock(&arm_smmu_asid_lock);
> - return ret;
> + return 0;
> }
>
> static int arm_smmu_attach_dev_ste(struct device *dev,
> --
> 2.43.2
>
I believe this is fine, I couldn’t break it. With the comment on the previous
patch, where we explicitly allocate the CD here and not inside
arm_smmu_get_cd_ptr().
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-23 13:02 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-23 13:02 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:53PM -0400, Jason Gunthorpe wrote:
> CD table entries and STE's have the same essential programming sequence,
> just with different types and sizes.
>
> Have arm_smmu_write_ctx_desc() generate a target CD and call
> arm_smmu_write_entry() to do the programming. Due to the way the
> target CD is generated by modifying the existing CD this alone is not
> enough for the CD callers to be freed of the ordering requirements.
>
> The following patches will make the rest of the CD flow mirror the STE
> flow with precise CD contents generated in all cases.
>
> Currently the logic can't ensure that the CD always conforms to the used
> requirements until all the CD generation is moved to the new method. Add a
> temporary no_used_check to disable the assertions.
>
I am still going through the patches, but is it possible to
reorder/squash to avoid that, so it is easier to review?
> Signed-off-by: Michael Shavit <mshavit@google.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 101 ++++++++++++++------
> 1 file changed, 74 insertions(+), 27 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index b7f947e36f596f..237fd6d92c880b 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -57,11 +57,14 @@ struct arm_smmu_entry_writer {
> struct arm_smmu_entry_writer_ops {
> unsigned int num_entry_qwords;
> __le64 v_bit;
> + bool no_used_check;
> void (*get_used)(const __le64 *entry, __le64 *used);
> void (*sync)(struct arm_smmu_entry_writer *writer);
> };
>
> -#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
> +#define NUM_ENTRY_QWORDS \
> + (max(sizeof(struct arm_smmu_ste), sizeof(struct arm_smmu_cd)) / \
> + sizeof(u64))
>
> static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
> [EVTQ_MSI_INDEX] = {
> @@ -1056,7 +1059,8 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> * allowed to set a bit to 1 if the used function doesn't say it
> * is used.
> */
> - WARN_ON_ONCE(target[i] & ~target_used[i]);
> + if (!writer->ops->no_used_check)
> + WARN_ON_ONCE(target[i] & ~target_used[i]);
>
> /* Bits can change because they are not currently being used */
> unused_update[i] = (entry[i] & cur_used[i]) |
> @@ -1065,7 +1069,8 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> * Each bit indicates that a used bit in a qword needs to be
> * changed after unused_update is applied.
> */
> - if ((unused_update[i] & target_used[i]) != target[i])
> + if ((unused_update[i] & target_used[i]) !=
> + (target[i] & target_used[i]))
> used_qword_diff |= 1 << i;
> }
> return used_qword_diff;
> @@ -1161,8 +1166,11 @@ static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer,
> * in the entry. The target was already sanity checked by
> * compute_qword_diff().
> */
> - WARN_ON_ONCE(
> - entry_set(writer, entry, target, 0, num_entry_qwords));
> + if (writer->ops->no_used_check)
> + entry_set(writer, entry, target, 0, num_entry_qwords);
> + else
> + WARN_ON_ONCE(entry_set(writer, entry, target, 0,
> + num_entry_qwords));
> }
> }
>
> @@ -1242,6 +1250,59 @@ static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> return &l1_desc->l2ptr[idx];
> }
>
> +struct arm_smmu_cd_writer {
> + struct arm_smmu_entry_writer writer;
> + unsigned int ssid;
> +};
> +
> +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
> +{
> + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
> + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
> + return;
> + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
This is a slightly different approach than what the driver does for STEs,
where it explicitly sets the used bits. Is there a reason for that?
> +
> + /* EPD0 means T0SZ/TG0/IR0/OR0/SH0/TTB0 are IGNORED */
> + if (ent[0] & cpu_to_le64(CTXDESC_CD_0_TCR_EPD0)) {
> + used_bits[0] &= ~cpu_to_le64(
> + CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 |
> + CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 |
> + CTXDESC_CD_0_TCR_SH0);
> + used_bits[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK);
> + }
> +}
We should add a comment about EPD1 maybe?
> +
> +static void arm_smmu_cd_writer_sync_entry(struct arm_smmu_entry_writer *writer)
> +{
> + struct arm_smmu_cd_writer *cd_writer =
> + container_of(writer, struct arm_smmu_cd_writer, writer);
> +
> + arm_smmu_sync_cd(writer->master, cd_writer->ssid, true);
> +}
> +
> +static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
> + .sync = arm_smmu_cd_writer_sync_entry,
> + .get_used = arm_smmu_get_cd_used,
> + .v_bit = cpu_to_le64(CTXDESC_CD_0_V),
> + .no_used_check = true,
> + .num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
> +};
> +
> +static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> + struct arm_smmu_cd *cdptr,
> + const struct arm_smmu_cd *target)
> +{
> + struct arm_smmu_cd_writer cd_writer = {
> + .writer = {
> + .ops = &arm_smmu_cd_writer_ops,
> + .master = master,
> + },
> + .ssid = ssid,
> + };
> +
> + arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
> +}
> +
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_ctx_desc *cd)
> {
> @@ -1258,17 +1319,20 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> */
> u64 val;
> bool cd_live;
> - struct arm_smmu_cd *cdptr;
> + struct arm_smmu_cd target;
> + struct arm_smmu_cd *cdptr = ⌖
> + struct arm_smmu_cd *cd_table_entry;
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
> struct arm_smmu_device *smmu = master->smmu;
>
> if (WARN_ON(ssid >= (1 << cd_table->s1cdmax)))
> return -E2BIG;
>
> - cdptr = arm_smmu_get_cd_ptr(master, ssid);
> - if (!cdptr)
> + cd_table_entry = arm_smmu_get_cd_ptr(master, ssid);
> + if (!cd_table_entry)
> return -ENOMEM;
>
> + target = *cd_table_entry;
> val = le64_to_cpu(cdptr->data[0]);
> cd_live = !!(val & CTXDESC_CD_0_V);
>
> @@ -1290,13 +1354,6 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> cdptr->data[2] = 0;
> cdptr->data[3] = cpu_to_le64(cd->mair);
>
> - /*
> - * STE may be live, and the SMMU might read dwords of this CD in any
> - * order. Ensure that it observes valid values before reading
> - * V=1.
> - */
> - arm_smmu_sync_cd(master, ssid, true);
> -
> val = cd->tcr |
> #ifdef __BIG_ENDIAN
> CTXDESC_CD_0_ENDI |
> @@ -1310,18 +1367,8 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> if (cd_table->stall_enabled)
> val |= CTXDESC_CD_0_S;
> }
> -
> - /*
> - * The SMMU accesses 64-bit values atomically. See IHI0070Ca 3.21.3
> - * "Configuration structures and configuration invalidation completion"
> - *
> - * The size of single-copy atomic reads made by the SMMU is
> - * IMPLEMENTATION DEFINED but must be at least 64 bits. Any single
> - * field within an aligned 64-bit span of a structure can be altered
> - * without first making the structure invalid.
> - */
> - WRITE_ONCE(cdptr->data[0], cpu_to_le64(val));
> - arm_smmu_sync_cd(master, ssid, true);
> + cdptr->data[0] = cpu_to_le64(val);
> + arm_smmu_write_cd_entry(master, ssid, cd_table_entry, &target);
> return 0;
> }
>
> --
> 2.43.2
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
@ 2024-03-23 13:02 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-23 13:02 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:53PM -0400, Jason Gunthorpe wrote:
> CD table entries and STE's have the same essential programming sequence,
> just with different types and sizes.
>
> Have arm_smmu_write_ctx_desc() generate a target CD and call
> arm_smmu_write_entry() to do the programming. Due to the way the
> target CD is generated by modifying the existing CD this alone is not
> enough for the CD callers to be freed of the ordering requirements.
>
> The following patches will make the rest of the CD flow mirror the STE
> flow with precise CD contents generated in all cases.
>
> Currently the logic can't ensure that the CD always conforms to the used
> requirements until all the CD generation is moved to the new method. Add a
> temporary no_used_check to disable the assertions.
>
I am still going through the patches, but is it possible to
reorder/squash to avoid that, so it is easier to review?
> Signed-off-by: Michael Shavit <mshavit@google.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 101 ++++++++++++++------
> 1 file changed, 74 insertions(+), 27 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index b7f947e36f596f..237fd6d92c880b 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -57,11 +57,14 @@ struct arm_smmu_entry_writer {
> struct arm_smmu_entry_writer_ops {
> unsigned int num_entry_qwords;
> __le64 v_bit;
> + bool no_used_check;
> void (*get_used)(const __le64 *entry, __le64 *used);
> void (*sync)(struct arm_smmu_entry_writer *writer);
> };
>
> -#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64))
> +#define NUM_ENTRY_QWORDS \
> + (max(sizeof(struct arm_smmu_ste), sizeof(struct arm_smmu_cd)) / \
> + sizeof(u64))
>
> static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
> [EVTQ_MSI_INDEX] = {
> @@ -1056,7 +1059,8 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> * allowed to set a bit to 1 if the used function doesn't say it
> * is used.
> */
> - WARN_ON_ONCE(target[i] & ~target_used[i]);
> + if (!writer->ops->no_used_check)
> + WARN_ON_ONCE(target[i] & ~target_used[i]);
>
> /* Bits can change because they are not currently being used */
> unused_update[i] = (entry[i] & cur_used[i]) |
> @@ -1065,7 +1069,8 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> * Each bit indicates that a used bit in a qword needs to be
> * changed after unused_update is applied.
> */
> - if ((unused_update[i] & target_used[i]) != target[i])
> + if ((unused_update[i] & target_used[i]) !=
> + (target[i] & target_used[i]))
> used_qword_diff |= 1 << i;
> }
> return used_qword_diff;
> @@ -1161,8 +1166,11 @@ static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer,
> * in the entry. The target was already sanity checked by
> * compute_qword_diff().
> */
> - WARN_ON_ONCE(
> - entry_set(writer, entry, target, 0, num_entry_qwords));
> + if (writer->ops->no_used_check)
> + entry_set(writer, entry, target, 0, num_entry_qwords);
> + else
> + WARN_ON_ONCE(entry_set(writer, entry, target, 0,
> + num_entry_qwords));
> }
> }
>
> @@ -1242,6 +1250,59 @@ static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> return &l1_desc->l2ptr[idx];
> }
>
> +struct arm_smmu_cd_writer {
> + struct arm_smmu_entry_writer writer;
> + unsigned int ssid;
> +};
> +
> +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
> +{
> + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
> + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
> + return;
> + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
This is a slightly different approach than what the driver does for STEs,
where it explicitly sets the used bits. Is there a reason for that?
> +
> + /* EPD0 means T0SZ/TG0/IR0/OR0/SH0/TTB0 are IGNORED */
> + if (ent[0] & cpu_to_le64(CTXDESC_CD_0_TCR_EPD0)) {
> + used_bits[0] &= ~cpu_to_le64(
> + CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 |
> + CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 |
> + CTXDESC_CD_0_TCR_SH0);
> + used_bits[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK);
> + }
> +}
We should add a comment about EPD1 maybe?
> +
> +static void arm_smmu_cd_writer_sync_entry(struct arm_smmu_entry_writer *writer)
> +{
> + struct arm_smmu_cd_writer *cd_writer =
> + container_of(writer, struct arm_smmu_cd_writer, writer);
> +
> + arm_smmu_sync_cd(writer->master, cd_writer->ssid, true);
> +}
> +
> +static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
> + .sync = arm_smmu_cd_writer_sync_entry,
> + .get_used = arm_smmu_get_cd_used,
> + .v_bit = cpu_to_le64(CTXDESC_CD_0_V),
> + .no_used_check = true,
> + .num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
> +};
> +
> +static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> + struct arm_smmu_cd *cdptr,
> + const struct arm_smmu_cd *target)
> +{
> + struct arm_smmu_cd_writer cd_writer = {
> + .writer = {
> + .ops = &arm_smmu_cd_writer_ops,
> + .master = master,
> + },
> + .ssid = ssid,
> + };
> +
> + arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
> +}
> +
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_ctx_desc *cd)
> {
> @@ -1258,17 +1319,20 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> */
> u64 val;
> bool cd_live;
> - struct arm_smmu_cd *cdptr;
> + struct arm_smmu_cd target;
> + struct arm_smmu_cd *cdptr = ⌖
> + struct arm_smmu_cd *cd_table_entry;
> struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table;
> struct arm_smmu_device *smmu = master->smmu;
>
> if (WARN_ON(ssid >= (1 << cd_table->s1cdmax)))
> return -E2BIG;
>
> - cdptr = arm_smmu_get_cd_ptr(master, ssid);
> - if (!cdptr)
> + cd_table_entry = arm_smmu_get_cd_ptr(master, ssid);
> + if (!cd_table_entry)
> return -ENOMEM;
>
> + target = *cd_table_entry;
> val = le64_to_cpu(cdptr->data[0]);
> cd_live = !!(val & CTXDESC_CD_0_V);
>
> @@ -1290,13 +1354,6 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> cdptr->data[2] = 0;
> cdptr->data[3] = cpu_to_le64(cd->mair);
>
> - /*
> - * STE may be live, and the SMMU might read dwords of this CD in any
> - * order. Ensure that it observes valid values before reading
> - * V=1.
> - */
> - arm_smmu_sync_cd(master, ssid, true);
> -
> val = cd->tcr |
> #ifdef __BIG_ENDIAN
> CTXDESC_CD_0_ENDI |
> @@ -1310,18 +1367,8 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid,
> if (cd_table->stall_enabled)
> val |= CTXDESC_CD_0_S;
> }
> -
> - /*
> - * The SMMU accesses 64-bit values atomically. See IHI0070Ca 3.21.3
> - * "Configuration structures and configuration invalidation completion"
> - *
> - * The size of single-copy atomic reads made by the SMMU is
> - * IMPLEMENTATION DEFINED but must be at least 64 bits. Any single
> - * field within an aligned 64-bit span of a structure can be altered
> - * without first making the structure invalid.
> - */
> - WRITE_ONCE(cdptr->data[0], cpu_to_le64(val));
> - arm_smmu_sync_cd(master, ssid, true);
> + cdptr->data[0] = cpu_to_le64(val);
> + arm_smmu_write_cd_entry(master, ssid, cd_table_entry, &target);
> return 0;
> }
>
> --
> 2.43.2
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 07/27] iommu/arm-smmu-v3: Move the CD generation for S1 domains into a function
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-23 13:11 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-23 13:11 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:55PM -0400, Jason Gunthorpe wrote:
> Introduce arm_smmu_make_s1_cd() to build the CD from the paging S1 domain,
> and reorganize all the places programming S1 domain CD table entries to
> call it.
>
> Split arm_smmu_update_s1_domain_cd_entry() from
> arm_smmu_update_ctx_desc_devices() so that the S1 path has its own call
> chain separate from the unrelated SVA path.
>
> arm_smmu_update_s1_domain_cd_entry() only works on S1 domains
> attached to RIDs and refreshes all their CDs.
>
> Remove the forced clear of the CD during S1 domain attach,
> arm_smmu_write_cd_entry() will do this automatically if necessary.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 25 +++++++-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 60 +++++++++++++------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 +++
> 3 files changed, 75 insertions(+), 18 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index bb9bb6fd7914ce..6acc65f6d00a71 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -54,6 +54,29 @@ static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> }
>
> +static void
> +arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> +{
> + struct arm_smmu_master *master;
> + struct arm_smmu_cd target_cd;
> + unsigned long flags;
> +
> + spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> + list_for_each_entry(master, &smmu_domain->devices, domain_head) {
> + struct arm_smmu_cd *cdptr;
> +
> + /* S1 domains only support RID attachment right now */
> + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> + if (WARN_ON(!cdptr))
This should never hit, no? Otherwise that means this path can allocate memory
with a spinlock.
> + continue;
> +
> + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> + arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> + &target_cd);
> + }
> + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> +}
> +
> /*
> * Check if the CPU ASID is available on the SMMU side. If a private context
> * descriptor is using it, try to replace it.
> @@ -97,7 +120,7 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid)
> * be some overlap between use of both ASIDs, until we invalidate the
> * TLB.
> */
> - arm_smmu_update_ctx_desc_devices(smmu_domain, IOMMU_NO_PASID, cd);
> + arm_smmu_update_s1_domain_cd_entry(smmu_domain);
>
> /* Invalidate TLB entries previously associated with that context */
> arm_smmu_tlb_inv_asid(smmu, asid);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 3fb4a1523d1d3f..e25dbb982feeee 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1222,8 +1222,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
> WRITE_ONCE(*dst, cpu_to_le64(val));
> }
>
> -static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> - u32 ssid)
> +struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> + u32 ssid)
> {
> __le64 *l1ptr;
> unsigned int idx;
> @@ -1288,9 +1288,9 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
> .num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
> };
>
> -static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> - struct arm_smmu_cd *cdptr,
> - const struct arm_smmu_cd *target)
> +void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> + struct arm_smmu_cd *cdptr,
> + const struct arm_smmu_cd *target)
> {
> struct arm_smmu_cd_writer cd_writer = {
> .writer = {
> @@ -1303,6 +1303,32 @@ static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
> }
>
> +void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> + struct arm_smmu_master *master,
> + struct arm_smmu_domain *smmu_domain)
> +{
> + struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
> +
> + memset(target, 0, sizeof(*target));
> +
> + target->data[0] = cpu_to_le64(
> + cd->tcr |
> +#ifdef __BIG_ENDIAN
> + CTXDESC_CD_0_ENDI |
> +#endif
> + CTXDESC_CD_0_V |
> + CTXDESC_CD_0_AA64 |
> + (master->stall_enabled ? CTXDESC_CD_0_S : 0) |
> + CTXDESC_CD_0_R |
> + CTXDESC_CD_0_A |
> + CTXDESC_CD_0_ASET |
> + FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid)
> + );
> +
> + target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> + target->data[3] = cpu_to_le64(cd->mair);
> +}
> +
That seems to duplicate some logic from arm_smmu_write_ctx_desc(),
can that be consolidated?
> void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> {
> struct arm_smmu_cd target = {};
> @@ -2689,29 +2715,29 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
>
> switch (smmu_domain->stage) {
> - case ARM_SMMU_DOMAIN_S1:
> + case ARM_SMMU_DOMAIN_S1: {
> + struct arm_smmu_cd target_cd;
> + struct arm_smmu_cd *cdptr;
> +
> if (!master->cd_table.cdtab) {
> ret = arm_smmu_alloc_cd_tables(master);
> if (ret)
> goto out_list_del;
> - } else {
> - /*
> - * arm_smmu_write_ctx_desc() relies on the entry being
> - * invalid to work, clear any existing entry.
> - */
> - ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID,
> - NULL);
> - if (ret)
> - goto out_list_del;
> }
>
> - ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, &smmu_domain->cd);
> - if (ret)
> + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> + if (!cdptr) {
> + ret = -ENOMEM;
> goto out_list_del;
> + }
>
> + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> + arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> + &target_cd);
> arm_smmu_make_cdtable_ste(&target, master);
> arm_smmu_install_ste_for_dev(master, &target);
> break;
> + }
> case ARM_SMMU_DOMAIN_S2:
> arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
> arm_smmu_install_ste_for_dev(master, &target);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 87a7b57f566fbc..d32da11058aab6 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -750,6 +750,14 @@ extern struct mutex arm_smmu_asid_lock;
> extern struct arm_smmu_ctx_desc quiet_cd;
>
> void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
> +struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> + u32 ssid);
> +void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> + struct arm_smmu_master *master,
> + struct arm_smmu_domain *smmu_domain);
> +void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> + struct arm_smmu_cd *cdptr,
> + const struct arm_smmu_cd *target);
>
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid,
> struct arm_smmu_ctx_desc *cd);
> --
> 2.43.2
>
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 07/27] iommu/arm-smmu-v3: Move the CD generation for S1 domains into a function
@ 2024-03-23 13:11 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-23 13:11 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 04, 2024 at 07:43:55PM -0400, Jason Gunthorpe wrote:
> Introduce arm_smmu_make_s1_cd() to build the CD from the paging S1 domain,
> and reorganize all the places programming S1 domain CD table entries to
> call it.
>
> Split arm_smmu_update_s1_domain_cd_entry() from
> arm_smmu_update_ctx_desc_devices() so that the S1 path has its own call
> chain separate from the unrelated SVA path.
>
> arm_smmu_update_s1_domain_cd_entry() only works on S1 domains
> attached to RIDs and refreshes all their CDs.
>
> Remove the forced clear of the CD during S1 domain attach,
> arm_smmu_write_cd_entry() will do this automatically if necessary.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 25 +++++++-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 60 +++++++++++++------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 +++
> 3 files changed, 75 insertions(+), 18 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index bb9bb6fd7914ce..6acc65f6d00a71 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -54,6 +54,29 @@ static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> }
>
> +static void
> +arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> +{
> + struct arm_smmu_master *master;
> + struct arm_smmu_cd target_cd;
> + unsigned long flags;
> +
> + spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> + list_for_each_entry(master, &smmu_domain->devices, domain_head) {
> + struct arm_smmu_cd *cdptr;
> +
> + /* S1 domains only support RID attachment right now */
> + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> + if (WARN_ON(!cdptr))
This should never hit, no? Otherwise that means this path can allocate memory
with a spinlock.
> + continue;
> +
> + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> + arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> + &target_cd);
> + }
> + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> +}
> +
> /*
> * Check if the CPU ASID is available on the SMMU side. If a private context
> * descriptor is using it, try to replace it.
> @@ -97,7 +120,7 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid)
> * be some overlap between use of both ASIDs, until we invalidate the
> * TLB.
> */
> - arm_smmu_update_ctx_desc_devices(smmu_domain, IOMMU_NO_PASID, cd);
> + arm_smmu_update_s1_domain_cd_entry(smmu_domain);
>
> /* Invalidate TLB entries previously associated with that context */
> arm_smmu_tlb_inv_asid(smmu, asid);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 3fb4a1523d1d3f..e25dbb982feeee 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1222,8 +1222,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst,
> WRITE_ONCE(*dst, cpu_to_le64(val));
> }
>
> -static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> - u32 ssid)
> +struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> + u32 ssid)
> {
> __le64 *l1ptr;
> unsigned int idx;
> @@ -1288,9 +1288,9 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
> .num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
> };
>
> -static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> - struct arm_smmu_cd *cdptr,
> - const struct arm_smmu_cd *target)
> +void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> + struct arm_smmu_cd *cdptr,
> + const struct arm_smmu_cd *target)
> {
> struct arm_smmu_cd_writer cd_writer = {
> .writer = {
> @@ -1303,6 +1303,32 @@ static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data);
> }
>
> +void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> + struct arm_smmu_master *master,
> + struct arm_smmu_domain *smmu_domain)
> +{
> + struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
> +
> + memset(target, 0, sizeof(*target));
> +
> + target->data[0] = cpu_to_le64(
> + cd->tcr |
> +#ifdef __BIG_ENDIAN
> + CTXDESC_CD_0_ENDI |
> +#endif
> + CTXDESC_CD_0_V |
> + CTXDESC_CD_0_AA64 |
> + (master->stall_enabled ? CTXDESC_CD_0_S : 0) |
> + CTXDESC_CD_0_R |
> + CTXDESC_CD_0_A |
> + CTXDESC_CD_0_ASET |
> + FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid)
> + );
> +
> + target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> + target->data[3] = cpu_to_le64(cd->mair);
> +}
> +
That seems to duplicate some logic from arm_smmu_write_ctx_desc(),
can that be consolidated?
> void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> {
> struct arm_smmu_cd target = {};
> @@ -2689,29 +2715,29 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
> spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
>
> switch (smmu_domain->stage) {
> - case ARM_SMMU_DOMAIN_S1:
> + case ARM_SMMU_DOMAIN_S1: {
> + struct arm_smmu_cd target_cd;
> + struct arm_smmu_cd *cdptr;
> +
> if (!master->cd_table.cdtab) {
> ret = arm_smmu_alloc_cd_tables(master);
> if (ret)
> goto out_list_del;
> - } else {
> - /*
> - * arm_smmu_write_ctx_desc() relies on the entry being
> - * invalid to work, clear any existing entry.
> - */
> - ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID,
> - NULL);
> - if (ret)
> - goto out_list_del;
> }
>
> - ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, &smmu_domain->cd);
> - if (ret)
> + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> + if (!cdptr) {
> + ret = -ENOMEM;
> goto out_list_del;
> + }
>
> + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
> + arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
> + &target_cd);
> arm_smmu_make_cdtable_ste(&target, master);
> arm_smmu_install_ste_for_dev(master, &target);
> break;
> + }
> case ARM_SMMU_DOMAIN_S2:
> arm_smmu_make_s2_domain_ste(&target, master, smmu_domain);
> arm_smmu_install_ste_for_dev(master, &target);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 87a7b57f566fbc..d32da11058aab6 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -750,6 +750,14 @@ extern struct mutex arm_smmu_asid_lock;
> extern struct arm_smmu_ctx_desc quiet_cd;
>
> void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid);
> +struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
> + u32 ssid);
> +void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> + struct arm_smmu_master *master,
> + struct arm_smmu_domain *smmu_domain);
> +void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> + struct arm_smmu_cd *cdptr,
> + const struct arm_smmu_cd *target);
>
> int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid,
> struct arm_smmu_ctx_desc *cd);
> --
> 2.43.2
>
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 11/27] iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd()
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-23 13:20 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-23 13:20 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:59PM -0400, Jason Gunthorpe wrote:
> Half the code was living in arm_smmu_domain_finalise_s1(), just move it
> here and take the values directly from the pgtbl_ops instead of storing
> copies.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 47 ++++++++-------------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 --
> 2 files changed, 18 insertions(+), 32 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 50d17e3ce0a956..dfdd48cf217c4e 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1301,15 +1301,25 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> struct arm_smmu_domain *smmu_domain)
> {
> struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
> + const struct io_pgtable_cfg *pgtbl_cfg =
> + &io_pgtable_ops_to_pgtable(smmu_domain->pgtbl_ops)->cfg;
> + typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr =
> + &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
>
> memset(target, 0, sizeof(*target));
>
> target->data[0] = cpu_to_le64(
> - cd->tcr |
> + FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) |
> + FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) |
> + FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) |
> + FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) |
> + FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) |
> + CTXDESC_CD_0_TCR_EPD1 |
> #ifdef __BIG_ENDIAN
> CTXDESC_CD_0_ENDI |
> #endif
> CTXDESC_CD_0_V |
> + FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) |
> CTXDESC_CD_0_AA64 |
> (master->stall_enabled ? CTXDESC_CD_0_S : 0) |
> CTXDESC_CD_0_R |
> @@ -1317,9 +1327,9 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> CTXDESC_CD_0_ASET |
> FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid)
> );
> -
> - target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> - target->data[3] = cpu_to_le64(cd->mair);
> + target->data[1] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.ttbr &
> + CTXDESC_CD_1_TTB0_MASK);
> + target->data[3] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.mair);
> }
>
> void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> @@ -2305,13 +2315,11 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
> }
>
> static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
> - struct arm_smmu_domain *smmu_domain,
> - struct io_pgtable_cfg *pgtbl_cfg)
> + struct arm_smmu_domain *smmu_domain)
> {
> int ret;
> u32 asid;
> struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
> - typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
>
> refcount_set(&cd->refs, 1);
>
> @@ -2319,31 +2327,13 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
> mutex_lock(&arm_smmu_asid_lock);
> ret = xa_alloc(&arm_smmu_asid_xa, &asid, cd,
> XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> - if (ret)
> - goto out_unlock;
> -
> cd->asid = (u16)asid;
> - cd->ttbr = pgtbl_cfg->arm_lpae_s1_cfg.ttbr;
> - cd->tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) |
> - CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64;
> - cd->mair = pgtbl_cfg->arm_lpae_s1_cfg.mair;
> -
> - mutex_unlock(&arm_smmu_asid_lock);
> - return 0;
> -
> -out_unlock:
> mutex_unlock(&arm_smmu_asid_lock);
> return ret;
> }
>
> static int arm_smmu_domain_finalise_s2(struct arm_smmu_device *smmu,
> - struct arm_smmu_domain *smmu_domain,
> - struct io_pgtable_cfg *pgtbl_cfg)
> + struct arm_smmu_domain *smmu_domain)
> {
> int vmid;
> struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
> @@ -2367,8 +2357,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
> struct io_pgtable_cfg pgtbl_cfg;
> struct io_pgtable_ops *pgtbl_ops;
> int (*finalise_stage_fn)(struct arm_smmu_device *smmu,
> - struct arm_smmu_domain *smmu_domain,
> - struct io_pgtable_cfg *pgtbl_cfg);
> + struct arm_smmu_domain *smmu_domain);
>
> /* Restrict the stage to what we can actually support */
> if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S1))
> @@ -2411,7 +2400,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
> smmu_domain->domain.geometry.aperture_end = (1UL << pgtbl_cfg.ias) - 1;
> smmu_domain->domain.geometry.force_aperture = true;
>
> - ret = finalise_stage_fn(smmu, smmu_domain, &pgtbl_cfg);
> + ret = finalise_stage_fn(smmu, smmu_domain);
> if (ret < 0) {
> free_io_pgtable_ops(pgtbl_ops);
> return ret;
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 8eabcccb9420ba..468cd33b80ac35 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -587,9 +587,6 @@ struct arm_smmu_strtab_l1_desc {
>
> struct arm_smmu_ctx_desc {
> u16 asid;
> - u64 ttbr;
> - u64 tcr;
> - u64 mair;
>
> refcount_t refs;
> struct mm_struct *mm;
> --
> 2.43.2
>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 11/27] iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd()
@ 2024-03-23 13:20 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-23 13:20 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:59PM -0400, Jason Gunthorpe wrote:
> Half the code was living in arm_smmu_domain_finalise_s1(), just move it
> here and take the values directly from the pgtbl_ops instead of storing
> copies.
>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 47 ++++++++-------------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 --
> 2 files changed, 18 insertions(+), 32 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 50d17e3ce0a956..dfdd48cf217c4e 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1301,15 +1301,25 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> struct arm_smmu_domain *smmu_domain)
> {
> struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
> + const struct io_pgtable_cfg *pgtbl_cfg =
> + &io_pgtable_ops_to_pgtable(smmu_domain->pgtbl_ops)->cfg;
> + typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr =
> + &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
>
> memset(target, 0, sizeof(*target));
>
> target->data[0] = cpu_to_le64(
> - cd->tcr |
> + FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) |
> + FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) |
> + FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) |
> + FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) |
> + FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) |
> + CTXDESC_CD_0_TCR_EPD1 |
> #ifdef __BIG_ENDIAN
> CTXDESC_CD_0_ENDI |
> #endif
> CTXDESC_CD_0_V |
> + FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) |
> CTXDESC_CD_0_AA64 |
> (master->stall_enabled ? CTXDESC_CD_0_S : 0) |
> CTXDESC_CD_0_R |
> @@ -1317,9 +1327,9 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> CTXDESC_CD_0_ASET |
> FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid)
> );
> -
> - target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> - target->data[3] = cpu_to_le64(cd->mair);
> + target->data[1] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.ttbr &
> + CTXDESC_CD_1_TTB0_MASK);
> + target->data[3] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.mair);
> }
>
> void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> @@ -2305,13 +2315,11 @@ static void arm_smmu_domain_free(struct iommu_domain *domain)
> }
>
> static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
> - struct arm_smmu_domain *smmu_domain,
> - struct io_pgtable_cfg *pgtbl_cfg)
> + struct arm_smmu_domain *smmu_domain)
> {
> int ret;
> u32 asid;
> struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
> - typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr;
>
> refcount_set(&cd->refs, 1);
>
> @@ -2319,31 +2327,13 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
> mutex_lock(&arm_smmu_asid_lock);
> ret = xa_alloc(&arm_smmu_asid_xa, &asid, cd,
> XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
> - if (ret)
> - goto out_unlock;
> -
> cd->asid = (u16)asid;
> - cd->ttbr = pgtbl_cfg->arm_lpae_s1_cfg.ttbr;
> - cd->tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) |
> - FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) |
> - CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64;
> - cd->mair = pgtbl_cfg->arm_lpae_s1_cfg.mair;
> -
> - mutex_unlock(&arm_smmu_asid_lock);
> - return 0;
> -
> -out_unlock:
> mutex_unlock(&arm_smmu_asid_lock);
> return ret;
> }
>
> static int arm_smmu_domain_finalise_s2(struct arm_smmu_device *smmu,
> - struct arm_smmu_domain *smmu_domain,
> - struct io_pgtable_cfg *pgtbl_cfg)
> + struct arm_smmu_domain *smmu_domain)
> {
> int vmid;
> struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
> @@ -2367,8 +2357,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
> struct io_pgtable_cfg pgtbl_cfg;
> struct io_pgtable_ops *pgtbl_ops;
> int (*finalise_stage_fn)(struct arm_smmu_device *smmu,
> - struct arm_smmu_domain *smmu_domain,
> - struct io_pgtable_cfg *pgtbl_cfg);
> + struct arm_smmu_domain *smmu_domain);
>
> /* Restrict the stage to what we can actually support */
> if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S1))
> @@ -2411,7 +2400,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
> smmu_domain->domain.geometry.aperture_end = (1UL << pgtbl_cfg.ias) - 1;
> smmu_domain->domain.geometry.force_aperture = true;
>
> - ret = finalise_stage_fn(smmu, smmu_domain, &pgtbl_cfg);
> + ret = finalise_stage_fn(smmu, smmu_domain);
> if (ret < 0) {
> free_io_pgtable_ops(pgtbl_ops);
> return ret;
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 8eabcccb9420ba..468cd33b80ac35 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -587,9 +587,6 @@ struct arm_smmu_strtab_l1_desc {
>
> struct arm_smmu_ctx_desc {
> u16 asid;
> - u64 ttbr;
> - u64 tcr;
> - u64 mair;
>
> refcount_t refs;
> struct mm_struct *mm;
> --
> 2.43.2
>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-23 13:38 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-23 13:38 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:48PM -0400, Jason Gunthorpe wrote:
> Continuing the work of part 1 this focuses on the CD, PASID and SVA
> components:
>
> - attach_dev failure does not change the HW configuration.
>
> - Full PASID API support including:
> - S1/SVA domains attached to PASIDs
I am still going through the series, but I see at the end the main SMMUv3
driver has set_dev_pasid operation, are there any in-tree drivers that
use that? (and how can I test it).
> - IDENTITY/BLOCKED/S1 attached to RID
> - Change of the RID domain while PASIDs are attached
>
> - Streamlined SVA support using the core infrastructure
>
> - Hitless, whenever possible, change between two domains
Can you please clarify what cases are expected to be hitless?
From what I see if ASID and TTB0 changes that would break the CD.
>
> Making the CD programming work like the new STE programming allows
> untangling some of the confusing SVA flows. From there the focus is on
> building out the core infrastructure for dealing with PASID and CD
> entries, then keeping track of unique SSID's for ATS invalidation.
>
> The ATS ordering is generalized so that the PASID flow can use it and put
> into a form where it is fully hitless, whenever possible. Care is taken to
> ensure that ATC flushes are present after any change in translation.
>
> Finally we simply kill the entire outdated SVA mmu_notifier implementation
> in one shot and switch it over to the newly created generic PASID & CD
> code. This avoids the messy and confusing approach of trying to
> incrementally untangle this in place. The new code is small and simple
> enough this is much better than trying to figure out smaller steps.
>
> Once SVA is resting on the right CD code it is straightforward to make the
> PASID interface functionally complete.
>
> It achieves the same goals as the several series from Michael and the S1DSS
> series from Nicolin that were trying to improve portions of the API.
>
> This is on github:
> https://github.com/jgunthorpe/linux/commits/smmuv3_newapi
>
> v5:
> - Rebase on v6.8-rc7 & Will's tree
> - Accomdate the SVA rc patch removing the master list iteration
> - Move the kfree(to_smmu_domain(domain)) hunk to the right patch
> - Move S1DSS get_used hunk to "Allow IDENTITY/BLOCKED to be set while
> PASID is used"
> v4: https://lore.kernel.org/r/0-v4-e7091cdd9e8d+43b1-smmuv3_newapi_p2_jgg@nvidia.com
> - Rebase on v6.8-rc1, adjust to use mm_get_enqcmd_pasid() and eventually
> remove all references from ARM. Move the new ARM_SMMU_FEAT_STALL_FORCE
> stuff to arm_smmu_make_sva_cd()
> - Adjust to use the new shared STE/CD writer logic. Disable some of the
> sanity checks for the interior of the series
> - Return ERR_PTR from domain_alloc functions
> - Move the ATS disablement flow into arm_smmu_attach_prepare()/commit()
> which lets all the STE update flows use the same sequence. This is
> needed for nesting in part 3
> - Put ssid in attach_state
> - Replace to_smmu_domain_safe() with to_smmu_domain_devices()
> v3: https://lore.kernel.org/r/0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com
> - Rebase on the latest part 1
> - update comments and commit messages
> - Fix error exit in arm_smmu_set_pasid()
> - Fix inverted logic for btm_invalidation
> - Add missing ATC invalidation on mm release
> - Add a big comment explaining that BTM is not enabled and what is
> missing to enable it.
> v2: https://lore.kernel.org/r/0-v2-16665a652079+5947-smmuv3_newapi_p2_jgg@nvidia.com
> - Rebased on iommmufd + Joerg's tree
> - Use sid_smmu_domain consistently to refer to the domain attached to the
> device (eg the PCIe RID)
> - Rework how arm_smmu_attach_*() and callers flow to be more careful
> about ordering around ATC invalidation. The ATC must be invalidated
> after it is impossible to establish stale entires.
> - ATS disable is now entirely part of arm_smmu_attach_dev_ste(), which is
> the only STE type that ever disables ATS.
> - Remove the 'existing_master_domain' optimization, the code is
> functionally fine without it.
> - Whitespace, spelling, and checkpatch related items
> - Fixed wrong value stored in the xa for the BTM flows
> - Use pasid more consistently instead of id
> v1: https://lore.kernel.org/r/0-v1-afbb86647bbd+5-smmuv3_newapi_p2_jgg@nvidia.com
>
> Jason Gunthorpe (27):
> iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong
> PASID
> iommu/arm-smmu-v3: Do not ATC invalidate the entire domain
> iommu/arm-smmu-v3: Add a type for the CD entry
> iommu/arm-smmu-v3: Add an ops indirection to the STE code
> iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
> iommu/arm-smmu-v3: Consolidate clearing a CD table entry
> iommu/arm-smmu-v3: Move the CD generation for S1 domains into a
> function
> iommu/arm-smmu-v3: Move allocation of the cdtable into
> arm_smmu_get_cd_ptr()
> iommu/arm-smmu-v3: Allocate the CD table entry in advance
> iommu/arm-smmu-v3: Move the CD generation for SVA into a function
> iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd()
> iommu/arm-smmu-v3: Start building a generic PASID layer
> iommu/arm-smmu-v3: Make smmu_domain->devices into an allocated list
> iommu/arm-smmu-v3: Make changing domains be hitless for ATS
> iommu/arm-smmu-v3: Add ssid to struct arm_smmu_master_domain
> iommu/arm-smmu-v3: Keep track of valid CD entries in the cd_table
> iommu/arm-smmu-v3: Thread SSID through the arm_smmu_attach_*()
> interface
> iommu/arm-smmu-v3: Make SVA allocate a normal arm_smmu_domain
> iommu/arm-smmu-v3: Keep track of arm_smmu_master_domain for SVA
> iommu: Add ops->domain_alloc_sva()
> iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain
> iommu/arm-smmu-v3: Consolidate freeing the ASID/VMID
> iommu/arm-smmu-v3: Move the arm_smmu_asid_xa to per-smmu like vmid
> iommu/arm-smmu-v3: Bring back SVA BTM support
> iommu/arm-smmu-v3: Allow IDENTITY/BLOCKED to be set while PASID is
> used
> iommu/arm-smmu-v3: Allow a PASID to be set when RID is
> IDENTITY/BLOCKED
> iommu/arm-smmu-v3: Allow setting a S1 domain to a PASID
>
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 639 +++++-----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 1036 +++++++++++------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 79 +-
> drivers/iommu/iommu-sva.c | 4 +-
> drivers/iommu/iommu.c | 12 +-
> include/linux/iommu.h | 3 +
> 6 files changed, 1024 insertions(+), 749 deletions(-)
>
>
> base-commit: 98b23ebb0c84657a135957d727eedebd1280cbbf
> --
> 2.43.2
>
Thansks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
@ 2024-03-23 13:38 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-23 13:38 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:48PM -0400, Jason Gunthorpe wrote:
> Continuing the work of part 1 this focuses on the CD, PASID and SVA
> components:
>
> - attach_dev failure does not change the HW configuration.
>
> - Full PASID API support including:
> - S1/SVA domains attached to PASIDs
I am still going through the series, but I see at the end the main SMMUv3
driver has set_dev_pasid operation, are there any in-tree drivers that
use that? (and how can I test it).
> - IDENTITY/BLOCKED/S1 attached to RID
> - Change of the RID domain while PASIDs are attached
>
> - Streamlined SVA support using the core infrastructure
>
> - Hitless, whenever possible, change between two domains
Can you please clarify what cases are expected to be hitless?
From what I see if ASID and TTB0 changes that would break the CD.
>
> Making the CD programming work like the new STE programming allows
> untangling some of the confusing SVA flows. From there the focus is on
> building out the core infrastructure for dealing with PASID and CD
> entries, then keeping track of unique SSID's for ATS invalidation.
>
> The ATS ordering is generalized so that the PASID flow can use it and put
> into a form where it is fully hitless, whenever possible. Care is taken to
> ensure that ATC flushes are present after any change in translation.
>
> Finally we simply kill the entire outdated SVA mmu_notifier implementation
> in one shot and switch it over to the newly created generic PASID & CD
> code. This avoids the messy and confusing approach of trying to
> incrementally untangle this in place. The new code is small and simple
> enough this is much better than trying to figure out smaller steps.
>
> Once SVA is resting on the right CD code it is straightforward to make the
> PASID interface functionally complete.
>
> It achieves the same goals as the several series from Michael and the S1DSS
> series from Nicolin that were trying to improve portions of the API.
>
> This is on github:
> https://github.com/jgunthorpe/linux/commits/smmuv3_newapi
>
> v5:
> - Rebase on v6.8-rc7 & Will's tree
> - Accomdate the SVA rc patch removing the master list iteration
> - Move the kfree(to_smmu_domain(domain)) hunk to the right patch
> - Move S1DSS get_used hunk to "Allow IDENTITY/BLOCKED to be set while
> PASID is used"
> v4: https://lore.kernel.org/r/0-v4-e7091cdd9e8d+43b1-smmuv3_newapi_p2_jgg@nvidia.com
> - Rebase on v6.8-rc1, adjust to use mm_get_enqcmd_pasid() and eventually
> remove all references from ARM. Move the new ARM_SMMU_FEAT_STALL_FORCE
> stuff to arm_smmu_make_sva_cd()
> - Adjust to use the new shared STE/CD writer logic. Disable some of the
> sanity checks for the interior of the series
> - Return ERR_PTR from domain_alloc functions
> - Move the ATS disablement flow into arm_smmu_attach_prepare()/commit()
> which lets all the STE update flows use the same sequence. This is
> needed for nesting in part 3
> - Put ssid in attach_state
> - Replace to_smmu_domain_safe() with to_smmu_domain_devices()
> v3: https://lore.kernel.org/r/0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com
> - Rebase on the latest part 1
> - update comments and commit messages
> - Fix error exit in arm_smmu_set_pasid()
> - Fix inverted logic for btm_invalidation
> - Add missing ATC invalidation on mm release
> - Add a big comment explaining that BTM is not enabled and what is
> missing to enable it.
> v2: https://lore.kernel.org/r/0-v2-16665a652079+5947-smmuv3_newapi_p2_jgg@nvidia.com
> - Rebased on iommmufd + Joerg's tree
> - Use sid_smmu_domain consistently to refer to the domain attached to the
> device (eg the PCIe RID)
> - Rework how arm_smmu_attach_*() and callers flow to be more careful
> about ordering around ATC invalidation. The ATC must be invalidated
> after it is impossible to establish stale entires.
> - ATS disable is now entirely part of arm_smmu_attach_dev_ste(), which is
> the only STE type that ever disables ATS.
> - Remove the 'existing_master_domain' optimization, the code is
> functionally fine without it.
> - Whitespace, spelling, and checkpatch related items
> - Fixed wrong value stored in the xa for the BTM flows
> - Use pasid more consistently instead of id
> v1: https://lore.kernel.org/r/0-v1-afbb86647bbd+5-smmuv3_newapi_p2_jgg@nvidia.com
>
> Jason Gunthorpe (27):
> iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong
> PASID
> iommu/arm-smmu-v3: Do not ATC invalidate the entire domain
> iommu/arm-smmu-v3: Add a type for the CD entry
> iommu/arm-smmu-v3: Add an ops indirection to the STE code
> iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
> iommu/arm-smmu-v3: Consolidate clearing a CD table entry
> iommu/arm-smmu-v3: Move the CD generation for S1 domains into a
> function
> iommu/arm-smmu-v3: Move allocation of the cdtable into
> arm_smmu_get_cd_ptr()
> iommu/arm-smmu-v3: Allocate the CD table entry in advance
> iommu/arm-smmu-v3: Move the CD generation for SVA into a function
> iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd()
> iommu/arm-smmu-v3: Start building a generic PASID layer
> iommu/arm-smmu-v3: Make smmu_domain->devices into an allocated list
> iommu/arm-smmu-v3: Make changing domains be hitless for ATS
> iommu/arm-smmu-v3: Add ssid to struct arm_smmu_master_domain
> iommu/arm-smmu-v3: Keep track of valid CD entries in the cd_table
> iommu/arm-smmu-v3: Thread SSID through the arm_smmu_attach_*()
> interface
> iommu/arm-smmu-v3: Make SVA allocate a normal arm_smmu_domain
> iommu/arm-smmu-v3: Keep track of arm_smmu_master_domain for SVA
> iommu: Add ops->domain_alloc_sva()
> iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain
> iommu/arm-smmu-v3: Consolidate freeing the ASID/VMID
> iommu/arm-smmu-v3: Move the arm_smmu_asid_xa to per-smmu like vmid
> iommu/arm-smmu-v3: Bring back SVA BTM support
> iommu/arm-smmu-v3: Allow IDENTITY/BLOCKED to be set while PASID is
> used
> iommu/arm-smmu-v3: Allow a PASID to be set when RID is
> IDENTITY/BLOCKED
> iommu/arm-smmu-v3: Allow setting a S1 domain to a PASID
>
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 639 +++++-----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 1036 +++++++++++------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 79 +-
> drivers/iommu/iommu-sva.c | 4 +-
> drivers/iommu/iommu.c | 12 +-
> include/linux/iommu.h | 3 +
> 6 files changed, 1024 insertions(+), 749 deletions(-)
>
>
> base-commit: 98b23ebb0c84657a135957d727eedebd1280cbbf
> --
> 2.43.2
>
Thansks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
2024-03-04 23:43 ` Jason Gunthorpe
@ 2024-03-25 10:22 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-25 10:22 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:48PM -0400, Jason Gunthorpe wrote:
> Continuing the work of part 1 this focuses on the CD, PASID and SVA
> components:
>
> - attach_dev failure does not change the HW configuration.
>
> - Full PASID API support including:
> - S1/SVA domains attached to PASIDs
> - IDENTITY/BLOCKED/S1 attached to RID
> - Change of the RID domain while PASIDs are attached
>
> - Streamlined SVA support using the core infrastructure
>
> - Hitless, whenever possible, change between two domains
>
> Making the CD programming work like the new STE programming allows
> untangling some of the confusing SVA flows. From there the focus is on
> building out the core infrastructure for dealing with PASID and CD
> entries, then keeping track of unique SSID's for ATS invalidation.
>
> The ATS ordering is generalized so that the PASID flow can use it and put
> into a form where it is fully hitless, whenever possible. Care is taken to
> ensure that ATC flushes are present after any change in translation.
>
> Finally we simply kill the entire outdated SVA mmu_notifier implementation
> in one shot and switch it over to the newly created generic PASID & CD
> code. This avoids the messy and confusing approach of trying to
> incrementally untangle this in place. The new code is small and simple
> enough this is much better than trying to figure out smaller steps.
>
> Once SVA is resting on the right CD code it is straightforward to make the
> PASID interface functionally complete.
>
> It achieves the same goals as the several series from Michael and the S1DSS
> series from Nicolin that were trying to improve portions of the API.
>
> This is on github:
> https://github.com/jgunthorpe/linux/commits/smmuv3_newapi
Testing on qemu[1], with the same VMM Shameer tested with[2]:
qemu/build/qemu-system-aarch64 -M virt -machine virt,gic-version=3,iommu=nested-smmuv3,iommufd=iommufd0 \
-cpu cortex-a53,pmu=off -smp 1 -m 2048 \
-kernel Image \
-drive file=rootfs.ext4,if=virtio,format=raw \
-object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0 -nographic \
-append 'console=ttyAMA0 rootwait root=/dev/vda' \
-device virtio-scsi-pci,id=scsi0 \
-device ioh3420,id=pcie.1,chassis=1 \
-object iommufd,id=iommufd0 \
-device vfio-pci,host=0000:00:03.0,iommufd=iommufd0
I see the following panic:
[ 155.141233] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[ 155.142416] Mem abort info:
[ 155.142722] ESR = 0x0000000086000004
[ 155.143106] EC = 0x21: IABT (current EL), IL = 32 bits
[ 155.143827] SET = 0, FnV = 0
[ 155.144266] EA = 0, S1PTW = 0
[ 155.144721] FSC = 0x04: level 0 translation fault
[ 155.145432] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000101059000
[ 155.146234] [0000000000000000] pgd=0000000000000000, p4d=0000000000000000
[ 155.148162] Internal error: Oops: 0000000086000004 [#1] PREEMPT SMP
[ 155.149399] Modules linked in:
[ 155.150366] CPU: 2 PID: 371 Comm: qemu-system-aar Not tainted 6.8.0-rc7-gde77230ac23a #9
[ 155.151728] Hardware name: linux,dummy-virt (DT)
[ 155.152770] pstate: 81400809 (Nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=-c)
[ 155.153895] pc : 0x0
[ 155.154889] lr : iommufd_hwpt_invalidate+0xa4/0x204
[ 155.156272] sp : ffff800080f3bcc0
[ 155.156971] x29: ffff800080f3bcf0 x28: ffff0000c369b300 x27: 0000000000000000
[ 155.158135] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[ 155.159175] x23: 0000000000000000 x22: 00000000c1e334a0 x21: ffff0000c1e334a0
[ 155.160343] x20: ffff800080f3bd38 x19: ffff800080f3bd58 x18: 0000000000000000
[ 155.161298] x17: 0000000000000000 x16: 0000000000000000 x15: 0000ffff8240d6d8
[ 155.162355] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 155.163463] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[ 155.164947] x8 : 0000001000000002 x7 : 0000fffeac1ec950 x6 : 0000000000000000
[ 155.166057] x5 : ffff800080f3bd78 x4 : 0000000000000003 x3 : 0000000000000002
[ 155.167343] x2 : 0000000000000000 x1 : ffff800080f3bcc8 x0 : ffff0000c6034d80
[ 155.168851] Call trace:
[ 155.169738] 0x0
[ 155.170623] iommufd_fops_ioctl+0x154/0x274
[ 155.171555] __arm64_sys_ioctl+0xac/0xf0
[ 155.172095] invoke_syscall+0x48/0x110
[ 155.172633] el0_svc_common.constprop.0+0x40/0xe0
[ 155.173277] do_el0_svc+0x1c/0x28
[ 155.173847] el0_svc+0x34/0xb4
[ 155.174312] el0t_64_sync_handler+0x120/0x12c
[ 155.174969] el0t_64_sync+0x190/0x194
[ 155.176006] Code: ???????? ???????? ???????? ???????? (????????)
[ 155.178349] ---[ end trace 0000000000000000 ]---
The core IOMMUFD code calls domain->ops->cache_invalidate_user
unconditionally from IOCTL:IOMMU_HWPT_INVALIDATE and the SMMUv3 driver
doesn't implement it, that seems missing as otherwise the VMM can't
invalidate S1 mappings, or I a missing something?
[1] https://lore.kernel.org/all/20240325101442.1306300-1-smostafa@google.com/
[2] https://github.com/nicolinc/qemu/commits/wip/iommufd_vsmmu-02292024/
>
> v5:
> - Rebase on v6.8-rc7 & Will's tree
> - Accomdate the SVA rc patch removing the master list iteration
> - Move the kfree(to_smmu_domain(domain)) hunk to the right patch
> - Move S1DSS get_used hunk to "Allow IDENTITY/BLOCKED to be set while
> PASID is used"
> v4: https://lore.kernel.org/r/0-v4-e7091cdd9e8d+43b1-smmuv3_newapi_p2_jgg@nvidia.com
> - Rebase on v6.8-rc1, adjust to use mm_get_enqcmd_pasid() and eventually
> remove all references from ARM. Move the new ARM_SMMU_FEAT_STALL_FORCE
> stuff to arm_smmu_make_sva_cd()
> - Adjust to use the new shared STE/CD writer logic. Disable some of the
> sanity checks for the interior of the series
> - Return ERR_PTR from domain_alloc functions
> - Move the ATS disablement flow into arm_smmu_attach_prepare()/commit()
> which lets all the STE update flows use the same sequence. This is
> needed for nesting in part 3
> - Put ssid in attach_state
> - Replace to_smmu_domain_safe() with to_smmu_domain_devices()
> v3: https://lore.kernel.org/r/0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com
> - Rebase on the latest part 1
> - update comments and commit messages
> - Fix error exit in arm_smmu_set_pasid()
> - Fix inverted logic for btm_invalidation
> - Add missing ATC invalidation on mm release
> - Add a big comment explaining that BTM is not enabled and what is
> missing to enable it.
> v2: https://lore.kernel.org/r/0-v2-16665a652079+5947-smmuv3_newapi_p2_jgg@nvidia.com
> - Rebased on iommmufd + Joerg's tree
> - Use sid_smmu_domain consistently to refer to the domain attached to the
> device (eg the PCIe RID)
> - Rework how arm_smmu_attach_*() and callers flow to be more careful
> about ordering around ATC invalidation. The ATC must be invalidated
> after it is impossible to establish stale entires.
> - ATS disable is now entirely part of arm_smmu_attach_dev_ste(), which is
> the only STE type that ever disables ATS.
> - Remove the 'existing_master_domain' optimization, the code is
> functionally fine without it.
> - Whitespace, spelling, and checkpatch related items
> - Fixed wrong value stored in the xa for the BTM flows
> - Use pasid more consistently instead of id
> v1: https://lore.kernel.org/r/0-v1-afbb86647bbd+5-smmuv3_newapi_p2_jgg@nvidia.com
>
> Jason Gunthorpe (27):
> iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong
> PASID
> iommu/arm-smmu-v3: Do not ATC invalidate the entire domain
> iommu/arm-smmu-v3: Add a type for the CD entry
> iommu/arm-smmu-v3: Add an ops indirection to the STE code
> iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
> iommu/arm-smmu-v3: Consolidate clearing a CD table entry
> iommu/arm-smmu-v3: Move the CD generation for S1 domains into a
> function
> iommu/arm-smmu-v3: Move allocation of the cdtable into
> arm_smmu_get_cd_ptr()
> iommu/arm-smmu-v3: Allocate the CD table entry in advance
> iommu/arm-smmu-v3: Move the CD generation for SVA into a function
> iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd()
> iommu/arm-smmu-v3: Start building a generic PASID layer
> iommu/arm-smmu-v3: Make smmu_domain->devices into an allocated list
> iommu/arm-smmu-v3: Make changing domains be hitless for ATS
> iommu/arm-smmu-v3: Add ssid to struct arm_smmu_master_domain
> iommu/arm-smmu-v3: Keep track of valid CD entries in the cd_table
> iommu/arm-smmu-v3: Thread SSID through the arm_smmu_attach_*()
> interface
> iommu/arm-smmu-v3: Make SVA allocate a normal arm_smmu_domain
> iommu/arm-smmu-v3: Keep track of arm_smmu_master_domain for SVA
> iommu: Add ops->domain_alloc_sva()
> iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain
> iommu/arm-smmu-v3: Consolidate freeing the ASID/VMID
> iommu/arm-smmu-v3: Move the arm_smmu_asid_xa to per-smmu like vmid
> iommu/arm-smmu-v3: Bring back SVA BTM support
> iommu/arm-smmu-v3: Allow IDENTITY/BLOCKED to be set while PASID is
> used
> iommu/arm-smmu-v3: Allow a PASID to be set when RID is
> IDENTITY/BLOCKED
> iommu/arm-smmu-v3: Allow setting a S1 domain to a PASID
>
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 639 +++++-----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 1036 +++++++++++------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 79 +-
> drivers/iommu/iommu-sva.c | 4 +-
> drivers/iommu/iommu.c | 12 +-
> include/linux/iommu.h | 3 +
> 6 files changed, 1024 insertions(+), 749 deletions(-)
>
>
> base-commit: 98b23ebb0c84657a135957d727eedebd1280cbbf
> --
> 2.43.2
>
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
@ 2024-03-25 10:22 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-25 10:22 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
Hi Jason,
On Mon, Mar 04, 2024 at 07:43:48PM -0400, Jason Gunthorpe wrote:
> Continuing the work of part 1 this focuses on the CD, PASID and SVA
> components:
>
> - attach_dev failure does not change the HW configuration.
>
> - Full PASID API support including:
> - S1/SVA domains attached to PASIDs
> - IDENTITY/BLOCKED/S1 attached to RID
> - Change of the RID domain while PASIDs are attached
>
> - Streamlined SVA support using the core infrastructure
>
> - Hitless, whenever possible, change between two domains
>
> Making the CD programming work like the new STE programming allows
> untangling some of the confusing SVA flows. From there the focus is on
> building out the core infrastructure for dealing with PASID and CD
> entries, then keeping track of unique SSID's for ATS invalidation.
>
> The ATS ordering is generalized so that the PASID flow can use it and put
> into a form where it is fully hitless, whenever possible. Care is taken to
> ensure that ATC flushes are present after any change in translation.
>
> Finally we simply kill the entire outdated SVA mmu_notifier implementation
> in one shot and switch it over to the newly created generic PASID & CD
> code. This avoids the messy and confusing approach of trying to
> incrementally untangle this in place. The new code is small and simple
> enough this is much better than trying to figure out smaller steps.
>
> Once SVA is resting on the right CD code it is straightforward to make the
> PASID interface functionally complete.
>
> It achieves the same goals as the several series from Michael and the S1DSS
> series from Nicolin that were trying to improve portions of the API.
>
> This is on github:
> https://github.com/jgunthorpe/linux/commits/smmuv3_newapi
Testing on qemu[1], with the same VMM Shameer tested with[2]:
qemu/build/qemu-system-aarch64 -M virt -machine virt,gic-version=3,iommu=nested-smmuv3,iommufd=iommufd0 \
-cpu cortex-a53,pmu=off -smp 1 -m 2048 \
-kernel Image \
-drive file=rootfs.ext4,if=virtio,format=raw \
-object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0 -nographic \
-append 'console=ttyAMA0 rootwait root=/dev/vda' \
-device virtio-scsi-pci,id=scsi0 \
-device ioh3420,id=pcie.1,chassis=1 \
-object iommufd,id=iommufd0 \
-device vfio-pci,host=0000:00:03.0,iommufd=iommufd0
I see the following panic:
[ 155.141233] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[ 155.142416] Mem abort info:
[ 155.142722] ESR = 0x0000000086000004
[ 155.143106] EC = 0x21: IABT (current EL), IL = 32 bits
[ 155.143827] SET = 0, FnV = 0
[ 155.144266] EA = 0, S1PTW = 0
[ 155.144721] FSC = 0x04: level 0 translation fault
[ 155.145432] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000101059000
[ 155.146234] [0000000000000000] pgd=0000000000000000, p4d=0000000000000000
[ 155.148162] Internal error: Oops: 0000000086000004 [#1] PREEMPT SMP
[ 155.149399] Modules linked in:
[ 155.150366] CPU: 2 PID: 371 Comm: qemu-system-aar Not tainted 6.8.0-rc7-gde77230ac23a #9
[ 155.151728] Hardware name: linux,dummy-virt (DT)
[ 155.152770] pstate: 81400809 (Nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=-c)
[ 155.153895] pc : 0x0
[ 155.154889] lr : iommufd_hwpt_invalidate+0xa4/0x204
[ 155.156272] sp : ffff800080f3bcc0
[ 155.156971] x29: ffff800080f3bcf0 x28: ffff0000c369b300 x27: 0000000000000000
[ 155.158135] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[ 155.159175] x23: 0000000000000000 x22: 00000000c1e334a0 x21: ffff0000c1e334a0
[ 155.160343] x20: ffff800080f3bd38 x19: ffff800080f3bd58 x18: 0000000000000000
[ 155.161298] x17: 0000000000000000 x16: 0000000000000000 x15: 0000ffff8240d6d8
[ 155.162355] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 155.163463] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[ 155.164947] x8 : 0000001000000002 x7 : 0000fffeac1ec950 x6 : 0000000000000000
[ 155.166057] x5 : ffff800080f3bd78 x4 : 0000000000000003 x3 : 0000000000000002
[ 155.167343] x2 : 0000000000000000 x1 : ffff800080f3bcc8 x0 : ffff0000c6034d80
[ 155.168851] Call trace:
[ 155.169738] 0x0
[ 155.170623] iommufd_fops_ioctl+0x154/0x274
[ 155.171555] __arm64_sys_ioctl+0xac/0xf0
[ 155.172095] invoke_syscall+0x48/0x110
[ 155.172633] el0_svc_common.constprop.0+0x40/0xe0
[ 155.173277] do_el0_svc+0x1c/0x28
[ 155.173847] el0_svc+0x34/0xb4
[ 155.174312] el0t_64_sync_handler+0x120/0x12c
[ 155.174969] el0t_64_sync+0x190/0x194
[ 155.176006] Code: ???????? ???????? ???????? ???????? (????????)
[ 155.178349] ---[ end trace 0000000000000000 ]---
The core IOMMUFD code calls domain->ops->cache_invalidate_user
unconditionally from IOCTL:IOMMU_HWPT_INVALIDATE and the SMMUv3 driver
doesn't implement it, that seems missing as otherwise the VMM can't
invalidate S1 mappings, or I a missing something?
[1] https://lore.kernel.org/all/20240325101442.1306300-1-smostafa@google.com/
[2] https://github.com/nicolinc/qemu/commits/wip/iommufd_vsmmu-02292024/
>
> v5:
> - Rebase on v6.8-rc7 & Will's tree
> - Accomdate the SVA rc patch removing the master list iteration
> - Move the kfree(to_smmu_domain(domain)) hunk to the right patch
> - Move S1DSS get_used hunk to "Allow IDENTITY/BLOCKED to be set while
> PASID is used"
> v4: https://lore.kernel.org/r/0-v4-e7091cdd9e8d+43b1-smmuv3_newapi_p2_jgg@nvidia.com
> - Rebase on v6.8-rc1, adjust to use mm_get_enqcmd_pasid() and eventually
> remove all references from ARM. Move the new ARM_SMMU_FEAT_STALL_FORCE
> stuff to arm_smmu_make_sva_cd()
> - Adjust to use the new shared STE/CD writer logic. Disable some of the
> sanity checks for the interior of the series
> - Return ERR_PTR from domain_alloc functions
> - Move the ATS disablement flow into arm_smmu_attach_prepare()/commit()
> which lets all the STE update flows use the same sequence. This is
> needed for nesting in part 3
> - Put ssid in attach_state
> - Replace to_smmu_domain_safe() with to_smmu_domain_devices()
> v3: https://lore.kernel.org/r/0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com
> - Rebase on the latest part 1
> - update comments and commit messages
> - Fix error exit in arm_smmu_set_pasid()
> - Fix inverted logic for btm_invalidation
> - Add missing ATC invalidation on mm release
> - Add a big comment explaining that BTM is not enabled and what is
> missing to enable it.
> v2: https://lore.kernel.org/r/0-v2-16665a652079+5947-smmuv3_newapi_p2_jgg@nvidia.com
> - Rebased on iommmufd + Joerg's tree
> - Use sid_smmu_domain consistently to refer to the domain attached to the
> device (eg the PCIe RID)
> - Rework how arm_smmu_attach_*() and callers flow to be more careful
> about ordering around ATC invalidation. The ATC must be invalidated
> after it is impossible to establish stale entires.
> - ATS disable is now entirely part of arm_smmu_attach_dev_ste(), which is
> the only STE type that ever disables ATS.
> - Remove the 'existing_master_domain' optimization, the code is
> functionally fine without it.
> - Whitespace, spelling, and checkpatch related items
> - Fixed wrong value stored in the xa for the BTM flows
> - Use pasid more consistently instead of id
> v1: https://lore.kernel.org/r/0-v1-afbb86647bbd+5-smmuv3_newapi_p2_jgg@nvidia.com
>
> Jason Gunthorpe (27):
> iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong
> PASID
> iommu/arm-smmu-v3: Do not ATC invalidate the entire domain
> iommu/arm-smmu-v3: Add a type for the CD entry
> iommu/arm-smmu-v3: Add an ops indirection to the STE code
> iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
> iommu/arm-smmu-v3: Consolidate clearing a CD table entry
> iommu/arm-smmu-v3: Move the CD generation for S1 domains into a
> function
> iommu/arm-smmu-v3: Move allocation of the cdtable into
> arm_smmu_get_cd_ptr()
> iommu/arm-smmu-v3: Allocate the CD table entry in advance
> iommu/arm-smmu-v3: Move the CD generation for SVA into a function
> iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd()
> iommu/arm-smmu-v3: Start building a generic PASID layer
> iommu/arm-smmu-v3: Make smmu_domain->devices into an allocated list
> iommu/arm-smmu-v3: Make changing domains be hitless for ATS
> iommu/arm-smmu-v3: Add ssid to struct arm_smmu_master_domain
> iommu/arm-smmu-v3: Keep track of valid CD entries in the cd_table
> iommu/arm-smmu-v3: Thread SSID through the arm_smmu_attach_*()
> interface
> iommu/arm-smmu-v3: Make SVA allocate a normal arm_smmu_domain
> iommu/arm-smmu-v3: Keep track of arm_smmu_master_domain for SVA
> iommu: Add ops->domain_alloc_sva()
> iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain
> iommu/arm-smmu-v3: Consolidate freeing the ASID/VMID
> iommu/arm-smmu-v3: Move the arm_smmu_asid_xa to per-smmu like vmid
> iommu/arm-smmu-v3: Bring back SVA BTM support
> iommu/arm-smmu-v3: Allow IDENTITY/BLOCKED to be set while PASID is
> used
> iommu/arm-smmu-v3: Allow a PASID to be set when RID is
> IDENTITY/BLOCKED
> iommu/arm-smmu-v3: Allow setting a S1 domain to a PASID
>
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 639 +++++-----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 1036 +++++++++++------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 79 +-
> drivers/iommu/iommu-sva.c | 4 +-
> drivers/iommu/iommu.c | 12 +-
> include/linux/iommu.h | 3 +
> 6 files changed, 1024 insertions(+), 749 deletions(-)
>
>
> base-commit: 98b23ebb0c84657a135957d727eedebd1280cbbf
> --
> 2.43.2
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* RE: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
2024-03-25 10:22 ` Mostafa Saleh
@ 2024-03-25 10:44 ` Shameerali Kolothum Thodi
-1 siblings, 0 replies; 232+ messages in thread
From: Shameerali Kolothum Thodi @ 2024-03-25 10:44 UTC (permalink / raw)
To: Mostafa Saleh, Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches
> -----Original Message-----
> From: Mostafa Saleh <smostafa@google.com>
> Sent: Monday, March 25, 2024 10:22 AM
> To: Jason Gunthorpe <jgg@nvidia.com>
> Cc: iommu@lists.linux.dev; Joerg Roedel <joro@8bytes.org>; linux-arm-
> kernel@lists.infradead.org; Robin Murphy <robin.murphy@arm.com>; Will
> Deacon <will@kernel.org>; Eric Auger <eric.auger@redhat.com>; Jean-
> Philippe Brucker <jean-philippe@linaro.org>; Moritz Fischer
> <mdf@kernel.org>; Michael Shavit <mshavit@google.com>; Nicolin Chen
> <nicolinc@nvidia.com>; patches@lists.linux.dev; Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>
> Subject: Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API
> (part 2/3)
>
> Hi Jason,
>
> On Mon, Mar 04, 2024 at 07:43:48PM -0400, Jason Gunthorpe wrote:
> > Continuing the work of part 1 this focuses on the CD, PASID and SVA
> > components:
> >
> > - attach_dev failure does not change the HW configuration.
> >
> > - Full PASID API support including:
> > - S1/SVA domains attached to PASIDs
> > - IDENTITY/BLOCKED/S1 attached to RID
> > - Change of the RID domain while PASIDs are attached
> >
> > - Streamlined SVA support using the core infrastructure
> >
> > - Hitless, whenever possible, change between two domains
> >
> > Making the CD programming work like the new STE programming allows
> > untangling some of the confusing SVA flows. From there the focus is on
> > building out the core infrastructure for dealing with PASID and CD
> > entries, then keeping track of unique SSID's for ATS invalidation.
> >
> > The ATS ordering is generalized so that the PASID flow can use it and put
> > into a form where it is fully hitless, whenever possible. Care is taken to
> > ensure that ATC flushes are present after any change in translation.
> >
> > Finally we simply kill the entire outdated SVA mmu_notifier
> implementation
> > in one shot and switch it over to the newly created generic PASID & CD
> > code. This avoids the messy and confusing approach of trying to
> > incrementally untangle this in place. The new code is small and simple
> > enough this is much better than trying to figure out smaller steps.
> >
> > Once SVA is resting on the right CD code it is straightforward to make the
> > PASID interface functionally complete.
> >
> > It achieves the same goals as the several series from Michael and the S1DSS
> > series from Nicolin that were trying to improve portions of the API.
> >
> > This is on github:
> > https://github.com/jgunthorpe/linux/commits/smmuv3_newapi
>
> Testing on qemu[1], with the same VMM Shameer tested with[2]:
> qemu/build/qemu-system-aarch64 -M virt -machine virt,gic-
> version=3,iommu=nested-smmuv3,iommufd=iommufd0 \
> -cpu cortex-a53,pmu=off -smp 1 -m 2048 \
> -kernel Image \
> -drive file=rootfs.ext4,if=virtio,format=raw \
> -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-
> pci,rng=rng0 -nographic \
> -append 'console=ttyAMA0 rootwait root=/dev/vda' \
> -device virtio-scsi-pci,id=scsi0 \
> -device ioh3420,id=pcie.1,chassis=1 \
> -object iommufd,id=iommufd0 \
> -device vfio-pci,host=0000:00:03.0,iommufd=iommufd0
>
> I see the following panic:
I think that is probably because you are testing with "nested-smmuv3". This
series not yet fully enable that. For that, I think you are missing few patches
from Nicolin's iommufd branch,
https://github.com/nicolinc/iommufd/commits/wip/iommufd_nesting-03112024/
Thanks,
Shameer
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* RE: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
@ 2024-03-25 10:44 ` Shameerali Kolothum Thodi
0 siblings, 0 replies; 232+ messages in thread
From: Shameerali Kolothum Thodi @ 2024-03-25 10:44 UTC (permalink / raw)
To: Mostafa Saleh, Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches
> -----Original Message-----
> From: Mostafa Saleh <smostafa@google.com>
> Sent: Monday, March 25, 2024 10:22 AM
> To: Jason Gunthorpe <jgg@nvidia.com>
> Cc: iommu@lists.linux.dev; Joerg Roedel <joro@8bytes.org>; linux-arm-
> kernel@lists.infradead.org; Robin Murphy <robin.murphy@arm.com>; Will
> Deacon <will@kernel.org>; Eric Auger <eric.auger@redhat.com>; Jean-
> Philippe Brucker <jean-philippe@linaro.org>; Moritz Fischer
> <mdf@kernel.org>; Michael Shavit <mshavit@google.com>; Nicolin Chen
> <nicolinc@nvidia.com>; patches@lists.linux.dev; Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>
> Subject: Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API
> (part 2/3)
>
> Hi Jason,
>
> On Mon, Mar 04, 2024 at 07:43:48PM -0400, Jason Gunthorpe wrote:
> > Continuing the work of part 1 this focuses on the CD, PASID and SVA
> > components:
> >
> > - attach_dev failure does not change the HW configuration.
> >
> > - Full PASID API support including:
> > - S1/SVA domains attached to PASIDs
> > - IDENTITY/BLOCKED/S1 attached to RID
> > - Change of the RID domain while PASIDs are attached
> >
> > - Streamlined SVA support using the core infrastructure
> >
> > - Hitless, whenever possible, change between two domains
> >
> > Making the CD programming work like the new STE programming allows
> > untangling some of the confusing SVA flows. From there the focus is on
> > building out the core infrastructure for dealing with PASID and CD
> > entries, then keeping track of unique SSID's for ATS invalidation.
> >
> > The ATS ordering is generalized so that the PASID flow can use it and put
> > into a form where it is fully hitless, whenever possible. Care is taken to
> > ensure that ATC flushes are present after any change in translation.
> >
> > Finally we simply kill the entire outdated SVA mmu_notifier
> implementation
> > in one shot and switch it over to the newly created generic PASID & CD
> > code. This avoids the messy and confusing approach of trying to
> > incrementally untangle this in place. The new code is small and simple
> > enough this is much better than trying to figure out smaller steps.
> >
> > Once SVA is resting on the right CD code it is straightforward to make the
> > PASID interface functionally complete.
> >
> > It achieves the same goals as the several series from Michael and the S1DSS
> > series from Nicolin that were trying to improve portions of the API.
> >
> > This is on github:
> > https://github.com/jgunthorpe/linux/commits/smmuv3_newapi
>
> Testing on qemu[1], with the same VMM Shameer tested with[2]:
> qemu/build/qemu-system-aarch64 -M virt -machine virt,gic-
> version=3,iommu=nested-smmuv3,iommufd=iommufd0 \
> -cpu cortex-a53,pmu=off -smp 1 -m 2048 \
> -kernel Image \
> -drive file=rootfs.ext4,if=virtio,format=raw \
> -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-
> pci,rng=rng0 -nographic \
> -append 'console=ttyAMA0 rootwait root=/dev/vda' \
> -device virtio-scsi-pci,id=scsi0 \
> -device ioh3420,id=pcie.1,chassis=1 \
> -object iommufd,id=iommufd0 \
> -device vfio-pci,host=0000:00:03.0,iommufd=iommufd0
>
> I see the following panic:
I think that is probably because you are testing with "nested-smmuv3". This
series not yet fully enable that. For that, I think you are missing few patches
from Nicolin's iommufd branch,
https://github.com/nicolinc/iommufd/commits/wip/iommufd_nesting-03112024/
Thanks,
Shameer
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
2024-03-25 10:44 ` Shameerali Kolothum Thodi
@ 2024-03-25 11:22 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-25 11:22 UTC (permalink / raw)
To: Shameerali Kolothum Thodi
Cc: Jason Gunthorpe, iommu, Joerg Roedel, linux-arm-kernel,
Robin Murphy, Will Deacon, Eric Auger, Jean-Philippe Brucker,
Moritz Fischer, Michael Shavit, Nicolin Chen, patches
Hi Shameer,
On Mon, Mar 25, 2024 at 10:44 AM Shameerali Kolothum Thodi
<shameerali.kolothum.thodi@huawei.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Mostafa Saleh <smostafa@google.com>
> > Sent: Monday, March 25, 2024 10:22 AM
> > To: Jason Gunthorpe <jgg@nvidia.com>
> > Cc: iommu@lists.linux.dev; Joerg Roedel <joro@8bytes.org>; linux-arm-
> > kernel@lists.infradead.org; Robin Murphy <robin.murphy@arm.com>; Will
> > Deacon <will@kernel.org>; Eric Auger <eric.auger@redhat.com>; Jean-
> > Philippe Brucker <jean-philippe@linaro.org>; Moritz Fischer
> > <mdf@kernel.org>; Michael Shavit <mshavit@google.com>; Nicolin Chen
> > <nicolinc@nvidia.com>; patches@lists.linux.dev; Shameerali Kolothum Thodi
> > <shameerali.kolothum.thodi@huawei.com>
> > Subject: Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API
> > (part 2/3)
> >
> > Hi Jason,
> >
> > On Mon, Mar 04, 2024 at 07:43:48PM -0400, Jason Gunthorpe wrote:
> > > Continuing the work of part 1 this focuses on the CD, PASID and SVA
> > > components:
> > >
> > > - attach_dev failure does not change the HW configuration.
> > >
> > > - Full PASID API support including:
> > > - S1/SVA domains attached to PASIDs
> > > - IDENTITY/BLOCKED/S1 attached to RID
> > > - Change of the RID domain while PASIDs are attached
> > >
> > > - Streamlined SVA support using the core infrastructure
> > >
> > > - Hitless, whenever possible, change between two domains
> > >
> > > Making the CD programming work like the new STE programming allows
> > > untangling some of the confusing SVA flows. From there the focus is on
> > > building out the core infrastructure for dealing with PASID and CD
> > > entries, then keeping track of unique SSID's for ATS invalidation.
> > >
> > > The ATS ordering is generalized so that the PASID flow can use it and put
> > > into a form where it is fully hitless, whenever possible. Care is taken to
> > > ensure that ATC flushes are present after any change in translation.
> > >
> > > Finally we simply kill the entire outdated SVA mmu_notifier
> > implementation
> > > in one shot and switch it over to the newly created generic PASID & CD
> > > code. This avoids the messy and confusing approach of trying to
> > > incrementally untangle this in place. The new code is small and simple
> > > enough this is much better than trying to figure out smaller steps.
> > >
> > > Once SVA is resting on the right CD code it is straightforward to make the
> > > PASID interface functionally complete.
> > >
> > > It achieves the same goals as the several series from Michael and the S1DSS
> > > series from Nicolin that were trying to improve portions of the API.
> > >
> > > This is on github:
> > > https://github.com/jgunthorpe/linux/commits/smmuv3_newapi
> >
> > Testing on qemu[1], with the same VMM Shameer tested with[2]:
> > qemu/build/qemu-system-aarch64 -M virt -machine virt,gic-
> > version=3,iommu=nested-smmuv3,iommufd=iommufd0 \
> > -cpu cortex-a53,pmu=off -smp 1 -m 2048 \
> > -kernel Image \
> > -drive file=rootfs.ext4,if=virtio,format=raw \
> > -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-
> > pci,rng=rng0 -nographic \
> > -append 'console=ttyAMA0 rootwait root=/dev/vda' \
> > -device virtio-scsi-pci,id=scsi0 \
> > -device ioh3420,id=pcie.1,chassis=1 \
> > -object iommufd,id=iommufd0 \
> > -device vfio-pci,host=0000:00:03.0,iommufd=iommufd0
> >
> > I see the following panic:
>
> I think that is probably because you are testing with "nested-smmuv3". This
> series not yet fully enable that. For that, I think you are missing few patches
> from Nicolin's iommufd branch,
> https://github.com/nicolinc/iommufd/commits/wip/iommufd_nesting-03112024/
I see, thanks for clarifying. I think we shouldn't still crash the
kernel, but that's a problem for part 3.
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
@ 2024-03-25 11:22 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-25 11:22 UTC (permalink / raw)
To: Shameerali Kolothum Thodi
Cc: Jason Gunthorpe, iommu, Joerg Roedel, linux-arm-kernel,
Robin Murphy, Will Deacon, Eric Auger, Jean-Philippe Brucker,
Moritz Fischer, Michael Shavit, Nicolin Chen, patches
Hi Shameer,
On Mon, Mar 25, 2024 at 10:44 AM Shameerali Kolothum Thodi
<shameerali.kolothum.thodi@huawei.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Mostafa Saleh <smostafa@google.com>
> > Sent: Monday, March 25, 2024 10:22 AM
> > To: Jason Gunthorpe <jgg@nvidia.com>
> > Cc: iommu@lists.linux.dev; Joerg Roedel <joro@8bytes.org>; linux-arm-
> > kernel@lists.infradead.org; Robin Murphy <robin.murphy@arm.com>; Will
> > Deacon <will@kernel.org>; Eric Auger <eric.auger@redhat.com>; Jean-
> > Philippe Brucker <jean-philippe@linaro.org>; Moritz Fischer
> > <mdf@kernel.org>; Michael Shavit <mshavit@google.com>; Nicolin Chen
> > <nicolinc@nvidia.com>; patches@lists.linux.dev; Shameerali Kolothum Thodi
> > <shameerali.kolothum.thodi@huawei.com>
> > Subject: Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API
> > (part 2/3)
> >
> > Hi Jason,
> >
> > On Mon, Mar 04, 2024 at 07:43:48PM -0400, Jason Gunthorpe wrote:
> > > Continuing the work of part 1 this focuses on the CD, PASID and SVA
> > > components:
> > >
> > > - attach_dev failure does not change the HW configuration.
> > >
> > > - Full PASID API support including:
> > > - S1/SVA domains attached to PASIDs
> > > - IDENTITY/BLOCKED/S1 attached to RID
> > > - Change of the RID domain while PASIDs are attached
> > >
> > > - Streamlined SVA support using the core infrastructure
> > >
> > > - Hitless, whenever possible, change between two domains
> > >
> > > Making the CD programming work like the new STE programming allows
> > > untangling some of the confusing SVA flows. From there the focus is on
> > > building out the core infrastructure for dealing with PASID and CD
> > > entries, then keeping track of unique SSID's for ATS invalidation.
> > >
> > > The ATS ordering is generalized so that the PASID flow can use it and put
> > > into a form where it is fully hitless, whenever possible. Care is taken to
> > > ensure that ATC flushes are present after any change in translation.
> > >
> > > Finally we simply kill the entire outdated SVA mmu_notifier
> > implementation
> > > in one shot and switch it over to the newly created generic PASID & CD
> > > code. This avoids the messy and confusing approach of trying to
> > > incrementally untangle this in place. The new code is small and simple
> > > enough this is much better than trying to figure out smaller steps.
> > >
> > > Once SVA is resting on the right CD code it is straightforward to make the
> > > PASID interface functionally complete.
> > >
> > > It achieves the same goals as the several series from Michael and the S1DSS
> > > series from Nicolin that were trying to improve portions of the API.
> > >
> > > This is on github:
> > > https://github.com/jgunthorpe/linux/commits/smmuv3_newapi
> >
> > Testing on qemu[1], with the same VMM Shameer tested with[2]:
> > qemu/build/qemu-system-aarch64 -M virt -machine virt,gic-
> > version=3,iommu=nested-smmuv3,iommufd=iommufd0 \
> > -cpu cortex-a53,pmu=off -smp 1 -m 2048 \
> > -kernel Image \
> > -drive file=rootfs.ext4,if=virtio,format=raw \
> > -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-
> > pci,rng=rng0 -nographic \
> > -append 'console=ttyAMA0 rootwait root=/dev/vda' \
> > -device virtio-scsi-pci,id=scsi0 \
> > -device ioh3420,id=pcie.1,chassis=1 \
> > -object iommufd,id=iommufd0 \
> > -device vfio-pci,host=0000:00:03.0,iommufd=iommufd0
> >
> > I see the following panic:
>
> I think that is probably because you are testing with "nested-smmuv3". This
> series not yet fully enable that. For that, I think you are missing few patches
> from Nicolin's iommufd branch,
> https://github.com/nicolinc/iommufd/commits/wip/iommufd_nesting-03112024/
I see, thanks for clarifying. I think we shouldn't still crash the
kernel, but that's a problem for part 3.
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code
2024-03-22 18:14 ` Mostafa Saleh
@ 2024-03-25 14:11 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-25 14:11 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Fri, Mar 22, 2024 at 06:14:24PM +0000, Mostafa Saleh wrote:
> > @@ -1027,57 +1038,55 @@ static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent,
> > * unused_update is an intermediate value of entry that has unused bits set to
> > * their new values.
> > */
> > -static u8 arm_smmu_entry_qword_diff(const struct arm_smmu_ste *entry,
> > - const struct arm_smmu_ste *target,
> > - struct arm_smmu_ste *unused_update)
> > +static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> > + const __le64 *entry, const __le64 *target,
> > + __le64 *unused_update)
> > {
> > - struct arm_smmu_ste target_used = {};
> > - struct arm_smmu_ste cur_used = {};
> > + __le64 target_used[NUM_ENTRY_QWORDS] = {};
> > + __le64 cur_used[NUM_ENTRY_QWORDS] = {};
> This is confusing to me, the function was modified to be generic, so its has
> args are __le64 * instead of struct arm_smmu_ste *.
Right
> But NUM_ENTRY_QWORDS is defined as “(sizeof(struct arm_smmu_ste) / sizeof(u64))”
> and in the same function writer->ops->num_entry_qwords is used
> nterchangeably,
Right
> I understand that this not a constant and the compiler would complain.
> But since for any other num_entry_qwords larger than NUM_ENTRY_QWORDS it fails,
> and we know STEs and CDs both have the same size, we simplify the code and make
> it a constant everywhere.
So you say to get rid of num_entry_qwords and just use the constant?
> I see in the next patch, that this is redefined to be the max between STE and
> CD, but again, this hardware and it never changes, so my opinion is to simplify
> the code, as there is no need to generalize this part.
Yes, we need a constant.
It would look like this, it is a little bit simpler:
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index a54062faccde38..d015f41900d802 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -63,9 +63,9 @@ enum arm_smmu_msi_index {
ARM_SMMU_MAX_MSIS,
};
-#define NUM_ENTRY_QWORDS \
- (max(sizeof(struct arm_smmu_ste), sizeof(struct arm_smmu_cd)) / \
- sizeof(u64))
+#define NUM_ENTRY_QWORDS 8
+static_assert(sizeof(struct arm_smmu_ste) == NUM_ENTRY_QWORDS * sizeof(u64));
+static_assert(sizeof(struct arm_smmu_cd) == NUM_ENTRY_QWORDS * sizeof(u64));
static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
[EVTQ_MSI_INDEX] = {
@@ -1045,7 +1045,7 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
writer->ops->get_used(entry, cur_used);
writer->ops->get_used(target, target_used);
- for (i = 0; i != writer->ops->num_entry_qwords; i++) {
+ for (i = 0; i != NUM_ENTRY_QWORDS; i++) {
/*
* Check that masks are up to date, the make functions are not
* allowed to set a bit to 1 if the used function doesn't say it
@@ -1114,7 +1114,6 @@ static bool entry_set(struct arm_smmu_entry_writer *writer, __le64 *entry,
void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry,
const __le64 *target)
{
- unsigned int num_entry_qwords = writer->ops->num_entry_qwords;
__le64 unused_update[NUM_ENTRY_QWORDS];
u8 used_qword_diff;
@@ -1137,9 +1136,9 @@ void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry,
*/
unused_update[critical_qword_index] =
entry[critical_qword_index];
- entry_set(writer, entry, unused_update, 0, num_entry_qwords);
+ entry_set(writer, entry, unused_update, 0, NUM_ENTRY_QWORDS);
entry_set(writer, entry, target, critical_qword_index, 1);
- entry_set(writer, entry, target, 0, num_entry_qwords);
+ entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS);
} else if (used_qword_diff) {
/*
* At least two qwords need their inuse bits to be changed. This
@@ -1148,7 +1147,7 @@ void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry,
*/
unused_update[0] = entry[0] & (~writer->ops->v_bit);
entry_set(writer, entry, unused_update, 0, 1);
- entry_set(writer, entry, target, 1, num_entry_qwords - 1);
+ entry_set(writer, entry, target, 1, NUM_ENTRY_QWORDS - 1);
entry_set(writer, entry, target, 0, 1);
} else {
/*
@@ -1157,7 +1156,7 @@ void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry,
* compute_qword_diff().
*/
WARN_ON_ONCE(
- entry_set(writer, entry, target, 0, num_entry_qwords));
+ entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS));
}
}
@@ -1272,7 +1271,6 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
.sync = arm_smmu_cd_writer_sync_entry,
.get_used = arm_smmu_get_cd_used,
.v_bit = cpu_to_le64(CTXDESC_CD_0_V),
- .num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
};
void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
@@ -1460,7 +1458,6 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_ste_writer_ops = {
.sync = arm_smmu_ste_writer_sync_entry,
.get_used = arm_smmu_get_ste_used,
.v_bit = cpu_to_le64(STRTAB_STE_0_V),
- .num_entry_qwords = sizeof(struct arm_smmu_ste) / sizeof(u64),
};
static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 8ba07b00bf6056..5936dc5f76786a 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -779,7 +779,6 @@ struct arm_smmu_entry_writer {
};
struct arm_smmu_entry_writer_ops {
- unsigned int num_entry_qwords;
__le64 v_bit;
void (*get_used)(const __le64 *entry, __le64 *used);
void (*sync)(struct arm_smmu_entry_writer *writer);
^ permalink raw reply related [flat|nested] 232+ messages in thread
* Re: [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code
@ 2024-03-25 14:11 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-25 14:11 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Fri, Mar 22, 2024 at 06:14:24PM +0000, Mostafa Saleh wrote:
> > @@ -1027,57 +1038,55 @@ static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent,
> > * unused_update is an intermediate value of entry that has unused bits set to
> > * their new values.
> > */
> > -static u8 arm_smmu_entry_qword_diff(const struct arm_smmu_ste *entry,
> > - const struct arm_smmu_ste *target,
> > - struct arm_smmu_ste *unused_update)
> > +static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> > + const __le64 *entry, const __le64 *target,
> > + __le64 *unused_update)
> > {
> > - struct arm_smmu_ste target_used = {};
> > - struct arm_smmu_ste cur_used = {};
> > + __le64 target_used[NUM_ENTRY_QWORDS] = {};
> > + __le64 cur_used[NUM_ENTRY_QWORDS] = {};
> This is confusing to me, the function was modified to be generic, so its has
> args are __le64 * instead of struct arm_smmu_ste *.
Right
> But NUM_ENTRY_QWORDS is defined as “(sizeof(struct arm_smmu_ste) / sizeof(u64))”
> and in the same function writer->ops->num_entry_qwords is used
> nterchangeably,
Right
> I understand that this not a constant and the compiler would complain.
> But since for any other num_entry_qwords larger than NUM_ENTRY_QWORDS it fails,
> and we know STEs and CDs both have the same size, we simplify the code and make
> it a constant everywhere.
So you say to get rid of num_entry_qwords and just use the constant?
> I see in the next patch, that this is redefined to be the max between STE and
> CD, but again, this hardware and it never changes, so my opinion is to simplify
> the code, as there is no need to generalize this part.
Yes, we need a constant.
It would look like this, it is a little bit simpler:
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index a54062faccde38..d015f41900d802 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -63,9 +63,9 @@ enum arm_smmu_msi_index {
ARM_SMMU_MAX_MSIS,
};
-#define NUM_ENTRY_QWORDS \
- (max(sizeof(struct arm_smmu_ste), sizeof(struct arm_smmu_cd)) / \
- sizeof(u64))
+#define NUM_ENTRY_QWORDS 8
+static_assert(sizeof(struct arm_smmu_ste) == NUM_ENTRY_QWORDS * sizeof(u64));
+static_assert(sizeof(struct arm_smmu_cd) == NUM_ENTRY_QWORDS * sizeof(u64));
static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
[EVTQ_MSI_INDEX] = {
@@ -1045,7 +1045,7 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
writer->ops->get_used(entry, cur_used);
writer->ops->get_used(target, target_used);
- for (i = 0; i != writer->ops->num_entry_qwords; i++) {
+ for (i = 0; i != NUM_ENTRY_QWORDS; i++) {
/*
* Check that masks are up to date, the make functions are not
* allowed to set a bit to 1 if the used function doesn't say it
@@ -1114,7 +1114,6 @@ static bool entry_set(struct arm_smmu_entry_writer *writer, __le64 *entry,
void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry,
const __le64 *target)
{
- unsigned int num_entry_qwords = writer->ops->num_entry_qwords;
__le64 unused_update[NUM_ENTRY_QWORDS];
u8 used_qword_diff;
@@ -1137,9 +1136,9 @@ void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry,
*/
unused_update[critical_qword_index] =
entry[critical_qword_index];
- entry_set(writer, entry, unused_update, 0, num_entry_qwords);
+ entry_set(writer, entry, unused_update, 0, NUM_ENTRY_QWORDS);
entry_set(writer, entry, target, critical_qword_index, 1);
- entry_set(writer, entry, target, 0, num_entry_qwords);
+ entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS);
} else if (used_qword_diff) {
/*
* At least two qwords need their inuse bits to be changed. This
@@ -1148,7 +1147,7 @@ void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry,
*/
unused_update[0] = entry[0] & (~writer->ops->v_bit);
entry_set(writer, entry, unused_update, 0, 1);
- entry_set(writer, entry, target, 1, num_entry_qwords - 1);
+ entry_set(writer, entry, target, 1, NUM_ENTRY_QWORDS - 1);
entry_set(writer, entry, target, 0, 1);
} else {
/*
@@ -1157,7 +1156,7 @@ void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry,
* compute_qword_diff().
*/
WARN_ON_ONCE(
- entry_set(writer, entry, target, 0, num_entry_qwords));
+ entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS));
}
}
@@ -1272,7 +1271,6 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
.sync = arm_smmu_cd_writer_sync_entry,
.get_used = arm_smmu_get_cd_used,
.v_bit = cpu_to_le64(CTXDESC_CD_0_V),
- .num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
};
void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
@@ -1460,7 +1458,6 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_ste_writer_ops = {
.sync = arm_smmu_ste_writer_sync_entry,
.get_used = arm_smmu_get_ste_used,
.v_bit = cpu_to_le64(STRTAB_STE_0_V),
- .num_entry_qwords = sizeof(struct arm_smmu_ste) / sizeof(u64),
};
static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 8ba07b00bf6056..5936dc5f76786a 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -779,7 +779,6 @@ struct arm_smmu_entry_writer {
};
struct arm_smmu_entry_writer_ops {
- unsigned int num_entry_qwords;
__le64 v_bit;
void (*get_used)(const __le64 *entry, __le64 *used);
void (*sync)(struct arm_smmu_entry_writer *writer);
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 232+ messages in thread
* Re: [PATCH v5 06/27] iommu/arm-smmu-v3: Consolidate clearing a CD table entry
2024-03-22 18:36 ` Mostafa Saleh
@ 2024-03-25 14:14 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-25 14:14 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Fri, Mar 22, 2024 at 06:36:17PM +0000, Mostafa Saleh wrote:
> > +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> > +{
> > + struct arm_smmu_cd target = {};
> > + struct arm_smmu_cd *cdptr;
> > +
> > + if (!master->cd_table.cdtab)
> > + return;
> > + cdptr = arm_smmu_get_cd_ptr(master, ssid);
> > + if (WARN_ON(!cdptr))
> > + return;
>
> I don’t understand the SVA code enough, but AFAICT, arm_smmu_sva_set_dev_pasid
> can allocate the L2 CD table through arm_smmu_write_ctx_desc.
Yes
> And if it failed
> before allocating the CD table, then remove_dev_pasid would be
> called,
If it fails the core code should not call remove_dev_pasid() on a
domain that was never attached. There is an obscure error unwinding
issue in the core code Yi is looking to fix regarding multi-device
PASID groups, but it is not a driver issue..
> which warns here, the previous code would tolerate that, but that
> might regress on systems with panic_on_warn, so I am not sure if
> that is necessary.
Right, if it does hit it signals there is some error unwinding bug in
the core code that should be resolved.
> Otherwise, Reviewed-by: Mostafa Saleh <smostafa@google.com>
Thanks,
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 06/27] iommu/arm-smmu-v3: Consolidate clearing a CD table entry
@ 2024-03-25 14:14 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-25 14:14 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Fri, Mar 22, 2024 at 06:36:17PM +0000, Mostafa Saleh wrote:
> > +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> > +{
> > + struct arm_smmu_cd target = {};
> > + struct arm_smmu_cd *cdptr;
> > +
> > + if (!master->cd_table.cdtab)
> > + return;
> > + cdptr = arm_smmu_get_cd_ptr(master, ssid);
> > + if (WARN_ON(!cdptr))
> > + return;
>
> I don’t understand the SVA code enough, but AFAICT, arm_smmu_sva_set_dev_pasid
> can allocate the L2 CD table through arm_smmu_write_ctx_desc.
Yes
> And if it failed
> before allocating the CD table, then remove_dev_pasid would be
> called,
If it fails the core code should not call remove_dev_pasid() on a
domain that was never attached. There is an obscure error unwinding
issue in the core code Yi is looking to fix regarding multi-device
PASID groups, but it is not a driver issue..
> which warns here, the previous code would tolerate that, but that
> might regress on systems with panic_on_warn, so I am not sure if
> that is necessary.
Right, if it does hit it signals there is some error unwinding bug in
the core code that should be resolved.
> Otherwise, Reviewed-by: Mostafa Saleh <smostafa@google.com>
Thanks,
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 08/27] iommu/arm-smmu-v3: Move allocation of the cdtable into arm_smmu_get_cd_ptr()
2024-03-22 19:07 ` Mostafa Saleh
@ 2024-03-25 14:21 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-25 14:21 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Fri, Mar 22, 2024 at 07:07:10PM +0000, Mostafa Saleh wrote:
> Hi Jason,
>
> On Mon, Mar 04, 2024 at 07:43:56PM -0400, Jason Gunthorpe wrote:
> > No reason to force callers to do two steps. Make arm_smmu_get_cd_ptr()
> > able to return an entry in all cases except OOM
>
> I believe the current code is more clear, as it is explicit about which path
> is expected to allocate.
I think we had this allocate vs no allocate discussion before on
something else..
It would be good to make *full* allocate/noallocate variants of
get_cd_ptr() and the cases that must never allocate call the no
allocate variation. There are some issues with GFP_KERNEL/ATOMIC that
are a bit hard to understand as well.
This is a bigger issue than just the cd_table, as even the leafs
should not allocate.
> As there are many callers for arm_smmu_get_cd_ptr() directly and indirectly,
> and it read-modify-writes the cdtable, it would be a pain to debug not
> knowing which one could allocate, and this patch only abstracts one
> allocating call, so it is not much code less.
> For example, (again I don’t know much about SVA) I think there might be a
> race condition as follows:
> arm_smmu_attach_dev
> arm_smmu_domain_finalise() => set domain stage
> [....]
> arm_smmu_get_cd_ptr() => RMW master->cd_table
>
> arm_smmu_sva_set_dev_pasid
> __arm_smmu_sva_bind
> Check stage is valid
> [...]
> arm_smmu_write_ctx_desc
> arm_smmu_get_cd_ptr => RMW master->cd_table
>
> If this path is true though, I guess the in current code, we would need some
> barriers in arm_smmu_get_cd_ptr(), arm_smmu_get_cd_ptr()
Both of those functions are called under the group mutex held by the
core code.
The only valid condition to change the CD table backing memory is when
the group mutex is held. We now have iommu_group_mutex_assert() so an
alloc/noalloc split can call that test on the alloc side which is
motivation enough to do it, IMHO.
I will split the function and sort it all out, but I will still
integrate the cd_table allocation into the allocating version of
get_cd_ptr().
Thanks,
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 08/27] iommu/arm-smmu-v3: Move allocation of the cdtable into arm_smmu_get_cd_ptr()
@ 2024-03-25 14:21 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-25 14:21 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Fri, Mar 22, 2024 at 07:07:10PM +0000, Mostafa Saleh wrote:
> Hi Jason,
>
> On Mon, Mar 04, 2024 at 07:43:56PM -0400, Jason Gunthorpe wrote:
> > No reason to force callers to do two steps. Make arm_smmu_get_cd_ptr()
> > able to return an entry in all cases except OOM
>
> I believe the current code is more clear, as it is explicit about which path
> is expected to allocate.
I think we had this allocate vs no allocate discussion before on
something else..
It would be good to make *full* allocate/noallocate variants of
get_cd_ptr() and the cases that must never allocate call the no
allocate variation. There are some issues with GFP_KERNEL/ATOMIC that
are a bit hard to understand as well.
This is a bigger issue than just the cd_table, as even the leafs
should not allocate.
> As there are many callers for arm_smmu_get_cd_ptr() directly and indirectly,
> and it read-modify-writes the cdtable, it would be a pain to debug not
> knowing which one could allocate, and this patch only abstracts one
> allocating call, so it is not much code less.
> For example, (again I don’t know much about SVA) I think there might be a
> race condition as follows:
> arm_smmu_attach_dev
> arm_smmu_domain_finalise() => set domain stage
> [....]
> arm_smmu_get_cd_ptr() => RMW master->cd_table
>
> arm_smmu_sva_set_dev_pasid
> __arm_smmu_sva_bind
> Check stage is valid
> [...]
> arm_smmu_write_ctx_desc
> arm_smmu_get_cd_ptr => RMW master->cd_table
>
> If this path is true though, I guess the in current code, we would need some
> barriers in arm_smmu_get_cd_ptr(), arm_smmu_get_cd_ptr()
Both of those functions are called under the group mutex held by the
core code.
The only valid condition to change the CD table backing memory is when
the group mutex is held. We now have iommu_group_mutex_assert() so an
alloc/noalloc split can call that test on the alloc side which is
motivation enough to do it, IMHO.
I will split the function and sort it all out, but I will still
integrate the cd_table allocation into the allocating version of
get_cd_ptr().
Thanks,
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
2024-03-23 13:02 ` Mostafa Saleh
@ 2024-03-25 14:25 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-25 14:25 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Sat, Mar 23, 2024 at 01:02:15PM +0000, Mostafa Saleh wrote:
> Hi Jason,
>
> On Mon, Mar 04, 2024 at 07:43:53PM -0400, Jason Gunthorpe wrote:
> > CD table entries and STE's have the same essential programming sequence,
> > just with different types and sizes.
> >
> > Have arm_smmu_write_ctx_desc() generate a target CD and call
> > arm_smmu_write_entry() to do the programming. Due to the way the
> > target CD is generated by modifying the existing CD this alone is not
> > enough for the CD callers to be freed of the ordering requirements.
> >
> > The following patches will make the rest of the CD flow mirror the STE
> > flow with precise CD contents generated in all cases.
> >
> > Currently the logic can't ensure that the CD always conforms to the used
> > requirements until all the CD generation is moved to the new method. Add a
> > temporary no_used_check to disable the assertions.
> >
>
> I am still going through the patches, but is it possible to
> reorder/squash to avoid that, so it is easier to review?
After Nicolin's remark I changed this one use a temporary helper to 0
the unused bits then we delete the helper instead of touching the
machinery itself. It is much clearer.
We can't avoid this with patch ordering because the progression is to
move CD generation out of arm_smmu_write_ctx_desc() type-by-type then
delete it. In the mean time we need to use the new write logic in all
cases because I'm not sure the old/new schemes have compatible
assumptions for the existing arm_smmu_write_ctx_desc() to be safe.
Thanks,
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
@ 2024-03-25 14:25 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-25 14:25 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Sat, Mar 23, 2024 at 01:02:15PM +0000, Mostafa Saleh wrote:
> Hi Jason,
>
> On Mon, Mar 04, 2024 at 07:43:53PM -0400, Jason Gunthorpe wrote:
> > CD table entries and STE's have the same essential programming sequence,
> > just with different types and sizes.
> >
> > Have arm_smmu_write_ctx_desc() generate a target CD and call
> > arm_smmu_write_entry() to do the programming. Due to the way the
> > target CD is generated by modifying the existing CD this alone is not
> > enough for the CD callers to be freed of the ordering requirements.
> >
> > The following patches will make the rest of the CD flow mirror the STE
> > flow with precise CD contents generated in all cases.
> >
> > Currently the logic can't ensure that the CD always conforms to the used
> > requirements until all the CD generation is moved to the new method. Add a
> > temporary no_used_check to disable the assertions.
> >
>
> I am still going through the patches, but is it possible to
> reorder/squash to avoid that, so it is easier to review?
After Nicolin's remark I changed this one use a temporary helper to 0
the unused bits then we delete the helper instead of touching the
machinery itself. It is much clearer.
We can't avoid this with patch ordering because the progression is to
move CD generation out of arm_smmu_write_ctx_desc() type-by-type then
delete it. In the mean time we need to use the new write logic in all
cases because I'm not sure the old/new schemes have compatible
assumptions for the existing arm_smmu_write_ctx_desc() to be safe.
Thanks,
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 07/27] iommu/arm-smmu-v3: Move the CD generation for S1 domains into a function
2024-03-23 13:11 ` Mostafa Saleh
@ 2024-03-25 14:30 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-25 14:30 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Sat, Mar 23, 2024 at 01:11:49PM +0000, Mostafa Saleh wrote:
> On Mon, Mar 04, 2024 at 07:43:55PM -0400, Jason Gunthorpe wrote:
> > Introduce arm_smmu_make_s1_cd() to build the CD from the paging S1 domain,
> > and reorganize all the places programming S1 domain CD table entries to
> > call it.
> >
> > Split arm_smmu_update_s1_domain_cd_entry() from
> > arm_smmu_update_ctx_desc_devices() so that the S1 path has its own call
> > chain separate from the unrelated SVA path.
> >
> > arm_smmu_update_s1_domain_cd_entry() only works on S1 domains
> > attached to RIDs and refreshes all their CDs.
> >
> > Remove the forced clear of the CD during S1 domain attach,
> > arm_smmu_write_cd_entry() will do this automatically if necessary.
> >
> > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> > .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 25 +++++++-
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 60 +++++++++++++------
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 +++
> > 3 files changed, 75 insertions(+), 18 deletions(-)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > index bb9bb6fd7914ce..6acc65f6d00a71 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > @@ -54,6 +54,29 @@ static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain
> > spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> > }
> >
> > +static void
> > +arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> > +{
> > + struct arm_smmu_master *master;
> > + struct arm_smmu_cd target_cd;
> > + unsigned long flags;
> > +
> > + spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> > + list_for_each_entry(master, &smmu_domain->devices, domain_head) {
> > + struct arm_smmu_cd *cdptr;
> > +
> > + /* S1 domains only support RID attachment right now */
> > + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> > + if (WARN_ON(!cdptr))
>
> This should never hit, no? Otherwise that means this path can allocate memory
> with a spinlock.
Right, WARN_ON's should never be hit.
> > +void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> > + struct arm_smmu_master *master,
> > + struct arm_smmu_domain *smmu_domain)
> > +{
> > + struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
> > +
> > + memset(target, 0, sizeof(*target));
> > +
> > + target->data[0] = cpu_to_le64(
> > + cd->tcr |
> > +#ifdef __BIG_ENDIAN
> > + CTXDESC_CD_0_ENDI |
> > +#endif
> > + CTXDESC_CD_0_V |
> > + CTXDESC_CD_0_AA64 |
> > + (master->stall_enabled ? CTXDESC_CD_0_S : 0) |
> > + CTXDESC_CD_0_R |
> > + CTXDESC_CD_0_A |
> > + CTXDESC_CD_0_ASET |
> > + FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid)
> > + );
> > +
> > + target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> > + target->data[3] = cpu_to_le64(cd->mair);
> > +}
> > +
>
> That seems to duplicate some logic from arm_smmu_write_ctx_desc(),
> can that be consolidated?
Yes, it is consolidated by deleting arm_smmu_write_ctx_desc() in a few
more patches.
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 07/27] iommu/arm-smmu-v3: Move the CD generation for S1 domains into a function
@ 2024-03-25 14:30 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-25 14:30 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Sat, Mar 23, 2024 at 01:11:49PM +0000, Mostafa Saleh wrote:
> On Mon, Mar 04, 2024 at 07:43:55PM -0400, Jason Gunthorpe wrote:
> > Introduce arm_smmu_make_s1_cd() to build the CD from the paging S1 domain,
> > and reorganize all the places programming S1 domain CD table entries to
> > call it.
> >
> > Split arm_smmu_update_s1_domain_cd_entry() from
> > arm_smmu_update_ctx_desc_devices() so that the S1 path has its own call
> > chain separate from the unrelated SVA path.
> >
> > arm_smmu_update_s1_domain_cd_entry() only works on S1 domains
> > attached to RIDs and refreshes all their CDs.
> >
> > Remove the forced clear of the CD during S1 domain attach,
> > arm_smmu_write_cd_entry() will do this automatically if necessary.
> >
> > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> > .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 25 +++++++-
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 60 +++++++++++++------
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 +++
> > 3 files changed, 75 insertions(+), 18 deletions(-)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > index bb9bb6fd7914ce..6acc65f6d00a71 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > @@ -54,6 +54,29 @@ static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain
> > spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> > }
> >
> > +static void
> > +arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
> > +{
> > + struct arm_smmu_master *master;
> > + struct arm_smmu_cd target_cd;
> > + unsigned long flags;
> > +
> > + spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> > + list_for_each_entry(master, &smmu_domain->devices, domain_head) {
> > + struct arm_smmu_cd *cdptr;
> > +
> > + /* S1 domains only support RID attachment right now */
> > + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID);
> > + if (WARN_ON(!cdptr))
>
> This should never hit, no? Otherwise that means this path can allocate memory
> with a spinlock.
Right, WARN_ON's should never be hit.
> > +void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
> > + struct arm_smmu_master *master,
> > + struct arm_smmu_domain *smmu_domain)
> > +{
> > + struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
> > +
> > + memset(target, 0, sizeof(*target));
> > +
> > + target->data[0] = cpu_to_le64(
> > + cd->tcr |
> > +#ifdef __BIG_ENDIAN
> > + CTXDESC_CD_0_ENDI |
> > +#endif
> > + CTXDESC_CD_0_V |
> > + CTXDESC_CD_0_AA64 |
> > + (master->stall_enabled ? CTXDESC_CD_0_S : 0) |
> > + CTXDESC_CD_0_R |
> > + CTXDESC_CD_0_A |
> > + CTXDESC_CD_0_ASET |
> > + FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid)
> > + );
> > +
> > + target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK);
> > + target->data[3] = cpu_to_le64(cd->mair);
> > +}
> > +
>
> That seems to duplicate some logic from arm_smmu_write_ctx_desc(),
> can that be consolidated?
Yes, it is consolidated by deleting arm_smmu_write_ctx_desc() in a few
more patches.
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
2024-03-23 13:38 ` Mostafa Saleh
@ 2024-03-25 14:35 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-25 14:35 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Sat, Mar 23, 2024 at 01:38:04PM +0000, Mostafa Saleh wrote:
> Hi Jason,
>
> On Mon, Mar 04, 2024 at 07:43:48PM -0400, Jason Gunthorpe wrote:
> > Continuing the work of part 1 this focuses on the CD, PASID and SVA
> > components:
> >
> > - attach_dev failure does not change the HW configuration.
> >
> > - Full PASID API support including:
> > - S1/SVA domains attached to PASIDs
>
> I am still going through the series, but I see at the end the main SMMUv3
> driver has set_dev_pasid operation, are there any in-tree drivers that
> use that? (and how can I test it).
Not yet, but some will be coming. Currently only Intel driver supports
it, but Intel HW has other problems making it unusable..
A big part of the effort here is to enable the platform ecosystem so
devices and drivers can use it. Moritz has access to a device that
can exercise this, though we are still working on it.
> > - IDENTITY/BLOCKED/S1 attached to RID
> > - Change of the RID domain while PASIDs are attached
> >
> > - Streamlined SVA support using the core infrastructure
> >
> > - Hitless, whenever possible, change between two domains
>
> Can you please clarify what cases are expected to be hitless?
> From what I see if ASID and TTB0 changes that would break the CD.
Right. For CD it is only the SVA mm release flow, setting EPD0.
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
@ 2024-03-25 14:35 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-25 14:35 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Sat, Mar 23, 2024 at 01:38:04PM +0000, Mostafa Saleh wrote:
> Hi Jason,
>
> On Mon, Mar 04, 2024 at 07:43:48PM -0400, Jason Gunthorpe wrote:
> > Continuing the work of part 1 this focuses on the CD, PASID and SVA
> > components:
> >
> > - attach_dev failure does not change the HW configuration.
> >
> > - Full PASID API support including:
> > - S1/SVA domains attached to PASIDs
>
> I am still going through the series, but I see at the end the main SMMUv3
> driver has set_dev_pasid operation, are there any in-tree drivers that
> use that? (and how can I test it).
Not yet, but some will be coming. Currently only Intel driver supports
it, but Intel HW has other problems making it unusable..
A big part of the effort here is to enable the platform ecosystem so
devices and drivers can use it. Moritz has access to a device that
can exercise this, though we are still working on it.
> > - IDENTITY/BLOCKED/S1 attached to RID
> > - Change of the RID domain while PASIDs are attached
> >
> > - Streamlined SVA support using the core infrastructure
> >
> > - Hitless, whenever possible, change between two domains
>
> Can you please clarify what cases are expected to be hitless?
> From what I see if ASID and TTB0 changes that would break the CD.
Right. For CD it is only the SVA mm release flow, setting EPD0.
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
2024-03-25 11:22 ` Mostafa Saleh
@ 2024-03-25 16:47 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-25 16:47 UTC (permalink / raw)
To: Mostafa Saleh
Cc: Shameerali Kolothum Thodi, iommu, Joerg Roedel, linux-arm-kernel,
Robin Murphy, Will Deacon, Eric Auger, Jean-Philippe Brucker,
Moritz Fischer, Michael Shavit, Nicolin Chen, patches
On Mon, Mar 25, 2024 at 11:22:19AM +0000, Mostafa Saleh wrote:
> > I think that is probably because you are testing with "nested-smmuv3". This
> > series not yet fully enable that. For that, I think you are missing few patches
> > from Nicolin's iommufd branch,
> > https://github.com/nicolinc/iommufd/commits/wip/iommufd_nesting-03112024/
>
> I see, thanks for clarifying. I think we shouldn't still crash the
> kernel, but that's a problem for part 3.
Yeah, definately. Part 3 needs to include the invalidation bits too, I
haven't integrated them from Nicolin.
I'll send a patch like this for iommufd to stop the oops:
@@ -236,7 +236,8 @@ iommufd_hwpt_nested_alloc(struct iommufd_ctx *ictx,
}
hwpt->domain->owner = ops;
- if (WARN_ON_ONCE(hwpt->domain->type != IOMMU_DOMAIN_NESTED)) {
+ if (WARN_ON_ONCE(hwpt->domain->type != IOMMU_DOMAIN_NESTED ||
+ !hwpt->domain->ops->cache_invalidate_user)) {
rc = -EINVAL;
goto out_abort;
}
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
@ 2024-03-25 16:47 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-25 16:47 UTC (permalink / raw)
To: Mostafa Saleh
Cc: Shameerali Kolothum Thodi, iommu, Joerg Roedel, linux-arm-kernel,
Robin Murphy, Will Deacon, Eric Auger, Jean-Philippe Brucker,
Moritz Fischer, Michael Shavit, Nicolin Chen, patches
On Mon, Mar 25, 2024 at 11:22:19AM +0000, Mostafa Saleh wrote:
> > I think that is probably because you are testing with "nested-smmuv3". This
> > series not yet fully enable that. For that, I think you are missing few patches
> > from Nicolin's iommufd branch,
> > https://github.com/nicolinc/iommufd/commits/wip/iommufd_nesting-03112024/
>
> I see, thanks for clarifying. I think we shouldn't still crash the
> kernel, but that's a problem for part 3.
Yeah, definately. Part 3 needs to include the invalidation bits too, I
haven't integrated them from Nicolin.
I'll send a patch like this for iommufd to stop the oops:
@@ -236,7 +236,8 @@ iommufd_hwpt_nested_alloc(struct iommufd_ctx *ictx,
}
hwpt->domain->owner = ops;
- if (WARN_ON_ONCE(hwpt->domain->type != IOMMU_DOMAIN_NESTED)) {
+ if (WARN_ON_ONCE(hwpt->domain->type != IOMMU_DOMAIN_NESTED ||
+ !hwpt->domain->ops->cache_invalidate_user)) {
rc = -EINVAL;
goto out_abort;
}
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code
2024-03-25 14:11 ` Jason Gunthorpe
@ 2024-03-25 21:01 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-25 21:01 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 25, 2024 at 11:11:32AM -0300, Jason Gunthorpe wrote:
> On Fri, Mar 22, 2024 at 06:14:24PM +0000, Mostafa Saleh wrote:
> > > @@ -1027,57 +1038,55 @@ static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent,
> > > * unused_update is an intermediate value of entry that has unused bits set to
> > > * their new values.
> > > */
> > > -static u8 arm_smmu_entry_qword_diff(const struct arm_smmu_ste *entry,
> > > - const struct arm_smmu_ste *target,
> > > - struct arm_smmu_ste *unused_update)
> > > +static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> > > + const __le64 *entry, const __le64 *target,
> > > + __le64 *unused_update)
> > > {
> > > - struct arm_smmu_ste target_used = {};
> > > - struct arm_smmu_ste cur_used = {};
> > > + __le64 target_used[NUM_ENTRY_QWORDS] = {};
> > > + __le64 cur_used[NUM_ENTRY_QWORDS] = {};
> > This is confusing to me, the function was modified to be generic, so its has
> > args are __le64 * instead of struct arm_smmu_ste *.
>
> Right
>
> > But NUM_ENTRY_QWORDS is defined as “(sizeof(struct arm_smmu_ste) / sizeof(u64))”
> > and in the same function writer->ops->num_entry_qwords is used
> > nterchangeably,
>
> Right
>
> > I understand that this not a constant and the compiler would complain.
> > But since for any other num_entry_qwords larger than NUM_ENTRY_QWORDS it fails,
> > and we know STEs and CDs both have the same size, we simplify the code and make
> > it a constant everywhere.
>
> So you say to get rid of num_entry_qwords and just use the constant?
In my opinion, yes, that looks easier to understand, and avoids the MAX
stuff as there is no reason for the extra generalisation.
> > I see in the next patch, that this is redefined to be the max between STE and
> > CD, but again, this hardware and it never changes, so my opinion is to simplify
> > the code, as there is no need to generalize this part.
>
> Yes, we need a constant.
>
> It would look like this, it is a little bit simpler:
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index a54062faccde38..d015f41900d802 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -63,9 +63,9 @@ enum arm_smmu_msi_index {
> ARM_SMMU_MAX_MSIS,
> };
>
> -#define NUM_ENTRY_QWORDS \
> - (max(sizeof(struct arm_smmu_ste), sizeof(struct arm_smmu_cd)) / \
> - sizeof(u64))
> +#define NUM_ENTRY_QWORDS 8
> +static_assert(sizeof(struct arm_smmu_ste) == NUM_ENTRY_QWORDS * sizeof(u64));
> +static_assert(sizeof(struct arm_smmu_cd) == NUM_ENTRY_QWORDS * sizeof(u64));
>
> static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
> [EVTQ_MSI_INDEX] = {
> @@ -1045,7 +1045,7 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> writer->ops->get_used(entry, cur_used);
> writer->ops->get_used(target, target_used);
>
> - for (i = 0; i != writer->ops->num_entry_qwords; i++) {
> + for (i = 0; i != NUM_ENTRY_QWORDS; i++) {
> /*
> * Check that masks are up to date, the make functions are not
> * allowed to set a bit to 1 if the used function doesn't say it
> @@ -1114,7 +1114,6 @@ static bool entry_set(struct arm_smmu_entry_writer *writer, __le64 *entry,
> void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry,
> const __le64 *target)
> {
> - unsigned int num_entry_qwords = writer->ops->num_entry_qwords;
> __le64 unused_update[NUM_ENTRY_QWORDS];
> u8 used_qword_diff;
>
> @@ -1137,9 +1136,9 @@ void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry,
> */
> unused_update[critical_qword_index] =
> entry[critical_qword_index];
> - entry_set(writer, entry, unused_update, 0, num_entry_qwords);
> + entry_set(writer, entry, unused_update, 0, NUM_ENTRY_QWORDS);
> entry_set(writer, entry, target, critical_qword_index, 1);
> - entry_set(writer, entry, target, 0, num_entry_qwords);
> + entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS);
> } else if (used_qword_diff) {
> /*
> * At least two qwords need their inuse bits to be changed. This
> @@ -1148,7 +1147,7 @@ void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry,
> */
> unused_update[0] = entry[0] & (~writer->ops->v_bit);
> entry_set(writer, entry, unused_update, 0, 1);
> - entry_set(writer, entry, target, 1, num_entry_qwords - 1);
> + entry_set(writer, entry, target, 1, NUM_ENTRY_QWORDS - 1);
> entry_set(writer, entry, target, 0, 1);
> } else {
> /*
> @@ -1157,7 +1156,7 @@ void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry,
> * compute_qword_diff().
> */
> WARN_ON_ONCE(
> - entry_set(writer, entry, target, 0, num_entry_qwords));
> + entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS));
> }
> }
>
> @@ -1272,7 +1271,6 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
> .sync = arm_smmu_cd_writer_sync_entry,
> .get_used = arm_smmu_get_cd_used,
> .v_bit = cpu_to_le64(CTXDESC_CD_0_V),
> - .num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
> };
>
> void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> @@ -1460,7 +1458,6 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_ste_writer_ops = {
> .sync = arm_smmu_ste_writer_sync_entry,
> .get_used = arm_smmu_get_ste_used,
> .v_bit = cpu_to_le64(STRTAB_STE_0_V),
> - .num_entry_qwords = sizeof(struct arm_smmu_ste) / sizeof(u64),
> };
>
> static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 8ba07b00bf6056..5936dc5f76786a 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -779,7 +779,6 @@ struct arm_smmu_entry_writer {
> };
>
> struct arm_smmu_entry_writer_ops {
> - unsigned int num_entry_qwords;
> __le64 v_bit;
> void (*get_used)(const __le64 *entry, __le64 *used);
> void (*sync)(struct arm_smmu_entry_writer *writer);
>
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code
@ 2024-03-25 21:01 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-25 21:01 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 25, 2024 at 11:11:32AM -0300, Jason Gunthorpe wrote:
> On Fri, Mar 22, 2024 at 06:14:24PM +0000, Mostafa Saleh wrote:
> > > @@ -1027,57 +1038,55 @@ static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent,
> > > * unused_update is an intermediate value of entry that has unused bits set to
> > > * their new values.
> > > */
> > > -static u8 arm_smmu_entry_qword_diff(const struct arm_smmu_ste *entry,
> > > - const struct arm_smmu_ste *target,
> > > - struct arm_smmu_ste *unused_update)
> > > +static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> > > + const __le64 *entry, const __le64 *target,
> > > + __le64 *unused_update)
> > > {
> > > - struct arm_smmu_ste target_used = {};
> > > - struct arm_smmu_ste cur_used = {};
> > > + __le64 target_used[NUM_ENTRY_QWORDS] = {};
> > > + __le64 cur_used[NUM_ENTRY_QWORDS] = {};
> > This is confusing to me, the function was modified to be generic, so its has
> > args are __le64 * instead of struct arm_smmu_ste *.
>
> Right
>
> > But NUM_ENTRY_QWORDS is defined as “(sizeof(struct arm_smmu_ste) / sizeof(u64))”
> > and in the same function writer->ops->num_entry_qwords is used
> > nterchangeably,
>
> Right
>
> > I understand that this not a constant and the compiler would complain.
> > But since for any other num_entry_qwords larger than NUM_ENTRY_QWORDS it fails,
> > and we know STEs and CDs both have the same size, we simplify the code and make
> > it a constant everywhere.
>
> So you say to get rid of num_entry_qwords and just use the constant?
In my opinion, yes, that looks easier to understand, and avoids the MAX
stuff as there is no reason for the extra generalisation.
> > I see in the next patch, that this is redefined to be the max between STE and
> > CD, but again, this hardware and it never changes, so my opinion is to simplify
> > the code, as there is no need to generalize this part.
>
> Yes, we need a constant.
>
> It would look like this, it is a little bit simpler:
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index a54062faccde38..d015f41900d802 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -63,9 +63,9 @@ enum arm_smmu_msi_index {
> ARM_SMMU_MAX_MSIS,
> };
>
> -#define NUM_ENTRY_QWORDS \
> - (max(sizeof(struct arm_smmu_ste), sizeof(struct arm_smmu_cd)) / \
> - sizeof(u64))
> +#define NUM_ENTRY_QWORDS 8
> +static_assert(sizeof(struct arm_smmu_ste) == NUM_ENTRY_QWORDS * sizeof(u64));
> +static_assert(sizeof(struct arm_smmu_cd) == NUM_ENTRY_QWORDS * sizeof(u64));
>
> static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
> [EVTQ_MSI_INDEX] = {
> @@ -1045,7 +1045,7 @@ static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer,
> writer->ops->get_used(entry, cur_used);
> writer->ops->get_used(target, target_used);
>
> - for (i = 0; i != writer->ops->num_entry_qwords; i++) {
> + for (i = 0; i != NUM_ENTRY_QWORDS; i++) {
> /*
> * Check that masks are up to date, the make functions are not
> * allowed to set a bit to 1 if the used function doesn't say it
> @@ -1114,7 +1114,6 @@ static bool entry_set(struct arm_smmu_entry_writer *writer, __le64 *entry,
> void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry,
> const __le64 *target)
> {
> - unsigned int num_entry_qwords = writer->ops->num_entry_qwords;
> __le64 unused_update[NUM_ENTRY_QWORDS];
> u8 used_qword_diff;
>
> @@ -1137,9 +1136,9 @@ void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry,
> */
> unused_update[critical_qword_index] =
> entry[critical_qword_index];
> - entry_set(writer, entry, unused_update, 0, num_entry_qwords);
> + entry_set(writer, entry, unused_update, 0, NUM_ENTRY_QWORDS);
> entry_set(writer, entry, target, critical_qword_index, 1);
> - entry_set(writer, entry, target, 0, num_entry_qwords);
> + entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS);
> } else if (used_qword_diff) {
> /*
> * At least two qwords need their inuse bits to be changed. This
> @@ -1148,7 +1147,7 @@ void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry,
> */
> unused_update[0] = entry[0] & (~writer->ops->v_bit);
> entry_set(writer, entry, unused_update, 0, 1);
> - entry_set(writer, entry, target, 1, num_entry_qwords - 1);
> + entry_set(writer, entry, target, 1, NUM_ENTRY_QWORDS - 1);
> entry_set(writer, entry, target, 0, 1);
> } else {
> /*
> @@ -1157,7 +1156,7 @@ void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry,
> * compute_qword_diff().
> */
> WARN_ON_ONCE(
> - entry_set(writer, entry, target, 0, num_entry_qwords));
> + entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS));
> }
> }
>
> @@ -1272,7 +1271,6 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = {
> .sync = arm_smmu_cd_writer_sync_entry,
> .get_used = arm_smmu_get_cd_used,
> .v_bit = cpu_to_le64(CTXDESC_CD_0_V),
> - .num_entry_qwords = sizeof(struct arm_smmu_cd) / sizeof(u64),
> };
>
> void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> @@ -1460,7 +1458,6 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_ste_writer_ops = {
> .sync = arm_smmu_ste_writer_sync_entry,
> .get_used = arm_smmu_get_ste_used,
> .v_bit = cpu_to_le64(STRTAB_STE_0_V),
> - .num_entry_qwords = sizeof(struct arm_smmu_ste) / sizeof(u64),
> };
>
> static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 8ba07b00bf6056..5936dc5f76786a 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -779,7 +779,6 @@ struct arm_smmu_entry_writer {
> };
>
> struct arm_smmu_entry_writer_ops {
> - unsigned int num_entry_qwords;
> __le64 v_bit;
> void (*get_used)(const __le64 *entry, __le64 *used);
> void (*sync)(struct arm_smmu_entry_writer *writer);
>
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 06/27] iommu/arm-smmu-v3: Consolidate clearing a CD table entry
2024-03-25 14:14 ` Jason Gunthorpe
@ 2024-03-25 21:02 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-25 21:02 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 25, 2024 at 11:14:33AM -0300, Jason Gunthorpe wrote:
> On Fri, Mar 22, 2024 at 06:36:17PM +0000, Mostafa Saleh wrote:
> > > +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> > > +{
> > > + struct arm_smmu_cd target = {};
> > > + struct arm_smmu_cd *cdptr;
> > > +
> > > + if (!master->cd_table.cdtab)
> > > + return;
> > > + cdptr = arm_smmu_get_cd_ptr(master, ssid);
> > > + if (WARN_ON(!cdptr))
> > > + return;
> >
> > I don’t understand the SVA code enough, but AFAICT, arm_smmu_sva_set_dev_pasid
> > can allocate the L2 CD table through arm_smmu_write_ctx_desc.
>
> Yes
>
> > And if it failed
> > before allocating the CD table, then remove_dev_pasid would be
> > called,
>
> If it fails the core code should not call remove_dev_pasid() on a
> domain that was never attached. There is an obscure error unwinding
> issue in the core code Yi is looking to fix regarding multi-device
> PASID groups, but it is not a driver issue..
>
I see, thanks for clarifying.
> > which warns here, the previous code would tolerate that, but that
> > might regress on systems with panic_on_warn, so I am not sure if
> > that is necessary.
>
> Right, if it does hit it signals there is some error unwinding bug in
> the core code that should be resolved.
>
> > Otherwise, Reviewed-by: Mostafa Saleh <smostafa@google.com>
>
> Thanks,
> Jason
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 06/27] iommu/arm-smmu-v3: Consolidate clearing a CD table entry
@ 2024-03-25 21:02 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-25 21:02 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 25, 2024 at 11:14:33AM -0300, Jason Gunthorpe wrote:
> On Fri, Mar 22, 2024 at 06:36:17PM +0000, Mostafa Saleh wrote:
> > > +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid)
> > > +{
> > > + struct arm_smmu_cd target = {};
> > > + struct arm_smmu_cd *cdptr;
> > > +
> > > + if (!master->cd_table.cdtab)
> > > + return;
> > > + cdptr = arm_smmu_get_cd_ptr(master, ssid);
> > > + if (WARN_ON(!cdptr))
> > > + return;
> >
> > I don’t understand the SVA code enough, but AFAICT, arm_smmu_sva_set_dev_pasid
> > can allocate the L2 CD table through arm_smmu_write_ctx_desc.
>
> Yes
>
> > And if it failed
> > before allocating the CD table, then remove_dev_pasid would be
> > called,
>
> If it fails the core code should not call remove_dev_pasid() on a
> domain that was never attached. There is an obscure error unwinding
> issue in the core code Yi is looking to fix regarding multi-device
> PASID groups, but it is not a driver issue..
>
I see, thanks for clarifying.
> > which warns here, the previous code would tolerate that, but that
> > might regress on systems with panic_on_warn, so I am not sure if
> > that is necessary.
>
> Right, if it does hit it signals there is some error unwinding bug in
> the core code that should be resolved.
>
> > Otherwise, Reviewed-by: Mostafa Saleh <smostafa@google.com>
>
> Thanks,
> Jason
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 08/27] iommu/arm-smmu-v3: Move allocation of the cdtable into arm_smmu_get_cd_ptr()
2024-03-25 14:21 ` Jason Gunthorpe
@ 2024-03-25 21:03 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-25 21:03 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 25, 2024 at 11:21:28AM -0300, Jason Gunthorpe wrote:
> On Fri, Mar 22, 2024 at 07:07:10PM +0000, Mostafa Saleh wrote:
> > Hi Jason,
> >
> > On Mon, Mar 04, 2024 at 07:43:56PM -0400, Jason Gunthorpe wrote:
> > > No reason to force callers to do two steps. Make arm_smmu_get_cd_ptr()
> > > able to return an entry in all cases except OOM
> >
> > I believe the current code is more clear, as it is explicit about which path
> > is expected to allocate.
>
> I think we had this allocate vs no allocate discussion before on
> something else..
>
> It would be good to make *full* allocate/noallocate variants of
> get_cd_ptr() and the cases that must never allocate call the no
> allocate variation. There are some issues with GFP_KERNEL/ATOMIC that
> are a bit hard to understand as well.
>
> This is a bigger issue than just the cd_table, as even the leafs
> should not allocate.
>
> > As there are many callers for arm_smmu_get_cd_ptr() directly and indirectly,
> > and it read-modify-writes the cdtable, it would be a pain to debug not
> > knowing which one could allocate, and this patch only abstracts one
> > allocating call, so it is not much code less.
>
> > For example, (again I don’t know much about SVA) I think there might be a
> > race condition as follows:
> > arm_smmu_attach_dev
> > arm_smmu_domain_finalise() => set domain stage
> > [....]
> > arm_smmu_get_cd_ptr() => RMW master->cd_table
> >
> > arm_smmu_sva_set_dev_pasid
> > __arm_smmu_sva_bind
> > Check stage is valid
> > [...]
> > arm_smmu_write_ctx_desc
> > arm_smmu_get_cd_ptr => RMW master->cd_table
> >
> > If this path is true though, I guess the in current code, we would need some
> > barriers in arm_smmu_get_cd_ptr(), arm_smmu_get_cd_ptr()
>
> Both of those functions are called under the group mutex held by the
> core code.
>
> The only valid condition to change the CD table backing memory is when
> the group mutex is held. We now have iommu_group_mutex_assert() so an
> alloc/noalloc split can call that test on the alloc side which is
> motivation enough to do it, IMHO.
>
> I will split the function and sort it all out, but I will still
> integrate the cd_table allocation into the allocating version of
> get_cd_ptr().
I prefer that, as it makes it clear which paths expect to allocate
and which not.
> Thanks,
> Jason
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 08/27] iommu/arm-smmu-v3: Move allocation of the cdtable into arm_smmu_get_cd_ptr()
@ 2024-03-25 21:03 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-25 21:03 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 25, 2024 at 11:21:28AM -0300, Jason Gunthorpe wrote:
> On Fri, Mar 22, 2024 at 07:07:10PM +0000, Mostafa Saleh wrote:
> > Hi Jason,
> >
> > On Mon, Mar 04, 2024 at 07:43:56PM -0400, Jason Gunthorpe wrote:
> > > No reason to force callers to do two steps. Make arm_smmu_get_cd_ptr()
> > > able to return an entry in all cases except OOM
> >
> > I believe the current code is more clear, as it is explicit about which path
> > is expected to allocate.
>
> I think we had this allocate vs no allocate discussion before on
> something else..
>
> It would be good to make *full* allocate/noallocate variants of
> get_cd_ptr() and the cases that must never allocate call the no
> allocate variation. There are some issues with GFP_KERNEL/ATOMIC that
> are a bit hard to understand as well.
>
> This is a bigger issue than just the cd_table, as even the leafs
> should not allocate.
>
> > As there are many callers for arm_smmu_get_cd_ptr() directly and indirectly,
> > and it read-modify-writes the cdtable, it would be a pain to debug not
> > knowing which one could allocate, and this patch only abstracts one
> > allocating call, so it is not much code less.
>
> > For example, (again I don’t know much about SVA) I think there might be a
> > race condition as follows:
> > arm_smmu_attach_dev
> > arm_smmu_domain_finalise() => set domain stage
> > [....]
> > arm_smmu_get_cd_ptr() => RMW master->cd_table
> >
> > arm_smmu_sva_set_dev_pasid
> > __arm_smmu_sva_bind
> > Check stage is valid
> > [...]
> > arm_smmu_write_ctx_desc
> > arm_smmu_get_cd_ptr => RMW master->cd_table
> >
> > If this path is true though, I guess the in current code, we would need some
> > barriers in arm_smmu_get_cd_ptr(), arm_smmu_get_cd_ptr()
>
> Both of those functions are called under the group mutex held by the
> core code.
>
> The only valid condition to change the CD table backing memory is when
> the group mutex is held. We now have iommu_group_mutex_assert() so an
> alloc/noalloc split can call that test on the alloc side which is
> motivation enough to do it, IMHO.
>
> I will split the function and sort it all out, but I will still
> integrate the cd_table allocation into the allocating version of
> get_cd_ptr().
I prefer that, as it makes it clear which paths expect to allocate
and which not.
> Thanks,
> Jason
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
2024-03-25 14:35 ` Jason Gunthorpe
@ 2024-03-25 21:06 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-25 21:06 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 25, 2024 at 11:35:03AM -0300, Jason Gunthorpe wrote:
> On Sat, Mar 23, 2024 at 01:38:04PM +0000, Mostafa Saleh wrote:
> > Hi Jason,
> >
> > On Mon, Mar 04, 2024 at 07:43:48PM -0400, Jason Gunthorpe wrote:
> > > Continuing the work of part 1 this focuses on the CD, PASID and SVA
> > > components:
> > >
> > > - attach_dev failure does not change the HW configuration.
> > >
> > > - Full PASID API support including:
> > > - S1/SVA domains attached to PASIDs
> >
> > I am still going through the series, but I see at the end the main SMMUv3
> > driver has set_dev_pasid operation, are there any in-tree drivers that
> > use that? (and how can I test it).
>
> Not yet, but some will be coming. Currently only Intel driver supports
> it, but Intel HW has other problems making it unusable..
>
> A big part of the effort here is to enable the platform ecosystem so
> devices and drivers can use it. Moritz has access to a device that
> can exercise this, though we are still working on it.
>
Just out of curiosity, are there plans to upstream that driver?
> > > - IDENTITY/BLOCKED/S1 attached to RID
> > > - Change of the RID domain while PASIDs are attached
> > >
> > > - Streamlined SVA support using the core infrastructure
> > >
> > > - Hitless, whenever possible, change between two domains
> >
> > Can you please clarify what cases are expected to be hitless?
> > From what I see if ASID and TTB0 changes that would break the CD.
>
> Right. For CD it is only the SVA mm release flow, setting EPD0.
>
I see, thanks for confirming, I am still going through the series, but
I now wonder if this case is worth the extra complexity, unlike the STE
where the hitless transition was usefull in many cases.
Thanks,
Mostafa.
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
@ 2024-03-25 21:06 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-25 21:06 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 25, 2024 at 11:35:03AM -0300, Jason Gunthorpe wrote:
> On Sat, Mar 23, 2024 at 01:38:04PM +0000, Mostafa Saleh wrote:
> > Hi Jason,
> >
> > On Mon, Mar 04, 2024 at 07:43:48PM -0400, Jason Gunthorpe wrote:
> > > Continuing the work of part 1 this focuses on the CD, PASID and SVA
> > > components:
> > >
> > > - attach_dev failure does not change the HW configuration.
> > >
> > > - Full PASID API support including:
> > > - S1/SVA domains attached to PASIDs
> >
> > I am still going through the series, but I see at the end the main SMMUv3
> > driver has set_dev_pasid operation, are there any in-tree drivers that
> > use that? (and how can I test it).
>
> Not yet, but some will be coming. Currently only Intel driver supports
> it, but Intel HW has other problems making it unusable..
>
> A big part of the effort here is to enable the platform ecosystem so
> devices and drivers can use it. Moritz has access to a device that
> can exercise this, though we are still working on it.
>
Just out of curiosity, are there plans to upstream that driver?
> > > - IDENTITY/BLOCKED/S1 attached to RID
> > > - Change of the RID domain while PASIDs are attached
> > >
> > > - Streamlined SVA support using the core infrastructure
> > >
> > > - Hitless, whenever possible, change between two domains
> >
> > Can you please clarify what cases are expected to be hitless?
> > From what I see if ASID and TTB0 changes that would break the CD.
>
> Right. For CD it is only the SVA mm release flow, setting EPD0.
>
I see, thanks for confirming, I am still going through the series, but
I now wonder if this case is worth the extra complexity, unlike the STE
where the hitless transition was usefull in many cases.
Thanks,
Mostafa.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
2024-03-25 21:06 ` Mostafa Saleh
@ 2024-03-25 22:44 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-25 22:44 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 25, 2024 at 09:06:23PM +0000, Mostafa Saleh wrote:
> On Mon, Mar 25, 2024 at 11:35:03AM -0300, Jason Gunthorpe wrote:
> > On Sat, Mar 23, 2024 at 01:38:04PM +0000, Mostafa Saleh wrote:
> > > Hi Jason,
> > >
> > > On Mon, Mar 04, 2024 at 07:43:48PM -0400, Jason Gunthorpe wrote:
> > > > Continuing the work of part 1 this focuses on the CD, PASID and SVA
> > > > components:
> > > >
> > > > - attach_dev failure does not change the HW configuration.
> > > >
> > > > - Full PASID API support including:
> > > > - S1/SVA domains attached to PASIDs
> > >
> > > I am still going through the series, but I see at the end the main SMMUv3
> > > driver has set_dev_pasid operation, are there any in-tree drivers that
> > > use that? (and how can I test it).
> >
> > Not yet, but some will be coming. Currently only Intel driver supports
> > it, but Intel HW has other problems making it unusable..
> >
> > A big part of the effort here is to enable the platform ecosystem so
> > devices and drivers can use it. Moritz has access to a device that
> > can exercise this, though we are still working on it.
>
> Just out of curiosity, are there plans to upstream that driver?
I expect so, but until it passes out of the evaluation stage and into
a production stage it isn't something guaranteed. The team working on
it needs a HW/SW ecosystem to test the device on which is only now
just barely starting to exist.
> I see, thanks for confirming, I am still going through the series, but
> I now wonder if this case is worth the extra complexity, unlike the STE
> where the hitless transition was usefull in many cases.
Well, it is worth it to convert everything into 'make' functions for
sure.
At that point it is just re-using the complexity that already
exists. Implementing a special programming logic just for CD that did
the V/0=1 and EPD0 special case as open coded would be more code than
adding ops.
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3)
@ 2024-03-25 22:44 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-25 22:44 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Mon, Mar 25, 2024 at 09:06:23PM +0000, Mostafa Saleh wrote:
> On Mon, Mar 25, 2024 at 11:35:03AM -0300, Jason Gunthorpe wrote:
> > On Sat, Mar 23, 2024 at 01:38:04PM +0000, Mostafa Saleh wrote:
> > > Hi Jason,
> > >
> > > On Mon, Mar 04, 2024 at 07:43:48PM -0400, Jason Gunthorpe wrote:
> > > > Continuing the work of part 1 this focuses on the CD, PASID and SVA
> > > > components:
> > > >
> > > > - attach_dev failure does not change the HW configuration.
> > > >
> > > > - Full PASID API support including:
> > > > - S1/SVA domains attached to PASIDs
> > >
> > > I am still going through the series, but I see at the end the main SMMUv3
> > > driver has set_dev_pasid operation, are there any in-tree drivers that
> > > use that? (and how can I test it).
> >
> > Not yet, but some will be coming. Currently only Intel driver supports
> > it, but Intel HW has other problems making it unusable..
> >
> > A big part of the effort here is to enable the platform ecosystem so
> > devices and drivers can use it. Moritz has access to a device that
> > can exercise this, though we are still working on it.
>
> Just out of curiosity, are there plans to upstream that driver?
I expect so, but until it passes out of the evaluation stage and into
a production stage it isn't something guaranteed. The team working on
it needs a HW/SW ecosystem to test the device on which is only now
just barely starting to exist.
> I see, thanks for confirming, I am still going through the series, but
> I now wonder if this case is worth the extra complexity, unlike the STE
> where the hitless transition was usefull in many cases.
Well, it is worth it to convert everything into 'make' functions for
sure.
At that point it is just re-using the complexity that already
exists. Implementing a special programming logic just for CD that did
the V/0=1 and EPD0 special case as open coded would be more code than
adding ops.
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 01/27] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID
2024-03-22 17:48 ` Mostafa Saleh
@ 2024-03-26 18:30 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-26 18:30 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Fri, Mar 22, 2024 at 05:48:52PM +0000, Mostafa Saleh wrote:
> Hi Jason,
>
> On Mon, Mar 04, 2024 at 07:43:49PM -0400, Jason Gunthorpe wrote:
> > The SVA code is wired to assume that the SVA is programmed onto the
> > mm->pasid. The current core code always does this, so it is fine.
> >
> > Add a check for clarity.
> >
> > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++
> > 1 file changed, 3 insertions(+)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > index 2610e82c0ecd0d..347c2fdd865c1a 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > @@ -581,6 +581,9 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> > int ret = 0;
> > struct mm_struct *mm = domain->mm;
> >
> > + if (mm_get_enqcmd_pasid(mm) != id)
> > + return -EINVAL;
> > +
> I am not sure if that is needed, the only caller in the tree is the IOMMU code
> and it does the right thing, as that check is removed later anyway, I don’t
> think this patch adds much.
It really should be backported, when we get drivers that do other
things it creates a hazard. I've added a fixes line.
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 01/27] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID
@ 2024-03-26 18:30 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-26 18:30 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Fri, Mar 22, 2024 at 05:48:52PM +0000, Mostafa Saleh wrote:
> Hi Jason,
>
> On Mon, Mar 04, 2024 at 07:43:49PM -0400, Jason Gunthorpe wrote:
> > The SVA code is wired to assume that the SVA is programmed onto the
> > mm->pasid. The current core code always does this, so it is fine.
> >
> > Add a check for clarity.
> >
> > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++
> > 1 file changed, 3 insertions(+)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > index 2610e82c0ecd0d..347c2fdd865c1a 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > @@ -581,6 +581,9 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> > int ret = 0;
> > struct mm_struct *mm = domain->mm;
> >
> > + if (mm_get_enqcmd_pasid(mm) != id)
> > + return -EINVAL;
> > +
> I am not sure if that is needed, the only caller in the tree is the IOMMU code
> and it does the right thing, as that check is removed later anyway, I don’t
> think this patch adds much.
It really should be backported, when we get drivers that do other
things it creates a hazard. I've added a fixes line.
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
2024-03-23 13:02 ` Mostafa Saleh
@ 2024-03-26 18:30 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-26 18:30 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Sat, Mar 23, 2024 at 01:02:15PM +0000, Mostafa Saleh wrote:
> > +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
> > +{
> > + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
> > + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
> > + return;
> > + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
>
> This is a slightly different approach than what the driver does for STEs,
> where it explicitly sets the used bits. Is there a reason for that?
It is just more compact this way
> > + /* EPD0 means T0SZ/TG0/IR0/OR0/SH0/TTB0 are IGNORED */
> > + if (ent[0] & cpu_to_le64(CTXDESC_CD_0_TCR_EPD0)) {
> > + used_bits[0] &= ~cpu_to_le64(
> > + CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 |
> > + CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 |
> > + CTXDESC_CD_0_TCR_SH0);
> > + used_bits[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK);
> > + }
> > +}
>
> We should add a comment about EPD1 maybe?
Driver doesn't use TTB1?
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
@ 2024-03-26 18:30 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-26 18:30 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Sat, Mar 23, 2024 at 01:02:15PM +0000, Mostafa Saleh wrote:
> > +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
> > +{
> > + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
> > + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
> > + return;
> > + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
>
> This is a slightly different approach than what the driver does for STEs,
> where it explicitly sets the used bits. Is there a reason for that?
It is just more compact this way
> > + /* EPD0 means T0SZ/TG0/IR0/OR0/SH0/TTB0 are IGNORED */
> > + if (ent[0] & cpu_to_le64(CTXDESC_CD_0_TCR_EPD0)) {
> > + used_bits[0] &= ~cpu_to_le64(
> > + CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 |
> > + CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 |
> > + CTXDESC_CD_0_TCR_SH0);
> > + used_bits[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK);
> > + }
> > +}
>
> We should add a comment about EPD1 maybe?
Driver doesn't use TTB1?
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 01/27] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID
2024-03-26 18:30 ` Jason Gunthorpe
@ 2024-03-26 19:06 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-26 19:06 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Tue, Mar 26, 2024 at 03:30:16PM -0300, Jason Gunthorpe wrote:
> On Fri, Mar 22, 2024 at 05:48:52PM +0000, Mostafa Saleh wrote:
> > Hi Jason,
> >
> > On Mon, Mar 04, 2024 at 07:43:49PM -0400, Jason Gunthorpe wrote:
> > > The SVA code is wired to assume that the SVA is programmed onto the
> > > mm->pasid. The current core code always does this, so it is fine.
> > >
> > > Add a check for clarity.
> > >
> > > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > > ---
> > > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++
> > > 1 file changed, 3 insertions(+)
> > >
> > > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > > index 2610e82c0ecd0d..347c2fdd865c1a 100644
> > > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > > @@ -581,6 +581,9 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> > > int ret = 0;
> > > struct mm_struct *mm = domain->mm;
> > >
> > > + if (mm_get_enqcmd_pasid(mm) != id)
> > > + return -EINVAL;
> > > +
> > I am not sure if that is needed, the only caller in the tree is the IOMMU code
> > and it does the right thing, as that check is removed later anyway, I don’t
> > think this patch adds much.
>
> It really should be backported, when we get drivers that do other
> things it creates a hazard. I've added a fixes line.
Maybe I am misunderstanding the case, but AFAIU, the only caller for this is
iommu_sva_bind_device() which is the function populating the pasid, so this
condition should never hit with the current code.
And Linux won’t backport new drivers, so there is no need to do that?
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 01/27] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID
@ 2024-03-26 19:06 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-26 19:06 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Tue, Mar 26, 2024 at 03:30:16PM -0300, Jason Gunthorpe wrote:
> On Fri, Mar 22, 2024 at 05:48:52PM +0000, Mostafa Saleh wrote:
> > Hi Jason,
> >
> > On Mon, Mar 04, 2024 at 07:43:49PM -0400, Jason Gunthorpe wrote:
> > > The SVA code is wired to assume that the SVA is programmed onto the
> > > mm->pasid. The current core code always does this, so it is fine.
> > >
> > > Add a check for clarity.
> > >
> > > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > > ---
> > > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++
> > > 1 file changed, 3 insertions(+)
> > >
> > > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > > index 2610e82c0ecd0d..347c2fdd865c1a 100644
> > > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > > @@ -581,6 +581,9 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> > > int ret = 0;
> > > struct mm_struct *mm = domain->mm;
> > >
> > > + if (mm_get_enqcmd_pasid(mm) != id)
> > > + return -EINVAL;
> > > +
> > I am not sure if that is needed, the only caller in the tree is the IOMMU code
> > and it does the right thing, as that check is removed later anyway, I don’t
> > think this patch adds much.
>
> It really should be backported, when we get drivers that do other
> things it creates a hazard. I've added a fixes line.
Maybe I am misunderstanding the case, but AFAIU, the only caller for this is
iommu_sva_bind_device() which is the function populating the pasid, so this
condition should never hit with the current code.
And Linux won’t backport new drivers, so there is no need to do that?
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
2024-03-26 18:30 ` Jason Gunthorpe
@ 2024-03-26 19:12 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-26 19:12 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Tue, Mar 26, 2024 at 03:30:55PM -0300, Jason Gunthorpe wrote:
> On Sat, Mar 23, 2024 at 01:02:15PM +0000, Mostafa Saleh wrote:
> > > +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
> > > +{
> > > + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
> > > + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
> > > + return;
> > > + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
> >
> > This is a slightly different approach than what the driver does for STEs,
> > where it explicitly sets the used bits. Is there a reason for that?
>
> It is just more compact this way
IMHO, it seems too much to have this mechanism for CDs for just one
SVA case, but I'll need to go through the whole seires first to make
sure I am not missing anything.
> > > + /* EPD0 means T0SZ/TG0/IR0/OR0/SH0/TTB0 are IGNORED */
> > > + if (ent[0] & cpu_to_le64(CTXDESC_CD_0_TCR_EPD0)) {
> > > + used_bits[0] &= ~cpu_to_le64(
> > > + CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 |
> > > + CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 |
> > > + CTXDESC_CD_0_TCR_SH0);
> > > + used_bits[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK);
> > > + }
> > > +}
> >
> > We should add a comment about EPD1 maybe?
>
> Driver doesn't use TTB1?
Yes, it's not immediately obvious why we ignore EPD1, so maybe it's
worth a comment to highlight that, but no strong opinion.
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
@ 2024-03-26 19:12 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-26 19:12 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Tue, Mar 26, 2024 at 03:30:55PM -0300, Jason Gunthorpe wrote:
> On Sat, Mar 23, 2024 at 01:02:15PM +0000, Mostafa Saleh wrote:
> > > +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
> > > +{
> > > + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
> > > + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
> > > + return;
> > > + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
> >
> > This is a slightly different approach than what the driver does for STEs,
> > where it explicitly sets the used bits. Is there a reason for that?
>
> It is just more compact this way
IMHO, it seems too much to have this mechanism for CDs for just one
SVA case, but I'll need to go through the whole seires first to make
sure I am not missing anything.
> > > + /* EPD0 means T0SZ/TG0/IR0/OR0/SH0/TTB0 are IGNORED */
> > > + if (ent[0] & cpu_to_le64(CTXDESC_CD_0_TCR_EPD0)) {
> > > + used_bits[0] &= ~cpu_to_le64(
> > > + CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 |
> > > + CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 |
> > > + CTXDESC_CD_0_TCR_SH0);
> > > + used_bits[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK);
> > > + }
> > > +}
> >
> > We should add a comment about EPD1 maybe?
>
> Driver doesn't use TTB1?
Yes, it's not immediately obvious why we ignore EPD1, so maybe it's
worth a comment to highlight that, but no strong opinion.
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 01/27] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID
2024-03-26 19:06 ` Mostafa Saleh
@ 2024-03-26 22:10 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-26 22:10 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Tue, Mar 26, 2024 at 07:06:18PM +0000, Mostafa Saleh wrote:
> On Tue, Mar 26, 2024 at 03:30:16PM -0300, Jason Gunthorpe wrote:
> > On Fri, Mar 22, 2024 at 05:48:52PM +0000, Mostafa Saleh wrote:
> > > Hi Jason,
> > >
> > > On Mon, Mar 04, 2024 at 07:43:49PM -0400, Jason Gunthorpe wrote:
> > > > The SVA code is wired to assume that the SVA is programmed onto the
> > > > mm->pasid. The current core code always does this, so it is fine.
> > > >
> > > > Add a check for clarity.
> > > >
> > > > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > > > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > > > ---
> > > > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++
> > > > 1 file changed, 3 insertions(+)
> > > >
> > > > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > > > index 2610e82c0ecd0d..347c2fdd865c1a 100644
> > > > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > > > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > > > @@ -581,6 +581,9 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> > > > int ret = 0;
> > > > struct mm_struct *mm = domain->mm;
> > > >
> > > > + if (mm_get_enqcmd_pasid(mm) != id)
> > > > + return -EINVAL;
> > > > +
> > > I am not sure if that is needed, the only caller in the tree is the IOMMU code
> > > and it does the right thing, as that check is removed later anyway, I don’t
> > > think this patch adds much.
> >
> > It really should be backported, when we get drivers that do other
> > things it creates a hazard. I've added a fixes line.
>
> Maybe I am misunderstanding the case, but AFAIU, the only caller for this is
> iommu_sva_bind_device() which is the function populating the pasid, so this
> condition should never hit with the current code.
Yes, but iommufd is going to break this soon.
> And Linux won’t backport new drivers, so there is no need to do that?
Heh, you never know what -stable will backport and the distros are
quite a bit more lax. Seems nicer to fail quickly than get all
corrupted.
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 01/27] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID
@ 2024-03-26 22:10 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-26 22:10 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Tue, Mar 26, 2024 at 07:06:18PM +0000, Mostafa Saleh wrote:
> On Tue, Mar 26, 2024 at 03:30:16PM -0300, Jason Gunthorpe wrote:
> > On Fri, Mar 22, 2024 at 05:48:52PM +0000, Mostafa Saleh wrote:
> > > Hi Jason,
> > >
> > > On Mon, Mar 04, 2024 at 07:43:49PM -0400, Jason Gunthorpe wrote:
> > > > The SVA code is wired to assume that the SVA is programmed onto the
> > > > mm->pasid. The current core code always does this, so it is fine.
> > > >
> > > > Add a check for clarity.
> > > >
> > > > Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> > > > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > > > ---
> > > > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++
> > > > 1 file changed, 3 insertions(+)
> > > >
> > > > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > > > index 2610e82c0ecd0d..347c2fdd865c1a 100644
> > > > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > > > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> > > > @@ -581,6 +581,9 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
> > > > int ret = 0;
> > > > struct mm_struct *mm = domain->mm;
> > > >
> > > > + if (mm_get_enqcmd_pasid(mm) != id)
> > > > + return -EINVAL;
> > > > +
> > > I am not sure if that is needed, the only caller in the tree is the IOMMU code
> > > and it does the right thing, as that check is removed later anyway, I don’t
> > > think this patch adds much.
> >
> > It really should be backported, when we get drivers that do other
> > things it creates a hazard. I've added a fixes line.
>
> Maybe I am misunderstanding the case, but AFAIU, the only caller for this is
> iommu_sva_bind_device() which is the function populating the pasid, so this
> condition should never hit with the current code.
Yes, but iommufd is going to break this soon.
> And Linux won’t backport new drivers, so there is no need to do that?
Heh, you never know what -stable will backport and the distros are
quite a bit more lax. Seems nicer to fail quickly than get all
corrupted.
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
2024-03-26 19:12 ` Mostafa Saleh
@ 2024-03-26 22:27 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-26 22:27 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Tue, Mar 26, 2024 at 07:12:53PM +0000, Mostafa Saleh wrote:
> On Tue, Mar 26, 2024 at 03:30:55PM -0300, Jason Gunthorpe wrote:
> > On Sat, Mar 23, 2024 at 01:02:15PM +0000, Mostafa Saleh wrote:
> > > > +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
> > > > +{
> > > > + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
> > > > + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
> > > > + return;
> > > > + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
> > >
> > > This is a slightly different approach than what the driver does for STEs,
> > > where it explicitly sets the used bits. Is there a reason for that?
> >
> > It is just more compact this way
>
> IMHO, it seems too much to have this mechanism for CDs for just one
> SVA case, but I'll need to go through the whole seires first to make
> sure I am not missing anything.
It is pretty ugly if you try to do it that way. You still need to
create some ops because the entry_set should be re-used (I mean I
guess you could copy it as well). Then you have to open code the
logic. And then the EPD0 path is somewhat fragile. Something sort of
like this:
void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
struct arm_smmu_cd *cdptr,
const struct arm_smmu_cd *target)
{
bool target_valid = target->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
bool cur_valid = cdptr->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
struct arm_smmu_cd_writer cd_writer = {
.writer = {
.ops = &arm_smmu_cd_writer_ops,
.master = master,
},
.ssid = ssid,
};
if (ssid != IOMMU_NO_PASID && cur_valid != target_valid) {
if (cur_valid)
master->cd_table.used_ssids--;
else
master->cd_table.used_ssids++;
}
/* Force a V=0/V=1 update*/
__le64 update = target[0] & ~cpu_to_le64(CTXDESC_CD_0_V);
entry_set(&cd_writer.writer, cdptr->data, &update, 0, 1);
entry_set(&cd_writer.writer, cdptr->data, target->data, 1, NUM_ENTRY_QWORDS - 1);
entry_set(&cd_writer.writer, cdptr->data, target->data, 0, 1);
}
void arm_smmu_write_cd_entry_epd0(struct arm_smmu_master *master, int ssid,
struct arm_smmu_cd *cdptr,
const struct arm_smmu_cd *target)
{
struct arm_smmu_cd_writer cd_writer = {
.writer = {
.ops = &arm_smmu_cd_writer_ops,
.master = master,
},
.ssid = ssid,
};
/*
* Target must the EPD0 = 1 version of the existing CD entry, caller
* must enforce it. Assume used_ssids doesn't need updating
* for this reason.
*/
/* Update EPD0 */
entry_set(&cd_writer.writer, cdptr->data, target->data, 0, 1);
/* Update everthing else */
entry_set(&cd_writer.writer, cdptr->data, target->data, 0, NUM_ENTRY_QWORDS - 1);
}
IMOH, at this point it is saner to have just implemented the used
function and use the mechanism robustly. Less special cases, less
fragility, less duplication.
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
@ 2024-03-26 22:27 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-26 22:27 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Tue, Mar 26, 2024 at 07:12:53PM +0000, Mostafa Saleh wrote:
> On Tue, Mar 26, 2024 at 03:30:55PM -0300, Jason Gunthorpe wrote:
> > On Sat, Mar 23, 2024 at 01:02:15PM +0000, Mostafa Saleh wrote:
> > > > +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
> > > > +{
> > > > + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
> > > > + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
> > > > + return;
> > > > + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
> > >
> > > This is a slightly different approach than what the driver does for STEs,
> > > where it explicitly sets the used bits. Is there a reason for that?
> >
> > It is just more compact this way
>
> IMHO, it seems too much to have this mechanism for CDs for just one
> SVA case, but I'll need to go through the whole seires first to make
> sure I am not missing anything.
It is pretty ugly if you try to do it that way. You still need to
create some ops because the entry_set should be re-used (I mean I
guess you could copy it as well). Then you have to open code the
logic. And then the EPD0 path is somewhat fragile. Something sort of
like this:
void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
struct arm_smmu_cd *cdptr,
const struct arm_smmu_cd *target)
{
bool target_valid = target->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
bool cur_valid = cdptr->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
struct arm_smmu_cd_writer cd_writer = {
.writer = {
.ops = &arm_smmu_cd_writer_ops,
.master = master,
},
.ssid = ssid,
};
if (ssid != IOMMU_NO_PASID && cur_valid != target_valid) {
if (cur_valid)
master->cd_table.used_ssids--;
else
master->cd_table.used_ssids++;
}
/* Force a V=0/V=1 update*/
__le64 update = target[0] & ~cpu_to_le64(CTXDESC_CD_0_V);
entry_set(&cd_writer.writer, cdptr->data, &update, 0, 1);
entry_set(&cd_writer.writer, cdptr->data, target->data, 1, NUM_ENTRY_QWORDS - 1);
entry_set(&cd_writer.writer, cdptr->data, target->data, 0, 1);
}
void arm_smmu_write_cd_entry_epd0(struct arm_smmu_master *master, int ssid,
struct arm_smmu_cd *cdptr,
const struct arm_smmu_cd *target)
{
struct arm_smmu_cd_writer cd_writer = {
.writer = {
.ops = &arm_smmu_cd_writer_ops,
.master = master,
},
.ssid = ssid,
};
/*
* Target must the EPD0 = 1 version of the existing CD entry, caller
* must enforce it. Assume used_ssids doesn't need updating
* for this reason.
*/
/* Update EPD0 */
entry_set(&cd_writer.writer, cdptr->data, target->data, 0, 1);
/* Update everthing else */
entry_set(&cd_writer.writer, cdptr->data, target->data, 0, NUM_ENTRY_QWORDS - 1);
}
IMOH, at this point it is saner to have just implemented the used
function and use the mechanism robustly. Less special cases, less
fragility, less duplication.
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
2024-03-26 22:27 ` Jason Gunthorpe
@ 2024-03-27 9:45 ` Mostafa Saleh
-1 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-27 9:45 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Tue, Mar 26, 2024 at 07:27:58PM -0300, Jason Gunthorpe wrote:
> On Tue, Mar 26, 2024 at 07:12:53PM +0000, Mostafa Saleh wrote:
> > On Tue, Mar 26, 2024 at 03:30:55PM -0300, Jason Gunthorpe wrote:
> > > On Sat, Mar 23, 2024 at 01:02:15PM +0000, Mostafa Saleh wrote:
> > > > > +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
> > > > > +{
> > > > > + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
> > > > > + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
> > > > > + return;
> > > > > + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
> > > >
> > > > This is a slightly different approach than what the driver does for STEs,
> > > > where it explicitly sets the used bits. Is there a reason for that?
> > >
> > > It is just more compact this way
> >
> > IMHO, it seems too much to have this mechanism for CDs for just one
> > SVA case, but I'll need to go through the whole seires first to make
> > sure I am not missing anything.
>
> It is pretty ugly if you try to do it that way. You still need to
> create some ops because the entry_set should be re-used (I mean I
> guess you could copy it as well). Then you have to open code the
> logic. And then the EPD0 path is somewhat fragile. Something sort of
> like this:
>
> void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_cd *cdptr,
> const struct arm_smmu_cd *target)
> {
> bool target_valid = target->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
> bool cur_valid = cdptr->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
> struct arm_smmu_cd_writer cd_writer = {
> .writer = {
> .ops = &arm_smmu_cd_writer_ops,
> .master = master,
> },
> .ssid = ssid,
> };
>
> if (ssid != IOMMU_NO_PASID && cur_valid != target_valid) {
> if (cur_valid)
> master->cd_table.used_ssids--;
> else
> master->cd_table.used_ssids++;
> }
>
> /* Force a V=0/V=1 update*/
> __le64 update = target[0] & ~cpu_to_le64(CTXDESC_CD_0_V);
> entry_set(&cd_writer.writer, cdptr->data, &update, 0, 1);
> entry_set(&cd_writer.writer, cdptr->data, target->data, 1, NUM_ENTRY_QWORDS - 1);
> entry_set(&cd_writer.writer, cdptr->data, target->data, 0, 1);
> }
>
> void arm_smmu_write_cd_entry_epd0(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_cd *cdptr,
> const struct arm_smmu_cd *target)
> {
> struct arm_smmu_cd_writer cd_writer = {
> .writer = {
> .ops = &arm_smmu_cd_writer_ops,
> .master = master,
> },
> .ssid = ssid,
> };
>
> /*
> * Target must the EPD0 = 1 version of the existing CD entry, caller
> * must enforce it. Assume used_ssids doesn't need updating
> * for this reason.
> */
> /* Update EPD0 */
> entry_set(&cd_writer.writer, cdptr->data, target->data, 0, 1);
> /* Update everthing else */
> entry_set(&cd_writer.writer, cdptr->data, target->data, 0, NUM_ENTRY_QWORDS - 1);
> }
>
> IMOH, at this point it is saner to have just implemented the used
> function and use the mechanism robustly. Less special cases, less
> fragility, less duplication.
>
But that adds extra cost of adding ops, indirection, modifying STE
code..., for a case that is not common, so I think special casing it
is actually better for readability and maintainability.
But again, I need to finish going through the series to get the
full context.
Thanks,
Mostafa
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
@ 2024-03-27 9:45 ` Mostafa Saleh
0 siblings, 0 replies; 232+ messages in thread
From: Mostafa Saleh @ 2024-03-27 9:45 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Tue, Mar 26, 2024 at 07:27:58PM -0300, Jason Gunthorpe wrote:
> On Tue, Mar 26, 2024 at 07:12:53PM +0000, Mostafa Saleh wrote:
> > On Tue, Mar 26, 2024 at 03:30:55PM -0300, Jason Gunthorpe wrote:
> > > On Sat, Mar 23, 2024 at 01:02:15PM +0000, Mostafa Saleh wrote:
> > > > > +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
> > > > > +{
> > > > > + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
> > > > > + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
> > > > > + return;
> > > > > + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
> > > >
> > > > This is a slightly different approach than what the driver does for STEs,
> > > > where it explicitly sets the used bits. Is there a reason for that?
> > >
> > > It is just more compact this way
> >
> > IMHO, it seems too much to have this mechanism for CDs for just one
> > SVA case, but I'll need to go through the whole seires first to make
> > sure I am not missing anything.
>
> It is pretty ugly if you try to do it that way. You still need to
> create some ops because the entry_set should be re-used (I mean I
> guess you could copy it as well). Then you have to open code the
> logic. And then the EPD0 path is somewhat fragile. Something sort of
> like this:
>
> void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_cd *cdptr,
> const struct arm_smmu_cd *target)
> {
> bool target_valid = target->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
> bool cur_valid = cdptr->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
> struct arm_smmu_cd_writer cd_writer = {
> .writer = {
> .ops = &arm_smmu_cd_writer_ops,
> .master = master,
> },
> .ssid = ssid,
> };
>
> if (ssid != IOMMU_NO_PASID && cur_valid != target_valid) {
> if (cur_valid)
> master->cd_table.used_ssids--;
> else
> master->cd_table.used_ssids++;
> }
>
> /* Force a V=0/V=1 update*/
> __le64 update = target[0] & ~cpu_to_le64(CTXDESC_CD_0_V);
> entry_set(&cd_writer.writer, cdptr->data, &update, 0, 1);
> entry_set(&cd_writer.writer, cdptr->data, target->data, 1, NUM_ENTRY_QWORDS - 1);
> entry_set(&cd_writer.writer, cdptr->data, target->data, 0, 1);
> }
>
> void arm_smmu_write_cd_entry_epd0(struct arm_smmu_master *master, int ssid,
> struct arm_smmu_cd *cdptr,
> const struct arm_smmu_cd *target)
> {
> struct arm_smmu_cd_writer cd_writer = {
> .writer = {
> .ops = &arm_smmu_cd_writer_ops,
> .master = master,
> },
> .ssid = ssid,
> };
>
> /*
> * Target must the EPD0 = 1 version of the existing CD entry, caller
> * must enforce it. Assume used_ssids doesn't need updating
> * for this reason.
> */
> /* Update EPD0 */
> entry_set(&cd_writer.writer, cdptr->data, target->data, 0, 1);
> /* Update everthing else */
> entry_set(&cd_writer.writer, cdptr->data, target->data, 0, NUM_ENTRY_QWORDS - 1);
> }
>
> IMOH, at this point it is saner to have just implemented the used
> function and use the mechanism robustly. Less special cases, less
> fragility, less duplication.
>
But that adds extra cost of adding ops, indirection, modifying STE
code..., for a case that is not common, so I think special casing it
is actually better for readability and maintainability.
But again, I need to finish going through the series to get the
full context.
Thanks,
Mostafa
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
2024-03-27 9:45 ` Mostafa Saleh
@ 2024-03-27 16:42 ` Jason Gunthorpe
-1 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-27 16:42 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Wed, Mar 27, 2024 at 09:45:03AM +0000, Mostafa Saleh wrote:
> On Tue, Mar 26, 2024 at 07:27:58PM -0300, Jason Gunthorpe wrote:
> > On Tue, Mar 26, 2024 at 07:12:53PM +0000, Mostafa Saleh wrote:
> > > On Tue, Mar 26, 2024 at 03:30:55PM -0300, Jason Gunthorpe wrote:
> > > > On Sat, Mar 23, 2024 at 01:02:15PM +0000, Mostafa Saleh wrote:
> > > > > > +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
> > > > > > +{
> > > > > > + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
> > > > > > + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
> > > > > > + return;
> > > > > > + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
> > > > >
> > > > > This is a slightly different approach than what the driver does for STEs,
> > > > > where it explicitly sets the used bits. Is there a reason for that?
> > > >
> > > > It is just more compact this way
> > >
> > > IMHO, it seems too much to have this mechanism for CDs for just one
> > > SVA case, but I'll need to go through the whole seires first to make
> > > sure I am not missing anything.
> >
> > It is pretty ugly if you try to do it that way. You still need to
> > create some ops because the entry_set should be re-used (I mean I
> > guess you could copy it as well). Then you have to open code the
> > logic. And then the EPD0 path is somewhat fragile. Something sort of
> > like this:
> >
> > void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> > struct arm_smmu_cd *cdptr,
> > const struct arm_smmu_cd *target)
> > {
> > bool target_valid = target->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
> > bool cur_valid = cdptr->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
> > struct arm_smmu_cd_writer cd_writer = {
> > .writer = {
> > .ops = &arm_smmu_cd_writer_ops,
> > .master = master,
> > },
> > .ssid = ssid,
> > };
> >
> > if (ssid != IOMMU_NO_PASID && cur_valid != target_valid) {
> > if (cur_valid)
> > master->cd_table.used_ssids--;
> > else
> > master->cd_table.used_ssids++;
> > }
> >
> > /* Force a V=0/V=1 update*/
> > __le64 update = target[0] & ~cpu_to_le64(CTXDESC_CD_0_V);
> > entry_set(&cd_writer.writer, cdptr->data, &update, 0, 1);
> > entry_set(&cd_writer.writer, cdptr->data, target->data, 1, NUM_ENTRY_QWORDS - 1);
> > entry_set(&cd_writer.writer, cdptr->data, target->data, 0, 1);
> > }
> >
> > void arm_smmu_write_cd_entry_epd0(struct arm_smmu_master *master, int ssid,
> > struct arm_smmu_cd *cdptr,
> > const struct arm_smmu_cd *target)
> > {
> > struct arm_smmu_cd_writer cd_writer = {
> > .writer = {
> > .ops = &arm_smmu_cd_writer_ops,
> > .master = master,
> > },
> > .ssid = ssid,
> > };
> >
> > /*
> > * Target must the EPD0 = 1 version of the existing CD entry, caller
> > * must enforce it. Assume used_ssids doesn't need updating
> > * for this reason.
> > */
> > /* Update EPD0 */
> > entry_set(&cd_writer.writer, cdptr->data, target->data, 0, 1);
> > /* Update everthing else */
> > entry_set(&cd_writer.writer, cdptr->data, target->data, 0, NUM_ENTRY_QWORDS - 1);
> > }
> >
> > IMOH, at this point it is saner to have just implemented the used
> > function and use the mechanism robustly. Less special cases, less
> > fragility, less duplication.
> >
>
> But that adds extra cost of adding ops, indirection, modifying STE
> code..., for a case that is not common, so I think special casing it
> is actually better for readability and maintainability.
The above is what the special case would have to look like. This
programming work is not trivial. The programmer code was always
intended to be re-used for the CD and STE together so there is only
one logic, not two or three copies.
Reducing duplicated logic in such a tricky area is worth it, IMHO.
Jason
^ permalink raw reply [flat|nested] 232+ messages in thread
* Re: [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry()
@ 2024-03-27 16:42 ` Jason Gunthorpe
0 siblings, 0 replies; 232+ messages in thread
From: Jason Gunthorpe @ 2024-03-27 16:42 UTC (permalink / raw)
To: Mostafa Saleh
Cc: iommu, Joerg Roedel, linux-arm-kernel, Robin Murphy, Will Deacon,
Eric Auger, Jean-Philippe Brucker, Moritz Fischer,
Michael Shavit, Nicolin Chen, patches, Shameerali Kolothum Thodi
On Wed, Mar 27, 2024 at 09:45:03AM +0000, Mostafa Saleh wrote:
> On Tue, Mar 26, 2024 at 07:27:58PM -0300, Jason Gunthorpe wrote:
> > On Tue, Mar 26, 2024 at 07:12:53PM +0000, Mostafa Saleh wrote:
> > > On Tue, Mar 26, 2024 at 03:30:55PM -0300, Jason Gunthorpe wrote:
> > > > On Sat, Mar 23, 2024 at 01:02:15PM +0000, Mostafa Saleh wrote:
> > > > > > +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
> > > > > > +{
> > > > > > + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
> > > > > > + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
> > > > > > + return;
> > > > > > + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
> > > > >
> > > > > This is a slightly different approach than what the driver does for STEs,
> > > > > where it explicitly sets the used bits. Is there a reason for that?
> > > >
> > > > It is just more compact this way
> > >
> > > IMHO, it seems too much to have this mechanism for CDs for just one
> > > SVA case, but I'll need to go through the whole seires first to make
> > > sure I am not missing anything.
> >
> > It is pretty ugly if you try to do it that way. You still need to
> > create some ops because the entry_set should be re-used (I mean I
> > guess you could copy it as well). Then you have to open code the
> > logic. And then the EPD0 path is somewhat fragile. Something sort of
> > like this:
> >
> > void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
> > struct arm_smmu_cd *cdptr,
> > const struct arm_smmu_cd *target)
> > {
> > bool target_valid = target->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
> > bool cur_valid = cdptr->data[0] & cpu_to_le64(CTXDESC_CD_0_V);
> > struct arm_smmu_cd_writer cd_writer = {
> > .writer = {
> > .ops = &arm_smmu_cd_writer_ops,
> > .master = master,
> > },
> > .ssid = ssid,
> > };
> >
> > if (ssid != IOMMU_NO_PASID && cur_valid != target_valid) {
> > if (cur_valid)
> > master->cd_table.used_ssids--;
> > else
> > master->cd_table.used_ssids++;
> > }
> >
> > /* Force a V=0/V=1 update*/
> > __le64 update = target[0] & ~cpu_to_le64(CTXDESC_CD_0_V);
> > entry_set(&cd_writer.writer, cdptr->data, &update, 0, 1);
> > entry_set(&cd_writer.writer, cdptr->data, target->data, 1, NUM_ENTRY_QWORDS - 1);
> > entry_set(&cd_writer.writer, cdptr->data, target->data, 0, 1);
> > }
> >
> > void arm_smmu_write_cd_entry_epd0(struct arm_smmu_master *master, int ssid,
> > struct arm_smmu_cd *cdptr,
> > const struct arm_smmu_cd *target)
> > {
> > struct arm_smmu_cd_writer cd_writer = {
> > .writer = {
> > .ops = &arm_smmu_cd_writer_ops,
> > .master = master,
> > },
> > .ssid = ssid,
> > };
> >
> > /*
> > * Target must the EPD0 = 1 version of the existing CD entry, caller
> > * must enforce it. Assume used_ssids doesn't need updating
> > * for this reason.
> > */
> > /* Update EPD0 */
> > entry_set(&cd_writer.writer, cdptr->data, target->data, 0, 1);
> > /* Update everthing else */
> > entry_set(&cd_writer.writer, cdptr->data, target->data, 0, NUM_ENTRY_QWORDS - 1);
> > }
> >
> > IMOH, at this point it is saner to have just implemented the used
> > function and use the mechanism robustly. Less special cases, less
> > fragility, less duplication.
> >
>
> But that adds extra cost of adding ops, indirection, modifying STE
> code..., for a case that is not common, so I think special casing it
> is actually better for readability and maintainability.
The above is what the special case would have to look like. This
programming work is not trivial. The programmer code was always
intended to be re-used for the CD and STE together so there is only
one logic, not two or three copies.
Reducing duplicated logic in such a tricky area is worth it, IMHO.
Jason
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 232+ messages in thread
end of thread, other threads:[~2024-03-27 16:43 UTC | newest]
Thread overview: 232+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-03-04 23:43 [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3) Jason Gunthorpe
2024-03-04 23:43 ` Jason Gunthorpe
2024-03-04 23:43 ` [PATCH v5 01/27] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID Jason Gunthorpe
2024-03-04 23:43 ` Jason Gunthorpe
2024-03-15 3:38 ` Nicolin Chen
2024-03-15 3:38 ` Nicolin Chen
2024-03-18 18:16 ` Jason Gunthorpe
2024-03-18 18:16 ` Jason Gunthorpe
2024-03-22 17:48 ` Mostafa Saleh
2024-03-22 17:48 ` Mostafa Saleh
2024-03-26 18:30 ` Jason Gunthorpe
2024-03-26 18:30 ` Jason Gunthorpe
2024-03-26 19:06 ` Mostafa Saleh
2024-03-26 19:06 ` Mostafa Saleh
2024-03-26 22:10 ` Jason Gunthorpe
2024-03-26 22:10 ` Jason Gunthorpe
2024-03-04 23:43 ` [PATCH v5 02/27] iommu/arm-smmu-v3: Do not ATC invalidate the entire domain Jason Gunthorpe
2024-03-04 23:43 ` Jason Gunthorpe
2024-03-13 9:18 ` Michael Shavit
2024-03-13 9:18 ` Michael Shavit
2024-03-15 2:24 ` Nicolin Chen
2024-03-15 2:24 ` Nicolin Chen
2024-03-16 18:09 ` Moritz Fischer
2024-03-16 18:09 ` Moritz Fischer
2024-03-22 17:51 ` Mostafa Saleh
2024-03-22 17:51 ` Mostafa Saleh
2024-03-04 23:43 ` [PATCH v5 03/27] iommu/arm-smmu-v3: Add a type for the CD entry Jason Gunthorpe
2024-03-04 23:43 ` Jason Gunthorpe
2024-03-13 9:44 ` Michael Shavit
2024-03-13 9:44 ` Michael Shavit
2024-03-16 18:10 ` Moritz Fischer
2024-03-16 18:10 ` Moritz Fischer
2024-03-18 18:02 ` Jason Gunthorpe
2024-03-18 18:02 ` Jason Gunthorpe
2024-03-15 3:12 ` Nicolin Chen
2024-03-15 3:12 ` Nicolin Chen
2024-03-22 17:52 ` Mostafa Saleh
2024-03-22 17:52 ` Mostafa Saleh
2024-03-04 23:43 ` [PATCH v5 04/27] iommu/arm-smmu-v3: Add an ops indirection to the STE code Jason Gunthorpe
2024-03-04 23:43 ` Jason Gunthorpe
2024-03-13 11:30 ` Michael Shavit
2024-03-13 11:30 ` Michael Shavit
2024-03-15 4:22 ` Nicolin Chen
2024-03-15 4:22 ` Nicolin Chen
2024-03-15 5:20 ` Nicolin Chen
2024-03-15 5:20 ` Nicolin Chen
2024-03-18 18:06 ` Jason Gunthorpe
2024-03-18 18:06 ` Jason Gunthorpe
2024-03-22 18:14 ` Mostafa Saleh
2024-03-22 18:14 ` Mostafa Saleh
2024-03-25 14:11 ` Jason Gunthorpe
2024-03-25 14:11 ` Jason Gunthorpe
2024-03-25 21:01 ` Mostafa Saleh
2024-03-25 21:01 ` Mostafa Saleh
2024-03-04 23:43 ` [PATCH v5 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry() Jason Gunthorpe
2024-03-04 23:43 ` Jason Gunthorpe
2024-03-15 7:52 ` Nicolin Chen
2024-03-15 7:52 ` Nicolin Chen
2024-03-20 12:46 ` Jason Gunthorpe
2024-03-20 12:46 ` Jason Gunthorpe
2024-03-16 18:14 ` Moritz Fischer
2024-03-16 18:14 ` Moritz Fischer
2024-03-23 13:02 ` Mostafa Saleh
2024-03-23 13:02 ` Mostafa Saleh
2024-03-25 14:25 ` Jason Gunthorpe
2024-03-25 14:25 ` Jason Gunthorpe
2024-03-26 18:30 ` Jason Gunthorpe
2024-03-26 18:30 ` Jason Gunthorpe
2024-03-26 19:12 ` Mostafa Saleh
2024-03-26 19:12 ` Mostafa Saleh
2024-03-26 22:27 ` Jason Gunthorpe
2024-03-26 22:27 ` Jason Gunthorpe
2024-03-27 9:45 ` Mostafa Saleh
2024-03-27 9:45 ` Mostafa Saleh
2024-03-27 16:42 ` Jason Gunthorpe
2024-03-27 16:42 ` Jason Gunthorpe
2024-03-04 23:43 ` [PATCH v5 06/27] iommu/arm-smmu-v3: Consolidate clearing a CD table entry Jason Gunthorpe
2024-03-04 23:43 ` Jason Gunthorpe
2024-03-13 11:57 ` Michael Shavit
2024-03-13 11:57 ` Michael Shavit
2024-03-15 6:17 ` Nicolin Chen
2024-03-15 6:17 ` Nicolin Chen
2024-03-16 18:15 ` Moritz Fischer
2024-03-16 18:15 ` Moritz Fischer
2024-03-22 18:36 ` Mostafa Saleh
2024-03-22 18:36 ` Mostafa Saleh
2024-03-25 14:14 ` Jason Gunthorpe
2024-03-25 14:14 ` Jason Gunthorpe
2024-03-25 21:02 ` Mostafa Saleh
2024-03-25 21:02 ` Mostafa Saleh
2024-03-04 23:43 ` [PATCH v5 07/27] iommu/arm-smmu-v3: Move the CD generation for S1 domains into a function Jason Gunthorpe
2024-03-04 23:43 ` Jason Gunthorpe
2024-03-13 12:13 ` Michael Shavit
2024-03-13 12:13 ` Michael Shavit
2024-03-18 18:11 ` Jason Gunthorpe
2024-03-18 18:11 ` Jason Gunthorpe
2024-03-23 13:11 ` Mostafa Saleh
2024-03-23 13:11 ` Mostafa Saleh
2024-03-25 14:30 ` Jason Gunthorpe
2024-03-25 14:30 ` Jason Gunthorpe
2024-03-04 23:43 ` [PATCH v5 08/27] iommu/arm-smmu-v3: Move allocation of the cdtable into arm_smmu_get_cd_ptr() Jason Gunthorpe
2024-03-04 23:43 ` Jason Gunthorpe
2024-03-13 12:15 ` Michael Shavit
2024-03-13 12:15 ` Michael Shavit
2024-03-16 3:31 ` Nicolin Chen
2024-03-16 3:31 ` Nicolin Chen
2024-03-22 19:07 ` Mostafa Saleh
2024-03-22 19:07 ` Mostafa Saleh
2024-03-25 14:21 ` Jason Gunthorpe
2024-03-25 14:21 ` Jason Gunthorpe
2024-03-25 21:03 ` Mostafa Saleh
2024-03-25 21:03 ` Mostafa Saleh
2024-03-04 23:43 ` [PATCH v5 09/27] iommu/arm-smmu-v3: Allocate the CD table entry in advance Jason Gunthorpe
2024-03-04 23:43 ` Jason Gunthorpe
2024-03-13 12:17 ` Michael Shavit
2024-03-13 12:17 ` Michael Shavit
2024-03-16 4:16 ` Nicolin Chen
2024-03-16 4:16 ` Nicolin Chen
2024-03-18 18:14 ` Jason Gunthorpe
2024-03-18 18:14 ` Jason Gunthorpe
2024-03-22 19:15 ` Mostafa Saleh
2024-03-22 19:15 ` Mostafa Saleh
2024-03-04 23:43 ` [PATCH v5 10/27] iommu/arm-smmu-v3: Move the CD generation for SVA into a function Jason Gunthorpe
2024-03-04 23:43 ` Jason Gunthorpe
2024-03-16 5:19 ` Nicolin Chen
2024-03-16 5:19 ` Nicolin Chen
2024-03-20 13:09 ` Jason Gunthorpe
2024-03-20 13:09 ` Jason Gunthorpe
2024-03-04 23:43 ` [PATCH v5 11/27] iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd() Jason Gunthorpe
2024-03-04 23:43 ` Jason Gunthorpe
2024-03-15 10:04 ` Michael Shavit
2024-03-15 10:04 ` Michael Shavit
2024-03-20 12:50 ` Jason Gunthorpe
2024-03-20 12:50 ` Jason Gunthorpe
2024-03-23 13:20 ` Mostafa Saleh
2024-03-23 13:20 ` Mostafa Saleh
2024-03-04 23:44 ` [PATCH v5 12/27] iommu/arm-smmu-v3: Start building a generic PASID layer Jason Gunthorpe
2024-03-04 23:44 ` Jason Gunthorpe
2024-03-19 16:11 ` Michael Shavit
2024-03-19 16:11 ` Michael Shavit
2024-03-20 18:32 ` Jason Gunthorpe
2024-03-20 18:32 ` Jason Gunthorpe
2024-03-04 23:44 ` [PATCH v5 13/27] iommu/arm-smmu-v3: Make smmu_domain->devices into an allocated list Jason Gunthorpe
2024-03-04 23:44 ` Jason Gunthorpe
2024-03-19 13:09 ` Michael Shavit
2024-03-19 13:09 ` Michael Shavit
2024-03-04 23:44 ` [PATCH v5 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS Jason Gunthorpe
2024-03-04 23:44 ` Jason Gunthorpe
2024-03-21 12:26 ` Michael Shavit
2024-03-21 12:26 ` Michael Shavit
2024-03-21 13:28 ` Jason Gunthorpe
2024-03-21 13:28 ` Jason Gunthorpe
2024-03-21 14:53 ` Michael Shavit
2024-03-21 14:53 ` Michael Shavit
2024-03-21 14:57 ` Michael Shavit
2024-03-21 14:57 ` Michael Shavit
2024-03-21 17:32 ` Jason Gunthorpe
2024-03-21 17:32 ` Jason Gunthorpe
2024-03-04 23:44 ` [PATCH v5 15/27] iommu/arm-smmu-v3: Add ssid to struct arm_smmu_master_domain Jason Gunthorpe
2024-03-04 23:44 ` Jason Gunthorpe
2024-03-19 13:31 ` Michael Shavit
2024-03-19 13:31 ` Michael Shavit
2024-03-20 12:53 ` Jason Gunthorpe
2024-03-20 12:53 ` Jason Gunthorpe
2024-03-04 23:44 ` [PATCH v5 16/27] iommu/arm-smmu-v3: Keep track of valid CD entries in the cd_table Jason Gunthorpe
2024-03-04 23:44 ` Jason Gunthorpe
2024-03-19 13:55 ` Michael Shavit
2024-03-19 13:55 ` Michael Shavit
2024-03-20 18:21 ` Jason Gunthorpe
2024-03-20 18:21 ` Jason Gunthorpe
2024-03-04 23:44 ` [PATCH v5 17/27] iommu/arm-smmu-v3: Thread SSID through the arm_smmu_attach_*() interface Jason Gunthorpe
2024-03-04 23:44 ` Jason Gunthorpe
2024-03-04 23:44 ` [PATCH v5 18/27] iommu/arm-smmu-v3: Make SVA allocate a normal arm_smmu_domain Jason Gunthorpe
2024-03-04 23:44 ` Jason Gunthorpe
2024-03-19 14:52 ` Michael Shavit
2024-03-19 14:52 ` Michael Shavit
2024-03-20 23:20 ` Jason Gunthorpe
2024-03-20 23:20 ` Jason Gunthorpe
2024-03-04 23:44 ` [PATCH v5 19/27] iommu/arm-smmu-v3: Keep track of arm_smmu_master_domain for SVA Jason Gunthorpe
2024-03-04 23:44 ` Jason Gunthorpe
2024-03-21 10:47 ` Michael Shavit
2024-03-21 10:47 ` Michael Shavit
2024-03-21 13:55 ` Jason Gunthorpe
2024-03-21 13:55 ` Jason Gunthorpe
2024-03-04 23:44 ` [PATCH v5 20/27] iommu: Add ops->domain_alloc_sva() Jason Gunthorpe
2024-03-04 23:44 ` Jason Gunthorpe
2024-03-19 15:09 ` Michael Shavit
2024-03-19 15:09 ` Michael Shavit
2024-03-04 23:44 ` [PATCH v5 21/27] iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain Jason Gunthorpe
2024-03-04 23:44 ` Jason Gunthorpe
2024-03-19 16:23 ` Michael Shavit
2024-03-19 16:23 ` Michael Shavit
2024-03-20 18:35 ` Jason Gunthorpe
2024-03-20 18:35 ` Jason Gunthorpe
2024-03-04 23:44 ` [PATCH v5 22/27] iommu/arm-smmu-v3: Consolidate freeing the ASID/VMID Jason Gunthorpe
2024-03-04 23:44 ` Jason Gunthorpe
2024-03-19 16:44 ` Michael Shavit
2024-03-19 16:44 ` Michael Shavit
2024-03-19 18:37 ` Jason Gunthorpe
2024-03-19 18:37 ` Jason Gunthorpe
2024-03-04 23:44 ` [PATCH v5 23/27] iommu/arm-smmu-v3: Move the arm_smmu_asid_xa to per-smmu like vmid Jason Gunthorpe
2024-03-04 23:44 ` Jason Gunthorpe
2024-03-04 23:44 ` [PATCH v5 24/27] iommu/arm-smmu-v3: Bring back SVA BTM support Jason Gunthorpe
2024-03-04 23:44 ` Jason Gunthorpe
2024-03-19 17:07 ` Michael Shavit
2024-03-19 17:07 ` Michael Shavit
2024-03-20 13:05 ` Jason Gunthorpe
2024-03-20 13:05 ` Jason Gunthorpe
2024-03-04 23:44 ` [PATCH v5 25/27] iommu/arm-smmu-v3: Allow IDENTITY/BLOCKED to be set while PASID is used Jason Gunthorpe
2024-03-04 23:44 ` Jason Gunthorpe
2024-03-04 23:44 ` [PATCH v5 26/27] iommu/arm-smmu-v3: Allow a PASID to be set when RID is IDENTITY/BLOCKED Jason Gunthorpe
2024-03-04 23:44 ` Jason Gunthorpe
2024-03-04 23:44 ` [PATCH v5 27/27] iommu/arm-smmu-v3: Allow setting a S1 domain to a PASID Jason Gunthorpe
2024-03-04 23:44 ` Jason Gunthorpe
2024-03-15 10:40 ` [PATCH v5 00/27] Update SMMUv3 to the modern iommu API (part 2/3) Shameerali Kolothum Thodi
2024-03-15 10:40 ` Shameerali Kolothum Thodi
2024-03-23 13:38 ` Mostafa Saleh
2024-03-23 13:38 ` Mostafa Saleh
2024-03-25 14:35 ` Jason Gunthorpe
2024-03-25 14:35 ` Jason Gunthorpe
2024-03-25 21:06 ` Mostafa Saleh
2024-03-25 21:06 ` Mostafa Saleh
2024-03-25 22:44 ` Jason Gunthorpe
2024-03-25 22:44 ` Jason Gunthorpe
2024-03-25 10:22 ` Mostafa Saleh
2024-03-25 10:22 ` Mostafa Saleh
2024-03-25 10:44 ` Shameerali Kolothum Thodi
2024-03-25 10:44 ` Shameerali Kolothum Thodi
2024-03-25 11:22 ` Mostafa Saleh
2024-03-25 11:22 ` Mostafa Saleh
2024-03-25 16:47 ` Jason Gunthorpe
2024-03-25 16:47 ` Jason Gunthorpe
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.