* [PATCH v3 0/5] Add Tegra194 Dual ARM SMMU driver
@ 2018-12-04 1:36 Krishna Reddy
2018-12-04 1:36 ` [PATCH v3 1/6] iommu/arm-smmu: create library for ARM SMMU Krishna Reddy
` (5 more replies)
0 siblings, 6 replies; 9+ messages in thread
From: Krishna Reddy @ 2018-12-04 1:36 UTC (permalink / raw)
To: will.deacon, robin.murphy, joro
Cc: linux-arm-kernel, iommu, linux-kernel, linux-tegra, treding,
yhsu, snikam, praithatha, talho, avanbrunt, thomasz, olof,
jtukkinen, mperttunen, Krishna Reddy
NVIDIA's Xavier (Tegra194) SOC has two ARM SMMU(MMU-500) instances, which
are used as one SMMU device in HW. The IOVA accesses from HW devices are
interleaved across these two SMMU instances and need to be programmed identical.
The existing ARM SMMU driver can't be used in its current form for programming
the two SMMU instances identically. But, Most of the code can be shared between
ARM SMMU driver and Tegra194 SMMU driver. To allow sharing the code, Created
a libray based on the current ARM SMMU driver and added suppport to program
multiple ARM SMMU Instances identically. Upated Current ARM SMMU driver and
Tegra194 SMMU driver to use the library functions.
Please review the patches and provide feedback.
Changes in v2:
* Added CONFIG_ARM_SMMU_TEGRA to protect Tegra194 SMMU driver compilation
* Enabled CONFIG_ARM_SMMU_TEGRA in defconfig
* Added SMMU nodes in Tegra194 device tree
Changes in v3:
* Created library for ARM SMMU based on arm-smmu.c
* Added support to program multiple ARM SMMU instances identically
* Updated arm-smmu.c/tegra194-smmu.c to use ARM SMMU library functions
Krishna Reddy (6):
iommu/arm-smmu: create library for ARM SMMU
iommu/arm-smmu: Add support to program multiple ARM SMMU's identically
iommu/arm-smmu: update arm-smmu.c to use ARM SMMU library
iommu/tegra194_smmu: Add Tegra194 SMMU driver
arm64: defconfig: Enable ARM_SMMU_TEGRA
arm64: tegra: Add SMMU nodes to Tegra194 device tree
arch/arm64/boot/dts/nvidia/tegra194.dtsi | 148 +++
arch/arm64/configs/defconfig | 1 +
drivers/iommu/Kconfig | 11 +
drivers/iommu/Makefile | 2 +
drivers/iommu/arm-smmu.c | 1709 +----------------------------
drivers/iommu/lib-arm-smmu.c | 1768 ++++++++++++++++++++++++++++++
drivers/iommu/lib-arm-smmu.h | 161 +++
drivers/iommu/tegra194-smmu.c | 394 +++++++
8 files changed, 2496 insertions(+), 1698 deletions(-)
create mode 100644 drivers/iommu/lib-arm-smmu.c
create mode 100644 drivers/iommu/lib-arm-smmu.h
create mode 100644 drivers/iommu/tegra194-smmu.c
--
2.1.4
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v3 1/6] iommu/arm-smmu: create library for ARM SMMU
2018-12-04 1:36 [PATCH v3 0/5] Add Tegra194 Dual ARM SMMU driver Krishna Reddy
@ 2018-12-04 1:36 ` Krishna Reddy
2018-12-04 1:36 ` [PATCH v3 2/6] iommu/arm-smmu: Add support to program multiple ARM SMMU's identically Krishna Reddy
` (4 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: Krishna Reddy @ 2018-12-04 1:36 UTC (permalink / raw)
To: will.deacon, robin.murphy, joro
Cc: linux-arm-kernel, iommu, linux-kernel, linux-tegra, treding,
yhsu, snikam, praithatha, talho, avanbrunt, thomasz, olof,
jtukkinen, mperttunen, Krishna Reddy
Create library routines to share ARM SMMU programming
and common IOMMU API implementation for ARM SMMU v1 and v2
based architecture Implementations.
Signed-off-by: Krishna Reddy <vdumpa@nvidia.com>
---
drivers/iommu/Makefile | 1 +
drivers/iommu/lib-arm-smmu.c | 1671 ++++++++++++++++++++++++++++++++++++++++++
drivers/iommu/lib-arm-smmu.h | 161 ++++
3 files changed, 1833 insertions(+)
create mode 100644 drivers/iommu/lib-arm-smmu.c
create mode 100644 drivers/iommu/lib-arm-smmu.h
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index a158a68..ea87cae 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -14,6 +14,7 @@ obj-$(CONFIG_AMD_IOMMU) += amd_iommu.o amd_iommu_init.o
obj-$(CONFIG_AMD_IOMMU_DEBUGFS) += amd_iommu_debugfs.o
obj-$(CONFIG_AMD_IOMMU_V2) += amd_iommu_v2.o
obj-$(CONFIG_ARM_SMMU) += arm-smmu.o
+obj-$(CONFIG_ARM_SMMU) += lib-arm-smmu.o
obj-$(CONFIG_ARM_SMMU_V3) += arm-smmu-v3.o
obj-$(CONFIG_DMAR_TABLE) += dmar.o
obj-$(CONFIG_INTEL_IOMMU) += intel-iommu.o intel-pasid.o
diff --git a/drivers/iommu/lib-arm-smmu.c b/drivers/iommu/lib-arm-smmu.c
new file mode 100644
index 0000000..6aba5db
--- /dev/null
+++ b/drivers/iommu/lib-arm-smmu.c
@@ -0,0 +1,1671 @@
+/*
+ * Copyright (c) 2018, NVIDIA Corporation
+ * Author: Krishna Reddy <vdumpa@nvidia.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * Library for ARM architected v1 and v2 SMMU implementations.
+ * This library is created by reusing the code from arm-smmu.c which is
+ * authored by Will Deacon.
+ */
+
+#define pr_fmt(fmt) "lib-arm-smmu: " fmt
+
+#include <linux/atomic.h>
+#include <linux/delay.h>
+#include <linux/dma-iommu.h>
+#include <linux/dma-mapping.h>
+#include <linux/err.h>
+#include <linux/fsl/mc.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/io-64-nonatomic-hi-lo.h>
+#include <linux/iopoll.h>
+#include <linux/of.h>
+#include <linux/of_address.h>
+#include <linux/of_device.h>
+#include <linux/of_iommu.h>
+#include <linux/pci.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+
+#include "io-pgtable.h"
+#include "arm-smmu-regs.h"
+#include "lib-arm-smmu.h"
+
+#define ARM_MMU500_ACTLR_CPRE (1 << 1)
+
+#define ARM_MMU500_ACR_CACHE_LOCK (1 << 26)
+#define ARM_MMU500_ACR_S2CRB_TLBEN (1 << 10)
+#define ARM_MMU500_ACR_SMTNMB_TLBEN (1 << 8)
+
+#define TLB_LOOP_TIMEOUT 1000000 /* 1s! */
+#define TLB_SPIN_COUNT 10
+
+/* SMMU global address space */
+#define ARM_SMMU_GR0(smmu) ((smmu)->base)
+#define ARM_SMMU_GR1(smmu) ((smmu)->base + (1 << (smmu)->pgshift))
+
+/*
+ * SMMU global address space with conditional offset to access secure
+ * aliases of non-secure registers (e.g. nsCR0: 0x400, nsGFSR: 0x448,
+ * nsGFSYNR0: 0x450)
+ */
+#define ARM_SMMU_GR0_NS(smmu) \
+ ((smmu)->base + \
+ ((smmu->options & ARM_SMMU_OPT_SECURE_CFG_ACCESS) \
+ ? 0x400 : 0))
+
+/*
+ * Some 64-bit registers only make sense to write atomically, but in such
+ * cases all the data relevant to AArch32 formats lies within the lower word,
+ * therefore this actually makes more sense than it might first appear.
+ */
+#ifdef CONFIG_64BIT
+#define smmu_write_atomic_lq writeq_relaxed
+#else
+#define smmu_write_atomic_lq writel_relaxed
+#endif
+
+/* Translation context bank */
+#define ARM_SMMU_CB(smmu, n) ((smmu)->cb_base + ((n) << (smmu)->pgshift))
+
+#define MSI_IOVA_BASE 0x8000000
+#define MSI_IOVA_LENGTH 0x100000
+
+#define s2cr_init_val (struct arm_smmu_s2cr){ \
+ .type = smmu->disable_bypass ? S2CR_TYPE_FAULT : S2CR_TYPE_BYPASS, \
+}
+
+struct arm_smmu_master_cfg {
+ struct arm_smmu_device *smmu;
+ s16 smendx[];
+};
+#define INVALID_SMENDX -1
+#define __fwspec_cfg(fw) ((struct arm_smmu_master_cfg *)fw->iommu_priv)
+#define fwspec_smmu(fw) (__fwspec_cfg(fw)->smmu)
+#define fwspec_smendx(fw, i) \
+ (i >= fw->num_ids ? INVALID_SMENDX : __fwspec_cfg(fw)->smendx[i])
+#define for_each_cfg_sme(fw, i, idx) \
+ for (i = 0; idx = fwspec_smendx(fw, i), i < fw->num_ids; ++i)
+
+enum arm_smmu_context_fmt {
+ ARM_SMMU_CTX_FMT_NONE,
+ ARM_SMMU_CTX_FMT_AARCH64,
+ ARM_SMMU_CTX_FMT_AARCH32_L,
+ ARM_SMMU_CTX_FMT_AARCH32_S,
+};
+
+struct arm_smmu_cfg {
+ u8 cbndx;
+ u8 irptndx;
+ union {
+ u16 asid;
+ u16 vmid;
+ };
+ u32 cbar;
+ enum arm_smmu_context_fmt fmt;
+};
+#define INVALID_IRPTNDX 0xff
+
+enum arm_smmu_domain_stage {
+ ARM_SMMU_DOMAIN_S1 = 0,
+ ARM_SMMU_DOMAIN_S2,
+ ARM_SMMU_DOMAIN_NESTED,
+ ARM_SMMU_DOMAIN_BYPASS,
+};
+
+struct arm_smmu_domain {
+ struct arm_smmu_device *smmu;
+ struct io_pgtable_ops *pgtbl_ops;
+ const struct iommu_gather_ops *tlb_ops;
+ struct arm_smmu_cfg cfg;
+ enum arm_smmu_domain_stage stage;
+ bool non_strict;
+ struct mutex init_mutex; /* Protects smmu pointer */
+ /* Serialises ATS1* ops and TLB syncs */
+ spinlock_t cb_lock;
+ struct iommu_domain domain;
+};
+
+static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
+{
+ return container_of(dom, struct arm_smmu_domain, domain);
+}
+
+static int __arm_smmu_alloc_bitmap(unsigned long *map, int start, int end)
+{
+ int idx;
+
+ do {
+ idx = find_next_zero_bit(map, end, start);
+ if (idx == end)
+ return -ENOSPC;
+ } while (test_and_set_bit(idx, map));
+
+ return idx;
+}
+
+static void __arm_smmu_free_bitmap(unsigned long *map, int idx)
+{
+ clear_bit(idx, map);
+}
+
+/* Wait for any pending TLB invalidations to complete */
+static void __arm_smmu_tlb_sync(struct arm_smmu_device *smmu,
+ void __iomem *sync, void __iomem *status)
+{
+ unsigned int spin_cnt, delay;
+
+ writel_relaxed(0, sync);
+ for (delay = 1; delay < TLB_LOOP_TIMEOUT; delay *= 2) {
+ for (spin_cnt = TLB_SPIN_COUNT; spin_cnt > 0; spin_cnt--) {
+ if (!(readl_relaxed(status) & sTLBGSTATUS_GSACTIVE))
+ return;
+ cpu_relax();
+ }
+ udelay(delay);
+ }
+ dev_err_ratelimited(smmu->dev,
+ "TLB sync timed out -- SMMU may be deadlocked\n");
+}
+
+static void arm_smmu_tlb_sync_global(struct arm_smmu_device *smmu)
+{
+ void __iomem *base = ARM_SMMU_GR0(smmu);
+ unsigned long flags;
+
+ spin_lock_irqsave(&smmu->global_sync_lock, flags);
+ __arm_smmu_tlb_sync(smmu, base + ARM_SMMU_GR0_sTLBGSYNC,
+ base + ARM_SMMU_GR0_sTLBGSTATUS);
+ spin_unlock_irqrestore(&smmu->global_sync_lock, flags);
+}
+
+static void arm_smmu_tlb_sync_context(void *cookie)
+{
+ struct arm_smmu_domain *smmu_domain = cookie;
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
+ void __iomem *base = ARM_SMMU_CB(smmu, smmu_domain->cfg.cbndx);
+ unsigned long flags;
+
+ spin_lock_irqsave(&smmu_domain->cb_lock, flags);
+ __arm_smmu_tlb_sync(smmu, base + ARM_SMMU_CB_TLBSYNC,
+ base + ARM_SMMU_CB_TLBSTATUS);
+ spin_unlock_irqrestore(&smmu_domain->cb_lock, flags);
+}
+
+static void arm_smmu_tlb_sync_vmid(void *cookie)
+{
+ struct arm_smmu_domain *smmu_domain = cookie;
+
+ arm_smmu_tlb_sync_global(smmu_domain->smmu);
+}
+
+static void arm_smmu_tlb_inv_context_s1(void *cookie)
+{
+ struct arm_smmu_domain *smmu_domain = cookie;
+ struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
+ void __iomem *base = ARM_SMMU_CB(smmu_domain->smmu, cfg->cbndx);
+
+ /*
+ * NOTE: this is not a relaxed write; it needs to guarantee that PTEs
+ * cleared by the current CPU are visible to the SMMU before the TLBI.
+ */
+ writel(cfg->asid, base + ARM_SMMU_CB_S1_TLBIASID);
+ arm_smmu_tlb_sync_context(cookie);
+}
+
+static void arm_smmu_tlb_inv_context_s2(void *cookie)
+{
+ struct arm_smmu_domain *smmu_domain = cookie;
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
+ void __iomem *base = ARM_SMMU_GR0(smmu);
+
+ /* NOTE: see above */
+ writel(smmu_domain->cfg.vmid, base + ARM_SMMU_GR0_TLBIVMID);
+ arm_smmu_tlb_sync_global(smmu);
+}
+
+static void arm_smmu_tlb_inv_range_nosync(unsigned long iova, size_t size,
+ size_t granule, bool leaf, void *cookie)
+{
+ struct arm_smmu_domain *smmu_domain = cookie;
+ struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
+ bool stage1 = cfg->cbar != CBAR_TYPE_S2_TRANS;
+ void __iomem *reg = ARM_SMMU_CB(smmu_domain->smmu, cfg->cbndx);
+
+ if (smmu_domain->smmu->features & ARM_SMMU_FEAT_COHERENT_WALK)
+ wmb();
+
+ if (stage1) {
+ reg += leaf ? ARM_SMMU_CB_S1_TLBIVAL : ARM_SMMU_CB_S1_TLBIVA;
+
+ if (cfg->fmt != ARM_SMMU_CTX_FMT_AARCH64) {
+ iova &= ~12UL;
+ iova |= cfg->asid;
+ do {
+ writel_relaxed(iova, reg);
+ iova += granule;
+ } while (size -= granule);
+ } else {
+ iova >>= 12;
+ iova |= (u64)cfg->asid << 48;
+ do {
+ writeq_relaxed(iova, reg);
+ iova += granule >> 12;
+ } while (size -= granule);
+ }
+ } else {
+ reg += leaf ? ARM_SMMU_CB_S2_TLBIIPAS2L :
+ ARM_SMMU_CB_S2_TLBIIPAS2;
+ iova >>= 12;
+ do {
+ smmu_write_atomic_lq(iova, reg);
+ iova += granule >> 12;
+ } while (size -= granule);
+ }
+}
+
+/*
+ * On MMU-401 at least, the cost of firing off multiple TLBIVMIDs appears
+ * almost negligible, but the benefit of getting the first one in as far ahead
+ * of the sync as possible is significant, hence we don't just make this a
+ * no-op and set .tlb_sync to arm_smmu_inv_context_s2() as you might think.
+ */
+static void arm_smmu_tlb_inv_vmid_nosync(unsigned long iova, size_t size,
+ size_t granule, bool leaf, void *cookie)
+{
+ struct arm_smmu_domain *smmu_domain = cookie;
+ void __iomem *base = ARM_SMMU_GR0(smmu_domain->smmu);
+
+ if (smmu_domain->smmu->features & ARM_SMMU_FEAT_COHERENT_WALK)
+ wmb();
+
+ writel_relaxed(smmu_domain->cfg.vmid, base + ARM_SMMU_GR0_TLBIVMID);
+}
+
+static const struct iommu_gather_ops arm_smmu_s1_tlb_ops = {
+ .tlb_flush_all = arm_smmu_tlb_inv_context_s1,
+ .tlb_add_flush = arm_smmu_tlb_inv_range_nosync,
+ .tlb_sync = arm_smmu_tlb_sync_context,
+};
+
+static const struct iommu_gather_ops arm_smmu_s2_tlb_ops_v2 = {
+ .tlb_flush_all = arm_smmu_tlb_inv_context_s2,
+ .tlb_add_flush = arm_smmu_tlb_inv_range_nosync,
+ .tlb_sync = arm_smmu_tlb_sync_context,
+};
+
+static const struct iommu_gather_ops arm_smmu_s2_tlb_ops_v1 = {
+ .tlb_flush_all = arm_smmu_tlb_inv_context_s2,
+ .tlb_add_flush = arm_smmu_tlb_inv_vmid_nosync,
+ .tlb_sync = arm_smmu_tlb_sync_vmid,
+};
+
+irqreturn_t arm_smmu_context_fault(int irq, void *dev)
+{
+ u32 fsr, fsynr;
+ unsigned long iova;
+ struct iommu_domain *domain = dev;
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
+ void __iomem *cb_base;
+
+ cb_base = ARM_SMMU_CB(smmu, cfg->cbndx);
+ fsr = readl_relaxed(cb_base + ARM_SMMU_CB_FSR);
+
+ if (!(fsr & FSR_FAULT))
+ return IRQ_NONE;
+
+ fsynr = readl_relaxed(cb_base + ARM_SMMU_CB_FSYNR0);
+ iova = readq_relaxed(cb_base + ARM_SMMU_CB_FAR);
+
+ dev_err_ratelimited(smmu->dev,
+ "Unhandled context fault: fsr=0x%x, iova=0x%08lx, fsynr=0x%x, cb=%d\n",
+ fsr, iova, fsynr, cfg->cbndx);
+
+ writel(fsr, cb_base + ARM_SMMU_CB_FSR);
+ return IRQ_HANDLED;
+}
+
+irqreturn_t arm_smmu_global_fault(int irq, void *dev)
+{
+ u32 gfsr, gfsynr0, gfsynr1, gfsynr2;
+ struct arm_smmu_device *smmu = dev;
+ void __iomem *gr0_base = ARM_SMMU_GR0_NS(smmu);
+
+ gfsr = readl_relaxed(gr0_base + ARM_SMMU_GR0_sGFSR);
+ gfsynr0 = readl_relaxed(gr0_base + ARM_SMMU_GR0_sGFSYNR0);
+ gfsynr1 = readl_relaxed(gr0_base + ARM_SMMU_GR0_sGFSYNR1);
+ gfsynr2 = readl_relaxed(gr0_base + ARM_SMMU_GR0_sGFSYNR2);
+
+ if (!gfsr)
+ return IRQ_NONE;
+
+ dev_err_ratelimited(smmu->dev,
+ "Unexpected global fault, this could be serious\n");
+ dev_err_ratelimited(smmu->dev,
+ "\tGFSR 0x%08x, GFSYNR0 0x%08x, GFSYNR1 0x%08x, GFSYNR2 0x%08x\n",
+ gfsr, gfsynr0, gfsynr1, gfsynr2);
+
+ writel(gfsr, gr0_base + ARM_SMMU_GR0_sGFSR);
+ return IRQ_HANDLED;
+}
+
+static void arm_smmu_init_context_bank(struct arm_smmu_domain *smmu_domain,
+ struct io_pgtable_cfg *pgtbl_cfg)
+{
+ struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
+ struct arm_smmu_cb *cb = &smmu_domain->smmu->cbs[cfg->cbndx];
+ bool stage1 = cfg->cbar != CBAR_TYPE_S2_TRANS;
+
+ cb->cfg = cfg;
+
+ /* TTBCR */
+ if (stage1) {
+ if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH32_S) {
+ cb->tcr[0] = pgtbl_cfg->arm_v7s_cfg.tcr;
+ } else {
+ cb->tcr[0] = pgtbl_cfg->arm_lpae_s1_cfg.tcr;
+ cb->tcr[1] = pgtbl_cfg->arm_lpae_s1_cfg.tcr >> 32;
+ cb->tcr[1] |= TTBCR2_SEP_UPSTREAM;
+ if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH64)
+ cb->tcr[1] |= TTBCR2_AS;
+ }
+ } else {
+ cb->tcr[0] = pgtbl_cfg->arm_lpae_s2_cfg.vtcr;
+ }
+
+ /* TTBRs */
+ if (stage1) {
+ if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH32_S) {
+ cb->ttbr[0] = pgtbl_cfg->arm_v7s_cfg.ttbr[0];
+ cb->ttbr[1] = pgtbl_cfg->arm_v7s_cfg.ttbr[1];
+ } else {
+ cb->ttbr[0] = pgtbl_cfg->arm_lpae_s1_cfg.ttbr[0];
+ cb->ttbr[0] |= (u64)cfg->asid << TTBRn_ASID_SHIFT;
+ cb->ttbr[1] = pgtbl_cfg->arm_lpae_s1_cfg.ttbr[1];
+ cb->ttbr[1] |= (u64)cfg->asid << TTBRn_ASID_SHIFT;
+ }
+ } else {
+ cb->ttbr[0] = pgtbl_cfg->arm_lpae_s2_cfg.vttbr;
+ }
+
+ /* MAIRs (stage-1 only) */
+ if (stage1) {
+ if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH32_S) {
+ cb->mair[0] = pgtbl_cfg->arm_v7s_cfg.prrr;
+ cb->mair[1] = pgtbl_cfg->arm_v7s_cfg.nmrr;
+ } else {
+ cb->mair[0] = pgtbl_cfg->arm_lpae_s1_cfg.mair[0];
+ cb->mair[1] = pgtbl_cfg->arm_lpae_s1_cfg.mair[1];
+ }
+ }
+}
+
+static void arm_smmu_write_context_bank(struct arm_smmu_device *smmu, int idx)
+{
+ u32 reg;
+ bool stage1;
+ struct arm_smmu_cb *cb = &smmu->cbs[idx];
+ struct arm_smmu_cfg *cfg = cb->cfg;
+ void __iomem *cb_base, *gr1_base;
+
+ cb_base = ARM_SMMU_CB(smmu, idx);
+
+ /* Unassigned context banks only need disabling */
+ if (!cfg) {
+ writel_relaxed(0, cb_base + ARM_SMMU_CB_SCTLR);
+ return;
+ }
+
+ gr1_base = ARM_SMMU_GR1(smmu);
+ stage1 = cfg->cbar != CBAR_TYPE_S2_TRANS;
+
+ /* CBA2R */
+ if (smmu->version > ARM_SMMU_V1) {
+ if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH64)
+ reg = CBA2R_RW64_64BIT;
+ else
+ reg = CBA2R_RW64_32BIT;
+ /* 16-bit VMIDs live in CBA2R */
+ if (smmu->features & ARM_SMMU_FEAT_VMID16)
+ reg |= cfg->vmid << CBA2R_VMID_SHIFT;
+
+ writel_relaxed(reg, gr1_base + ARM_SMMU_GR1_CBA2R(idx));
+ }
+
+ /* CBAR */
+ reg = cfg->cbar;
+ if (smmu->version < ARM_SMMU_V2)
+ reg |= cfg->irptndx << CBAR_IRPTNDX_SHIFT;
+
+ /*
+ * Use the weakest shareability/memory types, so they are
+ * overridden by the ttbcr/pte.
+ */
+ if (stage1) {
+ reg |= (CBAR_S1_BPSHCFG_NSH << CBAR_S1_BPSHCFG_SHIFT) |
+ (CBAR_S1_MEMATTR_WB << CBAR_S1_MEMATTR_SHIFT);
+ } else if (!(smmu->features & ARM_SMMU_FEAT_VMID16)) {
+ /* 8-bit VMIDs live in CBAR */
+ reg |= cfg->vmid << CBAR_VMID_SHIFT;
+ }
+ writel_relaxed(reg, gr1_base + ARM_SMMU_GR1_CBAR(idx));
+
+ /*
+ * TTBCR
+ * We must write this before the TTBRs, since it determines the
+ * access behaviour of some fields (in particular, ASID[15:8]).
+ */
+ if (stage1 && smmu->version > ARM_SMMU_V1)
+ writel_relaxed(cb->tcr[1], cb_base + ARM_SMMU_CB_TTBCR2);
+ writel_relaxed(cb->tcr[0], cb_base + ARM_SMMU_CB_TTBCR);
+
+ /* TTBRs */
+ if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH32_S) {
+ writel_relaxed(cfg->asid, cb_base + ARM_SMMU_CB_CONTEXTIDR);
+ writel_relaxed(cb->ttbr[0], cb_base + ARM_SMMU_CB_TTBR0);
+ writel_relaxed(cb->ttbr[1], cb_base + ARM_SMMU_CB_TTBR1);
+ } else {
+ writeq_relaxed(cb->ttbr[0], cb_base + ARM_SMMU_CB_TTBR0);
+ if (stage1)
+ writeq_relaxed(cb->ttbr[1],
+ cb_base + ARM_SMMU_CB_TTBR1);
+ }
+
+ /* MAIRs (stage-1 only) */
+ if (stage1) {
+ writel_relaxed(cb->mair[0], cb_base + ARM_SMMU_CB_S1_MAIR0);
+ writel_relaxed(cb->mair[1], cb_base + ARM_SMMU_CB_S1_MAIR1);
+ }
+
+ /* SCTLR */
+ reg = SCTLR_CFIE | SCTLR_CFRE | SCTLR_AFE | SCTLR_TRE | SCTLR_M;
+ if (stage1)
+ reg |= SCTLR_S1_ASIDPNE;
+ if (IS_ENABLED(CONFIG_CPU_BIG_ENDIAN))
+ reg |= SCTLR_E;
+
+ writel_relaxed(reg, cb_base + ARM_SMMU_CB_SCTLR);
+}
+
+static int arm_smmu_init_domain_context(struct iommu_domain *domain,
+ struct arm_smmu_device *smmu)
+{
+ int irq, start, ret = 0;
+ unsigned long ias, oas;
+ struct io_pgtable_ops *pgtbl_ops;
+ struct io_pgtable_cfg pgtbl_cfg;
+ enum io_pgtable_fmt fmt;
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
+
+ mutex_lock(&smmu_domain->init_mutex);
+ if (smmu_domain->smmu)
+ goto out_unlock;
+
+ if (domain->type == IOMMU_DOMAIN_IDENTITY) {
+ smmu_domain->stage = ARM_SMMU_DOMAIN_BYPASS;
+ smmu_domain->smmu = smmu;
+ goto out_unlock;
+ }
+
+ /*
+ * Mapping the requested stage onto what we support is surprisingly
+ * complicated, mainly because the spec allows S1+S2 SMMUs without
+ * support for nested translation. That means we end up with the
+ * following table:
+ *
+ * Requested Supported Actual
+ * S1 N S1
+ * S1 S1+S2 S1
+ * S1 S2 S2
+ * S1 S1 S1
+ * N N N
+ * N S1+S2 S2
+ * N S2 S2
+ * N S1 S1
+ *
+ * Note that you can't actually request stage-2 mappings.
+ */
+ if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S1))
+ smmu_domain->stage = ARM_SMMU_DOMAIN_S2;
+ if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S2))
+ smmu_domain->stage = ARM_SMMU_DOMAIN_S1;
+
+ /*
+ * Choosing a suitable context format is even more fiddly. Until we
+ * grow some way for the caller to express a preference, and/or move
+ * the decision into the io-pgtable code where it arguably belongs,
+ * just aim for the closest thing to the rest of the system, and hope
+ * that the hardware isn't esoteric enough that we can't assume AArch64
+ * support to be a superset of AArch32 support...
+ */
+ if (smmu->features & ARM_SMMU_FEAT_FMT_AARCH32_L)
+ cfg->fmt = ARM_SMMU_CTX_FMT_AARCH32_L;
+ if (IS_ENABLED(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) &&
+ !IS_ENABLED(CONFIG_64BIT) && !IS_ENABLED(CONFIG_ARM_LPAE) &&
+ (smmu->features & ARM_SMMU_FEAT_FMT_AARCH32_S) &&
+ (smmu_domain->stage == ARM_SMMU_DOMAIN_S1))
+ cfg->fmt = ARM_SMMU_CTX_FMT_AARCH32_S;
+ if ((IS_ENABLED(CONFIG_64BIT) || cfg->fmt == ARM_SMMU_CTX_FMT_NONE) &&
+ (smmu->features & (ARM_SMMU_FEAT_FMT_AARCH64_64K |
+ ARM_SMMU_FEAT_FMT_AARCH64_16K |
+ ARM_SMMU_FEAT_FMT_AARCH64_4K)))
+ cfg->fmt = ARM_SMMU_CTX_FMT_AARCH64;
+
+ if (cfg->fmt == ARM_SMMU_CTX_FMT_NONE) {
+ ret = -EINVAL;
+ goto out_unlock;
+ }
+
+ switch (smmu_domain->stage) {
+ case ARM_SMMU_DOMAIN_S1:
+ cfg->cbar = CBAR_TYPE_S1_TRANS_S2_BYPASS;
+ start = smmu->num_s2_context_banks;
+ ias = smmu->va_size;
+ oas = smmu->ipa_size;
+ if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH64) {
+ fmt = ARM_64_LPAE_S1;
+ } else if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH32_L) {
+ fmt = ARM_32_LPAE_S1;
+ ias = min(ias, 32UL);
+ oas = min(oas, 40UL);
+ } else {
+ fmt = ARM_V7S;
+ ias = min(ias, 32UL);
+ oas = min(oas, 32UL);
+ }
+ smmu_domain->tlb_ops = &arm_smmu_s1_tlb_ops;
+ break;
+ case ARM_SMMU_DOMAIN_NESTED:
+ /*
+ * We will likely want to change this if/when KVM gets
+ * involved.
+ */
+ case ARM_SMMU_DOMAIN_S2:
+ cfg->cbar = CBAR_TYPE_S2_TRANS;
+ start = 0;
+ ias = smmu->ipa_size;
+ oas = smmu->pa_size;
+ if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH64) {
+ fmt = ARM_64_LPAE_S2;
+ } else {
+ fmt = ARM_32_LPAE_S2;
+ ias = min(ias, 40UL);
+ oas = min(oas, 40UL);
+ }
+ if (smmu->version == ARM_SMMU_V2)
+ smmu_domain->tlb_ops = &arm_smmu_s2_tlb_ops_v2;
+ else
+ smmu_domain->tlb_ops = &arm_smmu_s2_tlb_ops_v1;
+ break;
+ default:
+ ret = -EINVAL;
+ goto out_unlock;
+ }
+ ret = __arm_smmu_alloc_bitmap(smmu->context_map, start,
+ smmu->num_context_banks);
+ if (ret < 0)
+ goto out_unlock;
+
+ cfg->cbndx = ret;
+ if (smmu->version < ARM_SMMU_V2) {
+ cfg->irptndx = atomic_inc_return(&smmu->irptndx);
+ cfg->irptndx %= smmu->num_context_irqs;
+ } else {
+ cfg->irptndx = cfg->cbndx;
+ }
+
+ if (smmu_domain->stage == ARM_SMMU_DOMAIN_S2)
+ cfg->vmid = cfg->cbndx + 1 + smmu->cavium_id_base;
+ else
+ cfg->asid = cfg->cbndx + smmu->cavium_id_base;
+
+ pgtbl_cfg = (struct io_pgtable_cfg) {
+ .pgsize_bitmap = smmu->pgsize_bitmap,
+ .ias = ias,
+ .oas = oas,
+ .tlb = smmu_domain->tlb_ops,
+ .iommu_dev = smmu->dev,
+ };
+
+ if (smmu->features & ARM_SMMU_FEAT_COHERENT_WALK)
+ pgtbl_cfg.quirks = IO_PGTABLE_QUIRK_NO_DMA;
+
+ if (smmu_domain->non_strict)
+ pgtbl_cfg.quirks |= IO_PGTABLE_QUIRK_NON_STRICT;
+
+ smmu_domain->smmu = smmu;
+ pgtbl_ops = alloc_io_pgtable_ops(fmt, &pgtbl_cfg, smmu_domain);
+ if (!pgtbl_ops) {
+ ret = -ENOMEM;
+ goto out_clear_smmu;
+ }
+
+ /* Update the domain's page sizes to reflect the page table format */
+ domain->pgsize_bitmap = pgtbl_cfg.pgsize_bitmap;
+ domain->geometry.aperture_end = (1UL << ias) - 1;
+ domain->geometry.force_aperture = true;
+
+ /* Initialise the context bank with our page table cfg */
+ arm_smmu_init_context_bank(smmu_domain, &pgtbl_cfg);
+ arm_smmu_write_context_bank(smmu, cfg->cbndx);
+
+ /*
+ * Request context fault interrupt. Do this last to avoid the
+ * handler seeing a half-initialised domain state.
+ */
+ irq = smmu->irqs[smmu->num_global_irqs + cfg->irptndx];
+ ret = devm_request_irq(smmu->dev, irq, arm_smmu_context_fault,
+ IRQF_SHARED, "arm-smmu-context-fault", domain);
+ if (ret < 0) {
+ dev_err(smmu->dev, "failed to request context IRQ %d (%u)\n",
+ cfg->irptndx, irq);
+ cfg->irptndx = INVALID_IRPTNDX;
+ }
+
+ mutex_unlock(&smmu_domain->init_mutex);
+
+ /* Publish page table ops for map/unmap */
+ smmu_domain->pgtbl_ops = pgtbl_ops;
+ return 0;
+
+out_clear_smmu:
+ smmu_domain->smmu = NULL;
+out_unlock:
+ mutex_unlock(&smmu_domain->init_mutex);
+ return ret;
+}
+
+static void arm_smmu_destroy_domain_context(struct iommu_domain *domain)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
+ struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
+ int irq;
+
+ if (!smmu || domain->type == IOMMU_DOMAIN_IDENTITY)
+ return;
+
+ /*
+ * Disable the context bank and free the page tables before freeing
+ * it.
+ */
+ smmu->cbs[cfg->cbndx].cfg = NULL;
+ arm_smmu_write_context_bank(smmu, cfg->cbndx);
+
+ if (cfg->irptndx != INVALID_IRPTNDX) {
+ irq = smmu->irqs[smmu->num_global_irqs + cfg->irptndx];
+ devm_free_irq(smmu->dev, irq, domain);
+ }
+
+ free_io_pgtable_ops(smmu_domain->pgtbl_ops);
+ __arm_smmu_free_bitmap(smmu->context_map, cfg->cbndx);
+}
+
+struct iommu_domain *arm_smmu_domain_alloc_common(unsigned int type,
+ bool using_legacy_binding)
+{
+ struct arm_smmu_domain *smmu_domain;
+
+ if (type != IOMMU_DOMAIN_UNMANAGED &&
+ type != IOMMU_DOMAIN_DMA &&
+ type != IOMMU_DOMAIN_IDENTITY)
+ return NULL;
+ /*
+ * Allocate the domain and initialise some of its data structures.
+ * We can't really do anything meaningful until we've added a
+ * master.
+ */
+ smmu_domain = kzalloc(sizeof(*smmu_domain), GFP_KERNEL);
+ if (!smmu_domain)
+ return NULL;
+
+ if (type == IOMMU_DOMAIN_DMA && (using_legacy_binding ||
+ iommu_get_dma_cookie(&smmu_domain->domain))) {
+ kfree(smmu_domain);
+ return NULL;
+ }
+
+ mutex_init(&smmu_domain->init_mutex);
+ spin_lock_init(&smmu_domain->cb_lock);
+
+ return &smmu_domain->domain;
+}
+
+void arm_smmu_domain_free(struct iommu_domain *domain)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+
+ /*
+ * Free the domain resources. We assume that all devices have
+ * already been detached.
+ */
+ iommu_put_dma_cookie(domain);
+ arm_smmu_destroy_domain_context(domain);
+ kfree(smmu_domain);
+}
+
+static void arm_smmu_write_smr(struct arm_smmu_device *smmu, int idx)
+{
+ struct arm_smmu_smr *smr = smmu->smrs + idx;
+ u32 reg = smr->id << SMR_ID_SHIFT | smr->mask << SMR_MASK_SHIFT;
+
+ if (!(smmu->features & ARM_SMMU_FEAT_EXIDS) && smr->valid)
+ reg |= SMR_VALID;
+ writel_relaxed(reg, ARM_SMMU_GR0(smmu) + ARM_SMMU_GR0_SMR(idx));
+}
+
+static void arm_smmu_write_s2cr(struct arm_smmu_device *smmu, int idx)
+{
+ struct arm_smmu_s2cr *s2cr = smmu->s2crs + idx;
+ u32 reg = (s2cr->type & S2CR_TYPE_MASK) << S2CR_TYPE_SHIFT |
+ (s2cr->cbndx & S2CR_CBNDX_MASK) << S2CR_CBNDX_SHIFT |
+ (s2cr->privcfg & S2CR_PRIVCFG_MASK) << S2CR_PRIVCFG_SHIFT;
+
+ if (smmu->features & ARM_SMMU_FEAT_EXIDS && smmu->smrs &&
+ smmu->smrs[idx].valid)
+ reg |= S2CR_EXIDVALID;
+ writel_relaxed(reg, ARM_SMMU_GR0(smmu) + ARM_SMMU_GR0_S2CR(idx));
+}
+
+static void arm_smmu_write_sme(struct arm_smmu_device *smmu, int idx)
+{
+ arm_smmu_write_s2cr(smmu, idx);
+ if (smmu->smrs)
+ arm_smmu_write_smr(smmu, idx);
+}
+
+/*
+ * The width of SMR's mask field depends on sCR0_EXIDENABLE, so this function
+ * should be called after sCR0 is written.
+ */
+void arm_smmu_test_smr_masks(struct arm_smmu_device *smmu)
+{
+ void __iomem *gr0_base = ARM_SMMU_GR0(smmu);
+ u32 smr;
+
+ if (!smmu->smrs)
+ return;
+
+ /*
+ * SMR.ID bits may not be preserved if the corresponding MASK
+ * bits are set, so check each one separately. We can reject
+ * masters later if they try to claim IDs outside these masks.
+ */
+ smr = smmu->streamid_mask << SMR_ID_SHIFT;
+ writel_relaxed(smr, gr0_base + ARM_SMMU_GR0_SMR(0));
+ smr = readl_relaxed(gr0_base + ARM_SMMU_GR0_SMR(0));
+ smmu->streamid_mask = smr >> SMR_ID_SHIFT;
+
+ smr = smmu->streamid_mask << SMR_MASK_SHIFT;
+ writel_relaxed(smr, gr0_base + ARM_SMMU_GR0_SMR(0));
+ smr = readl_relaxed(gr0_base + ARM_SMMU_GR0_SMR(0));
+ smmu->smr_mask_mask = smr >> SMR_MASK_SHIFT;
+}
+
+static int arm_smmu_find_sme(struct arm_smmu_device *smmu, u16 id, u16 mask)
+{
+ struct arm_smmu_smr *smrs = smmu->smrs;
+ int i, free_idx = -ENOSPC;
+
+ /* Stream indexing is blissfully easy */
+ if (!smrs)
+ return id;
+
+ /* Validating SMRs is... less so */
+ for (i = 0; i < smmu->num_mapping_groups; ++i) {
+ if (!smrs[i].valid) {
+ /*
+ * Note the first free entry we come across, which
+ * we'll claim in the end if nothing else matches.
+ */
+ if (free_idx < 0)
+ free_idx = i;
+ continue;
+ }
+ /*
+ * If the new entry is _entirely_ matched by an existing entry,
+ * then reuse that, with the guarantee that there also cannot
+ * be any subsequent conflicting entries. In normal use we'd
+ * expect simply identical entries for this case, but there's
+ * no harm in accommodating the generalisation.
+ */
+ if ((mask & smrs[i].mask) == mask &&
+ !((id ^ smrs[i].id) & ~smrs[i].mask))
+ return i;
+ /*
+ * If the new entry has any other overlap with an existing one,
+ * though, then there always exists at least one stream ID
+ * which would cause a conflict, and we can't allow that risk.
+ */
+ if (!((id ^ smrs[i].id) & ~(smrs[i].mask | mask)))
+ return -EINVAL;
+ }
+
+ return free_idx;
+}
+
+static bool arm_smmu_free_sme(struct arm_smmu_device *smmu, int idx)
+{
+ if (--smmu->s2crs[idx].count)
+ return false;
+
+ smmu->s2crs[idx] = s2cr_init_val;
+ if (smmu->smrs)
+ smmu->smrs[idx].valid = false;
+
+ return true;
+}
+
+static int arm_smmu_master_alloc_smes(struct device *dev)
+{
+ struct iommu_fwspec *fwspec = dev->iommu_fwspec;
+ struct arm_smmu_master_cfg *cfg = fwspec->iommu_priv;
+ struct arm_smmu_device *smmu = cfg->smmu;
+ struct arm_smmu_smr *smrs = smmu->smrs;
+ struct iommu_group *group;
+ int i, idx, ret;
+
+ mutex_lock(&smmu->stream_map_mutex);
+ /* Figure out a viable stream map entry allocation */
+ for_each_cfg_sme(fwspec, i, idx) {
+ u16 sid = fwspec->ids[i];
+ u16 mask = fwspec->ids[i] >> SMR_MASK_SHIFT;
+
+ if (idx != INVALID_SMENDX) {
+ ret = -EEXIST;
+ goto out_err;
+ }
+
+ ret = arm_smmu_find_sme(smmu, sid, mask);
+ if (ret < 0)
+ goto out_err;
+
+ idx = ret;
+ if (smrs && smmu->s2crs[idx].count == 0) {
+ smrs[idx].id = sid;
+ smrs[idx].mask = mask;
+ smrs[idx].valid = true;
+ }
+ smmu->s2crs[idx].count++;
+ cfg->smendx[i] = (s16)idx;
+ }
+
+ group = iommu_group_get_for_dev(dev);
+ if (!group)
+ group = ERR_PTR(-ENOMEM);
+ if (IS_ERR(group)) {
+ ret = PTR_ERR(group);
+ goto out_err;
+ }
+ iommu_group_put(group);
+
+ /* It worked! Now, poke the actual hardware */
+ for_each_cfg_sme(fwspec, i, idx) {
+ arm_smmu_write_sme(smmu, idx);
+ smmu->s2crs[idx].group = group;
+ }
+
+ mutex_unlock(&smmu->stream_map_mutex);
+ return 0;
+
+out_err:
+ while (i--) {
+ arm_smmu_free_sme(smmu, cfg->smendx[i]);
+ cfg->smendx[i] = INVALID_SMENDX;
+ }
+ mutex_unlock(&smmu->stream_map_mutex);
+ return ret;
+}
+
+static void arm_smmu_master_free_smes(struct iommu_fwspec *fwspec)
+{
+ struct arm_smmu_device *smmu = fwspec_smmu(fwspec);
+ struct arm_smmu_master_cfg *cfg = fwspec->iommu_priv;
+ int i, idx;
+
+ mutex_lock(&smmu->stream_map_mutex);
+ for_each_cfg_sme(fwspec, i, idx) {
+ if (arm_smmu_free_sme(smmu, idx))
+ arm_smmu_write_sme(smmu, idx);
+ cfg->smendx[i] = INVALID_SMENDX;
+ }
+ mutex_unlock(&smmu->stream_map_mutex);
+}
+
+static int arm_smmu_domain_add_master(struct arm_smmu_domain *smmu_domain,
+ struct iommu_fwspec *fwspec)
+{
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
+ struct arm_smmu_s2cr *s2cr = smmu->s2crs;
+ u8 cbndx = smmu_domain->cfg.cbndx;
+ enum arm_smmu_s2cr_type type;
+ int i, idx;
+
+ if (smmu_domain->stage == ARM_SMMU_DOMAIN_BYPASS)
+ type = S2CR_TYPE_BYPASS;
+ else
+ type = S2CR_TYPE_TRANS;
+
+ for_each_cfg_sme(fwspec, i, idx) {
+ if (type == s2cr[idx].type && cbndx == s2cr[idx].cbndx)
+ continue;
+
+ s2cr[idx].type = type;
+ s2cr[idx].privcfg = S2CR_PRIVCFG_DEFAULT;
+ s2cr[idx].cbndx = cbndx;
+ arm_smmu_write_s2cr(smmu, idx);
+ }
+ return 0;
+}
+
+int arm_smmu_attach_dev_common(struct iommu_domain *domain,
+ struct device *dev, struct iommu_ops *ops)
+{
+ int ret;
+ struct iommu_fwspec *fwspec = dev->iommu_fwspec;
+ struct arm_smmu_device *smmu;
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+
+ if (!fwspec || fwspec->ops != ops) {
+ dev_err(dev, "cannot attach to SMMU, is it on the same bus?\n");
+ return -ENXIO;
+ }
+
+ /*
+ * FIXME: The arch/arm DMA API code tries to attach devices to its own
+ * domains between of_xlate() and add_device() - we have no way to cope
+ * with that, so until ARM gets converted to rely on groups and default
+ * domains, just say no (but more politely than by dereferencing NULL).
+ * This should be at least a WARN_ON once that's sorted.
+ */
+ if (!fwspec->iommu_priv)
+ return -ENODEV;
+
+ smmu = fwspec_smmu(fwspec);
+ /* Ensure that the domain is finalised */
+ ret = arm_smmu_init_domain_context(domain, smmu);
+ if (ret < 0)
+ return ret;
+
+ /*
+ * Sanity check the domain. We don't support domains across
+ * different SMMUs.
+ */
+ if (smmu_domain->smmu != smmu) {
+ dev_err(dev,
+ "cannot attach to SMMU %s whilst already attached to domain on SMMU %s\n",
+ dev_name(smmu_domain->smmu->dev), dev_name(smmu->dev));
+ return -EINVAL;
+ }
+
+ /* Looks ok, so add the device to the domain */
+ return arm_smmu_domain_add_master(smmu_domain, fwspec);
+}
+
+int arm_smmu_map(struct iommu_domain *domain, unsigned long iova,
+ phys_addr_t paddr, size_t size, int prot)
+{
+ struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops;
+
+ if (!ops)
+ return -ENODEV;
+
+ return ops->map(ops, iova, paddr, size, prot);
+}
+
+size_t arm_smmu_unmap(struct iommu_domain *domain, unsigned long iova,
+ size_t size)
+{
+ struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops;
+
+ if (!ops)
+ return 0;
+
+ return ops->unmap(ops, iova, size);
+}
+
+void arm_smmu_flush_iotlb_all(struct iommu_domain *domain)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+
+ if (smmu_domain->tlb_ops)
+ smmu_domain->tlb_ops->tlb_flush_all(smmu_domain);
+}
+
+void arm_smmu_iotlb_sync(struct iommu_domain *domain)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+
+ if (smmu_domain->tlb_ops)
+ smmu_domain->tlb_ops->tlb_sync(smmu_domain);
+}
+
+static phys_addr_t arm_smmu_iova_to_phys_hard(struct iommu_domain *domain,
+ dma_addr_t iova)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
+ struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
+ struct io_pgtable_ops *ops = smmu_domain->pgtbl_ops;
+ struct device *dev = smmu->dev;
+ void __iomem *cb_base;
+ u32 tmp;
+ u64 phys;
+ unsigned long va, flags;
+
+ cb_base = ARM_SMMU_CB(smmu, cfg->cbndx);
+
+ spin_lock_irqsave(&smmu_domain->cb_lock, flags);
+ /* ATS1 registers can only be written atomically */
+ va = iova & ~0xfffUL;
+ if (smmu->version == ARM_SMMU_V2)
+ smmu_write_atomic_lq(va, cb_base + ARM_SMMU_CB_ATS1PR);
+ else /* Register is only 32-bit in v1 */
+ writel_relaxed(va, cb_base + ARM_SMMU_CB_ATS1PR);
+
+ if (readl_poll_timeout_atomic(cb_base + ARM_SMMU_CB_ATSR, tmp,
+ !(tmp & ATSR_ACTIVE), 5, 50)) {
+ spin_unlock_irqrestore(&smmu_domain->cb_lock, flags);
+ dev_err(dev,
+ "iova to phys timed out on %pad. Falling back to software table walk.\n",
+ &iova);
+ return ops->iova_to_phys(ops, iova);
+ }
+
+ phys = readq_relaxed(cb_base + ARM_SMMU_CB_PAR);
+ spin_unlock_irqrestore(&smmu_domain->cb_lock, flags);
+ if (phys & CB_PAR_F) {
+ dev_err(dev, "translation fault!\n");
+ dev_err(dev, "PAR = 0x%llx\n", phys);
+ return 0;
+ }
+
+ return (phys & GENMASK_ULL(39, 12)) | (iova & 0xfff);
+}
+
+phys_addr_t arm_smmu_iova_to_phys(struct iommu_domain *domain,
+ dma_addr_t iova)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct io_pgtable_ops *ops = smmu_domain->pgtbl_ops;
+
+ if (domain->type == IOMMU_DOMAIN_IDENTITY)
+ return iova;
+
+ if (!ops)
+ return 0;
+
+ if (smmu_domain->smmu->features & ARM_SMMU_FEAT_TRANS_OPS &&
+ smmu_domain->stage == ARM_SMMU_DOMAIN_S1)
+ return arm_smmu_iova_to_phys_hard(domain, iova);
+
+ return ops->iova_to_phys(ops, iova);
+}
+
+bool arm_smmu_capable(enum iommu_cap cap)
+{
+ switch (cap) {
+ case IOMMU_CAP_CACHE_COHERENCY:
+ /*
+ * Return true here as the SMMU can always send out coherent
+ * requests.
+ */
+ return true;
+ case IOMMU_CAP_NOEXEC:
+ return true;
+ default:
+ return false;
+ }
+}
+
+int arm_smmu_add_device_common(struct device *dev, struct arm_smmu_device *smmu)
+{
+ struct arm_smmu_master_cfg *cfg;
+ struct iommu_fwspec *fwspec = dev->iommu_fwspec;
+ int i, ret = -EINVAL;
+
+ for (i = 0; i < fwspec->num_ids; i++) {
+ u16 sid = fwspec->ids[i];
+ u16 mask = fwspec->ids[i] >> SMR_MASK_SHIFT;
+
+ if (sid & ~smmu->streamid_mask) {
+ dev_err(dev, "stream ID 0x%x out of range for SMMU (0x%x)\n",
+ sid, smmu->streamid_mask);
+ goto out_free;
+ }
+ if (mask & ~smmu->smr_mask_mask) {
+ dev_err(dev, "SMR mask 0x%x out of range for SMMU (0x%x)\n",
+ mask, smmu->smr_mask_mask);
+ goto out_free;
+ }
+ }
+
+ ret = -ENOMEM;
+ cfg = kzalloc(offsetof(struct arm_smmu_master_cfg, smendx[i]),
+ GFP_KERNEL);
+ if (!cfg)
+ goto out_free;
+
+ cfg->smmu = smmu;
+ fwspec->iommu_priv = cfg;
+ while (i--)
+ cfg->smendx[i] = INVALID_SMENDX;
+
+ ret = arm_smmu_master_alloc_smes(dev);
+ if (ret)
+ goto out_cfg_free;
+
+ iommu_device_link(&smmu->iommu, dev);
+
+ return 0;
+
+out_cfg_free:
+ kfree(cfg);
+out_free:
+ iommu_fwspec_free(dev);
+ return ret;
+}
+
+void arm_smmu_remove_device_common(struct device *dev, struct iommu_ops *ops)
+{
+ struct iommu_fwspec *fwspec = dev->iommu_fwspec;
+ struct arm_smmu_master_cfg *cfg;
+ struct arm_smmu_device *smmu;
+
+ if (!fwspec || fwspec->ops != ops)
+ return;
+
+ cfg = fwspec->iommu_priv;
+ smmu = cfg->smmu;
+
+ iommu_device_unlink(&smmu->iommu, dev);
+ arm_smmu_master_free_smes(fwspec);
+ iommu_group_remove_device(dev);
+ kfree(fwspec->iommu_priv);
+ iommu_fwspec_free(dev);
+}
+
+struct iommu_group *arm_smmu_device_group(struct device *dev)
+{
+ struct iommu_fwspec *fwspec = dev->iommu_fwspec;
+ struct arm_smmu_device *smmu = fwspec_smmu(fwspec);
+ struct iommu_group *group = NULL;
+ int i, idx;
+
+ for_each_cfg_sme(fwspec, i, idx) {
+ if (group && smmu->s2crs[idx].group &&
+ group != smmu->s2crs[idx].group)
+ return ERR_PTR(-EINVAL);
+
+ group = smmu->s2crs[idx].group;
+ }
+
+ if (group)
+ return iommu_group_ref_get(group);
+
+ if (dev_is_pci(dev))
+ group = pci_device_group(dev);
+ else if (dev_is_fsl_mc(dev))
+ group = fsl_mc_device_group(dev);
+ else
+ group = generic_device_group(dev);
+
+ return group;
+}
+
+int arm_smmu_domain_get_attr(struct iommu_domain *domain,
+ enum iommu_attr attr, void *data)
+{
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+
+ switch (domain->type) {
+ case IOMMU_DOMAIN_UNMANAGED:
+ switch (attr) {
+ case DOMAIN_ATTR_NESTING:
+ *(int *)data = (smmu_domain->stage ==
+ ARM_SMMU_DOMAIN_NESTED);
+ return 0;
+ default:
+ return -ENODEV;
+ }
+ break;
+ case IOMMU_DOMAIN_DMA:
+ switch (attr) {
+ case DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE:
+ *(int *)data = smmu_domain->non_strict;
+ return 0;
+ default:
+ return -ENODEV;
+ }
+ break;
+ default:
+ return -EINVAL;
+ }
+}
+
+int arm_smmu_domain_set_attr(struct iommu_domain *domain,
+ enum iommu_attr attr, void *data)
+{
+ int ret = 0;
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+
+ mutex_lock(&smmu_domain->init_mutex);
+
+ switch (domain->type) {
+ case IOMMU_DOMAIN_UNMANAGED:
+ switch (attr) {
+ case DOMAIN_ATTR_NESTING:
+ if (smmu_domain->smmu) {
+ ret = -EPERM;
+ goto out_unlock;
+ }
+
+ if (*(int *)data)
+ smmu_domain->stage = ARM_SMMU_DOMAIN_NESTED;
+ else
+ smmu_domain->stage = ARM_SMMU_DOMAIN_S1;
+ break;
+ default:
+ ret = -ENODEV;
+ }
+ break;
+ case IOMMU_DOMAIN_DMA:
+ switch (attr) {
+ case DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE:
+ smmu_domain->non_strict = *(int *)data;
+ break;
+ default:
+ ret = -ENODEV;
+ }
+ break;
+ default:
+ ret = -EINVAL;
+ }
+out_unlock:
+ mutex_unlock(&smmu_domain->init_mutex);
+ return ret;
+}
+
+int arm_smmu_of_xlate(struct device *dev, struct of_phandle_args *args)
+{
+ u32 mask, fwid = 0;
+
+ if (args->args_count > 0)
+ fwid |= (u16)args->args[0];
+
+ if (args->args_count > 1)
+ fwid |= (u16)args->args[1] << SMR_MASK_SHIFT;
+ else if (!of_property_read_u32(args->np, "stream-match-mask", &mask))
+ fwid |= (u16)mask << SMR_MASK_SHIFT;
+
+ return iommu_fwspec_add_ids(dev, &fwid, 1);
+}
+
+void arm_smmu_get_resv_regions(struct device *dev,
+ struct list_head *head)
+{
+ struct iommu_resv_region *region;
+ int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
+
+ region = iommu_alloc_resv_region(MSI_IOVA_BASE, MSI_IOVA_LENGTH,
+ prot, IOMMU_RESV_SW_MSI);
+ if (!region)
+ return;
+
+ list_add_tail(®ion->list, head);
+
+ iommu_dma_get_resv_regions(dev, head);
+}
+
+void arm_smmu_put_resv_regions(struct device *dev,
+ struct list_head *head)
+{
+ struct iommu_resv_region *entry, *next;
+
+ list_for_each_entry_safe(entry, next, head, list)
+ kfree(entry);
+}
+
+void arm_smmu_device_reset(struct arm_smmu_device *smmu)
+{
+ void __iomem *gr0_base = ARM_SMMU_GR0(smmu);
+ int i;
+ u32 reg, major;
+
+ /* clear global FSR */
+ reg = readl_relaxed(ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sGFSR);
+ writel(reg, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sGFSR);
+
+ /*
+ * Reset stream mapping groups: Initial values mark all SMRn as
+ * invalid and all S2CRn as bypass unless overridden.
+ */
+ for (i = 0; i < smmu->num_mapping_groups; ++i)
+ arm_smmu_write_sme(smmu, i);
+
+ if (smmu->model == ARM_MMU500) {
+ /*
+ * Before clearing ARM_MMU500_ACTLR_CPRE, need to
+ * clear CACHE_LOCK bit of ACR first. And, CACHE_LOCK
+ * bit is only present in MMU-500r2 onwards.
+ */
+ reg = readl_relaxed(gr0_base + ARM_SMMU_GR0_ID7);
+ major = (reg >> ID7_MAJOR_SHIFT) & ID7_MAJOR_MASK;
+ reg = readl_relaxed(gr0_base + ARM_SMMU_GR0_sACR);
+ if (major >= 2)
+ reg &= ~ARM_MMU500_ACR_CACHE_LOCK;
+ /*
+ * Allow unmatched Stream IDs to allocate bypass
+ * TLB entries for reduced latency.
+ */
+ reg |= ARM_MMU500_ACR_SMTNMB_TLBEN | ARM_MMU500_ACR_S2CRB_TLBEN;
+ writel_relaxed(reg, gr0_base + ARM_SMMU_GR0_sACR);
+ }
+
+ /* Make sure all context banks are disabled and clear CB_FSR */
+ for (i = 0; i < smmu->num_context_banks; ++i) {
+ void __iomem *cb_base = ARM_SMMU_CB(smmu, i);
+
+ arm_smmu_write_context_bank(smmu, i);
+ writel_relaxed(FSR_FAULT, cb_base + ARM_SMMU_CB_FSR);
+ /*
+ * Disable MMU-500's not-particularly-beneficial next-page
+ * prefetcher for the sake of errata #841119 and #826419.
+ */
+ if (smmu->model == ARM_MMU500) {
+ reg = readl_relaxed(cb_base + ARM_SMMU_CB_ACTLR);
+ reg &= ~ARM_MMU500_ACTLR_CPRE;
+ writel_relaxed(reg, cb_base + ARM_SMMU_CB_ACTLR);
+ }
+ }
+
+ /* Invalidate the TLB, just in case */
+ writel_relaxed(0, gr0_base + ARM_SMMU_GR0_TLBIALLH);
+ writel_relaxed(0, gr0_base + ARM_SMMU_GR0_TLBIALLNSNH);
+
+ reg = readl_relaxed(ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
+
+ /* Enable fault reporting */
+ reg |= (sCR0_GFRE | sCR0_GFIE | sCR0_GCFGFRE | sCR0_GCFGFIE);
+
+ /* Disable TLB broadcasting. */
+ reg |= (sCR0_VMIDPNE | sCR0_PTM);
+
+ /* Enable client access, handling unmatched streams as appropriate */
+ reg &= ~sCR0_CLIENTPD;
+ if (smmu->disable_bypass)
+ reg |= sCR0_USFCFG;
+ else
+ reg &= ~sCR0_USFCFG;
+
+ /* Disable forced broadcasting */
+ reg &= ~sCR0_FB;
+
+ /* Don't upgrade barriers */
+ reg &= ~(sCR0_BSU_MASK << sCR0_BSU_SHIFT);
+
+ if (smmu->features & ARM_SMMU_FEAT_VMID16)
+ reg |= sCR0_VMID16EN;
+
+ if (smmu->features & ARM_SMMU_FEAT_EXIDS)
+ reg |= sCR0_EXIDENABLE;
+
+ /* Push the button */
+ arm_smmu_tlb_sync_global(smmu);
+ writel(reg, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
+}
+
+static int arm_smmu_id_size_to_bits(int size)
+{
+ switch (size) {
+ case 0:
+ return 32;
+ case 1:
+ return 36;
+ case 2:
+ return 40;
+ case 3:
+ return 42;
+ case 4:
+ return 44;
+ case 5:
+ default:
+ return 48;
+ }
+}
+
+int arm_smmu_device_cfg_probe_common(struct arm_smmu_device *smmu,
+ struct iommu_ops *ops, atomic_t *context_count, int force_stage)
+{
+ unsigned long size;
+ void __iomem *gr0_base = ARM_SMMU_GR0(smmu);
+ u32 id;
+ bool cttw_reg, cttw_fw = smmu->features & ARM_SMMU_FEAT_COHERENT_WALK;
+ int i;
+
+ dev_notice(smmu->dev, "probing hardware configuration...\n");
+ dev_notice(smmu->dev, "SMMUv%d with:\n",
+ smmu->version == ARM_SMMU_V2 ? 2 : 1);
+
+ /* ID0 */
+ id = readl_relaxed(gr0_base + ARM_SMMU_GR0_ID0);
+
+ /* Restrict available stages based on module parameter */
+ if (force_stage == 1)
+ id &= ~(ID0_S2TS | ID0_NTS);
+ else if (force_stage == 2)
+ id &= ~(ID0_S1TS | ID0_NTS);
+
+ if (id & ID0_S1TS) {
+ smmu->features |= ARM_SMMU_FEAT_TRANS_S1;
+ dev_notice(smmu->dev, "\tstage 1 translation\n");
+ }
+
+ if (id & ID0_S2TS) {
+ smmu->features |= ARM_SMMU_FEAT_TRANS_S2;
+ dev_notice(smmu->dev, "\tstage 2 translation\n");
+ }
+
+ if (id & ID0_NTS) {
+ smmu->features |= ARM_SMMU_FEAT_TRANS_NESTED;
+ dev_notice(smmu->dev, "\tnested translation\n");
+ }
+
+ if (!(smmu->features &
+ (ARM_SMMU_FEAT_TRANS_S1 | ARM_SMMU_FEAT_TRANS_S2))) {
+ dev_err(smmu->dev, "\tno translation support!\n");
+ return -ENODEV;
+ }
+
+ if ((id & ID0_S1TS) &&
+ ((smmu->version < ARM_SMMU_V2) || !(id & ID0_ATOSNS))) {
+ smmu->features |= ARM_SMMU_FEAT_TRANS_OPS;
+ dev_notice(smmu->dev, "\taddress translation ops\n");
+ }
+
+ /*
+ * In order for DMA API calls to work properly, we must defer to what
+ * the FW says about coherency, regardless of what the hardware claims.
+ * Fortunately, this also opens up a workaround for systems where the
+ * ID register value has ended up configured incorrectly.
+ */
+ cttw_reg = !!(id & ID0_CTTW);
+ if (cttw_fw || cttw_reg)
+ dev_notice(smmu->dev, "\t%scoherent table walk\n",
+ cttw_fw ? "" : "non-");
+ if (cttw_fw != cttw_reg)
+ dev_notice(smmu->dev,
+ "\t(IDR0.CTTW overridden by FW configuration)\n");
+
+ /* Max. number of entries we have for stream matching/indexing */
+ if (smmu->version == ARM_SMMU_V2 && id & ID0_EXIDS) {
+ smmu->features |= ARM_SMMU_FEAT_EXIDS;
+ size = 1 << 16;
+ } else {
+ size = 1 << ((id >> ID0_NUMSIDB_SHIFT) & ID0_NUMSIDB_MASK);
+ }
+ smmu->streamid_mask = size - 1;
+ if (id & ID0_SMS) {
+ smmu->features |= ARM_SMMU_FEAT_STREAM_MATCH;
+ size = (id >> ID0_NUMSMRG_SHIFT) & ID0_NUMSMRG_MASK;
+ if (size == 0) {
+ dev_err(smmu->dev,
+ "stream-matching supported, but no SMRs present!\n");
+ return -ENODEV;
+ }
+
+ /* Zero-initialised to mark as invalid */
+ smmu->smrs = devm_kcalloc(smmu->dev, size, sizeof(*smmu->smrs),
+ GFP_KERNEL);
+ if (!smmu->smrs)
+ return -ENOMEM;
+
+ dev_notice(smmu->dev,
+ "\tstream matching with %lu register groups", size);
+ }
+ /* s2cr->type == 0 means translation, so initialise explicitly */
+ smmu->s2crs = devm_kmalloc_array(smmu->dev, size, sizeof(*smmu->s2crs),
+ GFP_KERNEL);
+ if (!smmu->s2crs)
+ return -ENOMEM;
+ for (i = 0; i < size; i++)
+ smmu->s2crs[i] = s2cr_init_val;
+
+ smmu->num_mapping_groups = size;
+ mutex_init(&smmu->stream_map_mutex);
+ spin_lock_init(&smmu->global_sync_lock);
+
+ if (smmu->version < ARM_SMMU_V2 || !(id & ID0_PTFS_NO_AARCH32)) {
+ smmu->features |= ARM_SMMU_FEAT_FMT_AARCH32_L;
+ if (!(id & ID0_PTFS_NO_AARCH32S))
+ smmu->features |= ARM_SMMU_FEAT_FMT_AARCH32_S;
+ }
+
+ /* ID1 */
+ id = readl_relaxed(gr0_base + ARM_SMMU_GR0_ID1);
+ smmu->pgshift = (id & ID1_PAGESIZE) ? 16 : 12;
+
+ /* Check for size mismatch of SMMU address space from mapped region */
+ size = 1 << (((id >> ID1_NUMPAGENDXB_SHIFT) & ID1_NUMPAGENDXB_MASK) + 1);
+ size <<= smmu->pgshift;
+ if (smmu->cb_base != gr0_base + size)
+ dev_warn(smmu->dev,
+ "SMMU address space size (0x%lx) differs from mapped region size (0x%tx)!\n",
+ size * 2, (smmu->cb_base - gr0_base) * 2);
+
+ smmu->num_s2_context_banks = (id >> ID1_NUMS2CB_SHIFT) & ID1_NUMS2CB_MASK;
+ smmu->num_context_banks = (id >> ID1_NUMCB_SHIFT) & ID1_NUMCB_MASK;
+ if (smmu->num_s2_context_banks > smmu->num_context_banks) {
+ dev_err(smmu->dev, "impossible number of S2 context banks!\n");
+ return -ENODEV;
+ }
+ dev_notice(smmu->dev, "\t%u context banks (%u stage-2 only)\n",
+ smmu->num_context_banks, smmu->num_s2_context_banks);
+ /*
+ * Cavium CN88xx erratum #27704.
+ * Ensure ASID and VMID allocation is unique across all SMMUs in
+ * the system.
+ */
+ if (smmu->model == CAVIUM_SMMUV2) {
+ smmu->cavium_id_base = atomic_add_return(
+ smmu->num_context_banks, context_count);
+ smmu->cavium_id_base -= smmu->num_context_banks;
+ dev_notice(smmu->dev, "\tenabling workaround for Cavium erratum 27704\n");
+ }
+ smmu->cbs = devm_kcalloc(smmu->dev, smmu->num_context_banks,
+ sizeof(*smmu->cbs), GFP_KERNEL);
+ if (!smmu->cbs)
+ return -ENOMEM;
+
+ /* ID2 */
+ id = readl_relaxed(gr0_base + ARM_SMMU_GR0_ID2);
+ size = arm_smmu_id_size_to_bits((id >> ID2_IAS_SHIFT) & ID2_IAS_MASK);
+ smmu->ipa_size = size;
+
+ /* The output mask is also applied for bypass */
+ size = arm_smmu_id_size_to_bits((id >> ID2_OAS_SHIFT) & ID2_OAS_MASK);
+ smmu->pa_size = size;
+
+ if (id & ID2_VMID16)
+ smmu->features |= ARM_SMMU_FEAT_VMID16;
+
+ /*
+ * What the page table walker can address actually depends on which
+ * descriptor format is in use, but since a) we don't know that yet,
+ * and b) it can vary per context bank, this will have to do...
+ */
+ if (dma_set_mask_and_coherent(smmu->dev, DMA_BIT_MASK(size)))
+ dev_warn(smmu->dev,
+ "failed to set DMA mask for table walker\n");
+
+ if (smmu->version < ARM_SMMU_V2) {
+ smmu->va_size = smmu->ipa_size;
+ if (smmu->version == ARM_SMMU_V1_64K)
+ smmu->features |= ARM_SMMU_FEAT_FMT_AARCH64_64K;
+ } else {
+ size = (id >> ID2_UBS_SHIFT) & ID2_UBS_MASK;
+ smmu->va_size = arm_smmu_id_size_to_bits(size);
+ if (id & ID2_PTFS_4K)
+ smmu->features |= ARM_SMMU_FEAT_FMT_AARCH64_4K;
+ if (id & ID2_PTFS_16K)
+ smmu->features |= ARM_SMMU_FEAT_FMT_AARCH64_16K;
+ if (id & ID2_PTFS_64K)
+ smmu->features |= ARM_SMMU_FEAT_FMT_AARCH64_64K;
+ }
+
+ /* Now we've corralled the various formats, what'll it do? */
+ if (smmu->features & ARM_SMMU_FEAT_FMT_AARCH32_S)
+ smmu->pgsize_bitmap |= SZ_4K | SZ_64K | SZ_1M | SZ_16M;
+ if (smmu->features &
+ (ARM_SMMU_FEAT_FMT_AARCH32_L | ARM_SMMU_FEAT_FMT_AARCH64_4K))
+ smmu->pgsize_bitmap |= SZ_4K | SZ_2M | SZ_1G;
+ if (smmu->features & ARM_SMMU_FEAT_FMT_AARCH64_16K)
+ smmu->pgsize_bitmap |= SZ_16K | SZ_32M;
+ if (smmu->features & ARM_SMMU_FEAT_FMT_AARCH64_64K)
+ smmu->pgsize_bitmap |= SZ_64K | SZ_512M;
+
+ if (ops->pgsize_bitmap == -1UL)
+ ops->pgsize_bitmap = smmu->pgsize_bitmap;
+ else
+ ops->pgsize_bitmap |= smmu->pgsize_bitmap;
+ dev_notice(smmu->dev, "\tSupported page sizes: 0x%08lx\n",
+ smmu->pgsize_bitmap);
+
+
+ if (smmu->features & ARM_SMMU_FEAT_TRANS_S1)
+ dev_notice(smmu->dev, "\tStage-1: %lu-bit VA -> %lu-bit IPA\n",
+ smmu->va_size, smmu->ipa_size);
+
+ if (smmu->features & ARM_SMMU_FEAT_TRANS_S2)
+ dev_notice(smmu->dev, "\tStage-2: %lu-bit IPA -> %lu-bit PA\n",
+ smmu->ipa_size, smmu->pa_size);
+
+ return 0;
+}
+
+int arm_smmu_device_remove(struct platform_device *pdev)
+{
+ struct arm_smmu_device *smmu = platform_get_drvdata(pdev);
+
+ if (!smmu)
+ return -ENODEV;
+
+ if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS))
+ dev_err(&pdev->dev, "removing device with active domains!\n");
+
+ /* Turn the thing off */
+ writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
+ return 0;
+}
diff --git a/drivers/iommu/lib-arm-smmu.h b/drivers/iommu/lib-arm-smmu.h
new file mode 100644
index 0000000..d2f3980
--- /dev/null
+++ b/drivers/iommu/lib-arm-smmu.h
@@ -0,0 +1,161 @@
+/*
+ * Copyright (c) 2018, NVIDIA Corporation
+ * Author: Krishna Reddy <vdumpa@nvidia.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _LIB_ARM_SMMU_H
+#define _LIB_ARM_SMMU_H
+
+#include <linux/iommu.h>
+#include <linux/irqreturn.h>
+#include <linux/platform_device.h>
+#include <linux/types.h>
+
+/* Maximum number of context banks per SMMU */
+#define ARM_SMMU_MAX_CBS 128
+
+enum arm_smmu_arch_version {
+ ARM_SMMU_V1,
+ ARM_SMMU_V1_64K,
+ ARM_SMMU_V2,
+};
+
+enum arm_smmu_implementation {
+ GENERIC_SMMU,
+ ARM_MMU500,
+ CAVIUM_SMMUV2,
+};
+
+struct arm_smmu_s2cr {
+ struct iommu_group *group;
+ int count;
+ enum arm_smmu_s2cr_type type;
+ enum arm_smmu_s2cr_privcfg privcfg;
+ u8 cbndx;
+};
+
+struct arm_smmu_smr {
+ u16 mask;
+ u16 id;
+ bool valid;
+};
+
+struct arm_smmu_cb {
+ u64 ttbr[2];
+ u32 tcr[2];
+ u32 mair[2];
+ struct arm_smmu_cfg *cfg;
+};
+
+struct arm_smmu_device {
+ struct device *dev;
+
+ void __iomem *base;
+ void __iomem *cb_base;
+ /* Number of ARM SMMU's represented by arm_smmu_device
+ * Tegra194 uses two ARM SMMU's as one SMMU device.
+ */
+ u32 num_smmus;
+ /* To hold the multiple base addresses when num_smmus > 1 */
+ void __iomem **bases;
+ unsigned long pgshift;
+
+#define ARM_SMMU_FEAT_COHERENT_WALK (1 << 0)
+#define ARM_SMMU_FEAT_STREAM_MATCH (1 << 1)
+#define ARM_SMMU_FEAT_TRANS_S1 (1 << 2)
+#define ARM_SMMU_FEAT_TRANS_S2 (1 << 3)
+#define ARM_SMMU_FEAT_TRANS_NESTED (1 << 4)
+#define ARM_SMMU_FEAT_TRANS_OPS (1 << 5)
+#define ARM_SMMU_FEAT_VMID16 (1 << 6)
+#define ARM_SMMU_FEAT_FMT_AARCH64_4K (1 << 7)
+#define ARM_SMMU_FEAT_FMT_AARCH64_16K (1 << 8)
+#define ARM_SMMU_FEAT_FMT_AARCH64_64K (1 << 9)
+#define ARM_SMMU_FEAT_FMT_AARCH32_L (1 << 10)
+#define ARM_SMMU_FEAT_FMT_AARCH32_S (1 << 11)
+#define ARM_SMMU_FEAT_EXIDS (1 << 12)
+ u32 features;
+
+#define ARM_SMMU_OPT_SECURE_CFG_ACCESS (1 << 0)
+ u32 options;
+ enum arm_smmu_arch_version version;
+ enum arm_smmu_implementation model;
+
+ u32 num_context_banks;
+ u32 num_s2_context_banks;
+ DECLARE_BITMAP(context_map, ARM_SMMU_MAX_CBS);
+ struct arm_smmu_cb *cbs;
+ atomic_t irptndx;
+
+ u32 num_mapping_groups;
+ u16 streamid_mask;
+ u16 smr_mask_mask;
+ struct arm_smmu_smr *smrs;
+ struct arm_smmu_s2cr *s2crs;
+ struct mutex stream_map_mutex;
+
+ unsigned long va_size;
+ unsigned long ipa_size;
+ unsigned long pa_size;
+ unsigned long pgsize_bitmap;
+
+ u32 num_global_irqs;
+ u32 num_context_irqs;
+ unsigned int *irqs;
+
+ u32 cavium_id_base; /* Specific to Cavium */
+
+ spinlock_t global_sync_lock;
+ bool disable_bypass;
+
+ /* IOMMU core code handle */
+ struct iommu_device iommu;
+};
+
+/* Common programming functions */
+int arm_smmu_device_cfg_probe_common(
+ struct arm_smmu_device *smmu, struct iommu_ops *ops,
+ atomic_t *cavium_smmu_context_count, int force_stage);
+irqreturn_t arm_smmu_global_fault(int irq, void *dev);
+int arm_smmu_device_remove(struct platform_device *pdev);
+void arm_smmu_device_reset(struct arm_smmu_device *smmu);
+void arm_smmu_test_smr_masks(struct arm_smmu_device *smmu);
+
+/* For IOMMU ops */
+bool arm_smmu_capable(enum iommu_cap cap);
+struct iommu_domain *arm_smmu_domain_alloc_common(unsigned int type,
+ bool using_legacy_binding);
+void arm_smmu_domain_free(struct iommu_domain *domain);
+int arm_smmu_attach_dev_common(struct iommu_domain *domain, struct device *dev,
+ struct iommu_ops *ops);
+int arm_smmu_map(struct iommu_domain *domain, unsigned long iova,
+ phys_addr_t paddr, size_t size, int prot);
+size_t arm_smmu_unmap(struct iommu_domain *domain, unsigned long iova,
+ size_t size);
+void arm_smmu_flush_iotlb_all(struct iommu_domain *domain);
+void arm_smmu_iotlb_sync(struct iommu_domain *domain);
+phys_addr_t arm_smmu_iova_to_phys(struct iommu_domain *domain,
+ dma_addr_t iova);
+int arm_smmu_add_device_common(struct device *dev,
+ struct arm_smmu_device *smmu);
+void arm_smmu_remove_device_common(struct device *dev, struct iommu_ops *ops);
+struct iommu_group *arm_smmu_device_group(struct device *dev);
+int arm_smmu_domain_get_attr(struct iommu_domain *domain,
+ enum iommu_attr attr, void *data);
+int arm_smmu_domain_set_attr(struct iommu_domain *domain,
+ enum iommu_attr attr, void *data);
+int arm_smmu_of_xlate(struct device *dev, struct of_phandle_args *args);
+void arm_smmu_get_resv_regions(struct device *dev,
+ struct list_head *head);
+void arm_smmu_put_resv_regions(struct device *dev,
+ struct list_head *head);
+
+#endif
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v3 2/6] iommu/arm-smmu: Add support to program multiple ARM SMMU's identically
2018-12-04 1:36 [PATCH v3 0/5] Add Tegra194 Dual ARM SMMU driver Krishna Reddy
2018-12-04 1:36 ` [PATCH v3 1/6] iommu/arm-smmu: create library for ARM SMMU Krishna Reddy
@ 2018-12-04 1:36 ` Krishna Reddy
2018-12-19 18:07 ` Robin Murphy
2018-12-04 1:36 ` [PATCH v3 3/6] iommu/arm-smmu: update arm-smmu.c to use ARM SMMU library Krishna Reddy
` (3 subsequent siblings)
5 siblings, 1 reply; 9+ messages in thread
From: Krishna Reddy @ 2018-12-04 1:36 UTC (permalink / raw)
To: will.deacon, robin.murphy, joro
Cc: linux-arm-kernel, iommu, linux-kernel, linux-tegra, treding,
yhsu, snikam, praithatha, talho, avanbrunt, thomasz, olof,
jtukkinen, mperttunen, Krishna Reddy
Add support to program multiple ARM SMMU's identically as one SMMU device.
Tegra194 uses Two ARM SMMU's as one SMMU device and both ARM SMMU's need
to be programmed identically.
Signed-off-by: Krishna Reddy <vdumpa@nvidia.com>
---
drivers/iommu/lib-arm-smmu.c | 191 ++++++++++++++++++++++++++++++++-----------
1 file changed, 144 insertions(+), 47 deletions(-)
diff --git a/drivers/iommu/lib-arm-smmu.c b/drivers/iommu/lib-arm-smmu.c
index 6aba5db..7036763 100644
--- a/drivers/iommu/lib-arm-smmu.c
+++ b/drivers/iommu/lib-arm-smmu.c
@@ -69,9 +69,9 @@
* therefore this actually makes more sense than it might first appear.
*/
#ifdef CONFIG_64BIT
-#define smmu_write_atomic_lq writeq_relaxed
+#define smmu_write_atomic_lq writeq_all_relaxed
#else
-#define smmu_write_atomic_lq writel_relaxed
+#define smmu_write_atomic_lq writel_all_relaxed
#endif
/* Translation context bank */
@@ -135,6 +135,48 @@ struct arm_smmu_domain {
struct iommu_domain domain;
};
+#define to_smmu_intance(smmu, inst, addr) \
+ (addr - smmu->bases[0] + smmu->bases[inst])
+
+static void writel_all(struct arm_smmu_device *smmu,
+ u32 value, void __iomem *addr)
+{
+ int i;
+
+ writel(value, addr);
+ for (i = 1; i < smmu->num_smmus; i++) {
+ void __iomem *reg_addr = to_smmu_intance(smmu, i, addr);
+
+ writel(value, reg_addr);
+ }
+}
+
+static void writel_all_relaxed(struct arm_smmu_device *smmu,
+ u32 value, void __iomem *addr)
+{
+ int i;
+
+ writel_relaxed(value, addr);
+ for (i = 1; i < smmu->num_smmus; i++) {
+ void __iomem *reg_addr = to_smmu_intance(smmu, i, addr);
+
+ writel_relaxed(value, reg_addr);
+ }
+}
+
+static void writeq_all_relaxed(struct arm_smmu_device *smmu,
+ u64 value, void __iomem *addr)
+{
+ int i;
+
+ writeq_relaxed(value, addr);
+ for (i = 1; i < smmu->num_smmus; i++) {
+ void __iomem *reg_addr = to_smmu_intance(smmu, i, addr);
+
+ writeq_relaxed(value, reg_addr);
+ }
+}
+
static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
{
return container_of(dom, struct arm_smmu_domain, domain);
@@ -179,25 +221,37 @@ static void __arm_smmu_tlb_sync(struct arm_smmu_device *smmu,
static void arm_smmu_tlb_sync_global(struct arm_smmu_device *smmu)
{
- void __iomem *base = ARM_SMMU_GR0(smmu);
+ int i;
unsigned long flags;
spin_lock_irqsave(&smmu->global_sync_lock, flags);
- __arm_smmu_tlb_sync(smmu, base + ARM_SMMU_GR0_sTLBGSYNC,
- base + ARM_SMMU_GR0_sTLBGSTATUS);
+ for (i = 0; i < smmu->num_smmus; i++) {
+ void __iomem *base = ARM_SMMU_GR0(smmu);
+
+ if (i > 0)
+ base = to_smmu_intance(smmu, i, base);
+ __arm_smmu_tlb_sync(smmu, base + ARM_SMMU_GR0_sTLBGSYNC,
+ base + ARM_SMMU_GR0_sTLBGSTATUS);
+ }
spin_unlock_irqrestore(&smmu->global_sync_lock, flags);
}
static void arm_smmu_tlb_sync_context(void *cookie)
{
+ int i;
struct arm_smmu_domain *smmu_domain = cookie;
struct arm_smmu_device *smmu = smmu_domain->smmu;
- void __iomem *base = ARM_SMMU_CB(smmu, smmu_domain->cfg.cbndx);
unsigned long flags;
spin_lock_irqsave(&smmu_domain->cb_lock, flags);
- __arm_smmu_tlb_sync(smmu, base + ARM_SMMU_CB_TLBSYNC,
- base + ARM_SMMU_CB_TLBSTATUS);
+ for (i = 0; i < smmu->num_smmus; i++) {
+ void __iomem *base = ARM_SMMU_CB(smmu, smmu_domain->cfg.cbndx);
+
+ if (i > 0)
+ base = to_smmu_intance(smmu, i, base);
+ __arm_smmu_tlb_sync(smmu, base + ARM_SMMU_CB_TLBSYNC,
+ base + ARM_SMMU_CB_TLBSTATUS);
+ }
spin_unlock_irqrestore(&smmu_domain->cb_lock, flags);
}
@@ -212,13 +266,14 @@ static void arm_smmu_tlb_inv_context_s1(void *cookie)
{
struct arm_smmu_domain *smmu_domain = cookie;
struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
- void __iomem *base = ARM_SMMU_CB(smmu_domain->smmu, cfg->cbndx);
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
+ void __iomem *base = ARM_SMMU_CB(smmu, cfg->cbndx);
/*
* NOTE: this is not a relaxed write; it needs to guarantee that PTEs
* cleared by the current CPU are visible to the SMMU before the TLBI.
*/
- writel(cfg->asid, base + ARM_SMMU_CB_S1_TLBIASID);
+ writel_all(smmu, cfg->asid, base + ARM_SMMU_CB_S1_TLBIASID);
arm_smmu_tlb_sync_context(cookie);
}
@@ -229,7 +284,7 @@ static void arm_smmu_tlb_inv_context_s2(void *cookie)
void __iomem *base = ARM_SMMU_GR0(smmu);
/* NOTE: see above */
- writel(smmu_domain->cfg.vmid, base + ARM_SMMU_GR0_TLBIVMID);
+ writel_all(smmu, smmu_domain->cfg.vmid, base + ARM_SMMU_GR0_TLBIVMID);
arm_smmu_tlb_sync_global(smmu);
}
@@ -237,11 +292,12 @@ static void arm_smmu_tlb_inv_range_nosync(unsigned long iova, size_t size,
size_t granule, bool leaf, void *cookie)
{
struct arm_smmu_domain *smmu_domain = cookie;
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
bool stage1 = cfg->cbar != CBAR_TYPE_S2_TRANS;
- void __iomem *reg = ARM_SMMU_CB(smmu_domain->smmu, cfg->cbndx);
+ void __iomem *reg = ARM_SMMU_CB(smmu, cfg->cbndx);
- if (smmu_domain->smmu->features & ARM_SMMU_FEAT_COHERENT_WALK)
+ if (smmu->features & ARM_SMMU_FEAT_COHERENT_WALK)
wmb();
if (stage1) {
@@ -251,7 +307,7 @@ static void arm_smmu_tlb_inv_range_nosync(unsigned long iova, size_t size,
iova &= ~12UL;
iova |= cfg->asid;
do {
- writel_relaxed(iova, reg);
+ writel_all_relaxed(smmu, iova, reg);
iova += granule;
} while (size -= granule);
} else {
@@ -267,7 +323,7 @@ static void arm_smmu_tlb_inv_range_nosync(unsigned long iova, size_t size,
ARM_SMMU_CB_S2_TLBIIPAS2;
iova >>= 12;
do {
- smmu_write_atomic_lq(iova, reg);
+ smmu_write_atomic_lq(smmu, iova, reg);
iova += granule >> 12;
} while (size -= granule);
}
@@ -283,12 +339,13 @@ static void arm_smmu_tlb_inv_vmid_nosync(unsigned long iova, size_t size,
size_t granule, bool leaf, void *cookie)
{
struct arm_smmu_domain *smmu_domain = cookie;
- void __iomem *base = ARM_SMMU_GR0(smmu_domain->smmu);
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
+ void __iomem *base = ARM_SMMU_GR0(smmu);
- if (smmu_domain->smmu->features & ARM_SMMU_FEAT_COHERENT_WALK)
+ if (smmu->features & ARM_SMMU_FEAT_COHERENT_WALK)
wmb();
- writel_relaxed(smmu_domain->cfg.vmid, base + ARM_SMMU_GR0_TLBIVMID);
+ writel_all_relaxed(smmu, smmu_domain->cfg.vmid, base + ARM_SMMU_GR0_TLBIVMID);
}
static const struct iommu_gather_ops arm_smmu_s1_tlb_ops = {
@@ -309,7 +366,8 @@ static const struct iommu_gather_ops arm_smmu_s2_tlb_ops_v1 = {
.tlb_sync = arm_smmu_tlb_sync_vmid,
};
-irqreturn_t arm_smmu_context_fault(int irq, void *dev)
+static irqreturn_t __arm_smmu_context_fault(int irq, void *dev,
+ void __iomem *cb_base)
{
u32 fsr, fsynr;
unsigned long iova;
@@ -317,9 +375,7 @@ irqreturn_t arm_smmu_context_fault(int irq, void *dev)
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
struct arm_smmu_device *smmu = smmu_domain->smmu;
- void __iomem *cb_base;
- cb_base = ARM_SMMU_CB(smmu, cfg->cbndx);
fsr = readl_relaxed(cb_base + ARM_SMMU_CB_FSR);
if (!(fsr & FSR_FAULT))
@@ -336,11 +392,33 @@ irqreturn_t arm_smmu_context_fault(int irq, void *dev)
return IRQ_HANDLED;
}
-irqreturn_t arm_smmu_global_fault(int irq, void *dev)
+static irqreturn_t arm_smmu_context_fault(int irq, void *dev)
+{
+ int i;
+ irqreturn_t irq_ret = IRQ_NONE;
+ struct iommu_domain *domain = dev;
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
+
+ for (i = 0; i < smmu->num_smmus; i++) {
+ void __iomem *cb_base = ARM_SMMU_CB(smmu, cfg->cbndx);
+
+ if (i > 0)
+ cb_base = to_smmu_intance(smmu, i, cb_base);
+ irq_ret = __arm_smmu_context_fault(irq, dev, cb_base);
+ if (irq_ret == IRQ_HANDLED)
+ break;
+ }
+
+ return irq_ret;
+}
+
+static irqreturn_t __arm_smmu_global_fault(int irq, void *dev,
+ void __iomem *gr0_base)
{
- u32 gfsr, gfsynr0, gfsynr1, gfsynr2;
struct arm_smmu_device *smmu = dev;
- void __iomem *gr0_base = ARM_SMMU_GR0_NS(smmu);
+ u32 gfsr, gfsynr0, gfsynr1, gfsynr2;
gfsr = readl_relaxed(gr0_base + ARM_SMMU_GR0_sGFSR);
gfsynr0 = readl_relaxed(gr0_base + ARM_SMMU_GR0_sGFSYNR0);
@@ -360,6 +438,25 @@ irqreturn_t arm_smmu_global_fault(int irq, void *dev)
return IRQ_HANDLED;
}
+irqreturn_t arm_smmu_global_fault(int irq, void *dev)
+{
+ int i;
+ irqreturn_t irq_ret = IRQ_NONE;
+ struct arm_smmu_device *smmu = dev;
+
+ for (i = 0; i < smmu->num_smmus; i++) {
+ void __iomem *gr0_base = ARM_SMMU_GR0_NS(smmu);
+
+ if (i > 0)
+ gr0_base = to_smmu_intance(smmu, i, gr0_base);
+ irq_ret = __arm_smmu_global_fault(irq, dev, gr0_base);
+ if (irq_ret == IRQ_HANDLED)
+ break;
+ }
+
+ return irq_ret;
+}
+
static void arm_smmu_init_context_bank(struct arm_smmu_domain *smmu_domain,
struct io_pgtable_cfg *pgtbl_cfg)
{
@@ -423,7 +520,7 @@ static void arm_smmu_write_context_bank(struct arm_smmu_device *smmu, int idx)
/* Unassigned context banks only need disabling */
if (!cfg) {
- writel_relaxed(0, cb_base + ARM_SMMU_CB_SCTLR);
+ writel_all_relaxed(smmu, 0, cb_base + ARM_SMMU_CB_SCTLR);
return;
}
@@ -440,7 +537,7 @@ static void arm_smmu_write_context_bank(struct arm_smmu_device *smmu, int idx)
if (smmu->features & ARM_SMMU_FEAT_VMID16)
reg |= cfg->vmid << CBA2R_VMID_SHIFT;
- writel_relaxed(reg, gr1_base + ARM_SMMU_GR1_CBA2R(idx));
+ writel_all_relaxed(smmu, reg, gr1_base + ARM_SMMU_GR1_CBA2R(idx));
}
/* CBAR */
@@ -459,7 +556,7 @@ static void arm_smmu_write_context_bank(struct arm_smmu_device *smmu, int idx)
/* 8-bit VMIDs live in CBAR */
reg |= cfg->vmid << CBAR_VMID_SHIFT;
}
- writel_relaxed(reg, gr1_base + ARM_SMMU_GR1_CBAR(idx));
+ writel_all_relaxed(smmu, reg, gr1_base + ARM_SMMU_GR1_CBAR(idx));
/*
* TTBCR
@@ -467,14 +564,14 @@ static void arm_smmu_write_context_bank(struct arm_smmu_device *smmu, int idx)
* access behaviour of some fields (in particular, ASID[15:8]).
*/
if (stage1 && smmu->version > ARM_SMMU_V1)
- writel_relaxed(cb->tcr[1], cb_base + ARM_SMMU_CB_TTBCR2);
- writel_relaxed(cb->tcr[0], cb_base + ARM_SMMU_CB_TTBCR);
+ writel_all_relaxed(smmu, cb->tcr[1], cb_base + ARM_SMMU_CB_TTBCR2);
+ writel_all_relaxed(smmu, cb->tcr[0], cb_base + ARM_SMMU_CB_TTBCR);
/* TTBRs */
if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH32_S) {
- writel_relaxed(cfg->asid, cb_base + ARM_SMMU_CB_CONTEXTIDR);
- writel_relaxed(cb->ttbr[0], cb_base + ARM_SMMU_CB_TTBR0);
- writel_relaxed(cb->ttbr[1], cb_base + ARM_SMMU_CB_TTBR1);
+ writel_all_relaxed(smmu, cfg->asid, cb_base + ARM_SMMU_CB_CONTEXTIDR);
+ writel_all_relaxed(smmu, cb->ttbr[0], cb_base + ARM_SMMU_CB_TTBR0);
+ writel_all_relaxed(smmu, cb->ttbr[1], cb_base + ARM_SMMU_CB_TTBR1);
} else {
writeq_relaxed(cb->ttbr[0], cb_base + ARM_SMMU_CB_TTBR0);
if (stage1)
@@ -484,8 +581,8 @@ static void arm_smmu_write_context_bank(struct arm_smmu_device *smmu, int idx)
/* MAIRs (stage-1 only) */
if (stage1) {
- writel_relaxed(cb->mair[0], cb_base + ARM_SMMU_CB_S1_MAIR0);
- writel_relaxed(cb->mair[1], cb_base + ARM_SMMU_CB_S1_MAIR1);
+ writel_all_relaxed(smmu, cb->mair[0], cb_base + ARM_SMMU_CB_S1_MAIR0);
+ writel_all_relaxed(smmu, cb->mair[1], cb_base + ARM_SMMU_CB_S1_MAIR1);
}
/* SCTLR */
@@ -495,7 +592,7 @@ static void arm_smmu_write_context_bank(struct arm_smmu_device *smmu, int idx)
if (IS_ENABLED(CONFIG_CPU_BIG_ENDIAN))
reg |= SCTLR_E;
- writel_relaxed(reg, cb_base + ARM_SMMU_CB_SCTLR);
+ writel_all_relaxed(smmu, reg, cb_base + ARM_SMMU_CB_SCTLR);
}
static int arm_smmu_init_domain_context(struct iommu_domain *domain,
@@ -763,7 +860,7 @@ static void arm_smmu_write_smr(struct arm_smmu_device *smmu, int idx)
if (!(smmu->features & ARM_SMMU_FEAT_EXIDS) && smr->valid)
reg |= SMR_VALID;
- writel_relaxed(reg, ARM_SMMU_GR0(smmu) + ARM_SMMU_GR0_SMR(idx));
+ writel_all_relaxed(smmu, reg, ARM_SMMU_GR0(smmu) + ARM_SMMU_GR0_SMR(idx));
}
static void arm_smmu_write_s2cr(struct arm_smmu_device *smmu, int idx)
@@ -776,7 +873,7 @@ static void arm_smmu_write_s2cr(struct arm_smmu_device *smmu, int idx)
if (smmu->features & ARM_SMMU_FEAT_EXIDS && smmu->smrs &&
smmu->smrs[idx].valid)
reg |= S2CR_EXIDVALID;
- writel_relaxed(reg, ARM_SMMU_GR0(smmu) + ARM_SMMU_GR0_S2CR(idx));
+ writel_all_relaxed(smmu, reg, ARM_SMMU_GR0(smmu) + ARM_SMMU_GR0_S2CR(idx));
}
static void arm_smmu_write_sme(struct arm_smmu_device *smmu, int idx)
@@ -1071,9 +1168,9 @@ static phys_addr_t arm_smmu_iova_to_phys_hard(struct iommu_domain *domain,
/* ATS1 registers can only be written atomically */
va = iova & ~0xfffUL;
if (smmu->version == ARM_SMMU_V2)
- smmu_write_atomic_lq(va, cb_base + ARM_SMMU_CB_ATS1PR);
+ smmu_write_atomic_lq(smmu, va, cb_base + ARM_SMMU_CB_ATS1PR);
else /* Register is only 32-bit in v1 */
- writel_relaxed(va, cb_base + ARM_SMMU_CB_ATS1PR);
+ writel_all_relaxed(smmu, va, cb_base + ARM_SMMU_CB_ATS1PR);
if (readl_poll_timeout_atomic(cb_base + ARM_SMMU_CB_ATSR, tmp,
!(tmp & ATSR_ACTIVE), 5, 50)) {
@@ -1346,7 +1443,7 @@ void arm_smmu_device_reset(struct arm_smmu_device *smmu)
/* clear global FSR */
reg = readl_relaxed(ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sGFSR);
- writel(reg, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sGFSR);
+ writel_all(smmu, reg, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sGFSR);
/*
* Reset stream mapping groups: Initial values mark all SMRn as
@@ -1371,7 +1468,7 @@ void arm_smmu_device_reset(struct arm_smmu_device *smmu)
* TLB entries for reduced latency.
*/
reg |= ARM_MMU500_ACR_SMTNMB_TLBEN | ARM_MMU500_ACR_S2CRB_TLBEN;
- writel_relaxed(reg, gr0_base + ARM_SMMU_GR0_sACR);
+ writel_all_relaxed(smmu, reg, gr0_base + ARM_SMMU_GR0_sACR);
}
/* Make sure all context banks are disabled and clear CB_FSR */
@@ -1379,7 +1476,7 @@ void arm_smmu_device_reset(struct arm_smmu_device *smmu)
void __iomem *cb_base = ARM_SMMU_CB(smmu, i);
arm_smmu_write_context_bank(smmu, i);
- writel_relaxed(FSR_FAULT, cb_base + ARM_SMMU_CB_FSR);
+ writel_all_relaxed(smmu, FSR_FAULT, cb_base + ARM_SMMU_CB_FSR);
/*
* Disable MMU-500's not-particularly-beneficial next-page
* prefetcher for the sake of errata #841119 and #826419.
@@ -1387,13 +1484,13 @@ void arm_smmu_device_reset(struct arm_smmu_device *smmu)
if (smmu->model == ARM_MMU500) {
reg = readl_relaxed(cb_base + ARM_SMMU_CB_ACTLR);
reg &= ~ARM_MMU500_ACTLR_CPRE;
- writel_relaxed(reg, cb_base + ARM_SMMU_CB_ACTLR);
+ writel_all_relaxed(smmu, reg, cb_base + ARM_SMMU_CB_ACTLR);
}
}
/* Invalidate the TLB, just in case */
- writel_relaxed(0, gr0_base + ARM_SMMU_GR0_TLBIALLH);
- writel_relaxed(0, gr0_base + ARM_SMMU_GR0_TLBIALLNSNH);
+ writel_all_relaxed(smmu, 0, gr0_base + ARM_SMMU_GR0_TLBIALLH);
+ writel_all_relaxed(smmu, 0, gr0_base + ARM_SMMU_GR0_TLBIALLNSNH);
reg = readl_relaxed(ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
@@ -1424,7 +1521,7 @@ void arm_smmu_device_reset(struct arm_smmu_device *smmu)
/* Push the button */
arm_smmu_tlb_sync_global(smmu);
- writel(reg, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
+ writel_all(smmu, reg, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
}
static int arm_smmu_id_size_to_bits(int size)
@@ -1666,6 +1763,6 @@ int arm_smmu_device_remove(struct platform_device *pdev)
dev_err(&pdev->dev, "removing device with active domains!\n");
/* Turn the thing off */
- writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
+ writel_all(smmu, sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
return 0;
}
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v3 3/6] iommu/arm-smmu: update arm-smmu.c to use ARM SMMU library
2018-12-04 1:36 [PATCH v3 0/5] Add Tegra194 Dual ARM SMMU driver Krishna Reddy
2018-12-04 1:36 ` [PATCH v3 1/6] iommu/arm-smmu: create library for ARM SMMU Krishna Reddy
2018-12-04 1:36 ` [PATCH v3 2/6] iommu/arm-smmu: Add support to program multiple ARM SMMU's identically Krishna Reddy
@ 2018-12-04 1:36 ` Krishna Reddy
2018-12-04 1:36 ` [PATCH v3 4/6] iommu/tegra194_smmu: Add Tegra194 SMMU driver Krishna Reddy
` (2 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: Krishna Reddy @ 2018-12-04 1:36 UTC (permalink / raw)
To: will.deacon, robin.murphy, joro
Cc: linux-arm-kernel, iommu, linux-kernel, linux-tegra, treding,
yhsu, snikam, praithatha, talho, avanbrunt, thomasz, olof,
jtukkinen, mperttunen, Krishna Reddy
Update ARM SMMU driver to use the ARM SMMU library routines.
Signed-off-by: Krishna Reddy <vdumpa@nvidia.com>
---
drivers/iommu/arm-smmu.c | 1709 +---------------------------------------------
1 file changed, 11 insertions(+), 1698 deletions(-)
diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 5a28ae8..453159b 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -1,5 +1,5 @@
/*
- * IOMMU API for ARM architected SMMU implementations.
+ * ARM SMMU driver for ARM architected V1 and V2 SMMU implementations.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
@@ -39,7 +39,6 @@
#include <linux/interrupt.h>
#include <linux/io.h>
#include <linux/io-64-nonatomic-hi-lo.h>
-#include <linux/iommu.h>
#include <linux/iopoll.h>
#include <linux/module.h>
#include <linux/of.h>
@@ -56,49 +55,7 @@
#include "io-pgtable.h"
#include "arm-smmu-regs.h"
-
-#define ARM_MMU500_ACTLR_CPRE (1 << 1)
-
-#define ARM_MMU500_ACR_CACHE_LOCK (1 << 26)
-#define ARM_MMU500_ACR_S2CRB_TLBEN (1 << 10)
-#define ARM_MMU500_ACR_SMTNMB_TLBEN (1 << 8)
-
-#define TLB_LOOP_TIMEOUT 1000000 /* 1s! */
-#define TLB_SPIN_COUNT 10
-
-/* Maximum number of context banks per SMMU */
-#define ARM_SMMU_MAX_CBS 128
-
-/* SMMU global address space */
-#define ARM_SMMU_GR0(smmu) ((smmu)->base)
-#define ARM_SMMU_GR1(smmu) ((smmu)->base + (1 << (smmu)->pgshift))
-
-/*
- * SMMU global address space with conditional offset to access secure
- * aliases of non-secure registers (e.g. nsCR0: 0x400, nsGFSR: 0x448,
- * nsGFSYNR0: 0x450)
- */
-#define ARM_SMMU_GR0_NS(smmu) \
- ((smmu)->base + \
- ((smmu->options & ARM_SMMU_OPT_SECURE_CFG_ACCESS) \
- ? 0x400 : 0))
-
-/*
- * Some 64-bit registers only make sense to write atomically, but in such
- * cases all the data relevant to AArch32 formats lies within the lower word,
- * therefore this actually makes more sense than it might first appear.
- */
-#ifdef CONFIG_64BIT
-#define smmu_write_atomic_lq writeq_relaxed
-#else
-#define smmu_write_atomic_lq writel_relaxed
-#endif
-
-/* Translation context bank */
-#define ARM_SMMU_CB(smmu, n) ((smmu)->cb_base + ((n) << (smmu)->pgshift))
-
-#define MSI_IOVA_BASE 0x8000000
-#define MSI_IOVA_LENGTH 0x100000
+#include "lib-arm-smmu.h"
static int force_stage;
module_param(force_stage, int, S_IRUGO);
@@ -109,150 +66,6 @@ module_param(disable_bypass, bool, S_IRUGO);
MODULE_PARM_DESC(disable_bypass,
"Disable bypass streams such that incoming transactions from devices that are not attached to an iommu domain will report an abort back to the device and will not be allowed to pass through the SMMU.");
-enum arm_smmu_arch_version {
- ARM_SMMU_V1,
- ARM_SMMU_V1_64K,
- ARM_SMMU_V2,
-};
-
-enum arm_smmu_implementation {
- GENERIC_SMMU,
- ARM_MMU500,
- CAVIUM_SMMUV2,
-};
-
-struct arm_smmu_s2cr {
- struct iommu_group *group;
- int count;
- enum arm_smmu_s2cr_type type;
- enum arm_smmu_s2cr_privcfg privcfg;
- u8 cbndx;
-};
-
-#define s2cr_init_val (struct arm_smmu_s2cr){ \
- .type = disable_bypass ? S2CR_TYPE_FAULT : S2CR_TYPE_BYPASS, \
-}
-
-struct arm_smmu_smr {
- u16 mask;
- u16 id;
- bool valid;
-};
-
-struct arm_smmu_cb {
- u64 ttbr[2];
- u32 tcr[2];
- u32 mair[2];
- struct arm_smmu_cfg *cfg;
-};
-
-struct arm_smmu_master_cfg {
- struct arm_smmu_device *smmu;
- s16 smendx[];
-};
-#define INVALID_SMENDX -1
-#define __fwspec_cfg(fw) ((struct arm_smmu_master_cfg *)fw->iommu_priv)
-#define fwspec_smmu(fw) (__fwspec_cfg(fw)->smmu)
-#define fwspec_smendx(fw, i) \
- (i >= fw->num_ids ? INVALID_SMENDX : __fwspec_cfg(fw)->smendx[i])
-#define for_each_cfg_sme(fw, i, idx) \
- for (i = 0; idx = fwspec_smendx(fw, i), i < fw->num_ids; ++i)
-
-struct arm_smmu_device {
- struct device *dev;
-
- void __iomem *base;
- void __iomem *cb_base;
- unsigned long pgshift;
-
-#define ARM_SMMU_FEAT_COHERENT_WALK (1 << 0)
-#define ARM_SMMU_FEAT_STREAM_MATCH (1 << 1)
-#define ARM_SMMU_FEAT_TRANS_S1 (1 << 2)
-#define ARM_SMMU_FEAT_TRANS_S2 (1 << 3)
-#define ARM_SMMU_FEAT_TRANS_NESTED (1 << 4)
-#define ARM_SMMU_FEAT_TRANS_OPS (1 << 5)
-#define ARM_SMMU_FEAT_VMID16 (1 << 6)
-#define ARM_SMMU_FEAT_FMT_AARCH64_4K (1 << 7)
-#define ARM_SMMU_FEAT_FMT_AARCH64_16K (1 << 8)
-#define ARM_SMMU_FEAT_FMT_AARCH64_64K (1 << 9)
-#define ARM_SMMU_FEAT_FMT_AARCH32_L (1 << 10)
-#define ARM_SMMU_FEAT_FMT_AARCH32_S (1 << 11)
-#define ARM_SMMU_FEAT_EXIDS (1 << 12)
- u32 features;
-
-#define ARM_SMMU_OPT_SECURE_CFG_ACCESS (1 << 0)
- u32 options;
- enum arm_smmu_arch_version version;
- enum arm_smmu_implementation model;
-
- u32 num_context_banks;
- u32 num_s2_context_banks;
- DECLARE_BITMAP(context_map, ARM_SMMU_MAX_CBS);
- struct arm_smmu_cb *cbs;
- atomic_t irptndx;
-
- u32 num_mapping_groups;
- u16 streamid_mask;
- u16 smr_mask_mask;
- struct arm_smmu_smr *smrs;
- struct arm_smmu_s2cr *s2crs;
- struct mutex stream_map_mutex;
-
- unsigned long va_size;
- unsigned long ipa_size;
- unsigned long pa_size;
- unsigned long pgsize_bitmap;
-
- u32 num_global_irqs;
- u32 num_context_irqs;
- unsigned int *irqs;
-
- u32 cavium_id_base; /* Specific to Cavium */
-
- spinlock_t global_sync_lock;
-
- /* IOMMU core code handle */
- struct iommu_device iommu;
-};
-
-enum arm_smmu_context_fmt {
- ARM_SMMU_CTX_FMT_NONE,
- ARM_SMMU_CTX_FMT_AARCH64,
- ARM_SMMU_CTX_FMT_AARCH32_L,
- ARM_SMMU_CTX_FMT_AARCH32_S,
-};
-
-struct arm_smmu_cfg {
- u8 cbndx;
- u8 irptndx;
- union {
- u16 asid;
- u16 vmid;
- };
- u32 cbar;
- enum arm_smmu_context_fmt fmt;
-};
-#define INVALID_IRPTNDX 0xff
-
-enum arm_smmu_domain_stage {
- ARM_SMMU_DOMAIN_S1 = 0,
- ARM_SMMU_DOMAIN_S2,
- ARM_SMMU_DOMAIN_NESTED,
- ARM_SMMU_DOMAIN_BYPASS,
-};
-
-struct arm_smmu_domain {
- struct arm_smmu_device *smmu;
- struct io_pgtable_ops *pgtbl_ops;
- const struct iommu_gather_ops *tlb_ops;
- struct arm_smmu_cfg cfg;
- enum arm_smmu_domain_stage stage;
- bool non_strict;
- struct mutex init_mutex; /* Protects smmu pointer */
- spinlock_t cb_lock; /* Serialises ATS1* ops and TLB syncs */
- struct iommu_domain domain;
-};
-
struct arm_smmu_option_prop {
u32 opt;
const char *prop;
@@ -267,11 +80,6 @@ static struct arm_smmu_option_prop arm_smmu_options[] = {
{ 0, NULL},
};
-static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
-{
- return container_of(dom, struct arm_smmu_domain, domain);
-}
-
static void parse_driver_options(struct arm_smmu_device *smmu)
{
int i = 0;
@@ -375,991 +183,14 @@ static int arm_smmu_register_legacy_master(struct device *dev,
return err;
}
-static int __arm_smmu_alloc_bitmap(unsigned long *map, int start, int end)
-{
- int idx;
-
- do {
- idx = find_next_zero_bit(map, end, start);
- if (idx == end)
- return -ENOSPC;
- } while (test_and_set_bit(idx, map));
-
- return idx;
-}
-
-static void __arm_smmu_free_bitmap(unsigned long *map, int idx)
-{
- clear_bit(idx, map);
-}
-
-/* Wait for any pending TLB invalidations to complete */
-static void __arm_smmu_tlb_sync(struct arm_smmu_device *smmu,
- void __iomem *sync, void __iomem *status)
-{
- unsigned int spin_cnt, delay;
-
- writel_relaxed(0, sync);
- for (delay = 1; delay < TLB_LOOP_TIMEOUT; delay *= 2) {
- for (spin_cnt = TLB_SPIN_COUNT; spin_cnt > 0; spin_cnt--) {
- if (!(readl_relaxed(status) & sTLBGSTATUS_GSACTIVE))
- return;
- cpu_relax();
- }
- udelay(delay);
- }
- dev_err_ratelimited(smmu->dev,
- "TLB sync timed out -- SMMU may be deadlocked\n");
-}
-
-static void arm_smmu_tlb_sync_global(struct arm_smmu_device *smmu)
-{
- void __iomem *base = ARM_SMMU_GR0(smmu);
- unsigned long flags;
-
- spin_lock_irqsave(&smmu->global_sync_lock, flags);
- __arm_smmu_tlb_sync(smmu, base + ARM_SMMU_GR0_sTLBGSYNC,
- base + ARM_SMMU_GR0_sTLBGSTATUS);
- spin_unlock_irqrestore(&smmu->global_sync_lock, flags);
-}
-
-static void arm_smmu_tlb_sync_context(void *cookie)
-{
- struct arm_smmu_domain *smmu_domain = cookie;
- struct arm_smmu_device *smmu = smmu_domain->smmu;
- void __iomem *base = ARM_SMMU_CB(smmu, smmu_domain->cfg.cbndx);
- unsigned long flags;
-
- spin_lock_irqsave(&smmu_domain->cb_lock, flags);
- __arm_smmu_tlb_sync(smmu, base + ARM_SMMU_CB_TLBSYNC,
- base + ARM_SMMU_CB_TLBSTATUS);
- spin_unlock_irqrestore(&smmu_domain->cb_lock, flags);
-}
-
-static void arm_smmu_tlb_sync_vmid(void *cookie)
-{
- struct arm_smmu_domain *smmu_domain = cookie;
-
- arm_smmu_tlb_sync_global(smmu_domain->smmu);
-}
-
-static void arm_smmu_tlb_inv_context_s1(void *cookie)
-{
- struct arm_smmu_domain *smmu_domain = cookie;
- struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
- void __iomem *base = ARM_SMMU_CB(smmu_domain->smmu, cfg->cbndx);
-
- /*
- * NOTE: this is not a relaxed write; it needs to guarantee that PTEs
- * cleared by the current CPU are visible to the SMMU before the TLBI.
- */
- writel(cfg->asid, base + ARM_SMMU_CB_S1_TLBIASID);
- arm_smmu_tlb_sync_context(cookie);
-}
-
-static void arm_smmu_tlb_inv_context_s2(void *cookie)
-{
- struct arm_smmu_domain *smmu_domain = cookie;
- struct arm_smmu_device *smmu = smmu_domain->smmu;
- void __iomem *base = ARM_SMMU_GR0(smmu);
-
- /* NOTE: see above */
- writel(smmu_domain->cfg.vmid, base + ARM_SMMU_GR0_TLBIVMID);
- arm_smmu_tlb_sync_global(smmu);
-}
-
-static void arm_smmu_tlb_inv_range_nosync(unsigned long iova, size_t size,
- size_t granule, bool leaf, void *cookie)
-{
- struct arm_smmu_domain *smmu_domain = cookie;
- struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
- bool stage1 = cfg->cbar != CBAR_TYPE_S2_TRANS;
- void __iomem *reg = ARM_SMMU_CB(smmu_domain->smmu, cfg->cbndx);
-
- if (smmu_domain->smmu->features & ARM_SMMU_FEAT_COHERENT_WALK)
- wmb();
-
- if (stage1) {
- reg += leaf ? ARM_SMMU_CB_S1_TLBIVAL : ARM_SMMU_CB_S1_TLBIVA;
-
- if (cfg->fmt != ARM_SMMU_CTX_FMT_AARCH64) {
- iova &= ~12UL;
- iova |= cfg->asid;
- do {
- writel_relaxed(iova, reg);
- iova += granule;
- } while (size -= granule);
- } else {
- iova >>= 12;
- iova |= (u64)cfg->asid << 48;
- do {
- writeq_relaxed(iova, reg);
- iova += granule >> 12;
- } while (size -= granule);
- }
- } else {
- reg += leaf ? ARM_SMMU_CB_S2_TLBIIPAS2L :
- ARM_SMMU_CB_S2_TLBIIPAS2;
- iova >>= 12;
- do {
- smmu_write_atomic_lq(iova, reg);
- iova += granule >> 12;
- } while (size -= granule);
- }
-}
-
-/*
- * On MMU-401 at least, the cost of firing off multiple TLBIVMIDs appears
- * almost negligible, but the benefit of getting the first one in as far ahead
- * of the sync as possible is significant, hence we don't just make this a
- * no-op and set .tlb_sync to arm_smmu_inv_context_s2() as you might think.
- */
-static void arm_smmu_tlb_inv_vmid_nosync(unsigned long iova, size_t size,
- size_t granule, bool leaf, void *cookie)
-{
- struct arm_smmu_domain *smmu_domain = cookie;
- void __iomem *base = ARM_SMMU_GR0(smmu_domain->smmu);
-
- if (smmu_domain->smmu->features & ARM_SMMU_FEAT_COHERENT_WALK)
- wmb();
-
- writel_relaxed(smmu_domain->cfg.vmid, base + ARM_SMMU_GR0_TLBIVMID);
-}
-
-static const struct iommu_gather_ops arm_smmu_s1_tlb_ops = {
- .tlb_flush_all = arm_smmu_tlb_inv_context_s1,
- .tlb_add_flush = arm_smmu_tlb_inv_range_nosync,
- .tlb_sync = arm_smmu_tlb_sync_context,
-};
-
-static const struct iommu_gather_ops arm_smmu_s2_tlb_ops_v2 = {
- .tlb_flush_all = arm_smmu_tlb_inv_context_s2,
- .tlb_add_flush = arm_smmu_tlb_inv_range_nosync,
- .tlb_sync = arm_smmu_tlb_sync_context,
-};
-
-static const struct iommu_gather_ops arm_smmu_s2_tlb_ops_v1 = {
- .tlb_flush_all = arm_smmu_tlb_inv_context_s2,
- .tlb_add_flush = arm_smmu_tlb_inv_vmid_nosync,
- .tlb_sync = arm_smmu_tlb_sync_vmid,
-};
-
-static irqreturn_t arm_smmu_context_fault(int irq, void *dev)
-{
- u32 fsr, fsynr;
- unsigned long iova;
- struct iommu_domain *domain = dev;
- struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
- struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
- struct arm_smmu_device *smmu = smmu_domain->smmu;
- void __iomem *cb_base;
-
- cb_base = ARM_SMMU_CB(smmu, cfg->cbndx);
- fsr = readl_relaxed(cb_base + ARM_SMMU_CB_FSR);
-
- if (!(fsr & FSR_FAULT))
- return IRQ_NONE;
-
- fsynr = readl_relaxed(cb_base + ARM_SMMU_CB_FSYNR0);
- iova = readq_relaxed(cb_base + ARM_SMMU_CB_FAR);
-
- dev_err_ratelimited(smmu->dev,
- "Unhandled context fault: fsr=0x%x, iova=0x%08lx, fsynr=0x%x, cb=%d\n",
- fsr, iova, fsynr, cfg->cbndx);
-
- writel(fsr, cb_base + ARM_SMMU_CB_FSR);
- return IRQ_HANDLED;
-}
-
-static irqreturn_t arm_smmu_global_fault(int irq, void *dev)
-{
- u32 gfsr, gfsynr0, gfsynr1, gfsynr2;
- struct arm_smmu_device *smmu = dev;
- void __iomem *gr0_base = ARM_SMMU_GR0_NS(smmu);
-
- gfsr = readl_relaxed(gr0_base + ARM_SMMU_GR0_sGFSR);
- gfsynr0 = readl_relaxed(gr0_base + ARM_SMMU_GR0_sGFSYNR0);
- gfsynr1 = readl_relaxed(gr0_base + ARM_SMMU_GR0_sGFSYNR1);
- gfsynr2 = readl_relaxed(gr0_base + ARM_SMMU_GR0_sGFSYNR2);
-
- if (!gfsr)
- return IRQ_NONE;
-
- dev_err_ratelimited(smmu->dev,
- "Unexpected global fault, this could be serious\n");
- dev_err_ratelimited(smmu->dev,
- "\tGFSR 0x%08x, GFSYNR0 0x%08x, GFSYNR1 0x%08x, GFSYNR2 0x%08x\n",
- gfsr, gfsynr0, gfsynr1, gfsynr2);
-
- writel(gfsr, gr0_base + ARM_SMMU_GR0_sGFSR);
- return IRQ_HANDLED;
-}
-
-static void arm_smmu_init_context_bank(struct arm_smmu_domain *smmu_domain,
- struct io_pgtable_cfg *pgtbl_cfg)
-{
- struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
- struct arm_smmu_cb *cb = &smmu_domain->smmu->cbs[cfg->cbndx];
- bool stage1 = cfg->cbar != CBAR_TYPE_S2_TRANS;
-
- cb->cfg = cfg;
-
- /* TTBCR */
- if (stage1) {
- if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH32_S) {
- cb->tcr[0] = pgtbl_cfg->arm_v7s_cfg.tcr;
- } else {
- cb->tcr[0] = pgtbl_cfg->arm_lpae_s1_cfg.tcr;
- cb->tcr[1] = pgtbl_cfg->arm_lpae_s1_cfg.tcr >> 32;
- cb->tcr[1] |= TTBCR2_SEP_UPSTREAM;
- if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH64)
- cb->tcr[1] |= TTBCR2_AS;
- }
- } else {
- cb->tcr[0] = pgtbl_cfg->arm_lpae_s2_cfg.vtcr;
- }
-
- /* TTBRs */
- if (stage1) {
- if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH32_S) {
- cb->ttbr[0] = pgtbl_cfg->arm_v7s_cfg.ttbr[0];
- cb->ttbr[1] = pgtbl_cfg->arm_v7s_cfg.ttbr[1];
- } else {
- cb->ttbr[0] = pgtbl_cfg->arm_lpae_s1_cfg.ttbr[0];
- cb->ttbr[0] |= (u64)cfg->asid << TTBRn_ASID_SHIFT;
- cb->ttbr[1] = pgtbl_cfg->arm_lpae_s1_cfg.ttbr[1];
- cb->ttbr[1] |= (u64)cfg->asid << TTBRn_ASID_SHIFT;
- }
- } else {
- cb->ttbr[0] = pgtbl_cfg->arm_lpae_s2_cfg.vttbr;
- }
-
- /* MAIRs (stage-1 only) */
- if (stage1) {
- if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH32_S) {
- cb->mair[0] = pgtbl_cfg->arm_v7s_cfg.prrr;
- cb->mair[1] = pgtbl_cfg->arm_v7s_cfg.nmrr;
- } else {
- cb->mair[0] = pgtbl_cfg->arm_lpae_s1_cfg.mair[0];
- cb->mair[1] = pgtbl_cfg->arm_lpae_s1_cfg.mair[1];
- }
- }
-}
-
-static void arm_smmu_write_context_bank(struct arm_smmu_device *smmu, int idx)
-{
- u32 reg;
- bool stage1;
- struct arm_smmu_cb *cb = &smmu->cbs[idx];
- struct arm_smmu_cfg *cfg = cb->cfg;
- void __iomem *cb_base, *gr1_base;
-
- cb_base = ARM_SMMU_CB(smmu, idx);
-
- /* Unassigned context banks only need disabling */
- if (!cfg) {
- writel_relaxed(0, cb_base + ARM_SMMU_CB_SCTLR);
- return;
- }
-
- gr1_base = ARM_SMMU_GR1(smmu);
- stage1 = cfg->cbar != CBAR_TYPE_S2_TRANS;
-
- /* CBA2R */
- if (smmu->version > ARM_SMMU_V1) {
- if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH64)
- reg = CBA2R_RW64_64BIT;
- else
- reg = CBA2R_RW64_32BIT;
- /* 16-bit VMIDs live in CBA2R */
- if (smmu->features & ARM_SMMU_FEAT_VMID16)
- reg |= cfg->vmid << CBA2R_VMID_SHIFT;
-
- writel_relaxed(reg, gr1_base + ARM_SMMU_GR1_CBA2R(idx));
- }
-
- /* CBAR */
- reg = cfg->cbar;
- if (smmu->version < ARM_SMMU_V2)
- reg |= cfg->irptndx << CBAR_IRPTNDX_SHIFT;
-
- /*
- * Use the weakest shareability/memory types, so they are
- * overridden by the ttbcr/pte.
- */
- if (stage1) {
- reg |= (CBAR_S1_BPSHCFG_NSH << CBAR_S1_BPSHCFG_SHIFT) |
- (CBAR_S1_MEMATTR_WB << CBAR_S1_MEMATTR_SHIFT);
- } else if (!(smmu->features & ARM_SMMU_FEAT_VMID16)) {
- /* 8-bit VMIDs live in CBAR */
- reg |= cfg->vmid << CBAR_VMID_SHIFT;
- }
- writel_relaxed(reg, gr1_base + ARM_SMMU_GR1_CBAR(idx));
-
- /*
- * TTBCR
- * We must write this before the TTBRs, since it determines the
- * access behaviour of some fields (in particular, ASID[15:8]).
- */
- if (stage1 && smmu->version > ARM_SMMU_V1)
- writel_relaxed(cb->tcr[1], cb_base + ARM_SMMU_CB_TTBCR2);
- writel_relaxed(cb->tcr[0], cb_base + ARM_SMMU_CB_TTBCR);
-
- /* TTBRs */
- if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH32_S) {
- writel_relaxed(cfg->asid, cb_base + ARM_SMMU_CB_CONTEXTIDR);
- writel_relaxed(cb->ttbr[0], cb_base + ARM_SMMU_CB_TTBR0);
- writel_relaxed(cb->ttbr[1], cb_base + ARM_SMMU_CB_TTBR1);
- } else {
- writeq_relaxed(cb->ttbr[0], cb_base + ARM_SMMU_CB_TTBR0);
- if (stage1)
- writeq_relaxed(cb->ttbr[1], cb_base + ARM_SMMU_CB_TTBR1);
- }
-
- /* MAIRs (stage-1 only) */
- if (stage1) {
- writel_relaxed(cb->mair[0], cb_base + ARM_SMMU_CB_S1_MAIR0);
- writel_relaxed(cb->mair[1], cb_base + ARM_SMMU_CB_S1_MAIR1);
- }
-
- /* SCTLR */
- reg = SCTLR_CFIE | SCTLR_CFRE | SCTLR_AFE | SCTLR_TRE | SCTLR_M;
- if (stage1)
- reg |= SCTLR_S1_ASIDPNE;
- if (IS_ENABLED(CONFIG_CPU_BIG_ENDIAN))
- reg |= SCTLR_E;
-
- writel_relaxed(reg, cb_base + ARM_SMMU_CB_SCTLR);
-}
-
-static int arm_smmu_init_domain_context(struct iommu_domain *domain,
- struct arm_smmu_device *smmu)
-{
- int irq, start, ret = 0;
- unsigned long ias, oas;
- struct io_pgtable_ops *pgtbl_ops;
- struct io_pgtable_cfg pgtbl_cfg;
- enum io_pgtable_fmt fmt;
- struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
- struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
-
- mutex_lock(&smmu_domain->init_mutex);
- if (smmu_domain->smmu)
- goto out_unlock;
-
- if (domain->type == IOMMU_DOMAIN_IDENTITY) {
- smmu_domain->stage = ARM_SMMU_DOMAIN_BYPASS;
- smmu_domain->smmu = smmu;
- goto out_unlock;
- }
-
- /*
- * Mapping the requested stage onto what we support is surprisingly
- * complicated, mainly because the spec allows S1+S2 SMMUs without
- * support for nested translation. That means we end up with the
- * following table:
- *
- * Requested Supported Actual
- * S1 N S1
- * S1 S1+S2 S1
- * S1 S2 S2
- * S1 S1 S1
- * N N N
- * N S1+S2 S2
- * N S2 S2
- * N S1 S1
- *
- * Note that you can't actually request stage-2 mappings.
- */
- if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S1))
- smmu_domain->stage = ARM_SMMU_DOMAIN_S2;
- if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S2))
- smmu_domain->stage = ARM_SMMU_DOMAIN_S1;
-
- /*
- * Choosing a suitable context format is even more fiddly. Until we
- * grow some way for the caller to express a preference, and/or move
- * the decision into the io-pgtable code where it arguably belongs,
- * just aim for the closest thing to the rest of the system, and hope
- * that the hardware isn't esoteric enough that we can't assume AArch64
- * support to be a superset of AArch32 support...
- */
- if (smmu->features & ARM_SMMU_FEAT_FMT_AARCH32_L)
- cfg->fmt = ARM_SMMU_CTX_FMT_AARCH32_L;
- if (IS_ENABLED(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) &&
- !IS_ENABLED(CONFIG_64BIT) && !IS_ENABLED(CONFIG_ARM_LPAE) &&
- (smmu->features & ARM_SMMU_FEAT_FMT_AARCH32_S) &&
- (smmu_domain->stage == ARM_SMMU_DOMAIN_S1))
- cfg->fmt = ARM_SMMU_CTX_FMT_AARCH32_S;
- if ((IS_ENABLED(CONFIG_64BIT) || cfg->fmt == ARM_SMMU_CTX_FMT_NONE) &&
- (smmu->features & (ARM_SMMU_FEAT_FMT_AARCH64_64K |
- ARM_SMMU_FEAT_FMT_AARCH64_16K |
- ARM_SMMU_FEAT_FMT_AARCH64_4K)))
- cfg->fmt = ARM_SMMU_CTX_FMT_AARCH64;
-
- if (cfg->fmt == ARM_SMMU_CTX_FMT_NONE) {
- ret = -EINVAL;
- goto out_unlock;
- }
-
- switch (smmu_domain->stage) {
- case ARM_SMMU_DOMAIN_S1:
- cfg->cbar = CBAR_TYPE_S1_TRANS_S2_BYPASS;
- start = smmu->num_s2_context_banks;
- ias = smmu->va_size;
- oas = smmu->ipa_size;
- if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH64) {
- fmt = ARM_64_LPAE_S1;
- } else if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH32_L) {
- fmt = ARM_32_LPAE_S1;
- ias = min(ias, 32UL);
- oas = min(oas, 40UL);
- } else {
- fmt = ARM_V7S;
- ias = min(ias, 32UL);
- oas = min(oas, 32UL);
- }
- smmu_domain->tlb_ops = &arm_smmu_s1_tlb_ops;
- break;
- case ARM_SMMU_DOMAIN_NESTED:
- /*
- * We will likely want to change this if/when KVM gets
- * involved.
- */
- case ARM_SMMU_DOMAIN_S2:
- cfg->cbar = CBAR_TYPE_S2_TRANS;
- start = 0;
- ias = smmu->ipa_size;
- oas = smmu->pa_size;
- if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH64) {
- fmt = ARM_64_LPAE_S2;
- } else {
- fmt = ARM_32_LPAE_S2;
- ias = min(ias, 40UL);
- oas = min(oas, 40UL);
- }
- if (smmu->version == ARM_SMMU_V2)
- smmu_domain->tlb_ops = &arm_smmu_s2_tlb_ops_v2;
- else
- smmu_domain->tlb_ops = &arm_smmu_s2_tlb_ops_v1;
- break;
- default:
- ret = -EINVAL;
- goto out_unlock;
- }
- ret = __arm_smmu_alloc_bitmap(smmu->context_map, start,
- smmu->num_context_banks);
- if (ret < 0)
- goto out_unlock;
-
- cfg->cbndx = ret;
- if (smmu->version < ARM_SMMU_V2) {
- cfg->irptndx = atomic_inc_return(&smmu->irptndx);
- cfg->irptndx %= smmu->num_context_irqs;
- } else {
- cfg->irptndx = cfg->cbndx;
- }
-
- if (smmu_domain->stage == ARM_SMMU_DOMAIN_S2)
- cfg->vmid = cfg->cbndx + 1 + smmu->cavium_id_base;
- else
- cfg->asid = cfg->cbndx + smmu->cavium_id_base;
-
- pgtbl_cfg = (struct io_pgtable_cfg) {
- .pgsize_bitmap = smmu->pgsize_bitmap,
- .ias = ias,
- .oas = oas,
- .tlb = smmu_domain->tlb_ops,
- .iommu_dev = smmu->dev,
- };
-
- if (smmu->features & ARM_SMMU_FEAT_COHERENT_WALK)
- pgtbl_cfg.quirks = IO_PGTABLE_QUIRK_NO_DMA;
-
- if (smmu_domain->non_strict)
- pgtbl_cfg.quirks |= IO_PGTABLE_QUIRK_NON_STRICT;
-
- smmu_domain->smmu = smmu;
- pgtbl_ops = alloc_io_pgtable_ops(fmt, &pgtbl_cfg, smmu_domain);
- if (!pgtbl_ops) {
- ret = -ENOMEM;
- goto out_clear_smmu;
- }
-
- /* Update the domain's page sizes to reflect the page table format */
- domain->pgsize_bitmap = pgtbl_cfg.pgsize_bitmap;
- domain->geometry.aperture_end = (1UL << ias) - 1;
- domain->geometry.force_aperture = true;
-
- /* Initialise the context bank with our page table cfg */
- arm_smmu_init_context_bank(smmu_domain, &pgtbl_cfg);
- arm_smmu_write_context_bank(smmu, cfg->cbndx);
-
- /*
- * Request context fault interrupt. Do this last to avoid the
- * handler seeing a half-initialised domain state.
- */
- irq = smmu->irqs[smmu->num_global_irqs + cfg->irptndx];
- ret = devm_request_irq(smmu->dev, irq, arm_smmu_context_fault,
- IRQF_SHARED, "arm-smmu-context-fault", domain);
- if (ret < 0) {
- dev_err(smmu->dev, "failed to request context IRQ %d (%u)\n",
- cfg->irptndx, irq);
- cfg->irptndx = INVALID_IRPTNDX;
- }
-
- mutex_unlock(&smmu_domain->init_mutex);
-
- /* Publish page table ops for map/unmap */
- smmu_domain->pgtbl_ops = pgtbl_ops;
- return 0;
-
-out_clear_smmu:
- smmu_domain->smmu = NULL;
-out_unlock:
- mutex_unlock(&smmu_domain->init_mutex);
- return ret;
-}
-
-static void arm_smmu_destroy_domain_context(struct iommu_domain *domain)
-{
- struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
- struct arm_smmu_device *smmu = smmu_domain->smmu;
- struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
- int irq;
-
- if (!smmu || domain->type == IOMMU_DOMAIN_IDENTITY)
- return;
-
- /*
- * Disable the context bank and free the page tables before freeing
- * it.
- */
- smmu->cbs[cfg->cbndx].cfg = NULL;
- arm_smmu_write_context_bank(smmu, cfg->cbndx);
-
- if (cfg->irptndx != INVALID_IRPTNDX) {
- irq = smmu->irqs[smmu->num_global_irqs + cfg->irptndx];
- devm_free_irq(smmu->dev, irq, domain);
- }
-
- free_io_pgtable_ops(smmu_domain->pgtbl_ops);
- __arm_smmu_free_bitmap(smmu->context_map, cfg->cbndx);
-}
-
static struct iommu_domain *arm_smmu_domain_alloc(unsigned type)
{
- struct arm_smmu_domain *smmu_domain;
-
- if (type != IOMMU_DOMAIN_UNMANAGED &&
- type != IOMMU_DOMAIN_DMA &&
- type != IOMMU_DOMAIN_IDENTITY)
- return NULL;
- /*
- * Allocate the domain and initialise some of its data structures.
- * We can't really do anything meaningful until we've added a
- * master.
- */
- smmu_domain = kzalloc(sizeof(*smmu_domain), GFP_KERNEL);
- if (!smmu_domain)
- return NULL;
-
- if (type == IOMMU_DOMAIN_DMA && (using_legacy_binding ||
- iommu_get_dma_cookie(&smmu_domain->domain))) {
- kfree(smmu_domain);
- return NULL;
- }
-
- mutex_init(&smmu_domain->init_mutex);
- spin_lock_init(&smmu_domain->cb_lock);
-
- return &smmu_domain->domain;
-}
-
-static void arm_smmu_domain_free(struct iommu_domain *domain)
-{
- struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
-
- /*
- * Free the domain resources. We assume that all devices have
- * already been detached.
- */
- iommu_put_dma_cookie(domain);
- arm_smmu_destroy_domain_context(domain);
- kfree(smmu_domain);
-}
-
-static void arm_smmu_write_smr(struct arm_smmu_device *smmu, int idx)
-{
- struct arm_smmu_smr *smr = smmu->smrs + idx;
- u32 reg = smr->id << SMR_ID_SHIFT | smr->mask << SMR_MASK_SHIFT;
-
- if (!(smmu->features & ARM_SMMU_FEAT_EXIDS) && smr->valid)
- reg |= SMR_VALID;
- writel_relaxed(reg, ARM_SMMU_GR0(smmu) + ARM_SMMU_GR0_SMR(idx));
-}
-
-static void arm_smmu_write_s2cr(struct arm_smmu_device *smmu, int idx)
-{
- struct arm_smmu_s2cr *s2cr = smmu->s2crs + idx;
- u32 reg = (s2cr->type & S2CR_TYPE_MASK) << S2CR_TYPE_SHIFT |
- (s2cr->cbndx & S2CR_CBNDX_MASK) << S2CR_CBNDX_SHIFT |
- (s2cr->privcfg & S2CR_PRIVCFG_MASK) << S2CR_PRIVCFG_SHIFT;
-
- if (smmu->features & ARM_SMMU_FEAT_EXIDS && smmu->smrs &&
- smmu->smrs[idx].valid)
- reg |= S2CR_EXIDVALID;
- writel_relaxed(reg, ARM_SMMU_GR0(smmu) + ARM_SMMU_GR0_S2CR(idx));
-}
-
-static void arm_smmu_write_sme(struct arm_smmu_device *smmu, int idx)
-{
- arm_smmu_write_s2cr(smmu, idx);
- if (smmu->smrs)
- arm_smmu_write_smr(smmu, idx);
-}
-
-/*
- * The width of SMR's mask field depends on sCR0_EXIDENABLE, so this function
- * should be called after sCR0 is written.
- */
-static void arm_smmu_test_smr_masks(struct arm_smmu_device *smmu)
-{
- void __iomem *gr0_base = ARM_SMMU_GR0(smmu);
- u32 smr;
-
- if (!smmu->smrs)
- return;
-
- /*
- * SMR.ID bits may not be preserved if the corresponding MASK
- * bits are set, so check each one separately. We can reject
- * masters later if they try to claim IDs outside these masks.
- */
- smr = smmu->streamid_mask << SMR_ID_SHIFT;
- writel_relaxed(smr, gr0_base + ARM_SMMU_GR0_SMR(0));
- smr = readl_relaxed(gr0_base + ARM_SMMU_GR0_SMR(0));
- smmu->streamid_mask = smr >> SMR_ID_SHIFT;
-
- smr = smmu->streamid_mask << SMR_MASK_SHIFT;
- writel_relaxed(smr, gr0_base + ARM_SMMU_GR0_SMR(0));
- smr = readl_relaxed(gr0_base + ARM_SMMU_GR0_SMR(0));
- smmu->smr_mask_mask = smr >> SMR_MASK_SHIFT;
-}
-
-static int arm_smmu_find_sme(struct arm_smmu_device *smmu, u16 id, u16 mask)
-{
- struct arm_smmu_smr *smrs = smmu->smrs;
- int i, free_idx = -ENOSPC;
-
- /* Stream indexing is blissfully easy */
- if (!smrs)
- return id;
-
- /* Validating SMRs is... less so */
- for (i = 0; i < smmu->num_mapping_groups; ++i) {
- if (!smrs[i].valid) {
- /*
- * Note the first free entry we come across, which
- * we'll claim in the end if nothing else matches.
- */
- if (free_idx < 0)
- free_idx = i;
- continue;
- }
- /*
- * If the new entry is _entirely_ matched by an existing entry,
- * then reuse that, with the guarantee that there also cannot
- * be any subsequent conflicting entries. In normal use we'd
- * expect simply identical entries for this case, but there's
- * no harm in accommodating the generalisation.
- */
- if ((mask & smrs[i].mask) == mask &&
- !((id ^ smrs[i].id) & ~smrs[i].mask))
- return i;
- /*
- * If the new entry has any other overlap with an existing one,
- * though, then there always exists at least one stream ID
- * which would cause a conflict, and we can't allow that risk.
- */
- if (!((id ^ smrs[i].id) & ~(smrs[i].mask | mask)))
- return -EINVAL;
- }
-
- return free_idx;
-}
-
-static bool arm_smmu_free_sme(struct arm_smmu_device *smmu, int idx)
-{
- if (--smmu->s2crs[idx].count)
- return false;
-
- smmu->s2crs[idx] = s2cr_init_val;
- if (smmu->smrs)
- smmu->smrs[idx].valid = false;
-
- return true;
-}
-
-static int arm_smmu_master_alloc_smes(struct device *dev)
-{
- struct iommu_fwspec *fwspec = dev->iommu_fwspec;
- struct arm_smmu_master_cfg *cfg = fwspec->iommu_priv;
- struct arm_smmu_device *smmu = cfg->smmu;
- struct arm_smmu_smr *smrs = smmu->smrs;
- struct iommu_group *group;
- int i, idx, ret;
-
- mutex_lock(&smmu->stream_map_mutex);
- /* Figure out a viable stream map entry allocation */
- for_each_cfg_sme(fwspec, i, idx) {
- u16 sid = fwspec->ids[i];
- u16 mask = fwspec->ids[i] >> SMR_MASK_SHIFT;
-
- if (idx != INVALID_SMENDX) {
- ret = -EEXIST;
- goto out_err;
- }
-
- ret = arm_smmu_find_sme(smmu, sid, mask);
- if (ret < 0)
- goto out_err;
-
- idx = ret;
- if (smrs && smmu->s2crs[idx].count == 0) {
- smrs[idx].id = sid;
- smrs[idx].mask = mask;
- smrs[idx].valid = true;
- }
- smmu->s2crs[idx].count++;
- cfg->smendx[i] = (s16)idx;
- }
-
- group = iommu_group_get_for_dev(dev);
- if (!group)
- group = ERR_PTR(-ENOMEM);
- if (IS_ERR(group)) {
- ret = PTR_ERR(group);
- goto out_err;
- }
- iommu_group_put(group);
-
- /* It worked! Now, poke the actual hardware */
- for_each_cfg_sme(fwspec, i, idx) {
- arm_smmu_write_sme(smmu, idx);
- smmu->s2crs[idx].group = group;
- }
-
- mutex_unlock(&smmu->stream_map_mutex);
- return 0;
-
-out_err:
- while (i--) {
- arm_smmu_free_sme(smmu, cfg->smendx[i]);
- cfg->smendx[i] = INVALID_SMENDX;
- }
- mutex_unlock(&smmu->stream_map_mutex);
- return ret;
-}
-
-static void arm_smmu_master_free_smes(struct iommu_fwspec *fwspec)
-{
- struct arm_smmu_device *smmu = fwspec_smmu(fwspec);
- struct arm_smmu_master_cfg *cfg = fwspec->iommu_priv;
- int i, idx;
-
- mutex_lock(&smmu->stream_map_mutex);
- for_each_cfg_sme(fwspec, i, idx) {
- if (arm_smmu_free_sme(smmu, idx))
- arm_smmu_write_sme(smmu, idx);
- cfg->smendx[i] = INVALID_SMENDX;
- }
- mutex_unlock(&smmu->stream_map_mutex);
-}
-
-static int arm_smmu_domain_add_master(struct arm_smmu_domain *smmu_domain,
- struct iommu_fwspec *fwspec)
-{
- struct arm_smmu_device *smmu = smmu_domain->smmu;
- struct arm_smmu_s2cr *s2cr = smmu->s2crs;
- u8 cbndx = smmu_domain->cfg.cbndx;
- enum arm_smmu_s2cr_type type;
- int i, idx;
-
- if (smmu_domain->stage == ARM_SMMU_DOMAIN_BYPASS)
- type = S2CR_TYPE_BYPASS;
- else
- type = S2CR_TYPE_TRANS;
-
- for_each_cfg_sme(fwspec, i, idx) {
- if (type == s2cr[idx].type && cbndx == s2cr[idx].cbndx)
- continue;
-
- s2cr[idx].type = type;
- s2cr[idx].privcfg = S2CR_PRIVCFG_DEFAULT;
- s2cr[idx].cbndx = cbndx;
- arm_smmu_write_s2cr(smmu, idx);
- }
- return 0;
+ return arm_smmu_domain_alloc_common(type, using_generic_binding);
}
static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
{
- int ret;
- struct iommu_fwspec *fwspec = dev->iommu_fwspec;
- struct arm_smmu_device *smmu;
- struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
-
- if (!fwspec || fwspec->ops != &arm_smmu_ops) {
- dev_err(dev, "cannot attach to SMMU, is it on the same bus?\n");
- return -ENXIO;
- }
-
- /*
- * FIXME: The arch/arm DMA API code tries to attach devices to its own
- * domains between of_xlate() and add_device() - we have no way to cope
- * with that, so until ARM gets converted to rely on groups and default
- * domains, just say no (but more politely than by dereferencing NULL).
- * This should be at least a WARN_ON once that's sorted.
- */
- if (!fwspec->iommu_priv)
- return -ENODEV;
-
- smmu = fwspec_smmu(fwspec);
- /* Ensure that the domain is finalised */
- ret = arm_smmu_init_domain_context(domain, smmu);
- if (ret < 0)
- return ret;
-
- /*
- * Sanity check the domain. We don't support domains across
- * different SMMUs.
- */
- if (smmu_domain->smmu != smmu) {
- dev_err(dev,
- "cannot attach to SMMU %s whilst already attached to domain on SMMU %s\n",
- dev_name(smmu_domain->smmu->dev), dev_name(smmu->dev));
- return -EINVAL;
- }
-
- /* Looks ok, so add the device to the domain */
- return arm_smmu_domain_add_master(smmu_domain, fwspec);
-}
-
-static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova,
- phys_addr_t paddr, size_t size, int prot)
-{
- struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops;
-
- if (!ops)
- return -ENODEV;
-
- return ops->map(ops, iova, paddr, size, prot);
-}
-
-static size_t arm_smmu_unmap(struct iommu_domain *domain, unsigned long iova,
- size_t size)
-{
- struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops;
-
- if (!ops)
- return 0;
-
- return ops->unmap(ops, iova, size);
-}
-
-static void arm_smmu_flush_iotlb_all(struct iommu_domain *domain)
-{
- struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
-
- if (smmu_domain->tlb_ops)
- smmu_domain->tlb_ops->tlb_flush_all(smmu_domain);
-}
-
-static void arm_smmu_iotlb_sync(struct iommu_domain *domain)
-{
- struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
-
- if (smmu_domain->tlb_ops)
- smmu_domain->tlb_ops->tlb_sync(smmu_domain);
-}
-
-static phys_addr_t arm_smmu_iova_to_phys_hard(struct iommu_domain *domain,
- dma_addr_t iova)
-{
- struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
- struct arm_smmu_device *smmu = smmu_domain->smmu;
- struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
- struct io_pgtable_ops *ops= smmu_domain->pgtbl_ops;
- struct device *dev = smmu->dev;
- void __iomem *cb_base;
- u32 tmp;
- u64 phys;
- unsigned long va, flags;
-
- cb_base = ARM_SMMU_CB(smmu, cfg->cbndx);
-
- spin_lock_irqsave(&smmu_domain->cb_lock, flags);
- /* ATS1 registers can only be written atomically */
- va = iova & ~0xfffUL;
- if (smmu->version == ARM_SMMU_V2)
- smmu_write_atomic_lq(va, cb_base + ARM_SMMU_CB_ATS1PR);
- else /* Register is only 32-bit in v1 */
- writel_relaxed(va, cb_base + ARM_SMMU_CB_ATS1PR);
-
- if (readl_poll_timeout_atomic(cb_base + ARM_SMMU_CB_ATSR, tmp,
- !(tmp & ATSR_ACTIVE), 5, 50)) {
- spin_unlock_irqrestore(&smmu_domain->cb_lock, flags);
- dev_err(dev,
- "iova to phys timed out on %pad. Falling back to software table walk.\n",
- &iova);
- return ops->iova_to_phys(ops, iova);
- }
-
- phys = readq_relaxed(cb_base + ARM_SMMU_CB_PAR);
- spin_unlock_irqrestore(&smmu_domain->cb_lock, flags);
- if (phys & CB_PAR_F) {
- dev_err(dev, "translation fault!\n");
- dev_err(dev, "PAR = 0x%llx\n", phys);
- return 0;
- }
-
- return (phys & GENMASK_ULL(39, 12)) | (iova & 0xfff);
-}
-
-static phys_addr_t arm_smmu_iova_to_phys(struct iommu_domain *domain,
- dma_addr_t iova)
-{
- struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
- struct io_pgtable_ops *ops = smmu_domain->pgtbl_ops;
-
- if (domain->type == IOMMU_DOMAIN_IDENTITY)
- return iova;
-
- if (!ops)
- return 0;
-
- if (smmu_domain->smmu->features & ARM_SMMU_FEAT_TRANS_OPS &&
- smmu_domain->stage == ARM_SMMU_DOMAIN_S1)
- return arm_smmu_iova_to_phys_hard(domain, iova);
-
- return ops->iova_to_phys(ops, iova);
-}
-
-static bool arm_smmu_capable(enum iommu_cap cap)
-{
- switch (cap) {
- case IOMMU_CAP_CACHE_COHERENCY:
- /*
- * Return true here as the SMMU can always send out coherent
- * requests.
- */
- return true;
- case IOMMU_CAP_NOEXEC:
- return true;
- default:
- return false;
- }
+ return arm_smmu_attach_dev_common(domain, dev, &arm_smmu_ops);
}
static int arm_smmu_match_node(struct device *dev, void *data)
@@ -1379,9 +210,8 @@ struct arm_smmu_device *arm_smmu_get_by_fwnode(struct fwnode_handle *fwnode)
static int arm_smmu_add_device(struct device *dev)
{
struct arm_smmu_device *smmu;
- struct arm_smmu_master_cfg *cfg;
struct iommu_fwspec *fwspec = dev->iommu_fwspec;
- int i, ret;
+ int ret;
if (using_legacy_binding) {
ret = arm_smmu_register_legacy_master(dev, &smmu);
@@ -1400,44 +230,7 @@ static int arm_smmu_add_device(struct device *dev)
return -ENODEV;
}
- ret = -EINVAL;
- for (i = 0; i < fwspec->num_ids; i++) {
- u16 sid = fwspec->ids[i];
- u16 mask = fwspec->ids[i] >> SMR_MASK_SHIFT;
-
- if (sid & ~smmu->streamid_mask) {
- dev_err(dev, "stream ID 0x%x out of range for SMMU (0x%x)\n",
- sid, smmu->streamid_mask);
- goto out_free;
- }
- if (mask & ~smmu->smr_mask_mask) {
- dev_err(dev, "SMR mask 0x%x out of range for SMMU (0x%x)\n",
- mask, smmu->smr_mask_mask);
- goto out_free;
- }
- }
-
- ret = -ENOMEM;
- cfg = kzalloc(offsetof(struct arm_smmu_master_cfg, smendx[i]),
- GFP_KERNEL);
- if (!cfg)
- goto out_free;
-
- cfg->smmu = smmu;
- fwspec->iommu_priv = cfg;
- while (i--)
- cfg->smendx[i] = INVALID_SMENDX;
-
- ret = arm_smmu_master_alloc_smes(dev);
- if (ret)
- goto out_cfg_free;
-
- iommu_device_link(&smmu->iommu, dev);
-
- return 0;
-
-out_cfg_free:
- kfree(cfg);
+ return arm_smmu_add_device_common(dev, smmu);
out_free:
iommu_fwspec_free(dev);
return ret;
@@ -1445,162 +238,7 @@ static int arm_smmu_add_device(struct device *dev)
static void arm_smmu_remove_device(struct device *dev)
{
- struct iommu_fwspec *fwspec = dev->iommu_fwspec;
- struct arm_smmu_master_cfg *cfg;
- struct arm_smmu_device *smmu;
-
-
- if (!fwspec || fwspec->ops != &arm_smmu_ops)
- return;
-
- cfg = fwspec->iommu_priv;
- smmu = cfg->smmu;
-
- iommu_device_unlink(&smmu->iommu, dev);
- arm_smmu_master_free_smes(fwspec);
- iommu_group_remove_device(dev);
- kfree(fwspec->iommu_priv);
- iommu_fwspec_free(dev);
-}
-
-static struct iommu_group *arm_smmu_device_group(struct device *dev)
-{
- struct iommu_fwspec *fwspec = dev->iommu_fwspec;
- struct arm_smmu_device *smmu = fwspec_smmu(fwspec);
- struct iommu_group *group = NULL;
- int i, idx;
-
- for_each_cfg_sme(fwspec, i, idx) {
- if (group && smmu->s2crs[idx].group &&
- group != smmu->s2crs[idx].group)
- return ERR_PTR(-EINVAL);
-
- group = smmu->s2crs[idx].group;
- }
-
- if (group)
- return iommu_group_ref_get(group);
-
- if (dev_is_pci(dev))
- group = pci_device_group(dev);
- else if (dev_is_fsl_mc(dev))
- group = fsl_mc_device_group(dev);
- else
- group = generic_device_group(dev);
-
- return group;
-}
-
-static int arm_smmu_domain_get_attr(struct iommu_domain *domain,
- enum iommu_attr attr, void *data)
-{
- struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
-
- switch(domain->type) {
- case IOMMU_DOMAIN_UNMANAGED:
- switch (attr) {
- case DOMAIN_ATTR_NESTING:
- *(int *)data = (smmu_domain->stage == ARM_SMMU_DOMAIN_NESTED);
- return 0;
- default:
- return -ENODEV;
- }
- break;
- case IOMMU_DOMAIN_DMA:
- switch (attr) {
- case DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE:
- *(int *)data = smmu_domain->non_strict;
- return 0;
- default:
- return -ENODEV;
- }
- break;
- default:
- return -EINVAL;
- }
-}
-
-static int arm_smmu_domain_set_attr(struct iommu_domain *domain,
- enum iommu_attr attr, void *data)
-{
- int ret = 0;
- struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
-
- mutex_lock(&smmu_domain->init_mutex);
-
- switch(domain->type) {
- case IOMMU_DOMAIN_UNMANAGED:
- switch (attr) {
- case DOMAIN_ATTR_NESTING:
- if (smmu_domain->smmu) {
- ret = -EPERM;
- goto out_unlock;
- }
-
- if (*(int *)data)
- smmu_domain->stage = ARM_SMMU_DOMAIN_NESTED;
- else
- smmu_domain->stage = ARM_SMMU_DOMAIN_S1;
- break;
- default:
- ret = -ENODEV;
- }
- break;
- case IOMMU_DOMAIN_DMA:
- switch (attr) {
- case DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE:
- smmu_domain->non_strict = *(int *)data;
- break;
- default:
- ret = -ENODEV;
- }
- break;
- default:
- ret = -EINVAL;
- }
-out_unlock:
- mutex_unlock(&smmu_domain->init_mutex);
- return ret;
-}
-
-static int arm_smmu_of_xlate(struct device *dev, struct of_phandle_args *args)
-{
- u32 mask, fwid = 0;
-
- if (args->args_count > 0)
- fwid |= (u16)args->args[0];
-
- if (args->args_count > 1)
- fwid |= (u16)args->args[1] << SMR_MASK_SHIFT;
- else if (!of_property_read_u32(args->np, "stream-match-mask", &mask))
- fwid |= (u16)mask << SMR_MASK_SHIFT;
-
- return iommu_fwspec_add_ids(dev, &fwid, 1);
-}
-
-static void arm_smmu_get_resv_regions(struct device *dev,
- struct list_head *head)
-{
- struct iommu_resv_region *region;
- int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
-
- region = iommu_alloc_resv_region(MSI_IOVA_BASE, MSI_IOVA_LENGTH,
- prot, IOMMU_RESV_SW_MSI);
- if (!region)
- return;
-
- list_add_tail(®ion->list, head);
-
- iommu_dma_get_resv_regions(dev, head);
-}
-
-static void arm_smmu_put_resv_regions(struct device *dev,
- struct list_head *head)
-{
- struct iommu_resv_region *entry, *next;
-
- list_for_each_entry_safe(entry, next, head, list)
- kfree(entry);
+ return arm_smmu_remove_device_common(dev, &arm_smmu_ops);
}
static struct iommu_ops arm_smmu_ops = {
@@ -1624,321 +262,10 @@ static struct iommu_ops arm_smmu_ops = {
.pgsize_bitmap = -1UL, /* Restricted during device attach */
};
-static void arm_smmu_device_reset(struct arm_smmu_device *smmu)
-{
- void __iomem *gr0_base = ARM_SMMU_GR0(smmu);
- int i;
- u32 reg, major;
-
- /* clear global FSR */
- reg = readl_relaxed(ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sGFSR);
- writel(reg, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sGFSR);
-
- /*
- * Reset stream mapping groups: Initial values mark all SMRn as
- * invalid and all S2CRn as bypass unless overridden.
- */
- for (i = 0; i < smmu->num_mapping_groups; ++i)
- arm_smmu_write_sme(smmu, i);
-
- if (smmu->model == ARM_MMU500) {
- /*
- * Before clearing ARM_MMU500_ACTLR_CPRE, need to
- * clear CACHE_LOCK bit of ACR first. And, CACHE_LOCK
- * bit is only present in MMU-500r2 onwards.
- */
- reg = readl_relaxed(gr0_base + ARM_SMMU_GR0_ID7);
- major = (reg >> ID7_MAJOR_SHIFT) & ID7_MAJOR_MASK;
- reg = readl_relaxed(gr0_base + ARM_SMMU_GR0_sACR);
- if (major >= 2)
- reg &= ~ARM_MMU500_ACR_CACHE_LOCK;
- /*
- * Allow unmatched Stream IDs to allocate bypass
- * TLB entries for reduced latency.
- */
- reg |= ARM_MMU500_ACR_SMTNMB_TLBEN | ARM_MMU500_ACR_S2CRB_TLBEN;
- writel_relaxed(reg, gr0_base + ARM_SMMU_GR0_sACR);
- }
-
- /* Make sure all context banks are disabled and clear CB_FSR */
- for (i = 0; i < smmu->num_context_banks; ++i) {
- void __iomem *cb_base = ARM_SMMU_CB(smmu, i);
-
- arm_smmu_write_context_bank(smmu, i);
- writel_relaxed(FSR_FAULT, cb_base + ARM_SMMU_CB_FSR);
- /*
- * Disable MMU-500's not-particularly-beneficial next-page
- * prefetcher for the sake of errata #841119 and #826419.
- */
- if (smmu->model == ARM_MMU500) {
- reg = readl_relaxed(cb_base + ARM_SMMU_CB_ACTLR);
- reg &= ~ARM_MMU500_ACTLR_CPRE;
- writel_relaxed(reg, cb_base + ARM_SMMU_CB_ACTLR);
- }
- }
-
- /* Invalidate the TLB, just in case */
- writel_relaxed(0, gr0_base + ARM_SMMU_GR0_TLBIALLH);
- writel_relaxed(0, gr0_base + ARM_SMMU_GR0_TLBIALLNSNH);
-
- reg = readl_relaxed(ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
-
- /* Enable fault reporting */
- reg |= (sCR0_GFRE | sCR0_GFIE | sCR0_GCFGFRE | sCR0_GCFGFIE);
-
- /* Disable TLB broadcasting. */
- reg |= (sCR0_VMIDPNE | sCR0_PTM);
-
- /* Enable client access, handling unmatched streams as appropriate */
- reg &= ~sCR0_CLIENTPD;
- if (disable_bypass)
- reg |= sCR0_USFCFG;
- else
- reg &= ~sCR0_USFCFG;
-
- /* Disable forced broadcasting */
- reg &= ~sCR0_FB;
-
- /* Don't upgrade barriers */
- reg &= ~(sCR0_BSU_MASK << sCR0_BSU_SHIFT);
-
- if (smmu->features & ARM_SMMU_FEAT_VMID16)
- reg |= sCR0_VMID16EN;
-
- if (smmu->features & ARM_SMMU_FEAT_EXIDS)
- reg |= sCR0_EXIDENABLE;
-
- /* Push the button */
- arm_smmu_tlb_sync_global(smmu);
- writel(reg, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
-}
-
-static int arm_smmu_id_size_to_bits(int size)
-{
- switch (size) {
- case 0:
- return 32;
- case 1:
- return 36;
- case 2:
- return 40;
- case 3:
- return 42;
- case 4:
- return 44;
- case 5:
- default:
- return 48;
- }
-}
-
static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu)
{
- unsigned long size;
- void __iomem *gr0_base = ARM_SMMU_GR0(smmu);
- u32 id;
- bool cttw_reg, cttw_fw = smmu->features & ARM_SMMU_FEAT_COHERENT_WALK;
- int i;
-
- dev_notice(smmu->dev, "probing hardware configuration...\n");
- dev_notice(smmu->dev, "SMMUv%d with:\n",
- smmu->version == ARM_SMMU_V2 ? 2 : 1);
-
- /* ID0 */
- id = readl_relaxed(gr0_base + ARM_SMMU_GR0_ID0);
-
- /* Restrict available stages based on module parameter */
- if (force_stage == 1)
- id &= ~(ID0_S2TS | ID0_NTS);
- else if (force_stage == 2)
- id &= ~(ID0_S1TS | ID0_NTS);
-
- if (id & ID0_S1TS) {
- smmu->features |= ARM_SMMU_FEAT_TRANS_S1;
- dev_notice(smmu->dev, "\tstage 1 translation\n");
- }
-
- if (id & ID0_S2TS) {
- smmu->features |= ARM_SMMU_FEAT_TRANS_S2;
- dev_notice(smmu->dev, "\tstage 2 translation\n");
- }
-
- if (id & ID0_NTS) {
- smmu->features |= ARM_SMMU_FEAT_TRANS_NESTED;
- dev_notice(smmu->dev, "\tnested translation\n");
- }
-
- if (!(smmu->features &
- (ARM_SMMU_FEAT_TRANS_S1 | ARM_SMMU_FEAT_TRANS_S2))) {
- dev_err(smmu->dev, "\tno translation support!\n");
- return -ENODEV;
- }
-
- if ((id & ID0_S1TS) &&
- ((smmu->version < ARM_SMMU_V2) || !(id & ID0_ATOSNS))) {
- smmu->features |= ARM_SMMU_FEAT_TRANS_OPS;
- dev_notice(smmu->dev, "\taddress translation ops\n");
- }
-
- /*
- * In order for DMA API calls to work properly, we must defer to what
- * the FW says about coherency, regardless of what the hardware claims.
- * Fortunately, this also opens up a workaround for systems where the
- * ID register value has ended up configured incorrectly.
- */
- cttw_reg = !!(id & ID0_CTTW);
- if (cttw_fw || cttw_reg)
- dev_notice(smmu->dev, "\t%scoherent table walk\n",
- cttw_fw ? "" : "non-");
- if (cttw_fw != cttw_reg)
- dev_notice(smmu->dev,
- "\t(IDR0.CTTW overridden by FW configuration)\n");
-
- /* Max. number of entries we have for stream matching/indexing */
- if (smmu->version == ARM_SMMU_V2 && id & ID0_EXIDS) {
- smmu->features |= ARM_SMMU_FEAT_EXIDS;
- size = 1 << 16;
- } else {
- size = 1 << ((id >> ID0_NUMSIDB_SHIFT) & ID0_NUMSIDB_MASK);
- }
- smmu->streamid_mask = size - 1;
- if (id & ID0_SMS) {
- smmu->features |= ARM_SMMU_FEAT_STREAM_MATCH;
- size = (id >> ID0_NUMSMRG_SHIFT) & ID0_NUMSMRG_MASK;
- if (size == 0) {
- dev_err(smmu->dev,
- "stream-matching supported, but no SMRs present!\n");
- return -ENODEV;
- }
-
- /* Zero-initialised to mark as invalid */
- smmu->smrs = devm_kcalloc(smmu->dev, size, sizeof(*smmu->smrs),
- GFP_KERNEL);
- if (!smmu->smrs)
- return -ENOMEM;
-
- dev_notice(smmu->dev,
- "\tstream matching with %lu register groups", size);
- }
- /* s2cr->type == 0 means translation, so initialise explicitly */
- smmu->s2crs = devm_kmalloc_array(smmu->dev, size, sizeof(*smmu->s2crs),
- GFP_KERNEL);
- if (!smmu->s2crs)
- return -ENOMEM;
- for (i = 0; i < size; i++)
- smmu->s2crs[i] = s2cr_init_val;
-
- smmu->num_mapping_groups = size;
- mutex_init(&smmu->stream_map_mutex);
- spin_lock_init(&smmu->global_sync_lock);
-
- if (smmu->version < ARM_SMMU_V2 || !(id & ID0_PTFS_NO_AARCH32)) {
- smmu->features |= ARM_SMMU_FEAT_FMT_AARCH32_L;
- if (!(id & ID0_PTFS_NO_AARCH32S))
- smmu->features |= ARM_SMMU_FEAT_FMT_AARCH32_S;
- }
-
- /* ID1 */
- id = readl_relaxed(gr0_base + ARM_SMMU_GR0_ID1);
- smmu->pgshift = (id & ID1_PAGESIZE) ? 16 : 12;
-
- /* Check for size mismatch of SMMU address space from mapped region */
- size = 1 << (((id >> ID1_NUMPAGENDXB_SHIFT) & ID1_NUMPAGENDXB_MASK) + 1);
- size <<= smmu->pgshift;
- if (smmu->cb_base != gr0_base + size)
- dev_warn(smmu->dev,
- "SMMU address space size (0x%lx) differs from mapped region size (0x%tx)!\n",
- size * 2, (smmu->cb_base - gr0_base) * 2);
-
- smmu->num_s2_context_banks = (id >> ID1_NUMS2CB_SHIFT) & ID1_NUMS2CB_MASK;
- smmu->num_context_banks = (id >> ID1_NUMCB_SHIFT) & ID1_NUMCB_MASK;
- if (smmu->num_s2_context_banks > smmu->num_context_banks) {
- dev_err(smmu->dev, "impossible number of S2 context banks!\n");
- return -ENODEV;
- }
- dev_notice(smmu->dev, "\t%u context banks (%u stage-2 only)\n",
- smmu->num_context_banks, smmu->num_s2_context_banks);
- /*
- * Cavium CN88xx erratum #27704.
- * Ensure ASID and VMID allocation is unique across all SMMUs in
- * the system.
- */
- if (smmu->model == CAVIUM_SMMUV2) {
- smmu->cavium_id_base =
- atomic_add_return(smmu->num_context_banks,
- &cavium_smmu_context_count);
- smmu->cavium_id_base -= smmu->num_context_banks;
- dev_notice(smmu->dev, "\tenabling workaround for Cavium erratum 27704\n");
- }
- smmu->cbs = devm_kcalloc(smmu->dev, smmu->num_context_banks,
- sizeof(*smmu->cbs), GFP_KERNEL);
- if (!smmu->cbs)
- return -ENOMEM;
-
- /* ID2 */
- id = readl_relaxed(gr0_base + ARM_SMMU_GR0_ID2);
- size = arm_smmu_id_size_to_bits((id >> ID2_IAS_SHIFT) & ID2_IAS_MASK);
- smmu->ipa_size = size;
-
- /* The output mask is also applied for bypass */
- size = arm_smmu_id_size_to_bits((id >> ID2_OAS_SHIFT) & ID2_OAS_MASK);
- smmu->pa_size = size;
-
- if (id & ID2_VMID16)
- smmu->features |= ARM_SMMU_FEAT_VMID16;
-
- /*
- * What the page table walker can address actually depends on which
- * descriptor format is in use, but since a) we don't know that yet,
- * and b) it can vary per context bank, this will have to do...
- */
- if (dma_set_mask_and_coherent(smmu->dev, DMA_BIT_MASK(size)))
- dev_warn(smmu->dev,
- "failed to set DMA mask for table walker\n");
-
- if (smmu->version < ARM_SMMU_V2) {
- smmu->va_size = smmu->ipa_size;
- if (smmu->version == ARM_SMMU_V1_64K)
- smmu->features |= ARM_SMMU_FEAT_FMT_AARCH64_64K;
- } else {
- size = (id >> ID2_UBS_SHIFT) & ID2_UBS_MASK;
- smmu->va_size = arm_smmu_id_size_to_bits(size);
- if (id & ID2_PTFS_4K)
- smmu->features |= ARM_SMMU_FEAT_FMT_AARCH64_4K;
- if (id & ID2_PTFS_16K)
- smmu->features |= ARM_SMMU_FEAT_FMT_AARCH64_16K;
- if (id & ID2_PTFS_64K)
- smmu->features |= ARM_SMMU_FEAT_FMT_AARCH64_64K;
- }
-
- /* Now we've corralled the various formats, what'll it do? */
- if (smmu->features & ARM_SMMU_FEAT_FMT_AARCH32_S)
- smmu->pgsize_bitmap |= SZ_4K | SZ_64K | SZ_1M | SZ_16M;
- if (smmu->features &
- (ARM_SMMU_FEAT_FMT_AARCH32_L | ARM_SMMU_FEAT_FMT_AARCH64_4K))
- smmu->pgsize_bitmap |= SZ_4K | SZ_2M | SZ_1G;
- if (smmu->features & ARM_SMMU_FEAT_FMT_AARCH64_16K)
- smmu->pgsize_bitmap |= SZ_16K | SZ_32M;
- if (smmu->features & ARM_SMMU_FEAT_FMT_AARCH64_64K)
- smmu->pgsize_bitmap |= SZ_64K | SZ_512M;
-
- if (arm_smmu_ops.pgsize_bitmap == -1UL)
- arm_smmu_ops.pgsize_bitmap = smmu->pgsize_bitmap;
- else
- arm_smmu_ops.pgsize_bitmap |= smmu->pgsize_bitmap;
- dev_notice(smmu->dev, "\tSupported page sizes: 0x%08lx\n",
- smmu->pgsize_bitmap);
-
-
- if (smmu->features & ARM_SMMU_FEAT_TRANS_S1)
- dev_notice(smmu->dev, "\tStage-1: %lu-bit VA -> %lu-bit IPA\n",
- smmu->va_size, smmu->ipa_size);
-
- if (smmu->features & ARM_SMMU_FEAT_TRANS_S2)
- dev_notice(smmu->dev, "\tStage-2: %lu-bit IPA -> %lu-bit PA\n",
- smmu->ipa_size, smmu->pa_size);
-
- return 0;
+ return arm_smmu_device_cfg_probe_common(smmu,
+ &arm_smmu_ops, &cavium_smmu_context_count, force_stage);
}
struct arm_smmu_match_data {
@@ -2119,6 +446,7 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
if (IS_ERR(smmu->base))
return PTR_ERR(smmu->base);
smmu->cb_base = smmu->base + resource_size(res) / 2;
+ smmu->num_smmus = 1;
num_irqs = 0;
while ((res = platform_get_resource(pdev, IORESOURCE_IRQ, num_irqs))) {
@@ -2224,21 +552,6 @@ static int arm_smmu_legacy_bus_init(void)
}
device_initcall_sync(arm_smmu_legacy_bus_init);
-static int arm_smmu_device_remove(struct platform_device *pdev)
-{
- struct arm_smmu_device *smmu = platform_get_drvdata(pdev);
-
- if (!smmu)
- return -ENODEV;
-
- if (!bitmap_empty(smmu->context_map, ARM_SMMU_MAX_CBS))
- dev_err(&pdev->dev, "removing device with active domains!\n");
-
- /* Turn the thing off */
- writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
- return 0;
-}
-
static void arm_smmu_device_shutdown(struct platform_device *pdev)
{
arm_smmu_device_remove(pdev);
@@ -2266,6 +579,6 @@ static struct platform_driver arm_smmu_driver = {
};
module_platform_driver(arm_smmu_driver);
-MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");
+MODULE_DESCRIPTION("Driver For ARM architected V1 & V2 SMMU implementations");
MODULE_AUTHOR("Will Deacon <will.deacon@arm.com>");
MODULE_LICENSE("GPL v2");
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v3 4/6] iommu/tegra194_smmu: Add Tegra194 SMMU driver
2018-12-04 1:36 [PATCH v3 0/5] Add Tegra194 Dual ARM SMMU driver Krishna Reddy
` (2 preceding siblings ...)
2018-12-04 1:36 ` [PATCH v3 3/6] iommu/arm-smmu: update arm-smmu.c to use ARM SMMU library Krishna Reddy
@ 2018-12-04 1:36 ` Krishna Reddy
2018-12-04 1:36 ` [PATCH v3 5/6] arm64: defconfig: Enable ARM_SMMU_TEGRA Krishna Reddy
2018-12-04 1:36 ` [PATCH v3 6/6] arm64: tegra: Add SMMU nodes to Tegra194 device tree Krishna Reddy
5 siblings, 0 replies; 9+ messages in thread
From: Krishna Reddy @ 2018-12-04 1:36 UTC (permalink / raw)
To: will.deacon, robin.murphy, joro
Cc: linux-arm-kernel, iommu, linux-kernel, linux-tegra, treding,
yhsu, snikam, praithatha, talho, avanbrunt, thomasz, olof,
jtukkinen, mperttunen, Krishna Reddy
Tegra194 SMMU driver supports Dual ARM SMMU configuration
necessary for Tegra194 SOC.
The IOVA accesses from HW devices are interleaved across two
ARM SMMU devices transparently.
Signed-off-by: Krishna Reddy <vdumpa@nvidia.com>
---
drivers/iommu/Kconfig | 11 ++
drivers/iommu/Makefile | 1 +
drivers/iommu/tegra194-smmu.c | 394 ++++++++++++++++++++++++++++++++++++++++++
3 files changed, 406 insertions(+)
create mode 100644 drivers/iommu/tegra194-smmu.c
diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index d9a2571..2dc08c7 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -357,6 +357,17 @@ config ARM_SMMU
Say Y here if your SoC includes an IOMMU device implementing
the ARM SMMU architecture.
+config ARM_SMMU_TEGRA
+ bool "Dual ARM SMMU(MMU-500) support on Tegra"
+ depends on ARM64 && MMU && ARCH_TEGRA
+ select IOMMU_API
+ select IOMMU_IO_PGTABLE_LPAE
+ help
+ Support for implementation of Dual ARM SMMU Instances as single
+ SMMU Device on Tegra194. IOVA accesses from devices are interleaved
+ across these two ARM SMMU Instances. Number of ARM SMMU Instances used
+ in the SMMU device is not known to HW devices.
+
config ARM_SMMU_V3
bool "ARM Ltd. System MMU Version 3 (SMMUv3) Support"
depends on ARM64
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index ea87cae..f1587bb 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -15,6 +15,7 @@ obj-$(CONFIG_AMD_IOMMU_DEBUGFS) += amd_iommu_debugfs.o
obj-$(CONFIG_AMD_IOMMU_V2) += amd_iommu_v2.o
obj-$(CONFIG_ARM_SMMU) += arm-smmu.o
obj-$(CONFIG_ARM_SMMU) += lib-arm-smmu.o
+obj-$(CONFIG_ARM_SMMU_TEGRA) += tegra194-smmu.o
obj-$(CONFIG_ARM_SMMU_V3) += arm-smmu-v3.o
obj-$(CONFIG_DMAR_TABLE) += dmar.o
obj-$(CONFIG_INTEL_IOMMU) += intel-iommu.o intel-pasid.o
diff --git a/drivers/iommu/tegra194-smmu.c b/drivers/iommu/tegra194-smmu.c
new file mode 100644
index 0000000..7b21670
--- /dev/null
+++ b/drivers/iommu/tegra194-smmu.c
@@ -0,0 +1,394 @@
+/*
+ * Copyright (c) 2018, NVIDIA Corporation
+ * Author: Krishna Reddy <vdumpa@nvidia.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#define pr_fmt(fmt) "tegra194-smmu: " fmt
+
+#include <linux/acpi.h>
+#include <linux/acpi_iort.h>
+#include <linux/atomic.h>
+#include <linux/delay.h>
+#include <linux/dma-iommu.h>
+#include <linux/dma-mapping.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/io-64-nonatomic-hi-lo.h>
+#include <linux/iopoll.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_address.h>
+#include <linux/of_device.h>
+#include <linux/of_iommu.h>
+#include <linux/pci.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+
+#include <linux/amba/bus.h>
+#include <linux/fsl/mc.h>
+
+#include "io-pgtable.h"
+#include "arm-smmu-regs.h"
+#include "lib-arm-smmu.h"
+
+static int force_stage;
+module_param(force_stage, int, S_IRUGO);
+MODULE_PARM_DESC(force_stage,
+ "Force SMMU mappings to be installed at a particular stage of translation. A value of '1' or '2' forces the corresponding stage. All other values are ignored (i.e. no stage is forced). Note that selecting a specific stage will disable support for nested translation.");
+static bool disable_bypass;
+module_param(disable_bypass, bool, S_IRUGO);
+MODULE_PARM_DESC(disable_bypass,
+ "Disable bypass streams such that incoming transactions from devices that are not attached to an iommu domain will report an abort back to the device and will not be allowed to pass through the SMMU.");
+
+static struct platform_driver tegra194_smmu_driver;
+static struct iommu_ops tegra194_smmu_ops;
+
+static struct iommu_domain *tegra194_smmu_domain_alloc(unsigned int type)
+{
+ return arm_smmu_domain_alloc_common(type, true);
+}
+
+static int tegra194_smmu_attach_dev(struct iommu_domain *domain,
+ struct device *dev)
+{
+ return arm_smmu_attach_dev_common(domain, dev, &tegra194_smmu_ops);
+}
+
+static int tegra194_smmu_match_node(struct device *dev, void *data)
+{
+ return dev->fwnode == data;
+}
+
+static
+struct arm_smmu_device *tegra194_smmu_get_by_fwnode(struct fwnode_handle *fwnode)
+{
+ struct device *dev = driver_find_device(&tegra194_smmu_driver.driver,
+ NULL, fwnode, tegra194_smmu_match_node);
+ put_device(dev);
+ return dev ? dev_get_drvdata(dev) : NULL;
+}
+
+static int tegra194_smmu_add_device(struct device *dev)
+{
+ struct arm_smmu_device *smmu;
+ struct iommu_fwspec *fwspec = dev->iommu_fwspec;
+
+ if (fwspec && fwspec->ops == &tegra194_smmu_ops)
+ smmu = tegra194_smmu_get_by_fwnode(fwspec->iommu_fwnode);
+ else
+ return -ENODEV;
+
+ return arm_smmu_add_device_common(dev, smmu);
+}
+
+static void tegra194_smmu_remove_device(struct device *dev)
+{
+ return arm_smmu_remove_device_common(dev, &tegra194_smmu_ops);
+}
+
+static struct iommu_ops tegra194_smmu_ops = {
+ .capable = arm_smmu_capable,
+ .domain_alloc = tegra194_smmu_domain_alloc,
+ .domain_free = arm_smmu_domain_free,
+ .attach_dev = tegra194_smmu_attach_dev,
+ .map = arm_smmu_map,
+ .unmap = arm_smmu_unmap,
+ .flush_iotlb_all = arm_smmu_flush_iotlb_all,
+ .iotlb_sync = arm_smmu_iotlb_sync,
+ .iova_to_phys = arm_smmu_iova_to_phys,
+ .add_device = tegra194_smmu_add_device,
+ .remove_device = tegra194_smmu_remove_device,
+ .device_group = arm_smmu_device_group,
+ .domain_get_attr = arm_smmu_domain_get_attr,
+ .domain_set_attr = arm_smmu_domain_set_attr,
+ .of_xlate = arm_smmu_of_xlate,
+ .get_resv_regions = arm_smmu_get_resv_regions,
+ .put_resv_regions = arm_smmu_put_resv_regions,
+ .pgsize_bitmap = -1UL, /* Restricted during device attach */
+};
+
+static int tegra194_smmu_device_cfg_probe(struct arm_smmu_device *smmu)
+{
+ return arm_smmu_device_cfg_probe_common(smmu,
+ &tegra194_smmu_ops, NULL, force_stage);
+}
+
+struct tegra194_smmu_match_data {
+ enum arm_smmu_arch_version version;
+ enum arm_smmu_implementation model;
+};
+
+#define TEGRA194_SMMU_MATCH_DATA(name, ver, imp) \
+static struct tegra194_smmu_match_data name = { .version = ver, .model = imp }
+
+TEGRA194_SMMU_MATCH_DATA(arm_mmu500, ARM_SMMU_V2, ARM_MMU500);
+
+static const struct of_device_id tegra194_smmu_of_match[] = {
+ { .compatible = "tegra194,arm,mmu-500", .data = &arm_mmu500 },
+ { },
+};
+MODULE_DEVICE_TABLE(of, tegra194_smmu_of_match);
+
+#ifdef CONFIG_ACPI
+static int acpi_smmu_get_data(u32 model, struct arm_smmu_device *smmu)
+{
+ int ret = 0;
+
+ switch (model) {
+ case ACPI_IORT_SMMU_CORELINK_MMU500:
+ smmu->version = ARM_SMMU_V2;
+ smmu->model = ARM_MMU500;
+ break;
+ default:
+ ret = -ENODEV;
+ }
+
+ return ret;
+}
+
+static int tegra194_smmu_device_acpi_probe(struct platform_device *pdev,
+ struct arm_smmu_device *smmu)
+{
+ struct device *dev = smmu->dev;
+ struct acpi_iort_node *node =
+ *(struct acpi_iort_node **)dev_get_platdata(dev);
+ struct acpi_iort_smmu *iort_smmu;
+ int ret;
+
+ /* Retrieve SMMU1/2 specific data */
+ iort_smmu = (struct acpi_iort_smmu *)node->node_data;
+
+ ret = acpi_smmu_get_data(iort_smmu->model, smmu);
+ if (ret < 0)
+ return ret;
+
+ /* Ignore the configuration access interrupt */
+ smmu->num_global_irqs = 1;
+
+ if (iort_smmu->flags & ACPI_IORT_SMMU_COHERENT_WALK)
+ smmu->features |= ARM_SMMU_FEAT_COHERENT_WALK;
+
+ return 0;
+}
+#else
+static inline int tegra194_smmu_device_acpi_probe(struct platform_device *pdev,
+ struct arm_smmu_device *smmu)
+{
+ return -ENODEV;
+}
+#endif
+
+static int tegra194_smmu_device_dt_probe(struct platform_device *pdev,
+ struct arm_smmu_device *smmu)
+{
+ const struct tegra194_smmu_match_data *data;
+ struct device *dev = &pdev->dev;
+
+ if (of_property_read_u32(dev->of_node, "#global-interrupts",
+ &smmu->num_global_irqs)) {
+ dev_err(dev, "missing #global-interrupts property\n");
+ return -ENODEV;
+ }
+
+ data = of_device_get_match_data(dev);
+ smmu->version = data->version;
+ smmu->model = data->model;
+
+ if (of_dma_is_coherent(dev->of_node))
+ smmu->features |= ARM_SMMU_FEAT_COHERENT_WALK;
+
+ return 0;
+}
+
+static void tegra194_smmu_bus_init(void)
+{
+ /* Oh, for a proper bus abstraction */
+ if (!iommu_present(&platform_bus_type))
+ bus_set_iommu(&platform_bus_type, &tegra194_smmu_ops);
+#ifdef CONFIG_ARM_AMBA
+ if (!iommu_present(&amba_bustype))
+ bus_set_iommu(&amba_bustype, &tegra194_smmu_ops);
+#endif
+#ifdef CONFIG_PCI
+ if (!iommu_present(&pci_bus_type)) {
+ pci_request_acs();
+ bus_set_iommu(&pci_bus_type, &tegra194_smmu_ops);
+ }
+#endif
+#ifdef CONFIG_FSL_MC_BUS
+ if (!iommu_present(&fsl_mc_bus_type))
+ bus_set_iommu(&fsl_mc_bus_type, &tegra194_smmu_ops);
+#endif
+}
+
+static int tegra194_smmu_device_probe(struct platform_device *pdev)
+{
+ struct resource *res;
+ resource_size_t ioaddr;
+ struct arm_smmu_device *smmu;
+ struct device *dev = &pdev->dev;
+ int num_irqs, i, err;
+
+ smmu = devm_kzalloc(dev, sizeof(*smmu), GFP_KERNEL);
+ if (!smmu) {
+ dev_err(dev, "failed to allocate arm_smmu_device\n");
+ return -ENOMEM;
+ }
+ smmu->dev = dev;
+ smmu->num_smmus = 2;
+ smmu->bases = devm_kzalloc(dev, sizeof(*smmu->bases) * smmu->num_smmus,
+ GFP_KERNEL);
+ if (!smmu->bases)
+ return -ENOMEM;
+
+ if (dev->of_node)
+ err = tegra194_smmu_device_dt_probe(pdev, smmu);
+ else
+ err = tegra194_smmu_device_acpi_probe(pdev, smmu);
+
+ if (err)
+ return err;
+
+ for (i = 0; i < smmu->num_smmus; i++) {
+ res = platform_get_resource(pdev, IORESOURCE_MEM, i);
+ ioaddr = res->start;
+ smmu->bases[i] = devm_ioremap_resource(dev, res);
+ if (IS_ERR(smmu->bases[i]))
+ return PTR_ERR(smmu->bases[i]);
+ if (i == 0) {
+ smmu->base = smmu->bases[i];
+ smmu->cb_base = smmu->base + resource_size(res) / 2;
+ }
+ }
+
+ num_irqs = 0;
+ while ((res = platform_get_resource(pdev, IORESOURCE_IRQ, num_irqs))) {
+ num_irqs++;
+ if (num_irqs > smmu->num_global_irqs)
+ smmu->num_context_irqs++;
+ }
+
+ if (!smmu->num_context_irqs) {
+ dev_err(dev, "found %d interrupts but expected at least %d\n",
+ num_irqs, smmu->num_global_irqs + 1);
+ return -ENODEV;
+ }
+
+ smmu->irqs = devm_kcalloc(dev, num_irqs, sizeof(*smmu->irqs),
+ GFP_KERNEL);
+ if (!smmu->irqs) {
+ dev_err(dev, "failed to allocate %d irqs\n", num_irqs);
+ return -ENOMEM;
+ }
+
+ for (i = 0; i < num_irqs; ++i) {
+ int irq = platform_get_irq(pdev, i);
+
+ if (irq < 0) {
+ dev_err(dev, "failed to get irq index %d\n", i);
+ return -ENODEV;
+ }
+ smmu->irqs[i] = irq;
+ }
+
+ err = tegra194_smmu_device_cfg_probe(smmu);
+ if (err)
+ return err;
+
+ if (smmu->version == ARM_SMMU_V2) {
+ if (smmu->num_context_banks > smmu->num_context_irqs) {
+ dev_err(dev,
+ "found only %d context irq(s) but %d required\n",
+ smmu->num_context_irqs, smmu->num_context_banks);
+ return -ENODEV;
+ }
+
+ /* Ignore superfluous interrupts */
+ smmu->num_context_irqs = smmu->num_context_banks;
+ }
+
+ for (i = 0; i < smmu->num_global_irqs; ++i) {
+ err = devm_request_irq(smmu->dev, smmu->irqs[i],
+ arm_smmu_global_fault,
+ IRQF_SHARED,
+ "arm-smmu global fault",
+ smmu);
+ if (err) {
+ dev_err(dev, "failed to request global IRQ %d (%u)\n",
+ i, smmu->irqs[i]);
+ return err;
+ }
+ }
+
+ err = iommu_device_sysfs_add(&smmu->iommu, smmu->dev, NULL,
+ "smmu.%pa", &ioaddr);
+ if (err) {
+ dev_err(dev, "Failed to register iommu in sysfs\n");
+ return err;
+ }
+
+ iommu_device_set_ops(&smmu->iommu, &tegra194_smmu_ops);
+ iommu_device_set_fwnode(&smmu->iommu, dev->fwnode);
+
+ err = iommu_device_register(&smmu->iommu);
+ if (err) {
+ dev_err(dev, "Failed to register iommu\n");
+ return err;
+ }
+
+ platform_set_drvdata(pdev, smmu);
+ arm_smmu_device_reset(smmu);
+ arm_smmu_test_smr_masks(smmu);
+
+ /*
+ * For ACPI and generic DT bindings, an SMMU will be probed before
+ * any device which might need it, so we want the bus ops in place
+ * ready to handle default domain setup as soon as any SMMU exists.
+ */
+ tegra194_smmu_bus_init();
+
+ return 0;
+}
+
+static void tegra194_smmu_device_shutdown(struct platform_device *pdev)
+{
+ arm_smmu_device_remove(pdev);
+}
+
+static int __maybe_unused tegra194_smmu_pm_resume(struct device *dev)
+{
+ struct arm_smmu_device *smmu = dev_get_drvdata(dev);
+
+ arm_smmu_device_reset(smmu);
+ return 0;
+}
+
+static SIMPLE_DEV_PM_OPS(tegra194_smmu_pm_ops, NULL, tegra194_smmu_pm_resume);
+
+static struct platform_driver tegra194_smmu_driver = {
+ .driver = {
+ .name = "tegra194-smmu",
+ .of_match_table = of_match_ptr(tegra194_smmu_of_match),
+ .pm = &tegra194_smmu_pm_ops,
+ },
+ .probe = tegra194_smmu_device_probe,
+ .remove = arm_smmu_device_remove,
+ .shutdown = tegra194_smmu_device_shutdown,
+};
+module_platform_driver(tegra194_smmu_driver);
+
+MODULE_DESCRIPTION("Tegra194 SMMU Driver");
+MODULE_AUTHOR("Krishna Reddy <vdumpa@nvidia.com>");
+MODULE_LICENSE("GPL v2");
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v3 5/6] arm64: defconfig: Enable ARM_SMMU_TEGRA
2018-12-04 1:36 [PATCH v3 0/5] Add Tegra194 Dual ARM SMMU driver Krishna Reddy
` (3 preceding siblings ...)
2018-12-04 1:36 ` [PATCH v3 4/6] iommu/tegra194_smmu: Add Tegra194 SMMU driver Krishna Reddy
@ 2018-12-04 1:36 ` Krishna Reddy
2018-12-04 1:36 ` [PATCH v3 6/6] arm64: tegra: Add SMMU nodes to Tegra194 device tree Krishna Reddy
5 siblings, 0 replies; 9+ messages in thread
From: Krishna Reddy @ 2018-12-04 1:36 UTC (permalink / raw)
To: will.deacon, robin.murphy, joro
Cc: linux-arm-kernel, iommu, linux-kernel, linux-tegra, treding,
yhsu, snikam, praithatha, talho, avanbrunt, thomasz, olof,
jtukkinen, mperttunen, Krishna Reddy
Enabling CONFIG_ARM_SMMU_TEGRA that is used
by Tegra194 SOC.
Signed-off-by: Krishna Reddy <vdumpa@nvidia.com>
---
arch/arm64/configs/defconfig | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index 5c2b1f6..e5ea13b 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -625,6 +625,7 @@ CONFIG_QCOM_APCS_IPC=y
CONFIG_ROCKCHIP_IOMMU=y
CONFIG_TEGRA_IOMMU_SMMU=y
CONFIG_ARM_SMMU=y
+CONFIG_ARM_SMMU_TEGRA=y
CONFIG_ARM_SMMU_V3=y
CONFIG_QCOM_IOMMU=y
CONFIG_REMOTEPROC=m
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v3 6/6] arm64: tegra: Add SMMU nodes to Tegra194 device tree
2018-12-04 1:36 [PATCH v3 0/5] Add Tegra194 Dual ARM SMMU driver Krishna Reddy
` (4 preceding siblings ...)
2018-12-04 1:36 ` [PATCH v3 5/6] arm64: defconfig: Enable ARM_SMMU_TEGRA Krishna Reddy
@ 2018-12-04 1:36 ` Krishna Reddy
5 siblings, 0 replies; 9+ messages in thread
From: Krishna Reddy @ 2018-12-04 1:36 UTC (permalink / raw)
To: will.deacon, robin.murphy, joro
Cc: linux-arm-kernel, iommu, linux-kernel, linux-tegra, treding,
yhsu, snikam, praithatha, talho, avanbrunt, thomasz, olof,
jtukkinen, mperttunen, Krishna Reddy
Add SMMU nodes and dma-ranges to Tegra194 device tree.
Tegra194 has three ARM SMMU Instances. Two of them are used
together to access IOVA interleaved. The third one is used
as regular ARM SMMU.
Signed-off-by: Krishna Reddy <vdumpa@nvidia.com>
---
arch/arm64/boot/dts/nvidia/tegra194.dtsi | 148 +++++++++++++++++++++++++++++++
1 file changed, 148 insertions(+)
diff --git a/arch/arm64/boot/dts/nvidia/tegra194.dtsi b/arch/arm64/boot/dts/nvidia/tegra194.dtsi
index f274562..9a3e08c 100644
--- a/arch/arm64/boot/dts/nvidia/tegra194.dtsi
+++ b/arch/arm64/boot/dts/nvidia/tegra194.dtsi
@@ -12,6 +12,7 @@
interrupt-parent = <&gic>;
#address-cells = <2>;
#size-cells = <2>;
+ dma-ranges = <0x0 0x0 0x0 0x0 0x8 0x0>;
/* control backbone */
cbb {
@@ -957,4 +958,151 @@
(GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_LOW)>;
interrupt-parent = <&gic>;
};
+
+ dualsmmu: iommu@12000000 {
+ compatible = "tegra194,arm,mmu-500";
+ reg = <0 0x12000000 0 0x800000>,
+ <0 0x11000000 0 0x800000>;
+ interrupts = <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>;
+ stream-match-mask = <0x7f80>;
+ #global-interrupts = <1>;
+ #iommu-cells = <1>;
+ };
+
+ smmu: iommu@10000000 {
+ compatible = "arm,mmu-500";
+ reg = <0 0x10000000 0 0x800000>;
+ interrupts = <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 170 IRQ_TYPE_LEVEL_HIGH>;
+ stream-match-mask = <0x7f80>;
+ #global-interrupts = <1>;
+ #iommu-cells = <1>;
+ };
};
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH v3 2/6] iommu/arm-smmu: Add support to program multiple ARM SMMU's identically
2018-12-04 1:36 ` [PATCH v3 2/6] iommu/arm-smmu: Add support to program multiple ARM SMMU's identically Krishna Reddy
@ 2018-12-19 18:07 ` Robin Murphy
2018-12-19 20:36 ` Krishna Reddy
0 siblings, 1 reply; 9+ messages in thread
From: Robin Murphy @ 2018-12-19 18:07 UTC (permalink / raw)
To: Krishna Reddy, will.deacon, joro
Cc: linux-arm-kernel, iommu, linux-kernel, linux-tegra, treding,
yhsu, snikam, praithatha, talho, avanbrunt, thomasz, olof,
jtukkinen, mperttunen, jon.he
Hi Krishna,
On 04/12/2018 01:36, Krishna Reddy wrote:
> Add support to program multiple ARM SMMU's identically as one SMMU device.
> Tegra194 uses Two ARM SMMU's as one SMMU device and both ARM SMMU's need
> to be programmed identically.
The whole point of the library idea was to factor out the code in such a
way that all the details specific to a particular implementation can be
kept together. But what this patch does is insert Tegra194-specific
handling all through the 'common' code, which is the exact opposite of
that concept and just makes more hard-to-maintain mess.
The amount of copy-paste duplication in patch #4 has the opposite
problem - about 95% of that isn't Tegra194-specific at all (I mean, how
many fsl_mc instances does it have?), and having multiple copies of
generic code with the potential to diverge is also not what anyone
wants. Plus I don't think ending up building multiple separate drivers
will even work in general - thanks to the current state of
bus_set_iommu() etc., you can't use the regular driver for your third
SMMU at the same time.
I think what really needs to be done is to conceptually split the driver
into "architecture" and "implementation" layers - at some point after
the holidays we're probably going to sit down and go through all the
various quirks and specifics we know about to try and figure out what
that should actually look like.
Robin.
> Signed-off-by: Krishna Reddy <vdumpa@nvidia.com>
> ---
> drivers/iommu/lib-arm-smmu.c | 191 ++++++++++++++++++++++++++++++++-----------
> 1 file changed, 144 insertions(+), 47 deletions(-)
>
> diff --git a/drivers/iommu/lib-arm-smmu.c b/drivers/iommu/lib-arm-smmu.c
> index 6aba5db..7036763 100644
> --- a/drivers/iommu/lib-arm-smmu.c
> +++ b/drivers/iommu/lib-arm-smmu.c
> @@ -69,9 +69,9 @@
> * therefore this actually makes more sense than it might first appear.
> */
> #ifdef CONFIG_64BIT
> -#define smmu_write_atomic_lq writeq_relaxed
> +#define smmu_write_atomic_lq writeq_all_relaxed
> #else
> -#define smmu_write_atomic_lq writel_relaxed
> +#define smmu_write_atomic_lq writel_all_relaxed
> #endif
>
> /* Translation context bank */
> @@ -135,6 +135,48 @@ struct arm_smmu_domain {
> struct iommu_domain domain;
> };
>
> +#define to_smmu_intance(smmu, inst, addr) \
> + (addr - smmu->bases[0] + smmu->bases[inst])
> +
> +static void writel_all(struct arm_smmu_device *smmu,
> + u32 value, void __iomem *addr)
> +{
> + int i;
> +
> + writel(value, addr);
> + for (i = 1; i < smmu->num_smmus; i++) {
> + void __iomem *reg_addr = to_smmu_intance(smmu, i, addr);
> +
> + writel(value, reg_addr);
> + }
> +}
> +
> +static void writel_all_relaxed(struct arm_smmu_device *smmu,
> + u32 value, void __iomem *addr)
> +{
> + int i;
> +
> + writel_relaxed(value, addr);
> + for (i = 1; i < smmu->num_smmus; i++) {
> + void __iomem *reg_addr = to_smmu_intance(smmu, i, addr);
> +
> + writel_relaxed(value, reg_addr);
> + }
> +}
> +
> +static void writeq_all_relaxed(struct arm_smmu_device *smmu,
> + u64 value, void __iomem *addr)
> +{
> + int i;
> +
> + writeq_relaxed(value, addr);
> + for (i = 1; i < smmu->num_smmus; i++) {
> + void __iomem *reg_addr = to_smmu_intance(smmu, i, addr);
> +
> + writeq_relaxed(value, reg_addr);
> + }
> +}
> +
> static struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom)
> {
> return container_of(dom, struct arm_smmu_domain, domain);
> @@ -179,25 +221,37 @@ static void __arm_smmu_tlb_sync(struct arm_smmu_device *smmu,
>
> static void arm_smmu_tlb_sync_global(struct arm_smmu_device *smmu)
> {
> - void __iomem *base = ARM_SMMU_GR0(smmu);
> + int i;
> unsigned long flags;
>
> spin_lock_irqsave(&smmu->global_sync_lock, flags);
> - __arm_smmu_tlb_sync(smmu, base + ARM_SMMU_GR0_sTLBGSYNC,
> - base + ARM_SMMU_GR0_sTLBGSTATUS);
> + for (i = 0; i < smmu->num_smmus; i++) {
> + void __iomem *base = ARM_SMMU_GR0(smmu);
> +
> + if (i > 0)
> + base = to_smmu_intance(smmu, i, base);
> + __arm_smmu_tlb_sync(smmu, base + ARM_SMMU_GR0_sTLBGSYNC,
> + base + ARM_SMMU_GR0_sTLBGSTATUS);
> + }
> spin_unlock_irqrestore(&smmu->global_sync_lock, flags);
> }
>
> static void arm_smmu_tlb_sync_context(void *cookie)
> {
> + int i;
> struct arm_smmu_domain *smmu_domain = cookie;
> struct arm_smmu_device *smmu = smmu_domain->smmu;
> - void __iomem *base = ARM_SMMU_CB(smmu, smmu_domain->cfg.cbndx);
> unsigned long flags;
>
> spin_lock_irqsave(&smmu_domain->cb_lock, flags);
> - __arm_smmu_tlb_sync(smmu, base + ARM_SMMU_CB_TLBSYNC,
> - base + ARM_SMMU_CB_TLBSTATUS);
> + for (i = 0; i < smmu->num_smmus; i++) {
> + void __iomem *base = ARM_SMMU_CB(smmu, smmu_domain->cfg.cbndx);
> +
> + if (i > 0)
> + base = to_smmu_intance(smmu, i, base);
> + __arm_smmu_tlb_sync(smmu, base + ARM_SMMU_CB_TLBSYNC,
> + base + ARM_SMMU_CB_TLBSTATUS);
> + }
> spin_unlock_irqrestore(&smmu_domain->cb_lock, flags);
> }
>
> @@ -212,13 +266,14 @@ static void arm_smmu_tlb_inv_context_s1(void *cookie)
> {
> struct arm_smmu_domain *smmu_domain = cookie;
> struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> - void __iomem *base = ARM_SMMU_CB(smmu_domain->smmu, cfg->cbndx);
> + struct arm_smmu_device *smmu = smmu_domain->smmu;
> + void __iomem *base = ARM_SMMU_CB(smmu, cfg->cbndx);
>
> /*
> * NOTE: this is not a relaxed write; it needs to guarantee that PTEs
> * cleared by the current CPU are visible to the SMMU before the TLBI.
> */
> - writel(cfg->asid, base + ARM_SMMU_CB_S1_TLBIASID);
> + writel_all(smmu, cfg->asid, base + ARM_SMMU_CB_S1_TLBIASID);
> arm_smmu_tlb_sync_context(cookie);
> }
>
> @@ -229,7 +284,7 @@ static void arm_smmu_tlb_inv_context_s2(void *cookie)
> void __iomem *base = ARM_SMMU_GR0(smmu);
>
> /* NOTE: see above */
> - writel(smmu_domain->cfg.vmid, base + ARM_SMMU_GR0_TLBIVMID);
> + writel_all(smmu, smmu_domain->cfg.vmid, base + ARM_SMMU_GR0_TLBIVMID);
> arm_smmu_tlb_sync_global(smmu);
> }
>
> @@ -237,11 +292,12 @@ static void arm_smmu_tlb_inv_range_nosync(unsigned long iova, size_t size,
> size_t granule, bool leaf, void *cookie)
> {
> struct arm_smmu_domain *smmu_domain = cookie;
> + struct arm_smmu_device *smmu = smmu_domain->smmu;
> struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> bool stage1 = cfg->cbar != CBAR_TYPE_S2_TRANS;
> - void __iomem *reg = ARM_SMMU_CB(smmu_domain->smmu, cfg->cbndx);
> + void __iomem *reg = ARM_SMMU_CB(smmu, cfg->cbndx);
>
> - if (smmu_domain->smmu->features & ARM_SMMU_FEAT_COHERENT_WALK)
> + if (smmu->features & ARM_SMMU_FEAT_COHERENT_WALK)
> wmb();
>
> if (stage1) {
> @@ -251,7 +307,7 @@ static void arm_smmu_tlb_inv_range_nosync(unsigned long iova, size_t size,
> iova &= ~12UL;
> iova |= cfg->asid;
> do {
> - writel_relaxed(iova, reg);
> + writel_all_relaxed(smmu, iova, reg);
> iova += granule;
> } while (size -= granule);
> } else {
> @@ -267,7 +323,7 @@ static void arm_smmu_tlb_inv_range_nosync(unsigned long iova, size_t size,
> ARM_SMMU_CB_S2_TLBIIPAS2;
> iova >>= 12;
> do {
> - smmu_write_atomic_lq(iova, reg);
> + smmu_write_atomic_lq(smmu, iova, reg);
> iova += granule >> 12;
> } while (size -= granule);
> }
> @@ -283,12 +339,13 @@ static void arm_smmu_tlb_inv_vmid_nosync(unsigned long iova, size_t size,
> size_t granule, bool leaf, void *cookie)
> {
> struct arm_smmu_domain *smmu_domain = cookie;
> - void __iomem *base = ARM_SMMU_GR0(smmu_domain->smmu);
> + struct arm_smmu_device *smmu = smmu_domain->smmu;
> + void __iomem *base = ARM_SMMU_GR0(smmu);
>
> - if (smmu_domain->smmu->features & ARM_SMMU_FEAT_COHERENT_WALK)
> + if (smmu->features & ARM_SMMU_FEAT_COHERENT_WALK)
> wmb();
>
> - writel_relaxed(smmu_domain->cfg.vmid, base + ARM_SMMU_GR0_TLBIVMID);
> + writel_all_relaxed(smmu, smmu_domain->cfg.vmid, base + ARM_SMMU_GR0_TLBIVMID);
> }
>
> static const struct iommu_gather_ops arm_smmu_s1_tlb_ops = {
> @@ -309,7 +366,8 @@ static const struct iommu_gather_ops arm_smmu_s2_tlb_ops_v1 = {
> .tlb_sync = arm_smmu_tlb_sync_vmid,
> };
>
> -irqreturn_t arm_smmu_context_fault(int irq, void *dev)
> +static irqreturn_t __arm_smmu_context_fault(int irq, void *dev,
> + void __iomem *cb_base)
> {
> u32 fsr, fsynr;
> unsigned long iova;
> @@ -317,9 +375,7 @@ irqreturn_t arm_smmu_context_fault(int irq, void *dev)
> struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> struct arm_smmu_device *smmu = smmu_domain->smmu;
> - void __iomem *cb_base;
>
> - cb_base = ARM_SMMU_CB(smmu, cfg->cbndx);
> fsr = readl_relaxed(cb_base + ARM_SMMU_CB_FSR);
>
> if (!(fsr & FSR_FAULT))
> @@ -336,11 +392,33 @@ irqreturn_t arm_smmu_context_fault(int irq, void *dev)
> return IRQ_HANDLED;
> }
>
> -irqreturn_t arm_smmu_global_fault(int irq, void *dev)
> +static irqreturn_t arm_smmu_context_fault(int irq, void *dev)
> +{
> + int i;
> + irqreturn_t irq_ret = IRQ_NONE;
> + struct iommu_domain *domain = dev;
> + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> + struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> + struct arm_smmu_device *smmu = smmu_domain->smmu;
> +
> + for (i = 0; i < smmu->num_smmus; i++) {
> + void __iomem *cb_base = ARM_SMMU_CB(smmu, cfg->cbndx);
> +
> + if (i > 0)
> + cb_base = to_smmu_intance(smmu, i, cb_base);
> + irq_ret = __arm_smmu_context_fault(irq, dev, cb_base);
> + if (irq_ret == IRQ_HANDLED)
> + break;
> + }
> +
> + return irq_ret;
> +}
> +
> +static irqreturn_t __arm_smmu_global_fault(int irq, void *dev,
> + void __iomem *gr0_base)
> {
> - u32 gfsr, gfsynr0, gfsynr1, gfsynr2;
> struct arm_smmu_device *smmu = dev;
> - void __iomem *gr0_base = ARM_SMMU_GR0_NS(smmu);
> + u32 gfsr, gfsynr0, gfsynr1, gfsynr2;
>
> gfsr = readl_relaxed(gr0_base + ARM_SMMU_GR0_sGFSR);
> gfsynr0 = readl_relaxed(gr0_base + ARM_SMMU_GR0_sGFSYNR0);
> @@ -360,6 +438,25 @@ irqreturn_t arm_smmu_global_fault(int irq, void *dev)
> return IRQ_HANDLED;
> }
>
> +irqreturn_t arm_smmu_global_fault(int irq, void *dev)
> +{
> + int i;
> + irqreturn_t irq_ret = IRQ_NONE;
> + struct arm_smmu_device *smmu = dev;
> +
> + for (i = 0; i < smmu->num_smmus; i++) {
> + void __iomem *gr0_base = ARM_SMMU_GR0_NS(smmu);
> +
> + if (i > 0)
> + gr0_base = to_smmu_intance(smmu, i, gr0_base);
> + irq_ret = __arm_smmu_global_fault(irq, dev, gr0_base);
> + if (irq_ret == IRQ_HANDLED)
> + break;
> + }
> +
> + return irq_ret;
> +}
> +
> static void arm_smmu_init_context_bank(struct arm_smmu_domain *smmu_domain,
> struct io_pgtable_cfg *pgtbl_cfg)
> {
> @@ -423,7 +520,7 @@ static void arm_smmu_write_context_bank(struct arm_smmu_device *smmu, int idx)
>
> /* Unassigned context banks only need disabling */
> if (!cfg) {
> - writel_relaxed(0, cb_base + ARM_SMMU_CB_SCTLR);
> + writel_all_relaxed(smmu, 0, cb_base + ARM_SMMU_CB_SCTLR);
> return;
> }
>
> @@ -440,7 +537,7 @@ static void arm_smmu_write_context_bank(struct arm_smmu_device *smmu, int idx)
> if (smmu->features & ARM_SMMU_FEAT_VMID16)
> reg |= cfg->vmid << CBA2R_VMID_SHIFT;
>
> - writel_relaxed(reg, gr1_base + ARM_SMMU_GR1_CBA2R(idx));
> + writel_all_relaxed(smmu, reg, gr1_base + ARM_SMMU_GR1_CBA2R(idx));
> }
>
> /* CBAR */
> @@ -459,7 +556,7 @@ static void arm_smmu_write_context_bank(struct arm_smmu_device *smmu, int idx)
> /* 8-bit VMIDs live in CBAR */
> reg |= cfg->vmid << CBAR_VMID_SHIFT;
> }
> - writel_relaxed(reg, gr1_base + ARM_SMMU_GR1_CBAR(idx));
> + writel_all_relaxed(smmu, reg, gr1_base + ARM_SMMU_GR1_CBAR(idx));
>
> /*
> * TTBCR
> @@ -467,14 +564,14 @@ static void arm_smmu_write_context_bank(struct arm_smmu_device *smmu, int idx)
> * access behaviour of some fields (in particular, ASID[15:8]).
> */
> if (stage1 && smmu->version > ARM_SMMU_V1)
> - writel_relaxed(cb->tcr[1], cb_base + ARM_SMMU_CB_TTBCR2);
> - writel_relaxed(cb->tcr[0], cb_base + ARM_SMMU_CB_TTBCR);
> + writel_all_relaxed(smmu, cb->tcr[1], cb_base + ARM_SMMU_CB_TTBCR2);
> + writel_all_relaxed(smmu, cb->tcr[0], cb_base + ARM_SMMU_CB_TTBCR);
>
> /* TTBRs */
> if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH32_S) {
> - writel_relaxed(cfg->asid, cb_base + ARM_SMMU_CB_CONTEXTIDR);
> - writel_relaxed(cb->ttbr[0], cb_base + ARM_SMMU_CB_TTBR0);
> - writel_relaxed(cb->ttbr[1], cb_base + ARM_SMMU_CB_TTBR1);
> + writel_all_relaxed(smmu, cfg->asid, cb_base + ARM_SMMU_CB_CONTEXTIDR);
> + writel_all_relaxed(smmu, cb->ttbr[0], cb_base + ARM_SMMU_CB_TTBR0);
> + writel_all_relaxed(smmu, cb->ttbr[1], cb_base + ARM_SMMU_CB_TTBR1);
> } else {
> writeq_relaxed(cb->ttbr[0], cb_base + ARM_SMMU_CB_TTBR0);
> if (stage1)
> @@ -484,8 +581,8 @@ static void arm_smmu_write_context_bank(struct arm_smmu_device *smmu, int idx)
>
> /* MAIRs (stage-1 only) */
> if (stage1) {
> - writel_relaxed(cb->mair[0], cb_base + ARM_SMMU_CB_S1_MAIR0);
> - writel_relaxed(cb->mair[1], cb_base + ARM_SMMU_CB_S1_MAIR1);
> + writel_all_relaxed(smmu, cb->mair[0], cb_base + ARM_SMMU_CB_S1_MAIR0);
> + writel_all_relaxed(smmu, cb->mair[1], cb_base + ARM_SMMU_CB_S1_MAIR1);
> }
>
> /* SCTLR */
> @@ -495,7 +592,7 @@ static void arm_smmu_write_context_bank(struct arm_smmu_device *smmu, int idx)
> if (IS_ENABLED(CONFIG_CPU_BIG_ENDIAN))
> reg |= SCTLR_E;
>
> - writel_relaxed(reg, cb_base + ARM_SMMU_CB_SCTLR);
> + writel_all_relaxed(smmu, reg, cb_base + ARM_SMMU_CB_SCTLR);
> }
>
> static int arm_smmu_init_domain_context(struct iommu_domain *domain,
> @@ -763,7 +860,7 @@ static void arm_smmu_write_smr(struct arm_smmu_device *smmu, int idx)
>
> if (!(smmu->features & ARM_SMMU_FEAT_EXIDS) && smr->valid)
> reg |= SMR_VALID;
> - writel_relaxed(reg, ARM_SMMU_GR0(smmu) + ARM_SMMU_GR0_SMR(idx));
> + writel_all_relaxed(smmu, reg, ARM_SMMU_GR0(smmu) + ARM_SMMU_GR0_SMR(idx));
> }
>
> static void arm_smmu_write_s2cr(struct arm_smmu_device *smmu, int idx)
> @@ -776,7 +873,7 @@ static void arm_smmu_write_s2cr(struct arm_smmu_device *smmu, int idx)
> if (smmu->features & ARM_SMMU_FEAT_EXIDS && smmu->smrs &&
> smmu->smrs[idx].valid)
> reg |= S2CR_EXIDVALID;
> - writel_relaxed(reg, ARM_SMMU_GR0(smmu) + ARM_SMMU_GR0_S2CR(idx));
> + writel_all_relaxed(smmu, reg, ARM_SMMU_GR0(smmu) + ARM_SMMU_GR0_S2CR(idx));
> }
>
> static void arm_smmu_write_sme(struct arm_smmu_device *smmu, int idx)
> @@ -1071,9 +1168,9 @@ static phys_addr_t arm_smmu_iova_to_phys_hard(struct iommu_domain *domain,
> /* ATS1 registers can only be written atomically */
> va = iova & ~0xfffUL;
> if (smmu->version == ARM_SMMU_V2)
> - smmu_write_atomic_lq(va, cb_base + ARM_SMMU_CB_ATS1PR);
> + smmu_write_atomic_lq(smmu, va, cb_base + ARM_SMMU_CB_ATS1PR);
> else /* Register is only 32-bit in v1 */
> - writel_relaxed(va, cb_base + ARM_SMMU_CB_ATS1PR);
> + writel_all_relaxed(smmu, va, cb_base + ARM_SMMU_CB_ATS1PR);
>
> if (readl_poll_timeout_atomic(cb_base + ARM_SMMU_CB_ATSR, tmp,
> !(tmp & ATSR_ACTIVE), 5, 50)) {
> @@ -1346,7 +1443,7 @@ void arm_smmu_device_reset(struct arm_smmu_device *smmu)
>
> /* clear global FSR */
> reg = readl_relaxed(ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sGFSR);
> - writel(reg, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sGFSR);
> + writel_all(smmu, reg, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sGFSR);
>
> /*
> * Reset stream mapping groups: Initial values mark all SMRn as
> @@ -1371,7 +1468,7 @@ void arm_smmu_device_reset(struct arm_smmu_device *smmu)
> * TLB entries for reduced latency.
> */
> reg |= ARM_MMU500_ACR_SMTNMB_TLBEN | ARM_MMU500_ACR_S2CRB_TLBEN;
> - writel_relaxed(reg, gr0_base + ARM_SMMU_GR0_sACR);
> + writel_all_relaxed(smmu, reg, gr0_base + ARM_SMMU_GR0_sACR);
> }
>
> /* Make sure all context banks are disabled and clear CB_FSR */
> @@ -1379,7 +1476,7 @@ void arm_smmu_device_reset(struct arm_smmu_device *smmu)
> void __iomem *cb_base = ARM_SMMU_CB(smmu, i);
>
> arm_smmu_write_context_bank(smmu, i);
> - writel_relaxed(FSR_FAULT, cb_base + ARM_SMMU_CB_FSR);
> + writel_all_relaxed(smmu, FSR_FAULT, cb_base + ARM_SMMU_CB_FSR);
> /*
> * Disable MMU-500's not-particularly-beneficial next-page
> * prefetcher for the sake of errata #841119 and #826419.
> @@ -1387,13 +1484,13 @@ void arm_smmu_device_reset(struct arm_smmu_device *smmu)
> if (smmu->model == ARM_MMU500) {
> reg = readl_relaxed(cb_base + ARM_SMMU_CB_ACTLR);
> reg &= ~ARM_MMU500_ACTLR_CPRE;
> - writel_relaxed(reg, cb_base + ARM_SMMU_CB_ACTLR);
> + writel_all_relaxed(smmu, reg, cb_base + ARM_SMMU_CB_ACTLR);
> }
> }
>
> /* Invalidate the TLB, just in case */
> - writel_relaxed(0, gr0_base + ARM_SMMU_GR0_TLBIALLH);
> - writel_relaxed(0, gr0_base + ARM_SMMU_GR0_TLBIALLNSNH);
> + writel_all_relaxed(smmu, 0, gr0_base + ARM_SMMU_GR0_TLBIALLH);
> + writel_all_relaxed(smmu, 0, gr0_base + ARM_SMMU_GR0_TLBIALLNSNH);
>
> reg = readl_relaxed(ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
>
> @@ -1424,7 +1521,7 @@ void arm_smmu_device_reset(struct arm_smmu_device *smmu)
>
> /* Push the button */
> arm_smmu_tlb_sync_global(smmu);
> - writel(reg, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
> + writel_all(smmu, reg, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
> }
>
> static int arm_smmu_id_size_to_bits(int size)
> @@ -1666,6 +1763,6 @@ int arm_smmu_device_remove(struct platform_device *pdev)
> dev_err(&pdev->dev, "removing device with active domains!\n");
>
> /* Turn the thing off */
> - writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
> + writel_all(smmu, sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
> return 0;
> }
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: [PATCH v3 2/6] iommu/arm-smmu: Add support to program multiple ARM SMMU's identically
2018-12-19 18:07 ` Robin Murphy
@ 2018-12-19 20:36 ` Krishna Reddy
0 siblings, 0 replies; 9+ messages in thread
From: Krishna Reddy @ 2018-12-19 20:36 UTC (permalink / raw)
To: Robin Murphy, will.deacon, joro
Cc: linux-arm-kernel, iommu, linux-kernel, linux-tegra,
Thierry Reding, Yu-Huan Hsu, Sachin Nikam, Pritesh Raithatha,
Timo Alho, Alexander Van Brunt, Thomas Zeng (SW-TEGRA),
olof, Juha Tukkinen, Mikko Perttunen, jon.he
Hi Robin,
Thanks for the feedback :)
>The whole point of the library idea was to factor out the code in such a way that all the details
>specific to a particular implementation can be kept together. But what this patch does is insert
>Tegra194-specific handling all through the 'common' code, which is the exact opposite of that concept
>and just makes more hard-to-maintain mess.
In an attempt to reuse most of the ARM SMMU implementation, which heavily relies on data from arm_smmu_device,
The library code has been added with some functionality only usable by Tegra194 SMMU driver.
In V4 patches, I am working on to add a mechanism to override writel/readl functions in library so that
Tegra smmu driver can override read/write functions and handle programming of multiple instances on its own.
>The amount of copy-paste duplication in patch #4 has the opposite problem - about 95% of that isn't
>Tegra194-specific at all (I mean, how many fsl_mc instances does it have?), and having multiple copies of
> generic code with the potential to diverge is also not what anyone wants.
I have split the code in a way that library only contains the code that deals with register programming.
And avoided platform driver code and DT parsing code getting into library, which can allow drivers changing
Independently if necessary in future.
> Plus I don't think ending up building
> multiple separate drivers will even work in general - thanks to the current state of
>bus_set_iommu() etc., you can't use the regular driver for your third SMMU at the same time.
Good point!
From code, platform_dma_configure/of_dma_configure/of_iommu_configure takes care
of setting right iommu_ops for devices based on the iommu DT node they have in iommus=<> entry.
If iommu.c is updated to use dev->bus->dma_configure(), then it doesn't really need to use dev->bus->iommu_ops.
dev->bus->dma_configure() can be used to set dev->dma_ops to the right one, if dev->dma_ops is not
already set.
If this approach looks good, I can make a patch to clean up bus->iommu_ops usage related code to allow
devices to use specific SMMU instance as they need.
>I think what really needs to be done is to conceptually split the driver into "architecture" and "implementation"
> layers - at some point after the holidays we're probably going to sit down and go through all the various quirks and
> specifics we know about to try and figure out what that should actually look like.
If you can provide some high level details on what to keep in library vs implementation after holidays, I would be
happy to rework the patches. Will look forward for further discussions on this.
-KR
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2018-12-19 20:37 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-12-04 1:36 [PATCH v3 0/5] Add Tegra194 Dual ARM SMMU driver Krishna Reddy
2018-12-04 1:36 ` [PATCH v3 1/6] iommu/arm-smmu: create library for ARM SMMU Krishna Reddy
2018-12-04 1:36 ` [PATCH v3 2/6] iommu/arm-smmu: Add support to program multiple ARM SMMU's identically Krishna Reddy
2018-12-19 18:07 ` Robin Murphy
2018-12-19 20:36 ` Krishna Reddy
2018-12-04 1:36 ` [PATCH v3 3/6] iommu/arm-smmu: update arm-smmu.c to use ARM SMMU library Krishna Reddy
2018-12-04 1:36 ` [PATCH v3 4/6] iommu/tegra194_smmu: Add Tegra194 SMMU driver Krishna Reddy
2018-12-04 1:36 ` [PATCH v3 5/6] arm64: defconfig: Enable ARM_SMMU_TEGRA Krishna Reddy
2018-12-04 1:36 ` [PATCH v3 6/6] arm64: tegra: Add SMMU nodes to Tegra194 device tree Krishna Reddy
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).