[PATCH V2 00/10] Add support for Renesas R-Car S4 IPMMU and other misc changes

[PATCH V2 00/10] Add support for Renesas R-Car S4 IPMMU and other misc changes

[Oleksandr Tyshchenko]

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

Hello all.

You can find the V1 patch series at [1].

The R-Car S4 is an automotive System-on-Chip (SoC) for Car Server/Communication
Gateway and is one of the first products in Renesas’ 4th-generation R-Car Family.

The integrated IOMMU HW is also VMSA-compatible and supports stage 2 translation
table format, therefore can be used with current R-Car Gen3 driver with slight
modifications.

In the context of Xen driver the main differences between Gen3 and Gen4 are
the following:
 - HW capacity was enlarged to support up to 16 IPMMU contexts (sets of page table)
   and up to 64 micro-TLBs per IPMMU device
 - the memory mapped registers have different bases and offset

The first part (commits #1-6) is a non-verbatim port of Linux upstream commits
needed to add support for S4 series easily (prereq work).
The second part (commits #7-8) is based on the code from the Renesas BSP and
actually introduces support for R-Car S4 IPMMU.
The third part (commits #9-10) is misc changes I have locally.

The patch series is based on the latest staging branch 
"f3999bc2e099 arm/efi: Handle Xen bootargs from both xen.cfg and DT"
and also available at [2].

Tested on Renesas Salvator-X board with H3 ES3.0 SoC (Gen3) and Renesas Spider
board with S4 SoC (Gen4).

[1] https://lore.kernel.org/all/1638035505-16931-1-git-send-email-olekstysh@gmail.com/
[2] https://github.com/otyshchenko1/xen/commits/s4_ipmmu_ml2


Oleksandr Tyshchenko (10):
  iommu/ipmmu-vmsa: Remove all unused register definitions
  iommu/ipmmu-vmsa: Add helper functions for MMU "context" registers
  iommu/ipmmu-vmsa: Add helper functions for "uTLB" registers
  iommu/ipmmu-vmsa: Add light version of Linux's ipmmu_features
  iommu/ipmmu-vmsa: Calculate context registers' offset instead of a
    macro
  iommu/ipmmu-vmsa: Add utlb_offset_base
  iommu/ipmmu-vmsa: Add Renesas R8A779F0 (R-Car S4) support
  iommu/ipmmu-vmsa: Set IPMMU bit IMSCTLR_USE_SECGRP to 0
  iommu/ipmmu-vmsa: Use refcount for the micro-TLBs
  iommu/arm: Remove code duplication in all IOMMU drivers

 xen/drivers/passthrough/Kconfig          |   6 +-
 xen/drivers/passthrough/arm/iommu.c      |   7 +
 xen/drivers/passthrough/arm/ipmmu-vmsa.c | 267 ++++++++++++++++++-------------
 xen/drivers/passthrough/arm/smmu-v3.c    |  10 --
 xen/drivers/passthrough/arm/smmu.c       |  10 --
 5 files changed, 167 insertions(+), 133 deletions(-)

-- 
2.7.4

[PATCH V2 01/10] iommu/ipmmu-vmsa: Remove all unused register definitions

[Oleksandr Tyshchenko]

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

This is a non-verbatim port of corresponding Linux upsteam commit:
77cf983892b2e0d40dc256b784930a9ffaad4fc8

Original commit message:
 commit 77cf983892b2e0d40dc256b784930a9ffaad4fc8
 Author: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
 Date:   Wed Nov 6 11:35:45 2019 +0900

  iommu/ipmmu-vmsa: Remove all unused register definitions

  To support different registers memory mapping hardware easily
  in the future, this patch removes all unused register
  definitions.

  Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
  Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
  Signed-off-by: Joerg Roedel <jroedel@suse.de>

**********

This is a prereq work needed to add support for S4 series easily
in the future.

Although Linux and Xen drivers have a lot in common, the main
differences are in translation stages (table formats), VMSAv8 modes,
supported SoC generations, etc, therefore that's why there is
a slight difference in registers/bits each driver considers unused.

No change in behavior.

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
Changes V1 -> V2:
   - add R-b
---
 xen/drivers/passthrough/arm/ipmmu-vmsa.c | 59 ++------------------------------
 1 file changed, 2 insertions(+), 57 deletions(-)

diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
index 1255b0d..4a8a974 100644
--- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
+++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
@@ -166,15 +166,11 @@ static DEFINE_SPINLOCK(ipmmu_devices_lock);
 
 #define IMCTR                0x0000
 /*
- * These fields are implemented in IPMMU-MM only. So, can be set for
+ * This field is implemented in IPMMU-MM only. So, can be set for
  * Root IPMMU only.
  */
 #define IMCTR_VA64           (1 << 29)
-#define IMCTR_TRE            (1 << 17)
-#define IMCTR_AFE            (1 << 16)
-#define IMCTR_RTSEL_MASK     (3 << 4)
-#define IMCTR_RTSEL_SHIFT    4
-#define IMCTR_TREN           (1 << 3)
+
 /*
  * These fields are common for all IPMMU devices. So, can be set for
  * Cache IPMMUs as well.
@@ -184,42 +180,9 @@ static DEFINE_SPINLOCK(ipmmu_devices_lock);
 #define IMCTR_MMUEN          (1 << 0)
 #define IMCTR_COMMON_MASK    (7 << 0)
 
-#define IMCAAR               0x0004
-
 #define IMTTBCR                        0x0008
 #define IMTTBCR_EAE                    (1U << 31)
 #define IMTTBCR_PMB                    (1 << 30)
-#define IMTTBCR_SH1_NON_SHAREABLE      (0 << 28)
-#define IMTTBCR_SH1_OUTER_SHAREABLE    (2 << 28)
-#define IMTTBCR_SH1_INNER_SHAREABLE    (3 << 28)
-#define IMTTBCR_SH1_MASK               (3 << 28)
-#define IMTTBCR_ORGN1_NC               (0 << 26)
-#define IMTTBCR_ORGN1_WB_WA            (1 << 26)
-#define IMTTBCR_ORGN1_WT               (2 << 26)
-#define IMTTBCR_ORGN1_WB               (3 << 26)
-#define IMTTBCR_ORGN1_MASK             (3 << 26)
-#define IMTTBCR_IRGN1_NC               (0 << 24)
-#define IMTTBCR_IRGN1_WB_WA            (1 << 24)
-#define IMTTBCR_IRGN1_WT               (2 << 24)
-#define IMTTBCR_IRGN1_WB               (3 << 24)
-#define IMTTBCR_IRGN1_MASK             (3 << 24)
-#define IMTTBCR_TSZ1_MASK              (0x1f << 16)
-#define IMTTBCR_TSZ1_SHIFT             16
-#define IMTTBCR_SH0_NON_SHAREABLE      (0 << 12)
-#define IMTTBCR_SH0_OUTER_SHAREABLE    (2 << 12)
-#define IMTTBCR_SH0_INNER_SHAREABLE    (3 << 12)
-#define IMTTBCR_SH0_MASK               (3 << 12)
-#define IMTTBCR_ORGN0_NC               (0 << 10)
-#define IMTTBCR_ORGN0_WB_WA            (1 << 10)
-#define IMTTBCR_ORGN0_WT               (2 << 10)
-#define IMTTBCR_ORGN0_WB               (3 << 10)
-#define IMTTBCR_ORGN0_MASK             (3 << 10)
-#define IMTTBCR_IRGN0_NC               (0 << 8)
-#define IMTTBCR_IRGN0_WB_WA            (1 << 8)
-#define IMTTBCR_IRGN0_WT               (2 << 8)
-#define IMTTBCR_IRGN0_WB               (3 << 8)
-#define IMTTBCR_IRGN0_MASK             (3 << 8)
-#define IMTTBCR_SL0_LVL_2              (0 << 6)
 #define IMTTBCR_SL0_LVL_1              (1 << 6)
 #define IMTTBCR_TSZ0_MASK              (0x1f << 0)
 #define IMTTBCR_TSZ0_SHIFT             0
@@ -228,18 +191,8 @@ static DEFINE_SPINLOCK(ipmmu_devices_lock);
 #define IMTTLBR0_TTBR_MASK    (0xfffff << 12)
 #define IMTTUBR0              0x0014
 #define IMTTUBR0_TTBR_MASK    (0xff << 0)
-#define IMTTLBR1              0x0018
-#define IMTTLBR1_TTBR_MASK    (0xfffff << 12)
-#define IMTTUBR1              0x001c
-#define IMTTUBR1_TTBR_MASK    (0xff << 0)
 
 #define IMSTR                          0x0020
-#define IMSTR_ERRLVL_MASK              (3 << 12)
-#define IMSTR_ERRLVL_SHIFT             12
-#define IMSTR_ERRCODE_TLB_FORMAT       (1 << 8)
-#define IMSTR_ERRCODE_ACCESS_PERM      (4 << 8)
-#define IMSTR_ERRCODE_SECURE_ACCESS    (5 << 8)
-#define IMSTR_ERRCODE_MASK             (7 << 8)
 #define IMSTR_MHIT                     (1 << 4)
 #define IMSTR_ABORT                    (1 << 2)
 #define IMSTR_PF                       (1 << 1)
@@ -251,11 +204,7 @@ static DEFINE_SPINLOCK(ipmmu_devices_lock);
 #define IMUCTR(n)              ((n) < 32 ? IMUCTR0(n) : IMUCTR32(n))
 #define IMUCTR0(n)             (0x0300 + ((n) * 16))
 #define IMUCTR32(n)            (0x0600 + (((n) - 32) * 16))
-#define IMUCTR_FIXADDEN        (1U << 31)
-#define IMUCTR_FIXADD_MASK     (0xff << 16)
-#define IMUCTR_FIXADD_SHIFT    16
 #define IMUCTR_TTSEL_MMU(n)    ((n) << 4)
-#define IMUCTR_TTSEL_PMB       (8 << 4)
 #define IMUCTR_TTSEL_MASK      (15 << 4)
 #define IMUCTR_TTSEL_SHIFT     4
 #define IMUCTR_FLUSH           (1 << 1)
@@ -264,10 +213,6 @@ static DEFINE_SPINLOCK(ipmmu_devices_lock);
 #define IMUASID(n)             ((n) < 32 ? IMUASID0(n) : IMUASID32(n))
 #define IMUASID0(n)            (0x0308 + ((n) * 16))
 #define IMUASID32(n)           (0x0608 + (((n) - 32) * 16))
-#define IMUASID_ASID8_MASK     (0xff << 8)
-#define IMUASID_ASID8_SHIFT    8
-#define IMUASID_ASID0_MASK     (0xff << 0)
-#define IMUASID_ASID0_SHIFT    0
 
 #define IMSAUXCTLR          0x0504
 #define IMSAUXCTLR_S2PTE    (1 << 3)
-- 
2.7.4

[PATCH V2 02/10] iommu/ipmmu-vmsa: Add helper functions for MMU "context" registers

[Oleksandr Tyshchenko]

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

This is a non-verbatim port of corresponding Linux upsteam commit:
16d9454f5e0447f9c19cbf350b35ed377b9f64eb

Original commit message:
 commit 16d9454f5e0447f9c19cbf350b35ed377b9f64eb
 Author: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
 Date:   Wed Nov 6 11:35:47 2019 +0900

  iommu/ipmmu-vmsa: Add helper functions for MMU "context" registers

  Since we will have changed memory mapping of the IPMMU in the future,
  This patch adds helper functions ipmmu_ctx_{reg,read,write}()
  for MMU "context" registers. No behavior change.

  Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
  Reviewed-by: Geert Uytterhoeven <geert+renesas@glider.be>
  Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
  Signed-off-by: Joerg Roedel <jroedel@suse.de>

**********

This is a prereq work needed to add support for S4 series easily
in the future.

Besides changes done in the original commit, we also need to update
an extra call sites which Linux driver doesn't have, but Xen driver
has such as ipmmu_ctx_write_cache(), etc.

No change in behavior.

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
Changes V1 -> V2:
   - add R-b
---
 xen/drivers/passthrough/arm/ipmmu-vmsa.c | 31 ++++++++++++++++++++++++-------
 1 file changed, 24 insertions(+), 7 deletions(-)

diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
index 4a8a974..ce5c3bc 100644
--- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
+++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
@@ -307,18 +307,35 @@ static void ipmmu_write(struct ipmmu_vmsa_device *mmu, uint32_t offset,
     writel(data, mmu->base + offset);
 }
 
+static unsigned int ipmmu_ctx_reg(struct ipmmu_vmsa_device *mmu,
+                                  unsigned int context_id, uint32_t reg)
+{
+    return context_id * IM_CTX_SIZE + reg;
+}
+
+static uint32_t ipmmu_ctx_read(struct ipmmu_vmsa_device *mmu,
+                               unsigned int context_id, uint32_t reg)
+{
+    return ipmmu_read(mmu, ipmmu_ctx_reg(mmu, context_id, reg));
+}
+
+static void ipmmu_ctx_write(struct ipmmu_vmsa_device *mmu,
+                            unsigned int context_id, uint32_t reg,
+                            uint32_t data)
+{
+    ipmmu_write(mmu, ipmmu_ctx_reg(mmu, context_id, reg), data);
+}
+
 static uint32_t ipmmu_ctx_read_root(struct ipmmu_vmsa_domain *domain,
                                     uint32_t reg)
 {
-    return ipmmu_read(domain->mmu->root,
-                      domain->context_id * IM_CTX_SIZE + reg);
+    return ipmmu_ctx_read(domain->mmu->root, domain->context_id, reg);
 }
 
 static void ipmmu_ctx_write_root(struct ipmmu_vmsa_domain *domain,
                                  uint32_t reg, uint32_t data)
 {
-    ipmmu_write(domain->mmu->root,
-                domain->context_id * IM_CTX_SIZE + reg, data);
+    ipmmu_ctx_write(domain->mmu->root, domain->context_id, reg, data);
 }
 
 static void ipmmu_ctx_write_cache(struct ipmmu_vmsa_domain *domain,
@@ -329,8 +346,8 @@ static void ipmmu_ctx_write_cache(struct ipmmu_vmsa_domain *domain,
 
     /* Mask fields which are implemented in IPMMU-MM only. */
     if ( !ipmmu_is_root(domain->mmu) )
-        ipmmu_write(domain->mmu, domain->context_id * IM_CTX_SIZE + reg,
-                    data & IMCTR_COMMON_MASK);
+        ipmmu_ctx_write(domain->mmu, domain->context_id, reg,
+                        data & IMCTR_COMMON_MASK);
 }
 
 /*
@@ -693,7 +710,7 @@ static void ipmmu_device_reset(struct ipmmu_vmsa_device *mmu)
 
     /* Disable all contexts. */
     for ( i = 0; i < mmu->num_ctx; ++i )
-        ipmmu_write(mmu, i * IM_CTX_SIZE + IMCTR, 0);
+        ipmmu_ctx_write(mmu, i, IMCTR, 0);
 }
 
 /* R-Car Gen3 SoCs product and cut information. */
-- 
2.7.4

[PATCH V2 03/10] iommu/ipmmu-vmsa: Add helper functions for "uTLB" registers

[Oleksandr Tyshchenko]

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

This is a non-verbatim port of corresponding Linux upsteam commit:
3667c9978b2911dc1ded77f5971df477885409c4

Original commit message:
 commit 3667c9978b2911dc1ded77f5971df477885409c4
 Author: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
 Date:   Wed Nov 6 11:35:49 2019 +0900

  iommu/ipmmu-vmsa: Add helper functions for "uTLB" registers

  Since we will have changed memory mapping of the IPMMU in the future,
  This patch adds helper functions ipmmu_utlb_reg() and
  ipmmu_imu{asid,ctr}_write() for "uTLB" registers. No behavior change.

  Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
  Reviewed-by: Geert Uytterhoeven <geert+renesas@glider.be>
  Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
  Signed-off-by: Joerg Roedel <jroedel@suse.de>

**********

This is a prereq work needed to add support for S4 series easily
in the future.

Besides changes done in the original commit, we also need to introduce
ipmmu_imuctr_read() since Xen driver contains an additional logic in
ipmmu_utlb_enable() to prevent the use cases where devices which use
the same micro-TLB are assigned to different Xen domains.

No change in behavior.

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
Changes V1 -> V2:
   - add R-b
---
 xen/drivers/passthrough/arm/ipmmu-vmsa.c | 33 +++++++++++++++++++++++++++-----
 1 file changed, 28 insertions(+), 5 deletions(-)

diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
index ce5c3bc..1b94af2 100644
--- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
+++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
@@ -366,6 +366,29 @@ static void ipmmu_ctx_write_all(struct ipmmu_vmsa_domain *domain,
     ipmmu_ctx_write_root(domain, reg, data);
 }
 
+static uint32_t ipmmu_utlb_reg(struct ipmmu_vmsa_device *mmu, uint32_t reg)
+{
+    return reg;
+}
+
+static void ipmmu_imuasid_write(struct ipmmu_vmsa_device *mmu,
+                                unsigned int utlb, uint32_t data)
+{
+    ipmmu_write(mmu, ipmmu_utlb_reg(mmu, IMUASID(utlb)), data);
+}
+
+static void ipmmu_imuctr_write(struct ipmmu_vmsa_device *mmu,
+                               unsigned int utlb, uint32_t data)
+{
+    ipmmu_write(mmu, ipmmu_utlb_reg(mmu, IMUCTR(utlb)), data);
+}
+
+static uint32_t ipmmu_imuctr_read(struct ipmmu_vmsa_device *mmu,
+                                  unsigned int utlb)
+{
+    return ipmmu_read(mmu, ipmmu_utlb_reg(mmu, IMUCTR(utlb)));
+}
+
 /* TLB and micro-TLB Management */
 
 /* Wait for any pending TLB invalidations to complete. */
@@ -413,7 +436,7 @@ static int ipmmu_utlb_enable(struct ipmmu_vmsa_domain *domain,
      * context_id for already enabled micro-TLB and prevent different context
      * bank from being set.
      */
-    imuctr = ipmmu_read(mmu, IMUCTR(utlb));
+    imuctr = ipmmu_imuctr_read(mmu, utlb);
     if ( imuctr & IMUCTR_MMUEN )
     {
         unsigned int context_id;
@@ -431,9 +454,9 @@ static int ipmmu_utlb_enable(struct ipmmu_vmsa_domain *domain,
      * TODO: Reference-count the micro-TLB as several bus masters can be
      * connected to the same micro-TLB.
      */
-    ipmmu_write(mmu, IMUASID(utlb), 0);
-    ipmmu_write(mmu, IMUCTR(utlb), imuctr |
-                IMUCTR_TTSEL_MMU(domain->context_id) | IMUCTR_MMUEN);
+    ipmmu_imuasid_write(mmu, utlb, 0);
+    ipmmu_imuctr_write(mmu, utlb, imuctr |
+                       IMUCTR_TTSEL_MMU(domain->context_id) | IMUCTR_MMUEN);
 
     return 0;
 }
@@ -444,7 +467,7 @@ static void ipmmu_utlb_disable(struct ipmmu_vmsa_domain *domain,
 {
     struct ipmmu_vmsa_device *mmu = domain->mmu;
 
-    ipmmu_write(mmu, IMUCTR(utlb), 0);
+    ipmmu_imuctr_write(mmu, utlb, 0);
 }
 
 /* Domain/Context Management */
-- 
2.7.4

[PATCH V2 04/10] iommu/ipmmu-vmsa: Add light version of Linux's ipmmu_features

[Oleksandr Tyshchenko]

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

This is a prereq work needed to add support for S4 series easily
in the future.

We don't need to pull the whole struct and all instances as Xen
driver doesn't support old Arm32 based Gen2 SoCs, so there is no
point in keeping all differences between Gen2 and Gen3 here.
All what we need is a minimal support to be able to operate with
Gen3 and new S4.

Add Gen3 specific info with only two fields (number_of_contexts and
num_utlbs) for now, the subsequent patches will add remaining bits.

No change in behavior.

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
Changes V1 -> V2:
   - add R-b
---
 xen/drivers/passthrough/arm/ipmmu-vmsa.c | 54 +++++++++++++++++++++++---------
 1 file changed, 40 insertions(+), 14 deletions(-)

diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
index 1b94af2..369be4c 100644
--- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
+++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
@@ -71,9 +71,9 @@
  * R-Car Gen3 SoCs make use of up to 8 IPMMU contexts (sets of page table) and
  * these can be managed independently. Each context is mapped to one Xen domain.
  */
-#define IPMMU_CTX_MAX     8
+#define IPMMU_CTX_MAX     8U
 /* R-Car Gen3 SoCs make use of up to 48 micro-TLBs per IPMMU device. */
-#define IPMMU_UTLB_MAX    48
+#define IPMMU_UTLB_MAX    48U
 
 /* IPMMU context supports IPA size up to 40 bit. */
 #define IPMMU_MAX_P2M_IPA_BITS    40
@@ -106,17 +106,22 @@ struct ipmmu_vmsa_xen_device {
     struct ipmmu_vmsa_device *mmu;
 };
 
+struct ipmmu_features {
+    unsigned int number_of_contexts;
+    unsigned int num_utlbs;
+};
+
 /* Root/Cache IPMMU device's information */
 struct ipmmu_vmsa_device {
     struct device *dev;
     void __iomem *base;
     struct ipmmu_vmsa_device *root;
     struct list_head list;
-    unsigned int num_utlbs;
     unsigned int num_ctx;
     spinlock_t lock;    /* Protects ctx and domains[] */
     DECLARE_BITMAP(ctx, IPMMU_CTX_MAX);
     struct ipmmu_vmsa_domain *domains[IPMMU_CTX_MAX];
+    const struct ipmmu_features *features;
 };
 
 /*
@@ -727,6 +732,11 @@ static int ipmmu_init_platform_device(struct device *dev,
     return 0;
 }
 
+static const struct ipmmu_features ipmmu_features_rcar_gen3 = {
+    .number_of_contexts = 8,
+    .num_utlbs = 48,
+};
+
 static void ipmmu_device_reset(struct ipmmu_vmsa_device *mmu)
 {
     unsigned int i;
@@ -798,6 +808,27 @@ static __init bool ipmmu_stage2_supported(void)
     return stage2_supported;
 }
 
+static const struct dt_device_match ipmmu_dt_match[] __initconst =
+{
+    {
+        .compatible = "renesas,ipmmu-r8a7795",
+        .data = &ipmmu_features_rcar_gen3,
+    },
+    {
+        .compatible = "renesas,ipmmu-r8a77965",
+        .data = &ipmmu_features_rcar_gen3,
+    },
+    {
+        .compatible = "renesas,ipmmu-r8a7796",
+        .data = &ipmmu_features_rcar_gen3,
+    },
+    {
+        .compatible = "renesas,ipmmu-r8a77961",
+        .data = &ipmmu_features_rcar_gen3,
+    },
+    { /* sentinel */ },
+};
+
 /*
  * This function relies on the fact that Root IPMMU device is being probed
  * the first. If not the case, it denies further Cache IPMMU device probes
@@ -806,6 +837,7 @@ static __init bool ipmmu_stage2_supported(void)
  */
 static int ipmmu_probe(struct dt_device_node *node)
 {
+    const struct dt_device_match *match;
     struct ipmmu_vmsa_device *mmu;
     uint64_t addr, size;
     int irq, ret;
@@ -817,9 +849,12 @@ static int ipmmu_probe(struct dt_device_node *node)
         return -ENOMEM;
     }
 
+    match = dt_match_node(ipmmu_dt_match, node);
+    ASSERT(match);
+    mmu->features = match->data;
+
     mmu->dev = &node->dev;
-    mmu->num_utlbs = IPMMU_UTLB_MAX;
-    mmu->num_ctx = IPMMU_CTX_MAX;
+    mmu->num_ctx = min(IPMMU_CTX_MAX, mmu->features->number_of_contexts);
     spin_lock_init(&mmu->lock);
     bitmap_zero(mmu->ctx, IPMMU_CTX_MAX);
 
@@ -1296,15 +1331,6 @@ static const struct iommu_ops ipmmu_iommu_ops =
     .add_device      = ipmmu_add_device,
 };
 
-static const struct dt_device_match ipmmu_dt_match[] __initconst =
-{
-    DT_MATCH_COMPATIBLE("renesas,ipmmu-r8a7795"),
-    DT_MATCH_COMPATIBLE("renesas,ipmmu-r8a77965"),
-    DT_MATCH_COMPATIBLE("renesas,ipmmu-r8a7796"),
-    DT_MATCH_COMPATIBLE("renesas,ipmmu-r8a77961"),
-    { /* sentinel */ },
-};
-
 static __init int ipmmu_init(struct dt_device_node *node, const void *data)
 {
     int ret;
-- 
2.7.4

[PATCH V2 05/10] iommu/ipmmu-vmsa: Calculate context registers' offset instead of a macro

[Oleksandr Tyshchenko]

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

This is a non-verbatim port of corresponding Linux upsteam commit:
3dc28d9f59eaae41461542b27afe70339347ebb3

Original commit message:
 commit 3dc28d9f59eaae41461542b27afe70339347ebb3
 Author: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
 Date:   Wed Nov 6 11:35:48 2019 +0900

  iommu/ipmmu-vmsa: Calculate context registers' offset instead of a macro

  Since we will have changed memory mapping of the IPMMU in the future,
  this patch uses ipmmu_features values instead of a macro to
  calculate context registers offset. No behavior change.

  Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
  Reviewed-by: Geert Uytterhoeven <geert+renesas@glider.be>
  Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
  Signed-off-by: Joerg Roedel <jroedel@suse.de>

**********

This is a prereq work needed to add support for S4 series easily
in the future.

Almost the same change as original commit makes, but without updating
struct ipmmu_features_default which Xen driver doesn't have (there is
no support of old Arm32 based Gen2 SoCs).

No change in behavior.

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
Changes V1 -> V2:
   - add R-b
---
 xen/drivers/passthrough/arm/ipmmu-vmsa.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
index 369be4c..ca33456 100644
--- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
+++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
@@ -109,6 +109,8 @@ struct ipmmu_vmsa_xen_device {
 struct ipmmu_features {
     unsigned int number_of_contexts;
     unsigned int num_utlbs;
+    unsigned int ctx_offset_base;
+    unsigned int ctx_offset_stride;
 };
 
 /* Root/Cache IPMMU device's information */
@@ -167,8 +169,6 @@ static DEFINE_SPINLOCK(ipmmu_devices_lock);
 #define TLB_LOOP_TIMEOUT    100 /* 100us */
 
 /* Registers Definition */
-#define IM_CTX_SIZE    0x40
-
 #define IMCTR                0x0000
 /*
  * This field is implemented in IPMMU-MM only. So, can be set for
@@ -315,7 +315,8 @@ static void ipmmu_write(struct ipmmu_vmsa_device *mmu, uint32_t offset,
 static unsigned int ipmmu_ctx_reg(struct ipmmu_vmsa_device *mmu,
                                   unsigned int context_id, uint32_t reg)
 {
-    return context_id * IM_CTX_SIZE + reg;
+    return mmu->features->ctx_offset_base +
+        context_id * mmu->features->ctx_offset_stride + reg;
 }
 
 static uint32_t ipmmu_ctx_read(struct ipmmu_vmsa_device *mmu,
@@ -735,6 +736,8 @@ static int ipmmu_init_platform_device(struct device *dev,
 static const struct ipmmu_features ipmmu_features_rcar_gen3 = {
     .number_of_contexts = 8,
     .num_utlbs = 48,
+    .ctx_offset_base = 0,
+    .ctx_offset_stride = 0x40,
 };
 
 static void ipmmu_device_reset(struct ipmmu_vmsa_device *mmu)
-- 
2.7.4

[PATCH V2 06/10] iommu/ipmmu-vmsa: Add utlb_offset_base

[Oleksandr Tyshchenko]

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

This is a non-verbatim port of corresponding Linux upsteam commit:
1289f7f15001c7ed36be6d23cb145c1d5feacdc8

Original commit message:
 commit 1289f7f15001c7ed36be6d23cb145c1d5feacdc8
 Author: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
 Date:   Wed Nov 6 11:35:50 2019 +0900

  iommu/ipmmu-vmsa: Add utlb_offset_base

  Since we will have changed memory mapping of the IPMMU in the future,
  this patch adds a utlb_offset_base into struct ipmmu_features
  for IMUCTR and IMUASID registers. No behavior change.

  Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
  Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
  Signed-off-by: Joerg Roedel <jroedel@suse.de>

**********

This is a prereq work needed to add support for S4 series easily
in the future.

Almost the same change as original commit makes, but without updating
struct ipmmu_features_default which Xen driver doesn't have (there is
no support of old Arm32 based Gen2 SoCs).

No change in behavior.

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
Changes V1 -> V2:
   - add R-b
---
 xen/drivers/passthrough/arm/ipmmu-vmsa.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
index ca33456..d8f96fc 100644
--- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
+++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
@@ -111,6 +111,7 @@ struct ipmmu_features {
     unsigned int num_utlbs;
     unsigned int ctx_offset_base;
     unsigned int ctx_offset_stride;
+    unsigned int utlb_offset_base;
 };
 
 /* Root/Cache IPMMU device's information */
@@ -374,7 +375,7 @@ static void ipmmu_ctx_write_all(struct ipmmu_vmsa_domain *domain,
 
 static uint32_t ipmmu_utlb_reg(struct ipmmu_vmsa_device *mmu, uint32_t reg)
 {
-    return reg;
+    return mmu->features->utlb_offset_base + reg;
 }
 
 static void ipmmu_imuasid_write(struct ipmmu_vmsa_device *mmu,
@@ -738,6 +739,7 @@ static const struct ipmmu_features ipmmu_features_rcar_gen3 = {
     .num_utlbs = 48,
     .ctx_offset_base = 0,
     .ctx_offset_stride = 0x40,
+    .utlb_offset_base = 0,
 };
 
 static void ipmmu_device_reset(struct ipmmu_vmsa_device *mmu)
-- 
2.7.4

[PATCH V2 07/10] iommu/ipmmu-vmsa: Add Renesas R8A779F0 (R-Car S4) support

[Oleksandr Tyshchenko]

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

Based on the following Linux upsteam commit:
7a62ced8ebd0e1b692c9dc4781a8d4ddb0f74792

Original commit message:
 commit 7a62ced8ebd0e1b692c9dc4781a8d4ddb0f74792
 Author: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
 Date:   Tue Sep 7 17:30:20 2021 +0900

  iommu/ipmmu-vmsa: Add support for r8a779a0

  Add support for r8a779a0 (R-Car V3U). The IPMMU hardware design
  of this SoC differs than others. So, add a new ipmmu_features for it.

  Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
  Reviewed-by: Geert Uytterhoeven <geert+renesas@glider.be>
  Link: https://lore.kernel.org/r/20210907083020.907648-3-yoshihiro.shimoda.uh@renesas.com
  Signed-off-by: Joerg Roedel <jroedel@suse.de>

**********

The R-Car S4 is an automotive System-on-Chip (SoC) for Car
Server/Communication Gateway and is one of the first products
in Renesas’ 4th-generation R-Car Family.

The integrated IOMMU HW is also VMSA-compatible and supports
stage 2 translation table format, therefore can be used with
current driver with slight modifications (thanks to the prereq
work).

In the context of Xen driver the main differences between Gen3
and Gen4 are the following:
- HW capacity was enlarged to support up to 16 IPMMU contexts
  (sets of page table) and up to 64 micro-TLBs per IPMMU device
- the memory mapped registers have different bases and offsets

Please note that Linux upstream doesn't support R-Car S4 SoC
yet unlike Renesas BSP [1], but it was decided to reuse upstream
patch for R-Car V3U anyway as the IPMMU HW settings are similar.

[1]
7003b9f732cf iommu/ipmmu-vmsa: Add Renesas R8A779F0 (R-Car S4) support
https://github.com/renesas-rcar/linux-bsp/tree/v5.10.41/rcar-5.1.3.rc5

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
---
Changes V1 -> V2:
   - base on the Linux upsteam patch instead of BSP's one
   - update patch description and comments in code
   - replace ipmmu_features_rcar_s4 with ipmmu_features_rcar_gen4
   - introduce control_offset_base field
---
 xen/drivers/passthrough/Kconfig          |  6 +--
 xen/drivers/passthrough/arm/ipmmu-vmsa.c | 63 +++++++++++++++++++++++---------
 2 files changed, 49 insertions(+), 20 deletions(-)

diff --git a/xen/drivers/passthrough/Kconfig b/xen/drivers/passthrough/Kconfig
index 09505aa..479d7de 100644
--- a/xen/drivers/passthrough/Kconfig
+++ b/xen/drivers/passthrough/Kconfig
@@ -25,14 +25,14 @@ config ARM_SMMU_V3
 	 the ARM SMMUv3 architecture.
 
 config IPMMU_VMSA
-	bool "Renesas IPMMU-VMSA found in R-Car Gen3 SoCs"
+	bool "Renesas IPMMU-VMSA found in R-Car Gen3/Gen4 SoCs"
 	depends on ARM_64
 	---help---
 	  Support for implementations of the Renesas IPMMU-VMSA found
-	  in R-Car Gen3 SoCs.
+	  in R-Car Gen3/Gen4 SoCs.
 
 	  Say Y here if you are using newest R-Car Gen3 SoCs revisions
-	  (H3 ES3.0, M3-W+, etc) which IPMMU hardware supports stage 2
+	  (H3 ES3.0, M3-W+, etc) or Gen4 SoCs which IPMMU hardware supports stage 2
 	  translation table format and is able to use CPU's P2M table as is.
 
 endif
diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
index d8f96fc..14848ce 100644
--- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
+++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
@@ -1,15 +1,15 @@
 /*
  * xen/drivers/passthrough/arm/ipmmu-vmsa.c
  *
- * Driver for the Renesas IPMMU-VMSA found in R-Car Gen3 SoCs.
+ * Driver for the Renesas IPMMU-VMSA found in R-Car Gen3/Gen4 SoCs.
  *
  * The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU)
  * which provides address translation and access protection functionalities
  * to processing units and interconnect networks.
  *
  * Please note, current driver is supposed to work only with newest
- * R-Car Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation
- * table format and is able to use CPU's P2M table as is.
+ * R-Car Gen3/Gen4 SoCs revisions which IPMMU hardware supports stage 2
+ * translation table format and is able to use CPU's P2M table as is.
  *
  * Based on Linux's IPMMU-VMSA driver from Renesas BSP:
  *    drivers/iommu/ipmmu-vmsa.c
@@ -20,9 +20,9 @@
  * and Xen's SMMU driver:
  *    xen/drivers/passthrough/arm/smmu.c
  *
- * Copyright (C) 2014-2019 Renesas Electronics Corporation
+ * Copyright (C) 2014-2021 Renesas Electronics Corporation
  *
- * Copyright (C) 2016-2019 EPAM Systems Inc.
+ * Copyright (C) 2016-2021 EPAM Systems Inc.
  *
  * This program is free software; you can redistribute it and/or
  * modify it under the terms and conditions of the GNU General Public
@@ -68,12 +68,13 @@
     dev_print(dev, XENLOG_ERR, fmt, ## __VA_ARGS__)
 
 /*
- * R-Car Gen3 SoCs make use of up to 8 IPMMU contexts (sets of page table) and
- * these can be managed independently. Each context is mapped to one Xen domain.
+ * R-Car Gen3/Gen4 SoCs make use of up to 16 IPMMU contexts (sets of page table)
+ * and these can be managed independently. Each context is mapped to one Xen
+ * domain.
  */
-#define IPMMU_CTX_MAX     8U
-/* R-Car Gen3 SoCs make use of up to 48 micro-TLBs per IPMMU device. */
-#define IPMMU_UTLB_MAX    48U
+#define IPMMU_CTX_MAX     16U
+/* R-Car Gen3/Gen4 SoCs make use of up to 64 micro-TLBs per IPMMU device. */
+#define IPMMU_UTLB_MAX    64U
 
 /* IPMMU context supports IPA size up to 40 bit. */
 #define IPMMU_MAX_P2M_IPA_BITS    40
@@ -112,6 +113,8 @@ struct ipmmu_features {
     unsigned int ctx_offset_base;
     unsigned int ctx_offset_stride;
     unsigned int utlb_offset_base;
+    unsigned int control_offset_base;
+    unsigned int imuctr_ttsel_mask;
 };
 
 /* Root/Cache IPMMU device's information */
@@ -211,7 +214,6 @@ static DEFINE_SPINLOCK(ipmmu_devices_lock);
 #define IMUCTR0(n)             (0x0300 + ((n) * 16))
 #define IMUCTR32(n)            (0x0600 + (((n) - 32) * 16))
 #define IMUCTR_TTSEL_MMU(n)    ((n) << 4)
-#define IMUCTR_TTSEL_MASK      (15 << 4)
 #define IMUCTR_TTSEL_SHIFT     4
 #define IMUCTR_FLUSH           (1 << 1)
 #define IMUCTR_MMUEN           (1 << 0)
@@ -316,8 +318,12 @@ static void ipmmu_write(struct ipmmu_vmsa_device *mmu, uint32_t offset,
 static unsigned int ipmmu_ctx_reg(struct ipmmu_vmsa_device *mmu,
                                   unsigned int context_id, uint32_t reg)
 {
-    return mmu->features->ctx_offset_base +
-        context_id * mmu->features->ctx_offset_stride + reg;
+    unsigned int base = mmu->features->ctx_offset_base;
+
+    if ( context_id > 7 )
+        base += 0x800 - 8 * 0x40;
+
+    return base + context_id * mmu->features->ctx_offset_stride + reg;
 }
 
 static uint32_t ipmmu_ctx_read(struct ipmmu_vmsa_device *mmu,
@@ -448,7 +454,8 @@ static int ipmmu_utlb_enable(struct ipmmu_vmsa_domain *domain,
     {
         unsigned int context_id;
 
-        context_id = (imuctr & IMUCTR_TTSEL_MASK) >> IMUCTR_TTSEL_SHIFT;
+        context_id = (imuctr & mmu->features->imuctr_ttsel_mask) >>
+            IMUCTR_TTSEL_SHIFT;
         if ( domain->context_id != context_id )
         {
             dev_err(mmu->dev, "Micro-TLB %u already assigned to IPMMU context %u\n",
@@ -740,6 +747,18 @@ static const struct ipmmu_features ipmmu_features_rcar_gen3 = {
     .ctx_offset_base = 0,
     .ctx_offset_stride = 0x40,
     .utlb_offset_base = 0,
+    .control_offset_base = 0,
+    .imuctr_ttsel_mask = (15 << 4),
+};
+
+static const struct ipmmu_features ipmmu_features_rcar_gen4 = {
+    .number_of_contexts = 16,
+    .num_utlbs = 64,
+    .ctx_offset_base = 0x10000,
+    .ctx_offset_stride = 0x1040,
+    .utlb_offset_base = 0x3000,
+    .control_offset_base = 0x1000,
+    .imuctr_ttsel_mask = (31 << 4),
 };
 
 static void ipmmu_device_reset(struct ipmmu_vmsa_device *mmu)
@@ -751,11 +770,12 @@ static void ipmmu_device_reset(struct ipmmu_vmsa_device *mmu)
         ipmmu_ctx_write(mmu, i, IMCTR, 0);
 }
 
-/* R-Car Gen3 SoCs product and cut information. */
+/* R-Car Gen3/Gen4 SoCs product and cut information. */
 #define RCAR_PRODUCT_MASK    0x00007F00
 #define RCAR_PRODUCT_H3      0x00004F00
 #define RCAR_PRODUCT_M3W     0x00005200
 #define RCAR_PRODUCT_M3N     0x00005500
+#define RCAR_PRODUCT_S4      0x00005A00
 #define RCAR_CUT_MASK        0x000000FF
 #define RCAR_CUT_VER30       0x00000020
 
@@ -803,6 +823,10 @@ static __init bool ipmmu_stage2_supported(void)
         stage2_supported = true;
         break;
 
+    case RCAR_PRODUCT_S4:
+        stage2_supported = true;
+        break;
+
     default:
         printk(XENLOG_ERR "ipmmu: Unsupported SoC version\n");
         break;
@@ -831,6 +855,10 @@ static const struct dt_device_match ipmmu_dt_match[] __initconst =
         .compatible = "renesas,ipmmu-r8a77961",
         .data = &ipmmu_features_rcar_gen3,
     },
+    {
+        .compatible = "renesas,ipmmu-r8a779f0",
+        .data = &ipmmu_features_rcar_gen4,
+    },
     { /* sentinel */ },
 };
 
@@ -845,6 +873,7 @@ static int ipmmu_probe(struct dt_device_node *node)
     const struct dt_device_match *match;
     struct ipmmu_vmsa_device *mmu;
     uint64_t addr, size;
+    uint32_t reg;
     int irq, ret;
 
     mmu = xzalloc(struct ipmmu_vmsa_device);
@@ -930,8 +959,8 @@ static int ipmmu_probe(struct dt_device_node *node)
          * Use stage 2 translation table format when stage 2 translation
          * enabled.
          */
-        ipmmu_write(mmu, IMSAUXCTLR,
-                    ipmmu_read(mmu, IMSAUXCTLR) | IMSAUXCTLR_S2PTE);
+        reg = IMSAUXCTLR + mmu->features->control_offset_base;
+        ipmmu_write(mmu, reg, ipmmu_read(mmu, reg) | IMSAUXCTLR_S2PTE);
 
         dev_info(&node->dev, "IPMMU context 0 is reserved\n");
         set_bit(0, mmu->ctx);
-- 
2.7.4

[PATCH V2 08/10] iommu/ipmmu-vmsa: Set IPMMU bit IMSCTLR_USE_SECGRP to 0

[Oleksandr Tyshchenko]

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

Based on the following commits from the Renesas BSP:
8fba83d97cca709a05139c38e29408e81ed4cf62
a8d93bc07da89a7fcf4d85f34d119a030310efa5
located at:
https://github.com/renesas-rcar/linux-bsp/tree/v5.10.41/rcar-5.1.3.rc5

Original commit messages:
 commit 8fba83d97cca709a05139c38e29408e81ed4cf62
 Author: Nam Nguyen <nam.nguyen.yh@renesas.com>
 Date:   Wed Apr 28 18:54:44 2021 +0700

  iommu/ipmmu-vmsa: Set IPMMU bit IMSCTLR_USE_SECGRP to 0

  Need to set bit IMSCTLR_USE_SECGRP to 0
  because H/W initial value is unknown, without this
  dma-transfer cannot be done due to address translation doesn't work.

  Signed-off-by: Nam Nguyen <nam.nguyen.yh@renesas.com>

 commit a8d93bc07da89a7fcf4d85f34d119a030310efa5
 Author: Nam Nguyen <nam.nguyen.yh@renesas.com>
 Date:   Tue Sep 7 14:46:12 2021 +0700

  iommu/ipmmu-vmsa: Update IMSCTLR register offset address for R-Car S4

  Update IMSCTLR register offset address to align with R-Car S4 H/W UM.

  Signed-off-by: Nam Nguyen <nam.nguyen.yh@renesas.com>

**********

It is still a question whether this really needs to be done in Xen,
rather in firmware, but better to be on the safe side. After all,
if firmware already takes care of clearing this bit, nothing bad
will happen.

Please note the following:
1. I decided to squash both commits since the first commit adds clearing
code and only the second one makes it functional on S4. Moreover, this is
not a direct port. So it would be better to introduce complete solution
by a single patch.
2. Although patch indeed does what it claims in the subject,
the implementation is different in comparison with original changes.
On Linux the clearing is done at runtime in ipmmu_domain_setup_context().
On Xen the clearing is done at boot time in ipmmu_probe().
The IMSCTLR is not a MMU "context" register at all, so I think there is
no point in performing the clearing each time we initialize the context,
instead perform the clearing at once during initialization. Also do not
abuse ctx_offset_stride_adj field for the register's offset calculation,
instead use recently added control_offset_base field.

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
---
Changes V1 -> V2:
   - add R-b
   - update patch description
   - replace ctx_offset_stride_adj with control_offset_base for
     register's offset calculation
---
 xen/drivers/passthrough/arm/ipmmu-vmsa.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
index 14848ce..649b9f6 100644
--- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
+++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
@@ -222,6 +222,9 @@ static DEFINE_SPINLOCK(ipmmu_devices_lock);
 #define IMUASID0(n)            (0x0308 + ((n) * 16))
 #define IMUASID32(n)           (0x0608 + (((n) - 32) * 16))
 
+#define IMSCTLR             0x0500
+#define IMSCTLR_USE_SECGRP  (1 << 28)
+
 #define IMSAUXCTLR          0x0504
 #define IMSAUXCTLR_S2PTE    (1 << 3)
 
@@ -966,6 +969,10 @@ static int ipmmu_probe(struct dt_device_node *node)
         set_bit(0, mmu->ctx);
     }
 
+    /* Do not use security group function. */
+    reg = IMSCTLR + mmu->features->control_offset_base;
+    ipmmu_write(mmu, reg, ipmmu_read(mmu, reg) & ~IMSCTLR_USE_SECGRP);
+
     spin_lock(&ipmmu_devices_lock);
     list_add(&mmu->list, &ipmmu_devices);
     spin_unlock(&ipmmu_devices_lock);
-- 
2.7.4

[PATCH V2 09/10] iommu/ipmmu-vmsa: Use refcount for the micro-TLBs

[Oleksandr Tyshchenko]

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

Reference-count the micro-TLBs as several bus masters can be
connected to the same micro-TLB (and drop TODO comment).
This wasn't an issue so far, since the platform devices
(this driver deals with) get assigned/deassigned together during
domain creation/destruction. But, in order to support PCI devices
(which are hot-pluggable) in the near future we will need to
take care of.

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
Changes V1 -> V2:
   - add R-b
   - add ASSERT() in ipmmu_utlb_disable()
---
 xen/drivers/passthrough/arm/ipmmu-vmsa.c | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
index 649b9f6..1224ea4 100644
--- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
+++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
@@ -127,6 +127,7 @@ struct ipmmu_vmsa_device {
     spinlock_t lock;    /* Protects ctx and domains[] */
     DECLARE_BITMAP(ctx, IPMMU_CTX_MAX);
     struct ipmmu_vmsa_domain *domains[IPMMU_CTX_MAX];
+    unsigned int utlb_refcount[IPMMU_UTLB_MAX];
     const struct ipmmu_features *features;
 };
 
@@ -467,13 +468,12 @@ static int ipmmu_utlb_enable(struct ipmmu_vmsa_domain *domain,
         }
     }
 
-    /*
-     * TODO: Reference-count the micro-TLB as several bus masters can be
-     * connected to the same micro-TLB.
-     */
-    ipmmu_imuasid_write(mmu, utlb, 0);
-    ipmmu_imuctr_write(mmu, utlb, imuctr |
-                       IMUCTR_TTSEL_MMU(domain->context_id) | IMUCTR_MMUEN);
+    if ( mmu->utlb_refcount[utlb]++ == 0 )
+    {
+        ipmmu_imuasid_write(mmu, utlb, 0);
+        ipmmu_imuctr_write(mmu, utlb, imuctr |
+                           IMUCTR_TTSEL_MMU(domain->context_id) | IMUCTR_MMUEN);
+    }
 
     return 0;
 }
@@ -484,7 +484,10 @@ static void ipmmu_utlb_disable(struct ipmmu_vmsa_domain *domain,
 {
     struct ipmmu_vmsa_device *mmu = domain->mmu;
 
-    ipmmu_imuctr_write(mmu, utlb, 0);
+    ASSERT(mmu->utlb_refcount[utlb] > 0);
+
+    if ( --mmu->utlb_refcount[utlb] == 0 )
+        ipmmu_imuctr_write(mmu, utlb, 0);
 }
 
 /* Domain/Context Management */
-- 
2.7.4

[PATCH V2 10/10] iommu/arm: Remove code duplication in all IOMMU drivers

[Oleksandr Tyshchenko]

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

All IOMMU drivers on Arm perform almost the same generic actions in
hwdom_init callback. Move this code to common arch_iommu_hwdom_init()
in order to get rid of code duplication.

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
Changes V1 -> V2:
   - add R-b
---
 xen/drivers/passthrough/arm/iommu.c      |  7 +++++++
 xen/drivers/passthrough/arm/ipmmu-vmsa.c |  8 --------
 xen/drivers/passthrough/arm/smmu-v3.c    | 10 ----------
 xen/drivers/passthrough/arm/smmu.c       | 10 ----------
 4 files changed, 7 insertions(+), 28 deletions(-)

diff --git a/xen/drivers/passthrough/arm/iommu.c b/xen/drivers/passthrough/arm/iommu.c
index ee653a9..fc45318 100644
--- a/xen/drivers/passthrough/arm/iommu.c
+++ b/xen/drivers/passthrough/arm/iommu.c
@@ -134,6 +134,13 @@ void arch_iommu_domain_destroy(struct domain *d)
 
 void __hwdom_init arch_iommu_hwdom_init(struct domain *d)
 {
+    /* Set to false options not supported on ARM. */
+    if ( iommu_hwdom_inclusive )
+        printk(XENLOG_WARNING "map-inclusive dom0-iommu option is not supported on ARM\n");
+    iommu_hwdom_inclusive = false;
+    if ( iommu_hwdom_reserved == 1 )
+        printk(XENLOG_WARNING "map-reserved dom0-iommu option is not supported on ARM\n");
+    iommu_hwdom_reserved = 0;
 }
 
 /*
diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
index 1224ea4..64d8ddc 100644
--- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
+++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
@@ -1325,14 +1325,6 @@ static int ipmmu_iommu_domain_init(struct domain *d)
 
 static void __hwdom_init ipmmu_iommu_hwdom_init(struct domain *d)
 {
-    /* Set to false options not supported on ARM. */
-    if ( iommu_hwdom_inclusive )
-        printk(XENLOG_WARNING "ipmmu: map-inclusive dom0-iommu option is not supported on ARM\n");
-    iommu_hwdom_inclusive = false;
-    if ( iommu_hwdom_reserved == 1 )
-        printk(XENLOG_WARNING "ipmmu: map-reserved dom0-iommu option is not supported on ARM\n");
-    iommu_hwdom_reserved = 0;
-
     arch_iommu_hwdom_init(d);
 }
 
diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
index d115df7..ca8b5c7 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.c
+++ b/xen/drivers/passthrough/arm/smmu-v3.c
@@ -3404,16 +3404,6 @@ static int arm_smmu_iommu_xen_domain_init(struct domain *d)
 
 static void __hwdom_init arm_smmu_iommu_hwdom_init(struct domain *d)
 {
-	/* Set to false options not supported on ARM. */
-	if (iommu_hwdom_inclusive)
-		printk(XENLOG_WARNING
-		"map-inclusive dom0-iommu option is not supported on ARM\n");
-	iommu_hwdom_inclusive = false;
-	if (iommu_hwdom_reserved == 1)
-		printk(XENLOG_WARNING
-		"map-reserved dom0-iommu option is not supported on ARM\n");
-	iommu_hwdom_reserved = 0;
-
 	arch_iommu_hwdom_init(d);
 }
 
diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c
index c9dfc4c..ec18df7 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -2851,16 +2851,6 @@ static int arm_smmu_iommu_domain_init(struct domain *d)
 
 static void __hwdom_init arm_smmu_iommu_hwdom_init(struct domain *d)
 {
-	/* Set to false options not supported on ARM. */
-	if ( iommu_hwdom_inclusive )
-		printk(XENLOG_WARNING
-		"map-inclusive dom0-iommu option is not supported on ARM\n");
-	iommu_hwdom_inclusive = false;
-	if ( iommu_hwdom_reserved == 1 )
-		printk(XENLOG_WARNING
-		"map-reserved dom0-iommu option is not supported on ARM\n");
-	iommu_hwdom_reserved = 0;
-
 	arch_iommu_hwdom_init(d);
 }
 
-- 
2.7.4

RE: [PATCH V2 07/10] iommu/ipmmu-vmsa: Add Renesas R8A779F0 (R-Car S4) support

[Yoshihiro Shimoda]

Hello Oleksandr-san,

> From: Oleksandr Tyshchenko, Sent: Tuesday, December 21, 2021 6:16 AM
> 
> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> 
> Based on the following Linux upsteam commit:
> 7a62ced8ebd0e1b692c9dc4781a8d4ddb0f74792
> 
> Original commit message:
>  commit 7a62ced8ebd0e1b692c9dc4781a8d4ddb0f74792
>  Author: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
>  Date:   Tue Sep 7 17:30:20 2021 +0900
> 
>   iommu/ipmmu-vmsa: Add support for r8a779a0
> 
>   Add support for r8a779a0 (R-Car V3U). The IPMMU hardware design
>   of this SoC differs than others. So, add a new ipmmu_features for it.
> 
>   Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
>   Reviewed-by: Geert Uytterhoeven <geert+renesas@glider.be>
>   Link:
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kernel.org%2Fr%2F20210907083020.907648-3-yosh
> ihiro.shimoda.uh%40renesas.com&amp;data=04%7C01%7Cyoshihiro.shimoda.uh%40renesas.com%7C995a1c66b6d841ccc79a08d9c3fdf
> 90a%7C53d82571da1947e49cb4625a166a4a2a%7C0%7C0%7C637756317833053012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=TirhJToNqy33Fzt0tSpukl9YFOj4iHgd3M1lX8W%2Bxlo%3D&amp;res
> erved=0
>   Signed-off-by: Joerg Roedel <jroedel@suse.de>
> 
> **********
> 
> The R-Car S4 is an automotive System-on-Chip (SoC) for Car
> Server/Communication Gateway and is one of the first products
> in Renesas' 4th-generation R-Car Family.
> 
> The integrated IOMMU HW is also VMSA-compatible and supports
> stage 2 translation table format, therefore can be used with
> current driver with slight modifications (thanks to the prereq
> work).
> 
> In the context of Xen driver the main differences between Gen3
> and Gen4 are the following:
> - HW capacity was enlarged to support up to 16 IPMMU contexts
>   (sets of page table) and up to 64 micro-TLBs per IPMMU device
> - the memory mapped registers have different bases and offsets
> 
> Please note that Linux upstream doesn't support R-Car S4 SoC
> yet unlike Renesas BSP [1], but it was decided to reuse upstream
> patch for R-Car V3U anyway as the IPMMU HW settings are similar.
> 
> [1]
> 7003b9f732cf iommu/ipmmu-vmsa: Add Renesas R8A779F0 (R-Car S4) support
<snip>
> 
> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>

Thank you for the patch!

Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>

Best regards,
Yoshihiro Shimoda

RE: [PATCH V2 08/10] iommu/ipmmu-vmsa: Set IPMMU bit IMSCTLR_USE_SECGRP to 0

[Yoshihiro Shimoda]

Hello Oleksandr-san,

> From: Oleksandr Tyshchenko, Sent: Tuesday, December 21, 2021 6:16 AM
> 
> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> 
> Based on the following commits from the Renesas BSP:
> 8fba83d97cca709a05139c38e29408e81ed4cf62
> a8d93bc07da89a7fcf4d85f34d119a030310efa5
> located at:
<snip>
> 
> Original commit messages:
>  commit 8fba83d97cca709a05139c38e29408e81ed4cf62
>  Author: Nam Nguyen <nam.nguyen.yh@renesas.com>
>  Date:   Wed Apr 28 18:54:44 2021 +0700
> 
>   iommu/ipmmu-vmsa: Set IPMMU bit IMSCTLR_USE_SECGRP to 0
> 
>   Need to set bit IMSCTLR_USE_SECGRP to 0
>   because H/W initial value is unknown, without this
>   dma-transfer cannot be done due to address translation doesn't work.
> 
>   Signed-off-by: Nam Nguyen <nam.nguyen.yh@renesas.com>
> 
>  commit a8d93bc07da89a7fcf4d85f34d119a030310efa5
>  Author: Nam Nguyen <nam.nguyen.yh@renesas.com>
>  Date:   Tue Sep 7 14:46:12 2021 +0700
> 
>   iommu/ipmmu-vmsa: Update IMSCTLR register offset address for R-Car S4
> 
>   Update IMSCTLR register offset address to align with R-Car S4 H/W UM.
> 
>   Signed-off-by: Nam Nguyen <nam.nguyen.yh@renesas.com>
> 
> **********
> 
> It is still a question whether this really needs to be done in Xen,
> rather in firmware, but better to be on the safe side. After all,
> if firmware already takes care of clearing this bit, nothing bad
> will happen.
> 
> Please note the following:
> 1. I decided to squash both commits since the first commit adds clearing
> code and only the second one makes it functional on S4. Moreover, this is
> not a direct port. So it would be better to introduce complete solution
> by a single patch.
> 2. Although patch indeed does what it claims in the subject,
> the implementation is different in comparison with original changes.
> On Linux the clearing is done at runtime in ipmmu_domain_setup_context().
> On Xen the clearing is done at boot time in ipmmu_probe().
> The IMSCTLR is not a MMU "context" register at all, so I think there is
> no point in performing the clearing each time we initialize the context,
> instead perform the clearing at once during initialization. Also do not
> abuse ctx_offset_stride_adj field for the register's offset calculation,
> instead use recently added control_offset_base field.
> 
> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>

Thank you for the patch!

Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>

Best regards,
Yoshihiro Shimoda

Re: [PATCH V2 09/10] iommu/ipmmu-vmsa: Use refcount for the micro-TLBs

[Julien Grall]

Hi Oleksandr,

On 20/12/2021 21:15, Oleksandr Tyshchenko wrote:
> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> 
> Reference-count the micro-TLBs as several bus masters can be
> connected to the same micro-TLB (and drop TODO comment).
> This wasn't an issue so far, since the platform devices
> (this driver deals with) get assigned/deassigned together during
> domain creation/destruction. But, in order to support PCI devices
> (which are hot-pluggable) in the near future we will need to
> take care of.

Looking at the code, it is not possible to share the micro-TLB between 
domains (or even Xen). So technically, we will still want to {, 
un}hotplug the devices using the same u-TLB together.

Therefore, I would clarify that this is necessary because even if we 
have to remove all the devices together, the IOMMU driver will be 
de-assigning them one-by-one.

I would add a similar comment in the code as well.

> 
> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> ---
> Changes V1 -> V2:
>     - add R-b
>     - add ASSERT() in ipmmu_utlb_disable()
> ---
>   xen/drivers/passthrough/arm/ipmmu-vmsa.c | 19 +++++++++++--------
>   1 file changed, 11 insertions(+), 8 deletions(-)
> 
> diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
> index 649b9f6..1224ea4 100644
> --- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
> +++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
> @@ -127,6 +127,7 @@ struct ipmmu_vmsa_device {
>       spinlock_t lock;    /* Protects ctx and domains[] */
>       DECLARE_BITMAP(ctx, IPMMU_CTX_MAX);
>       struct ipmmu_vmsa_domain *domains[IPMMU_CTX_MAX];
> +    unsigned int utlb_refcount[IPMMU_UTLB_MAX];
>       const struct ipmmu_features *features;
>   };
>   
> @@ -467,13 +468,12 @@ static int ipmmu_utlb_enable(struct ipmmu_vmsa_domain *domain,
>           }
>       }
>   
> -    /*
> -     * TODO: Reference-count the micro-TLB as several bus masters can be
> -     * connected to the same micro-TLB.
> -     */
> -    ipmmu_imuasid_write(mmu, utlb, 0);
> -    ipmmu_imuctr_write(mmu, utlb, imuctr |
> -                       IMUCTR_TTSEL_MMU(domain->context_id) | IMUCTR_MMUEN);
> +    if ( mmu->utlb_refcount[utlb]++ == 0 )
> +    {
> +        ipmmu_imuasid_write(mmu, utlb, 0);
> +        ipmmu_imuctr_write(mmu, utlb, imuctr |
> +                           IMUCTR_TTSEL_MMU(domain->context_id) | IMUCTR_MMUEN);
> +    }
>   
>       return 0;
>   }
> @@ -484,7 +484,10 @@ static void ipmmu_utlb_disable(struct ipmmu_vmsa_domain *domain,
>   {
>       struct ipmmu_vmsa_device *mmu = domain->mmu;
>   
> -    ipmmu_imuctr_write(mmu, utlb, 0);
> +    ASSERT(mmu->utlb_refcount[utlb] > 0);
> +
> +    if ( --mmu->utlb_refcount[utlb] == 0 )
> +        ipmmu_imuctr_write(mmu, utlb, 0);
>   }
>   
>   /* Domain/Context Management */

Cheers,

-- 
Julien Grall

Re: [PATCH V2 10/10] iommu/arm: Remove code duplication in all IOMMU drivers

[Julien Grall]

Hi,

On 20/12/2021 21:15, Oleksandr Tyshchenko wrote:
> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> 
> All IOMMU drivers on Arm perform almost the same generic actions in
> hwdom_init callback. Move this code to common arch_iommu_hwdom_init()
> in order to get rid of code duplication.
> 
> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
> Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> ---
> Changes V1 -> V2:
>     - add R-b
> ---
>   xen/drivers/passthrough/arm/iommu.c      |  7 +++++++
>   xen/drivers/passthrough/arm/ipmmu-vmsa.c |  8 --------
>   xen/drivers/passthrough/arm/smmu-v3.c    | 10 ----------
>   xen/drivers/passthrough/arm/smmu.c       | 10 ----------
>   4 files changed, 7 insertions(+), 28 deletions(-)
> 
> diff --git a/xen/drivers/passthrough/arm/iommu.c b/xen/drivers/passthrough/arm/iommu.c
> index ee653a9..fc45318 100644
> --- a/xen/drivers/passthrough/arm/iommu.c
> +++ b/xen/drivers/passthrough/arm/iommu.c
> @@ -134,6 +134,13 @@ void arch_iommu_domain_destroy(struct domain *d)
>   
>   void __hwdom_init arch_iommu_hwdom_init(struct domain *d)
>   {
> +    /* Set to false options not supported on ARM. */
> +    if ( iommu_hwdom_inclusive )
> +        printk(XENLOG_WARNING "map-inclusive dom0-iommu option is not supported on ARM\n");
> +    iommu_hwdom_inclusive = false;
> +    if ( iommu_hwdom_reserved == 1 )
> +        printk(XENLOG_WARNING "map-reserved dom0-iommu option is not supported on ARM\n");
> +    iommu_hwdom_reserved = 0;
>   }
>   
>   /*
> diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
> index 1224ea4..64d8ddc 100644
> --- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
> +++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
> @@ -1325,14 +1325,6 @@ static int ipmmu_iommu_domain_init(struct domain *d)
>   
>   static void __hwdom_init ipmmu_iommu_hwdom_init(struct domain *d)
>   {
> -    /* Set to false options not supported on ARM. */
> -    if ( iommu_hwdom_inclusive )
> -        printk(XENLOG_WARNING "ipmmu: map-inclusive dom0-iommu option is not supported on ARM\n");
> -    iommu_hwdom_inclusive = false;
> -    if ( iommu_hwdom_reserved == 1 )
> -        printk(XENLOG_WARNING "ipmmu: map-reserved dom0-iommu option is not supported on ARM\n");
> -    iommu_hwdom_reserved = 0;
> -
>       arch_iommu_hwdom_init(d);
>   }

With this change, ipmmu_iommu_domain_init() is just a call to 
arch_iommu_hwdom_init(). The two functions have the same prototype, so 
could we simply drop the former and update the .hwdom_init to directly 
point to the latter?

>   
> diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
> index d115df7..ca8b5c7 100644
> --- a/xen/drivers/passthrough/arm/smmu-v3.c
> +++ b/xen/drivers/passthrough/arm/smmu-v3.c
> @@ -3404,16 +3404,6 @@ static int arm_smmu_iommu_xen_domain_init(struct domain *d)
>   
>   static void __hwdom_init arm_smmu_iommu_hwdom_init(struct domain *d)
>   {
> -	/* Set to false options not supported on ARM. */
> -	if (iommu_hwdom_inclusive)
> -		printk(XENLOG_WARNING
> -		"map-inclusive dom0-iommu option is not supported on ARM\n");
> -	iommu_hwdom_inclusive = false;
> -	if (iommu_hwdom_reserved == 1)
> -		printk(XENLOG_WARNING
> -		"map-reserved dom0-iommu option is not supported on ARM\n");
> -	iommu_hwdom_reserved = 0;
> -
>   	arch_iommu_hwdom_init(d);
>   }
>   

Same here.

> diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c
> index c9dfc4c..ec18df7 100644
> --- a/xen/drivers/passthrough/arm/smmu.c
> +++ b/xen/drivers/passthrough/arm/smmu.c
> @@ -2851,16 +2851,6 @@ static int arm_smmu_iommu_domain_init(struct domain *d)
>   
>   static void __hwdom_init arm_smmu_iommu_hwdom_init(struct domain *d)
>   {
> -	/* Set to false options not supported on ARM. */
> -	if ( iommu_hwdom_inclusive )
> -		printk(XENLOG_WARNING
> -		"map-inclusive dom0-iommu option is not supported on ARM\n");
> -	iommu_hwdom_inclusive = false;
> -	if ( iommu_hwdom_reserved == 1 )
> -		printk(XENLOG_WARNING
> -		"map-reserved dom0-iommu option is not supported on ARM\n");
> -	iommu_hwdom_reserved = 0;
> -
>   	arch_iommu_hwdom_init(d);
>   }
>   

Same here.

Cheers,

-- 
Julien Grall

Re: [PATCH V2 10/10] iommu/arm: Remove code duplication in all IOMMU drivers

[Oleksandr]

On 27.01.22 13:54, Julien Grall wrote:
> Hi,


Hi Julien


>
> On 20/12/2021 21:15, Oleksandr Tyshchenko wrote:
>> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>>
>> All IOMMU drivers on Arm perform almost the same generic actions in
>> hwdom_init callback. Move this code to common arch_iommu_hwdom_init()
>> in order to get rid of code duplication.
>>
>> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>> Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
>> Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
>> ---
>> Changes V1 -> V2:
>>     - add R-b
>> ---
>>   xen/drivers/passthrough/arm/iommu.c      |  7 +++++++
>>   xen/drivers/passthrough/arm/ipmmu-vmsa.c |  8 --------
>>   xen/drivers/passthrough/arm/smmu-v3.c    | 10 ----------
>>   xen/drivers/passthrough/arm/smmu.c       | 10 ----------
>>   4 files changed, 7 insertions(+), 28 deletions(-)
>>
>> diff --git a/xen/drivers/passthrough/arm/iommu.c 
>> b/xen/drivers/passthrough/arm/iommu.c
>> index ee653a9..fc45318 100644
>> --- a/xen/drivers/passthrough/arm/iommu.c
>> +++ b/xen/drivers/passthrough/arm/iommu.c
>> @@ -134,6 +134,13 @@ void arch_iommu_domain_destroy(struct domain *d)
>>     void __hwdom_init arch_iommu_hwdom_init(struct domain *d)
>>   {
>> +    /* Set to false options not supported on ARM. */
>> +    if ( iommu_hwdom_inclusive )
>> +        printk(XENLOG_WARNING "map-inclusive dom0-iommu option is 
>> not supported on ARM\n");
>> +    iommu_hwdom_inclusive = false;
>> +    if ( iommu_hwdom_reserved == 1 )
>> +        printk(XENLOG_WARNING "map-reserved dom0-iommu option is not 
>> supported on ARM\n");
>> +    iommu_hwdom_reserved = 0;
>>   }
>>     /*
>> diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c 
>> b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
>> index 1224ea4..64d8ddc 100644
>> --- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
>> +++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
>> @@ -1325,14 +1325,6 @@ static int ipmmu_iommu_domain_init(struct 
>> domain *d)
>>     static void __hwdom_init ipmmu_iommu_hwdom_init(struct domain *d)
>>   {
>> -    /* Set to false options not supported on ARM. */
>> -    if ( iommu_hwdom_inclusive )
>> -        printk(XENLOG_WARNING "ipmmu: map-inclusive dom0-iommu 
>> option is not supported on ARM\n");
>> -    iommu_hwdom_inclusive = false;
>> -    if ( iommu_hwdom_reserved == 1 )
>> -        printk(XENLOG_WARNING "ipmmu: map-reserved dom0-iommu option 
>> is not supported on ARM\n");
>> -    iommu_hwdom_reserved = 0;
>> -
>>       arch_iommu_hwdom_init(d);
>>   }
>
> With this change, ipmmu_iommu_domain_init() is just a call to 
> arch_iommu_hwdom_init(). The two functions have the same prototype, so 
> could we simply drop the former and update the .hwdom_init to directly 
> point to the latter?

Yes, good idea, will update. Thank you.


>
>
>>   diff --git a/xen/drivers/passthrough/arm/smmu-v3.c 
>> b/xen/drivers/passthrough/arm/smmu-v3.c
>> index d115df7..ca8b5c7 100644
>> --- a/xen/drivers/passthrough/arm/smmu-v3.c
>> +++ b/xen/drivers/passthrough/arm/smmu-v3.c
>> @@ -3404,16 +3404,6 @@ static int 
>> arm_smmu_iommu_xen_domain_init(struct domain *d)
>>     static void __hwdom_init arm_smmu_iommu_hwdom_init(struct domain *d)
>>   {
>> -    /* Set to false options not supported on ARM. */
>> -    if (iommu_hwdom_inclusive)
>> -        printk(XENLOG_WARNING
>> -        "map-inclusive dom0-iommu option is not supported on ARM\n");
>> -    iommu_hwdom_inclusive = false;
>> -    if (iommu_hwdom_reserved == 1)
>> -        printk(XENLOG_WARNING
>> -        "map-reserved dom0-iommu option is not supported on ARM\n");
>> -    iommu_hwdom_reserved = 0;
>> -
>>       arch_iommu_hwdom_init(d);
>>   }
>
> Same here.
>
>> diff --git a/xen/drivers/passthrough/arm/smmu.c 
>> b/xen/drivers/passthrough/arm/smmu.c
>> index c9dfc4c..ec18df7 100644
>> --- a/xen/drivers/passthrough/arm/smmu.c
>> +++ b/xen/drivers/passthrough/arm/smmu.c
>> @@ -2851,16 +2851,6 @@ static int arm_smmu_iommu_domain_init(struct 
>> domain *d)
>>     static void __hwdom_init arm_smmu_iommu_hwdom_init(struct domain *d)
>>   {
>> -    /* Set to false options not supported on ARM. */
>> -    if ( iommu_hwdom_inclusive )
>> -        printk(XENLOG_WARNING
>> -        "map-inclusive dom0-iommu option is not supported on ARM\n");
>> -    iommu_hwdom_inclusive = false;
>> -    if ( iommu_hwdom_reserved == 1 )
>> -        printk(XENLOG_WARNING
>> -        "map-reserved dom0-iommu option is not supported on ARM\n");
>> -    iommu_hwdom_reserved = 0;
>> -
>>       arch_iommu_hwdom_init(d);
>>   }
>
> Same here.
>
> Cheers,
>
-- 
Regards,

Oleksandr Tyshchenko

Re: [PATCH V2 00/10] Add support for Renesas R-Car S4 IPMMU and other misc changes

[Julien Grall]

Hi,

On 20/12/2021 21:15, Oleksandr Tyshchenko wrote:
> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> 
> Hello all.
> 
> You can find the V1 patch series at [1].
> 
> The R-Car S4 is an automotive System-on-Chip (SoC) for Car Server/Communication
> Gateway and is one of the first products in Renesas’ 4th-generation R-Car Family.
> 
> The integrated IOMMU HW is also VMSA-compatible and supports stage 2 translation
> table format, therefore can be used with current R-Car Gen3 driver with slight
> modifications.
> 
> In the context of Xen driver the main differences between Gen3 and Gen4 are
> the following:
>   - HW capacity was enlarged to support up to 16 IPMMU contexts (sets of page table)
>     and up to 64 micro-TLBs per IPMMU device
>   - the memory mapped registers have different bases and offset
> 
> The first part (commits #1-6) is a non-verbatim port of Linux upstream commits
> needed to add support for S4 series easily (prereq work).
> The second part (commits #7-8) is based on the code from the Renesas BSP and
> actually introduces support for R-Car S4 IPMMU.
> The third part (commits #9-10) is misc changes I have locally.
> 
> The patch series is based on the latest staging branch
> "f3999bc2e099 arm/efi: Handle Xen bootargs from both xen.cfg and DT"
> and also available at [2].
> 
> Tested on Renesas Salvator-X board with H3 ES3.0 SoC (Gen3) and Renesas Spider
> board with S4 SoC (Gen4).
> 
> [1] https://lore.kernel.org/all/1638035505-16931-1-git-send-email-olekstysh@gmail.com/
> [2] https://github.com/otyshchenko1/xen/commits/s4_ipmmu_ml2
> 
> 
> Oleksandr Tyshchenko (10):
>    iommu/ipmmu-vmsa: Remove all unused register definitions
>    iommu/ipmmu-vmsa: Add helper functions for MMU "context" registers
>    iommu/ipmmu-vmsa: Add helper functions for "uTLB" registers
>    iommu/ipmmu-vmsa: Add light version of Linux's ipmmu_features
>    iommu/ipmmu-vmsa: Calculate context registers' offset instead of a
>      macro
>    iommu/ipmmu-vmsa: Add utlb_offset_base
>    iommu/ipmmu-vmsa: Add Renesas R8A779F0 (R-Car S4) support
>    iommu/ipmmu-vmsa: Set IPMMU bit IMSCTLR_USE_SECGRP to 0

For the first 8 patches:

Acked-by: Julien Grall <jgrall@amazon.com>

I have also committed them. The last..

>    iommu/ipmmu-vmsa: Use refcount for the micro-TLBs
>    iommu/arm: Remove code duplication in all IOMMU drivers

... 2 patches have some comments. So I have committed them yet.

Cheers,

-- 
Julien Grall

Re: [PATCH V2 00/10] Add support for Renesas R-Car S4 IPMMU and other misc changes

[Oleksandr]

On 27.01.22 14:36, Julien Grall wrote:
> Hi,


Hi Julien


>
> On 20/12/2021 21:15, Oleksandr Tyshchenko wrote:
>> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>>
>> Hello all.
>>
>> You can find the V1 patch series at [1].
>>
>> The R-Car S4 is an automotive System-on-Chip (SoC) for Car 
>> Server/Communication
>> Gateway and is one of the first products in Renesas’ 4th-generation 
>> R-Car Family.
>>
>> The integrated IOMMU HW is also VMSA-compatible and supports stage 2 
>> translation
>> table format, therefore can be used with current R-Car Gen3 driver 
>> with slight
>> modifications.
>>
>> In the context of Xen driver the main differences between Gen3 and 
>> Gen4 are
>> the following:
>>   - HW capacity was enlarged to support up to 16 IPMMU contexts (sets 
>> of page table)
>>     and up to 64 micro-TLBs per IPMMU device
>>   - the memory mapped registers have different bases and offset
>>
>> The first part (commits #1-6) is a non-verbatim port of Linux 
>> upstream commits
>> needed to add support for S4 series easily (prereq work).
>> The second part (commits #7-8) is based on the code from the Renesas 
>> BSP and
>> actually introduces support for R-Car S4 IPMMU.
>> The third part (commits #9-10) is misc changes I have locally.
>>
>> The patch series is based on the latest staging branch
>> "f3999bc2e099 arm/efi: Handle Xen bootargs from both xen.cfg and DT"
>> and also available at [2].
>>
>> Tested on Renesas Salvator-X board with H3 ES3.0 SoC (Gen3) and 
>> Renesas Spider
>> board with S4 SoC (Gen4).
>>
>> [1] 
>> https://lore.kernel.org/all/1638035505-16931-1-git-send-email-olekstysh@gmail.com/
>> [2] https://github.com/otyshchenko1/xen/commits/s4_ipmmu_ml2
>>
>>
>> Oleksandr Tyshchenko (10):
>>    iommu/ipmmu-vmsa: Remove all unused register definitions
>>    iommu/ipmmu-vmsa: Add helper functions for MMU "context" registers
>>    iommu/ipmmu-vmsa: Add helper functions for "uTLB" registers
>>    iommu/ipmmu-vmsa: Add light version of Linux's ipmmu_features
>>    iommu/ipmmu-vmsa: Calculate context registers' offset instead of a
>>      macro
>>    iommu/ipmmu-vmsa: Add utlb_offset_base
>>    iommu/ipmmu-vmsa: Add Renesas R8A779F0 (R-Car S4) support
>>    iommu/ipmmu-vmsa: Set IPMMU bit IMSCTLR_USE_SECGRP to 0
>
> For the first 8 patches:
>
> Acked-by: Julien Grall <jgrall@amazon.com>

Thank you!


>
> I have also committed them. The last..
>
>>    iommu/ipmmu-vmsa: Use refcount for the micro-TLBs
>>    iommu/arm: Remove code duplication in all IOMMU drivers
>
> ... 2 patches have some comments. So I have committed them yet.

ok, I will update them and re-push probably today.


>
>
> Cheers,
>
-- 
Regards,

Oleksandr Tyshchenko

Re: [PATCH V2 09/10] iommu/ipmmu-vmsa: Use refcount for the micro-TLBs

[Oleksandr]

On 27.01.22 13:48, Julien Grall wrote:
> Hi Oleksandr,

Hi Julien


>
> On 20/12/2021 21:15, Oleksandr Tyshchenko wrote:
>> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>>
>> Reference-count the micro-TLBs as several bus masters can be
>> connected to the same micro-TLB (and drop TODO comment).
>> This wasn't an issue so far, since the platform devices
>> (this driver deals with) get assigned/deassigned together during
>> domain creation/destruction. But, in order to support PCI devices
>> (which are hot-pluggable) in the near future we will need to
>> take care of.
>
> Looking at the code, it is not possible to share the micro-TLB between 
> domains (or even Xen). So technically, we will still want to {, 
> un}hotplug the devices using the same u-TLB together.
>
> Therefore, I would clarify that this is necessary because even if we 
> have to remove all the devices together, the IOMMU driver will be 
> de-assigning them one-by-one.
>
> I would add a similar comment in the code as well.

ok, will add.


>
>
>>
>> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>> Reviewed-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
>> ---
>> Changes V1 -> V2:
>>     - add R-b
>>     - add ASSERT() in ipmmu_utlb_disable()
>> ---
>>   xen/drivers/passthrough/arm/ipmmu-vmsa.c | 19 +++++++++++--------
>>   1 file changed, 11 insertions(+), 8 deletions(-)
>>
>> diff --git a/xen/drivers/passthrough/arm/ipmmu-vmsa.c 
>> b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
>> index 649b9f6..1224ea4 100644
>> --- a/xen/drivers/passthrough/arm/ipmmu-vmsa.c
>> +++ b/xen/drivers/passthrough/arm/ipmmu-vmsa.c
>> @@ -127,6 +127,7 @@ struct ipmmu_vmsa_device {
>>       spinlock_t lock;    /* Protects ctx and domains[] */
>>       DECLARE_BITMAP(ctx, IPMMU_CTX_MAX);
>>       struct ipmmu_vmsa_domain *domains[IPMMU_CTX_MAX];
>> +    unsigned int utlb_refcount[IPMMU_UTLB_MAX];
>>       const struct ipmmu_features *features;
>>   };
>>   @@ -467,13 +468,12 @@ static int ipmmu_utlb_enable(struct 
>> ipmmu_vmsa_domain *domain,
>>           }
>>       }
>>   -    /*
>> -     * TODO: Reference-count the micro-TLB as several bus masters 
>> can be
>> -     * connected to the same micro-TLB.
>> -     */
>> -    ipmmu_imuasid_write(mmu, utlb, 0);
>> -    ipmmu_imuctr_write(mmu, utlb, imuctr |
>> -                       IMUCTR_TTSEL_MMU(domain->context_id) | 
>> IMUCTR_MMUEN);
>> +    if ( mmu->utlb_refcount[utlb]++ == 0 )
>> +    {
>> +        ipmmu_imuasid_write(mmu, utlb, 0);
>> +        ipmmu_imuctr_write(mmu, utlb, imuctr |
>> + IMUCTR_TTSEL_MMU(domain->context_id) | IMUCTR_MMUEN);
>> +    }
>>         return 0;
>>   }
>> @@ -484,7 +484,10 @@ static void ipmmu_utlb_disable(struct 
>> ipmmu_vmsa_domain *domain,
>>   {
>>       struct ipmmu_vmsa_device *mmu = domain->mmu;
>>   -    ipmmu_imuctr_write(mmu, utlb, 0);
>> +    ASSERT(mmu->utlb_refcount[utlb] > 0);
>> +
>> +    if ( --mmu->utlb_refcount[utlb] == 0 )
>> +        ipmmu_imuctr_write(mmu, utlb, 0);
>>   }
>>     /* Domain/Context Management */
>
> Cheers,
>
-- 
Regards,

Oleksandr Tyshchenko