[PATCH 0/4] Apple M1 USB4/Thunderbolt DART support

[PATCH 0/4] Apple M1 USB4/Thunderbolt DART support

[Sven Peter]

Hi,

The M1 SoCs have a second slightly different variant of the regular
DART used for the USB4 PCIe ports. It supports 64 instead of 16 streams
which requires a minor change to the MMIO layout.
There seems to be no way to tell them apart from the regular DARTs by
just looking at the DART_PARAMs register so we have to add a new
compatible for those.

Best,

Sven

Sven Peter (4):
  dt-bindings: iommu: dart: Add t8103-usb4-dart compatible
  iommu: dart: Add flag to override bypass support
  iommu: dart: Write to all DART_T8020_STREAM_SELECT
  iommu: dart: Add support for t8103 USB4 DART

 .../devicetree/bindings/iommu/apple,dart.yaml |  1 +
 drivers/iommu/apple-dart.c                    | 42 ++++++++++++++++++-
 2 files changed, 41 insertions(+), 2 deletions(-)

-- 
2.25.1

[PATCH 1/4] dt-bindings: iommu: dart: Add t8103-usb4-dart compatible

[Sven Peter]

This DART variant is found in the t8103 (M1) SoCs and used for the
USB4/Thunderbolt PCIe ports. Unlike the regular t8103 DART these support
up to 64 SIDs and require a slightly different MMIO layout. This variant
is only found on the M1 SoCs.

Signed-off-by: Sven Peter <sven@svenpeter.dev>
---
 Documentation/devicetree/bindings/iommu/apple,dart.yaml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Documentation/devicetree/bindings/iommu/apple,dart.yaml b/Documentation/devicetree/bindings/iommu/apple,dart.yaml
index 903edf85d72e..f9c1843c074d 100644
--- a/Documentation/devicetree/bindings/iommu/apple,dart.yaml
+++ b/Documentation/devicetree/bindings/iommu/apple,dart.yaml
@@ -24,6 +24,7 @@ properties:
   compatible:
     enum:
       - apple,t8103-dart
+      - apple,t8103-dart-usb4
       - apple,t8110-dart
       - apple,t6000-dart
 
-- 
2.25.1

Re: [PATCH 1/4] dt-bindings: iommu: dart: Add t8103-usb4-dart compatible

[Hector Martin]

On 28/01/2023 20.11, Sven Peter wrote:
> This DART variant is found in the t8103 (M1) SoCs and used for the
> USB4/Thunderbolt PCIe ports. Unlike the regular t8103 DART these support
> up to 64 SIDs and require a slightly different MMIO layout. This variant
> is only found on the M1 SoCs.
> 
> Signed-off-by: Sven Peter <sven@svenpeter.dev>
> ---
>  Documentation/devicetree/bindings/iommu/apple,dart.yaml | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/Documentation/devicetree/bindings/iommu/apple,dart.yaml b/Documentation/devicetree/bindings/iommu/apple,dart.yaml
> index 903edf85d72e..f9c1843c074d 100644
> --- a/Documentation/devicetree/bindings/iommu/apple,dart.yaml
> +++ b/Documentation/devicetree/bindings/iommu/apple,dart.yaml
> @@ -24,6 +24,7 @@ properties:
>    compatible:
>      enum:
>        - apple,t8103-dart
> +      - apple,t8103-dart-usb4
>        - apple,t8110-dart
>        - apple,t6000-dart
>  

You probably wanted t8103-usb4-dart (as mentioned in the commit message).

Other than that,

Acked-by: Hector Martin <marcan@marcan.st>

- Hector

Re: [PATCH 1/4] dt-bindings: iommu: dart: Add t8103-usb4-dart compatible

[Rob Herring]

On Sat, Jan 28, 2023 at 12:11:11PM +0100, Sven Peter wrote:
> This DART variant is found in the t8103 (M1) SoCs and used for the
> USB4/Thunderbolt PCIe ports. Unlike the regular t8103 DART these support
> up to 64 SIDs and require a slightly different MMIO layout. This variant
> is only found on the M1 SoCs.

Looks fine to me other than the discrepancy in the name.

> Signed-off-by: Sven Peter <sven@svenpeter.dev>
> ---
>  Documentation/devicetree/bindings/iommu/apple,dart.yaml | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/Documentation/devicetree/bindings/iommu/apple,dart.yaml b/Documentation/devicetree/bindings/iommu/apple,dart.yaml
> index 903edf85d72e..f9c1843c074d 100644
> --- a/Documentation/devicetree/bindings/iommu/apple,dart.yaml
> +++ b/Documentation/devicetree/bindings/iommu/apple,dart.yaml
> @@ -24,6 +24,7 @@ properties:
>    compatible:
>      enum:
>        - apple,t8103-dart
> +      - apple,t8103-dart-usb4
>        - apple,t8110-dart
>        - apple,t6000-dart
>  
> -- 
> 2.25.1
> 

[PATCH 2/4] iommu: dart: Add flag to override bypass support

[Sven Peter]

The USB4 PCIe DARTs claim to support bypass but we never want to allow
that on externally facing ports.

Signed-off-by: Sven Peter <sven@svenpeter.dev>
---
 drivers/iommu/apple-dart.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/iommu/apple-dart.c b/drivers/iommu/apple-dart.c
index 42666617803d..7d8b2b90cdb6 100644
--- a/drivers/iommu/apple-dart.c
+++ b/drivers/iommu/apple-dart.c
@@ -182,6 +182,8 @@ struct apple_dart_hw {
 	u64 ttbr_addr_field_shift;
 	u64 ttbr_shift;
 	int ttbr_count;
+
+	bool disable_bypass;
 };
 
 /*
@@ -1075,6 +1077,9 @@ static int apple_dart_probe(struct platform_device *pdev)
 	dart->pgsize = 1 << FIELD_GET(DART_PARAMS1_PAGE_SHIFT, dart_params[0]);
 	dart->supports_bypass = dart_params[1] & DART_PARAMS2_BYPASS_SUPPORT;
 
+	if (dart->hw->disable_bypass)
+		dart->supports_bypass = 0;
+
 	switch (dart->hw->type) {
 	case DART_T8020:
 	case DART_T6000:
-- 
2.25.1

Re: [PATCH 2/4] iommu: dart: Add flag to override bypass support

[Hector Martin]

On 28/01/2023 20.11, Sven Peter wrote:
> The USB4 PCIe DARTs claim to support bypass but we never want to allow
> that on externally facing ports.
> 
> Signed-off-by: Sven Peter <sven@svenpeter.dev>
> ---
>  drivers/iommu/apple-dart.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/drivers/iommu/apple-dart.c b/drivers/iommu/apple-dart.c
> index 42666617803d..7d8b2b90cdb6 100644
> --- a/drivers/iommu/apple-dart.c
> +++ b/drivers/iommu/apple-dart.c
> @@ -182,6 +182,8 @@ struct apple_dart_hw {
>  	u64 ttbr_addr_field_shift;
>  	u64 ttbr_shift;
>  	int ttbr_count;
> +
> +	bool disable_bypass;
>  };
>  
>  /*
> @@ -1075,6 +1077,9 @@ static int apple_dart_probe(struct platform_device *pdev)
>  	dart->pgsize = 1 << FIELD_GET(DART_PARAMS1_PAGE_SHIFT, dart_params[0]);
>  	dart->supports_bypass = dart_params[1] & DART_PARAMS2_BYPASS_SUPPORT;
>  
> +	if (dart->hw->disable_bypass)
> +		dart->supports_bypass = 0;
> +
>  	switch (dart->hw->type) {
>  	case DART_T8020:
>  	case DART_T6000:


Acked-by: Hector Martin <marcan@marcan.st>

- Hector

[PATCH 3/4] iommu: dart: Write to all DART_T8020_STREAM_SELECT

[Sven Peter]

We're about to add support for a DART variant that use more than 16
streams and requires writing to two separate stream select registers
when issuing TLB flushes.

Signed-off-by: Sven Peter <sven@svenpeter.dev>
---
 drivers/iommu/apple-dart.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/apple-dart.c b/drivers/iommu/apple-dart.c
index 7d8b2b90cdb6..7579c97a9062 100644
--- a/drivers/iommu/apple-dart.c
+++ b/drivers/iommu/apple-dart.c
@@ -372,12 +372,14 @@ apple_dart_t8020_hw_stream_command(struct apple_dart_stream_map *stream_map,
 			     u32 command)
 {
 	unsigned long flags;
-	int ret;
+	int ret, i;
 	u32 command_reg;
 
 	spin_lock_irqsave(&stream_map->dart->lock, flags);
 
-	writel(stream_map->sidmap[0], stream_map->dart->regs + DART_T8020_STREAM_SELECT);
+	for (i = 0; i < BITS_TO_U32(stream_map->dart->num_streams); i++)
+		writel(stream_map->sidmap[i],
+		       stream_map->dart->regs + DART_T8020_STREAM_SELECT + 4 * i);
 	writel(command, stream_map->dart->regs + DART_T8020_STREAM_COMMAND);
 
 	ret = readl_poll_timeout_atomic(
-- 
2.25.1

Re: [PATCH 3/4] iommu: dart: Write to all DART_T8020_STREAM_SELECT

[Hector Martin]

On 28/01/2023 20.11, Sven Peter wrote:
> We're about to add support for a DART variant that use more than 16
> streams and requires writing to two separate stream select registers
> when issuing TLB flushes.
> 
> Signed-off-by: Sven Peter <sven@svenpeter.dev>
> ---
>  drivers/iommu/apple-dart.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/iommu/apple-dart.c b/drivers/iommu/apple-dart.c
> index 7d8b2b90cdb6..7579c97a9062 100644
> --- a/drivers/iommu/apple-dart.c
> +++ b/drivers/iommu/apple-dart.c
> @@ -372,12 +372,14 @@ apple_dart_t8020_hw_stream_command(struct apple_dart_stream_map *stream_map,
>  			     u32 command)
>  {
>  	unsigned long flags;
> -	int ret;
> +	int ret, i;
>  	u32 command_reg;
>  
>  	spin_lock_irqsave(&stream_map->dart->lock, flags);
>  
> -	writel(stream_map->sidmap[0], stream_map->dart->regs + DART_T8020_STREAM_SELECT);
> +	for (i = 0; i < BITS_TO_U32(stream_map->dart->num_streams); i++)
> +		writel(stream_map->sidmap[i],
> +		       stream_map->dart->regs + DART_T8020_STREAM_SELECT + 4 * i);
>  	writel(command, stream_map->dart->regs + DART_T8020_STREAM_COMMAND);
>  
>  	ret = readl_poll_timeout_atomic(

Acked-by: Hector Martin <marcan@marcan.st>

- Hector

[PATCH 4/4] iommu: dart: Add support for M1 USB4 PCIe DART

[Sven Peter]

This variant of the regular t8103 DART is used for the two
USB4/Thunderbolt PCIe controllers. It supports 64 instead of 16 streams
which requires a slightly different MMIO layout. We also disallow bypass
support since these DARTs will only ever be used for externally facing
devices on the USB4 ports.

Signed-off-by: Sven Peter <sven@svenpeter.dev>
---
 drivers/iommu/apple-dart.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/drivers/iommu/apple-dart.c b/drivers/iommu/apple-dart.c
index 7579c97a9062..a1304ba3639b 100644
--- a/drivers/iommu/apple-dart.c
+++ b/drivers/iommu/apple-dart.c
@@ -81,6 +81,7 @@
 #define DART_T8020_TCR_BYPASS_DAPF      BIT(12)
 
 #define DART_T8020_TTBR       0x200
+#define DART_T8020_USB4_TTBR  0x400
 #define DART_T8020_TTBR_VALID BIT(31)
 #define DART_T8020_TTBR_ADDR_FIELD_SHIFT 0
 #define DART_T8020_TTBR_SHIFT 12
@@ -1184,6 +1185,35 @@ static const struct apple_dart_hw apple_dart_hw_t8103 = {
 	.ttbr_shift = DART_T8020_TTBR_SHIFT,
 	.ttbr_count = 4,
 };
+
+static const struct apple_dart_hw apple_dart_hw_t8103_usb4 = {
+	.type = DART_T8020,
+	.irq_handler = apple_dart_t8020_irq,
+	.invalidate_tlb = apple_dart_t8020_hw_invalidate_tlb,
+	.oas = 36,
+	.fmt = APPLE_DART,
+	.max_sid_count = 64,
+
+	.enable_streams = DART_T8020_STREAMS_ENABLE,
+	.lock = DART_T8020_CONFIG,
+	.lock_bit = DART_T8020_CONFIG_LOCK,
+
+	.error = DART_T8020_ERROR,
+
+	.tcr = DART_T8020_TCR,
+	.tcr_enabled = DART_T8020_TCR_TRANSLATE_ENABLE,
+	.tcr_disabled = 0,
+	.tcr_bypass = 0,
+
+	.ttbr = DART_T8020_USB4_TTBR,
+	.ttbr_valid = DART_T8020_TTBR_VALID,
+	.ttbr_addr_field_shift = DART_T8020_TTBR_ADDR_FIELD_SHIFT,
+	.ttbr_shift = DART_T8020_TTBR_SHIFT,
+	.ttbr_count = 4,
+
+	.disable_bypass = true,
+};
+
 static const struct apple_dart_hw apple_dart_hw_t6000 = {
 	.type = DART_T6000,
 	.irq_handler = apple_dart_t8020_irq,
@@ -1276,6 +1306,7 @@ DEFINE_SIMPLE_DEV_PM_OPS(apple_dart_pm_ops, apple_dart_suspend, apple_dart_resum
 
 static const struct of_device_id apple_dart_of_match[] = {
 	{ .compatible = "apple,t8103-dart", .data = &apple_dart_hw_t8103 },
+	{ .compatible = "apple,t8103-dart-usb4", .data = &apple_dart_hw_t8103_usb4 },
 	{ .compatible = "apple,t8110-dart", .data = &apple_dart_hw_t8110 },
 	{ .compatible = "apple,t6000-dart", .data = &apple_dart_hw_t6000 },
 	{},
-- 
2.25.1

Re: [PATCH 4/4] iommu: dart: Add support for M1 USB4 PCIe DART

[Hector Martin]

On 28/01/2023 20.11, Sven Peter wrote:
> This variant of the regular t8103 DART is used for the two
> USB4/Thunderbolt PCIe controllers. It supports 64 instead of 16 streams
> which requires a slightly different MMIO layout. We also disallow bypass
> support since these DARTs will only ever be used for externally facing
> devices on the USB4 ports.
> 
> Signed-off-by: Sven Peter <sven@svenpeter.dev>
> ---
>  drivers/iommu/apple-dart.c | 31 +++++++++++++++++++++++++++++++
>  1 file changed, 31 insertions(+)
> 
> diff --git a/drivers/iommu/apple-dart.c b/drivers/iommu/apple-dart.c
> index 7579c97a9062..a1304ba3639b 100644
> --- a/drivers/iommu/apple-dart.c
> +++ b/drivers/iommu/apple-dart.c
> @@ -81,6 +81,7 @@
>  #define DART_T8020_TCR_BYPASS_DAPF      BIT(12)
>  
>  #define DART_T8020_TTBR       0x200
> +#define DART_T8020_USB4_TTBR  0x400
>  #define DART_T8020_TTBR_VALID BIT(31)
>  #define DART_T8020_TTBR_ADDR_FIELD_SHIFT 0
>  #define DART_T8020_TTBR_SHIFT 12
> @@ -1184,6 +1185,35 @@ static const struct apple_dart_hw apple_dart_hw_t8103 = {
>  	.ttbr_shift = DART_T8020_TTBR_SHIFT,
>  	.ttbr_count = 4,
>  };
> +
> +static const struct apple_dart_hw apple_dart_hw_t8103_usb4 = {
> +	.type = DART_T8020,
> +	.irq_handler = apple_dart_t8020_irq,
> +	.invalidate_tlb = apple_dart_t8020_hw_invalidate_tlb,
> +	.oas = 36,
> +	.fmt = APPLE_DART,
> +	.max_sid_count = 64,
> +
> +	.enable_streams = DART_T8020_STREAMS_ENABLE,
> +	.lock = DART_T8020_CONFIG,
> +	.lock_bit = DART_T8020_CONFIG_LOCK,
> +
> +	.error = DART_T8020_ERROR,
> +
> +	.tcr = DART_T8020_TCR,
> +	.tcr_enabled = DART_T8020_TCR_TRANSLATE_ENABLE,
> +	.tcr_disabled = 0,
> +	.tcr_bypass = 0,
> +
> +	.ttbr = DART_T8020_USB4_TTBR,
> +	.ttbr_valid = DART_T8020_TTBR_VALID,
> +	.ttbr_addr_field_shift = DART_T8020_TTBR_ADDR_FIELD_SHIFT,
> +	.ttbr_shift = DART_T8020_TTBR_SHIFT,
> +	.ttbr_count = 4,
> +
> +	.disable_bypass = true,
> +};
> +
>  static const struct apple_dart_hw apple_dart_hw_t6000 = {
>  	.type = DART_T6000,
>  	.irq_handler = apple_dart_t8020_irq,
> @@ -1276,6 +1306,7 @@ DEFINE_SIMPLE_DEV_PM_OPS(apple_dart_pm_ops, apple_dart_suspend, apple_dart_resum
>  
>  static const struct of_device_id apple_dart_of_match[] = {
>  	{ .compatible = "apple,t8103-dart", .data = &apple_dart_hw_t8103 },
> +	{ .compatible = "apple,t8103-dart-usb4", .data = &apple_dart_hw_t8103_usb4 },
>  	{ .compatible = "apple,t8110-dart", .data = &apple_dart_hw_t8110 },
>  	{ .compatible = "apple,t6000-dart", .data = &apple_dart_hw_t6000 },
>  	{},

Other than the compatible as per patch #1,

Acked-by: Hector Martin <marcan@marcan.st>

- Hector