[PATCH v3 0/3] MIPS: SiByte: Handle PCI DMA with 64-bit memory addressing

[PATCH v3 0/3] MIPS: SiByte: Handle PCI DMA with 64-bit memory addressing

[Maciej W. Rozycki]

Hi,

 This mini patch series enables correct support for DMA in the presence of 
memory outside the 32-bit address range with the Broadcom SiByte SOCs and 
the relevant development boards.

 There is a quirk in the BCM1250, BCM1125 and BCM1125H SOCs in that their 
onchip 32-bit PCI host bridge does not support DAC, however the HT link 
(where available) does support 40-bit addressing as per the HT spec.  
Therefore the first patch sets the bus mask accordingly, and then the 
second patch enables swiotlb.  See individual change descriptions for 
additional details; there's also a further discussion alongside.

 This has been verified with a Broadcom SWARM board equipped with 3200MiB 
of RAM (2176MiB of which the address decoder in the SOC maps above 4GiB), 
a pair of DEFPA FDDI adapters and an XHCI USB adapter.  There were also 
some other PCI and PCIe devices present in the system, though not actively 
used beyond being probed at boot, and none has shown any symptoms of 
breakage.

 I have come across commit 9d7a224b463e ("dma-direct: always allow dma 
mask <= physiscal memory size") and realised we do need ZONE_DMA32 for 
LittleSur.  Hence this v3, adding a third (second in the series) change 
for LittleSur.

 Also hopefully I'll have sorted out issues with threading in my MUA with 
this series update.

 Please apply.

  Maciej

[PATCH v3 1/3] MIPS: SiByte: Set 32-bit bus mask for BCM1250 PCI

[Maciej W. Rozycki]

The Broadcom SiByte BCM1250, BCM1125H and BCM1125 SOCs have an onchip 
32-bit PCI host bridge, and the two former SOCs also have an onchip HT 
host bridge.  The HT host bridge, where present, appears in the PCI 
configuration space as if it was a device on the 32-bit PCI bus behind 
the PCI host bridge, however at the hardware level its signals are 
routed separately, so these two devices are actually peer host bridges.

As documented[1] and observed in reality the 32-bit PCI host bridge does 
not support 64-bit addressing as it does not support the Dual Address 
Cycle (DAC) PCI command, and naturally, being 32-bit only, it has no 
means to carry the high 32 address bits otherwise.  However the DRAM 
controller also included in the SOC supports memory amounts of up to 
16GiB, and due to how the address decoder has been wired in the SOC any 
memory beyond 1GiB is actually mapped starting from 4GiB physical up, 
that is beyond the 32-bit addressable limit.  Consequently if the 
maximum amount of memory has been installed, then it will span up to 
19GiB.

Contrariwise, the HT host bridge does support full 40-bit addressing 
defined by the HyperTransport (formerly LDT) specification the bridge 
adheres to, depending on the peripherals revision of the SOC[2] either 
revision 0.17[3] or revision 1.03[4].  This allows addressing any and 
all memory installed, and well beyond.

Set the bus mask then to limit DMA addressing to 32 bits for all the 
devices down the 32-bit PCI host bridge, excluding however any devices 
that are down the HT host bridge.

References:

[1] "BCM1250/BCM1125/BCM1125H User Manual", Revision 1250_1125-UM100-R, 
    Broadcom Corporation, 21 Oct 2002, Section 8: "PCI Bus and 
    HyperTransport Fabric", "Introduction", p. 190

[2] same, Table 140: "HyperTransport Configuration Header (Type 1)", p. 
    245

[3] "Lightning Data Transport IO Specification", Revision 0.17, Advanced 
    Micro Devices, 21 Jan 2000, Section 3.2.1.2 "Command Packet", p. 8

[4] "HyperTransport I/O Link Specification", Revision 1.03, 
    HyperTransport Technology Consortium, 10 Oct 2001, Section 3.2.1.2 
    "Request Packet", pp. 27-28

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
---
No changes from v2.

Changes from v1:

- conditions restructured in `sb1250_bus_dma_mask' for clarity, no 
  functional change.
---
 arch/mips/pci/fixup-sb1250.c |   53 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)

linux-mips-sibyte-sb1250-bus-dma-mask.diff
Index: linux-20181104-swarm64-eb/arch/mips/pci/fixup-sb1250.c
===================================================================
--- linux-20181104-swarm64-eb.orig/arch/mips/pci/fixup-sb1250.c
+++ linux-20181104-swarm64-eb/arch/mips/pci/fixup-sb1250.c
@@ -1,6 +1,7 @@
 /*
  *	Copyright (C) 2004, 2006  MIPS Technologies, Inc.  All rights reserved.
  *	    Author:	Maciej W. Rozycki <macro@mips.com>
+ *	Copyright (C) 2018  Maciej W. Rozycki
  *
  *	This program is free software; you can redistribute it and/or
  *	modify it under the terms of the GNU General Public License
@@ -8,6 +9,7 @@
  *	2 of the License, or (at your option) any later version.
  */
 
+#include <linux/dma-mapping.h>
 #include <linux/pci.h>
 
 /*
@@ -22,6 +24,57 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_SI
 			quirk_sb1250_pci);
 
 /*
+ * The BCM1250, etc. PCI host bridge does not support DAC on its 32-bit
+ * bus, so we set the bus's DMA mask accordingly.  However the HT link
+ * down the artificial PCI-HT bridge supports 40-bit addressing and the
+ * SP1011 HT-PCI bridge downstream supports both DAC and a 64-bit bus
+ * width, so we record the PCI-HT bridge's secondary and subordinate bus
+ * numbers and do not set the mask for devices present in the inclusive
+ * range of those.
+ */
+struct sb1250_bus_dma_mask_exclude {
+	bool set;
+	unsigned char start;
+	unsigned char end;
+};
+
+static int sb1250_bus_dma_mask(struct pci_dev *dev, void *data)
+{
+	struct sb1250_bus_dma_mask_exclude *exclude = data;
+	bool exclude_this;
+	bool ht_bridge;
+
+	exclude_this = exclude->set && (dev->bus->number >= exclude->start &&
+					dev->bus->number <= exclude->end);
+	ht_bridge = !exclude->set && (dev->vendor == PCI_VENDOR_ID_SIBYTE &&
+				      dev->device == PCI_DEVICE_ID_BCM1250_HT);
+
+	if (exclude_this) {
+		dev_dbg(&dev->dev, "not disabling DAC for device");
+	} else if (ht_bridge) {
+		exclude->start = dev->subordinate->number;
+		exclude->end = pci_bus_max_busnr(dev->subordinate);
+		exclude->set = true;
+		dev_dbg(&dev->dev, "not disabling DAC for [bus %02x-%02x]",
+			exclude->start, exclude->end);
+	} else {
+		dev_dbg(&dev->dev, "disabling DAC for device");
+		dev->dev.bus_dma_mask = DMA_BIT_MASK(32);
+	}
+
+	return 0;
+}
+
+static void quirk_sb1250_pci_dac(struct pci_dev *dev)
+{
+	struct sb1250_bus_dma_mask_exclude exclude = { .set = false };
+
+	pci_walk_bus(dev->bus, sb1250_bus_dma_mask, &exclude);
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_SIBYTE, PCI_DEVICE_ID_BCM1250_PCI,
+			quirk_sb1250_pci_dac);
+
+/*
  * The BCM1250, etc. PCI/HT bridge reports as a host bridge.
  */
 static void quirk_sb1250_ht(struct pci_dev *dev)

Re: [PATCH v3 1/3] MIPS: SiByte: Set 32-bit bus mask for BCM1250 PCI

[Christoph Hellwig]

Looks fine,

Reviewed-by: Christoph Hellwig <hch@lst.de>

[PATCH v3 2/3] MIPS: SiByte: Enable ZONE_DMA32 for LittleSur

[Maciej W. Rozycki]

The LittleSur board is marked for high memory support and therefore 
clearly must provide a way to have enough memory installed for some to 
be present outside the low 4GiB physical address range.  With the memory 
map of the BCM1250 SOC it has been built around it means over 1GiB of 
actual DRAM, as only the first 1GiB is mapped in the low 4GiB physical 
address range[1].

Complement commit cce335ae47e2 ("[MIPS] 64-bit Sibyte kernels need 
DMA32.") then and also enable ZONE_DMA32 for LittleSur.

References:

[1] "BCM1250/BCM1125/BCM1125H User Manual", Revision 1250_1125-UM100-R,
    Broadcom Corporation, 21 Oct 2002, Section 3: "System Overview",
    "Memory Map", pp. 34-38

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Fixes: cce335ae47e2 ("[MIPS] 64-bit Sibyte kernels need DMA32.")
---
New in v3.
---
 arch/mips/Kconfig |    1 +
 1 file changed, 1 insertion(+)

linux-mips-sibyte-littlesur-zone-dma32.diff
Index: linux-20181104-littlesur64-defconfig/arch/mips/Kconfig
===================================================================
--- linux-20181104-littlesur64-defconfig.orig/arch/mips/Kconfig
+++ linux-20181104-littlesur64-defconfig/arch/mips/Kconfig
@@ -805,6 +805,7 @@ config SIBYTE_LITTLESUR
 	select SYS_SUPPORTS_BIG_ENDIAN
 	select SYS_SUPPORTS_HIGHMEM
 	select SYS_SUPPORTS_LITTLE_ENDIAN
+	select ZONE_DMA32 if 64BIT
 
 config SIBYTE_SENTOSA
 	bool "Sibyte BCM91250E-Sentosa"

Re: [PATCH v3 2/3] MIPS: SiByte: Enable ZONE_DMA32 for LittleSur

[Christoph Hellwig]

On Tue, Nov 13, 2018 at 10:42:37PM +0000, Maciej W. Rozycki wrote:
> The LittleSur board is marked for high memory support and therefore 
> clearly must provide a way to have enough memory installed for some to 
> be present outside the low 4GiB physical address range.  With the memory 
> map of the BCM1250 SOC it has been built around it means over 1GiB of 
> actual DRAM, as only the first 1GiB is mapped in the low 4GiB physical 
> address range[1].
> 
> Complement commit cce335ae47e2 ("[MIPS] 64-bit Sibyte kernels need 
> DMA32.") then and also enable ZONE_DMA32 for LittleSur.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

[PATCH v3 3/3] MIPS: SiByte: Enable swiotlb for SWARM, LittleSur and BigSur

[Maciej W. Rozycki]

The Broadcom SiByte BCM1250, BCM1125, and BCM1125H SOCs have an onchip 
DRAM controller that supports memory amounts of up to 16GiB, and due to 
how the address decoder has been wired in the SOC any memory beyond 1GiB 
is actually mapped starting from 4GiB physical up, that is beyond the 
32-bit addressable limit[1].  Consequently if the maximum amount of 
memory has been installed, then it will span up to 19GiB.

Many of the evaluation boards we support that are based on one of these 
SOCs have their memory soldered and the amount present fits in the 
32-bit address range.  The BCM91250A SWARM board however has actual DIMM 
slots and accepts, depending on the peripherals revision of the SOC, up 
to 4GiB or 8GiB of memory in commercially available JEDEC modules[2].  
I believe this is also the case with the BCM91250C2 LittleSur board. 
This means that up to either 3GiB or 7GiB of memory requires 64-bit 
addressing to access.

I believe the BCM91480B BigSur board, which has the BCM1480 SOC instead, 
accepts at least as much memory, although I have no documentation or 
actual hardware available to verify that.

Both systems have PCI slots installed for use by any PCI option boards, 
including ones that only support 32-bit addressing (additionally the 
32-bit PCI host bridge of the BCM1250, BCM1125, and BCM1125H SOCs limits 
addressing to 32-bits), and there is no IOMMU available.  Therefore for 
PCI DMA to work in the presence of memory beyond enable swiotlb for the
affected systems.

All the other SOC onchip DMA devices use 40-bit addressing and therefore 
can address the whole memory, so only enable swiotlb if PCI support and 
support for DMA beyond 4GiB have been both enabled in the configuration 
of the kernel.

This shows up as follows:

Broadcom SiByte BCM1250 B2 @ 800 MHz (SB1 rev 2)
Board type: SiByte BCM91250A (SWARM)
Determined physical RAM map:
 memory: 000000000fe7fe00 @ 0000000000000000 (usable)
 memory: 000000001ffffe00 @ 0000000080000000 (usable)
 memory: 000000000ffffe00 @ 00000000c0000000 (usable)
 memory: 0000000087fffe00 @ 0000000100000000 (usable)
software IO TLB: mapped [mem 0xcbffc000-0xcfffc000] (64MB)

in the bootstrap log and removes failures like these:

defxx 0000:02:00.0: dma_direct_map_page: overflow 0x0000000185bc6080+4608 of device mask ffffffff bus mask 0
fddi0: Receive buffer allocation failed
fddi0: Adapter open failed!
IP-Config: Failed to open fddi0
defxx 0000:09:08.0: dma_direct_map_page: overflow 0x0000000185bc6080+4608 of device mask ffffffff bus mask 0
fddi1: Receive buffer allocation failed
fddi1: Adapter open failed!
IP-Config: Failed to open fddi1

when memory beyond 4GiB is handed out to devices that can only do 32-bit 
addressing.

This updates commit cce335ae47e2 ("[MIPS] 64-bit Sibyte kernels need 
DMA32.").

References:

[1] "BCM1250/BCM1125/BCM1125H User Manual", Revision 1250_1125-UM100-R, 
    Broadcom Corporation, 21 Oct 2002, Section 3: "System Overview", 
    "Memory Map", pp. 34-38

[2] "BCM91250A User Manual", Revision 91250A-UM100-R, Broadcom 
    Corporation, 18 May 2004, Section 3: "Physical Description", 
    "Supported DRAM", p. 23

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
References: cce335ae47e2 ("[MIPS] 64-bit Sibyte kernels need DMA32.")
---
Changes from v2:

- missing reference to [1] added.

Changes from v1:

- title updated to include LittleSur.
---
 arch/mips/Kconfig                |    3 +++
 arch/mips/sibyte/common/Makefile |    1 +
 arch/mips/sibyte/common/dma.c    |   19 +++++++++++++++++++
 3 files changed, 23 insertions(+)

linux-mips-sibyte-swiotlb.diff
Index: linux-20181104-swarm64-eb/arch/mips/Kconfig
===================================================================
--- linux-20181104-swarm64-eb.orig/arch/mips/Kconfig
+++ linux-20181104-swarm64-eb/arch/mips/Kconfig
@@ -794,6 +794,7 @@ config SIBYTE_SWARM
 	select SYS_SUPPORTS_HIGHMEM
 	select SYS_SUPPORTS_LITTLE_ENDIAN
 	select ZONE_DMA32 if 64BIT
+	select SWIOTLB if ARCH_DMA_ADDR_T_64BIT && PCI
 
 config SIBYTE_LITTLESUR
 	bool "Sibyte BCM91250C2-LittleSur"
@@ -805,6 +806,7 @@ config SIBYTE_LITTLESUR
 	select SYS_SUPPORTS_BIG_ENDIAN
 	select SYS_SUPPORTS_HIGHMEM
 	select SYS_SUPPORTS_LITTLE_ENDIAN
+	select SWIOTLB if ARCH_DMA_ADDR_T_64BIT && PCI
 
 config SIBYTE_SENTOSA
 	bool "Sibyte BCM91250E-Sentosa"
@@ -826,6 +828,7 @@ config SIBYTE_BIGSUR
 	select SYS_SUPPORTS_HIGHMEM
 	select SYS_SUPPORTS_LITTLE_ENDIAN
 	select ZONE_DMA32 if 64BIT
+	select SWIOTLB if ARCH_DMA_ADDR_T_64BIT && PCI
 
 config SNI_RM
 	bool "SNI RM200/300/400"
Index: linux-20181104-swarm64-eb/arch/mips/sibyte/common/Makefile
===================================================================
--- linux-20181104-swarm64-eb.orig/arch/mips/sibyte/common/Makefile
+++ linux-20181104-swarm64-eb/arch/mips/sibyte/common/Makefile
@@ -1,4 +1,5 @@
 obj-y := cfe.o
+obj-$(CONFIG_SWIOTLB)			+= dma.o
 obj-$(CONFIG_SIBYTE_BUS_WATCHER)	+= bus_watcher.o
 obj-$(CONFIG_SIBYTE_CFE_CONSOLE)	+= cfe_console.o
 obj-$(CONFIG_SIBYTE_TBPROF)		+= sb_tbprof.o
Index: linux-20181104-swarm64-eb/arch/mips/sibyte/common/dma.c
===================================================================
--- /dev/null
+++ linux-20181104-swarm64-eb/arch/mips/sibyte/common/dma.c
@@ -0,0 +1,19 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ *	DMA support for Broadcom SiByte platforms.
+ *
+ *	Copyright (c) 2018  Maciej W. Rozycki
+ *
+ *	This program is free software; you can redistribute it and/or
+ *	modify it under the terms of the GNU General Public License
+ *	as published by the Free Software Foundation; either version
+ *	2 of the License, or (at your option) any later version.
+ */
+
+#include <linux/swiotlb.h>
+#include <asm/bootinfo.h>
+
+void __init plat_swiotlb_setup(void)
+{
+	swiotlb_init(1);
+}

Re: [PATCH v3 3/3] MIPS: SiByte: Enable swiotlb for SWARM, LittleSur and BigSur

[Christoph Hellwig]

On Tue, Nov 13, 2018 at 10:42:44PM +0000, Maciej W. Rozycki wrote:
> The Broadcom SiByte BCM1250, BCM1125, and BCM1125H SOCs have an onchip 
> DRAM controller that supports memory amounts of up to 16GiB, and due to 
> how the address decoder has been wired in the SOC any memory beyond 1GiB 
> is actually mapped starting from 4GiB physical up, that is beyond the 
> 32-bit addressable limit[1].  Consequently if the maximum amount of 
> memory has been installed, then it will span up to 19GiB.
> 
> Many of the evaluation boards we support that are based on one of these 
> SOCs have their memory soldered and the amount present fits in the 
> 32-bit address range.  The BCM91250A SWARM board however has actual DIMM 
> slots and accepts, depending on the peripherals revision of the SOC, up 
> to 4GiB or 8GiB of memory in commercially available JEDEC modules[2].  
> I believe this is also the case with the BCM91250C2 LittleSur board. 
> This means that up to either 3GiB or 7GiB of memory requires 64-bit 
> addressing to access.
> 
> I believe the BCM91480B BigSur board, which has the BCM1480 SOC instead, 
> accepts at least as much memory, although I have no documentation or 
> actual hardware available to verify that.
> 
> Both systems have PCI slots installed for use by any PCI option boards, 
> including ones that only support 32-bit addressing (additionally the 
> 32-bit PCI host bridge of the BCM1250, BCM1125, and BCM1125H SOCs limits 
> addressing to 32-bits), and there is no IOMMU available.  Therefore for 
> PCI DMA to work in the presence of memory beyond enable swiotlb for the
> affected systems.
> 
> All the other SOC onchip DMA devices use 40-bit addressing and therefore 
> can address the whole memory, so only enable swiotlb if PCI support and 
> support for DMA beyond 4GiB have been both enabled in the configuration 
> of the kernel.
> 
> This shows up as follows:
> 
> Broadcom SiByte BCM1250 B2 @ 800 MHz (SB1 rev 2)
> Board type: SiByte BCM91250A (SWARM)
> Determined physical RAM map:
>  memory: 000000000fe7fe00 @ 0000000000000000 (usable)
>  memory: 000000001ffffe00 @ 0000000080000000 (usable)
>  memory: 000000000ffffe00 @ 00000000c0000000 (usable)
>  memory: 0000000087fffe00 @ 0000000100000000 (usable)
> software IO TLB: mapped [mem 0xcbffc000-0xcfffc000] (64MB)
> 
> in the bootstrap log and removes failures like these:
> 
> defxx 0000:02:00.0: dma_direct_map_page: overflow 0x0000000185bc6080+4608 of device mask ffffffff bus mask 0
> fddi0: Receive buffer allocation failed
> fddi0: Adapter open failed!
> IP-Config: Failed to open fddi0
> defxx 0000:09:08.0: dma_direct_map_page: overflow 0x0000000185bc6080+4608 of device mask ffffffff bus mask 0
> fddi1: Receive buffer allocation failed
> fddi1: Adapter open failed!
> IP-Config: Failed to open fddi1
> 
> when memory beyond 4GiB is handed out to devices that can only do 32-bit 
> addressing.
> 
> This updates commit cce335ae47e2 ("[MIPS] 64-bit Sibyte kernels need 
> DMA32.").
> 
> References:
> 
> [1] "BCM1250/BCM1125/BCM1125H User Manual", Revision 1250_1125-UM100-R, 
>     Broadcom Corporation, 21 Oct 2002, Section 3: "System Overview", 
>     "Memory Map", pp. 34-38
> 
> [2] "BCM91250A User Manual", Revision 91250A-UM100-R, Broadcom 
>     Corporation, 18 May 2004, Section 3: "Physical Description", 
>     "Supported DRAM", p. 23
> 
> Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
> References: cce335ae47e2 ("[MIPS] 64-bit Sibyte kernels need DMA32.")
> ---
> Changes from v2:
> 
> - missing reference to [1] added.
> 
> Changes from v1:
> 
> - title updated to include LittleSur.
> ---
>  arch/mips/Kconfig                |    3 +++
>  arch/mips/sibyte/common/Makefile |    1 +
>  arch/mips/sibyte/common/dma.c    |   19 +++++++++++++++++++
>  3 files changed, 23 insertions(+)
> 
> linux-mips-sibyte-swiotlb.diff
> Index: linux-20181104-swarm64-eb/arch/mips/Kconfig
> ===================================================================
> --- linux-20181104-swarm64-eb.orig/arch/mips/Kconfig
> +++ linux-20181104-swarm64-eb/arch/mips/Kconfig
> @@ -794,6 +794,7 @@ config SIBYTE_SWARM
>  	select SYS_SUPPORTS_HIGHMEM
>  	select SYS_SUPPORTS_LITTLE_ENDIAN
>  	select ZONE_DMA32 if 64BIT
> +	select SWIOTLB if ARCH_DMA_ADDR_T_64BIT && PCI
>  
>  config SIBYTE_LITTLESUR
>  	bool "Sibyte BCM91250C2-LittleSur"
> @@ -805,6 +806,7 @@ config SIBYTE_LITTLESUR
>  	select SYS_SUPPORTS_BIG_ENDIAN
>  	select SYS_SUPPORTS_HIGHMEM
>  	select SYS_SUPPORTS_LITTLE_ENDIAN
> +	select SWIOTLB if ARCH_DMA_ADDR_T_64BIT && PCI
>  
>  config SIBYTE_SENTOSA
>  	bool "Sibyte BCM91250E-Sentosa"
> @@ -826,6 +828,7 @@ config SIBYTE_BIGSUR
>  	select SYS_SUPPORTS_HIGHMEM
>  	select SYS_SUPPORTS_LITTLE_ENDIAN
>  	select ZONE_DMA32 if 64BIT
> +	select SWIOTLB if ARCH_DMA_ADDR_T_64BIT && PCI
>  
>  config SNI_RM
>  	bool "SNI RM200/300/400"
> Index: linux-20181104-swarm64-eb/arch/mips/sibyte/common/Makefile
> ===================================================================
> --- linux-20181104-swarm64-eb.orig/arch/mips/sibyte/common/Makefile
> +++ linux-20181104-swarm64-eb/arch/mips/sibyte/common/Makefile
> @@ -1,4 +1,5 @@
>  obj-y := cfe.o
> +obj-$(CONFIG_SWIOTLB)			+= dma.o
>  obj-$(CONFIG_SIBYTE_BUS_WATCHER)	+= bus_watcher.o
>  obj-$(CONFIG_SIBYTE_CFE_CONSOLE)	+= cfe_console.o
>  obj-$(CONFIG_SIBYTE_TBPROF)		+= sb_tbprof.o
> Index: linux-20181104-swarm64-eb/arch/mips/sibyte/common/dma.c
> ===================================================================
> --- /dev/null
> +++ linux-20181104-swarm64-eb/arch/mips/sibyte/common/dma.c
> @@ -0,0 +1,19 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +/*
> + *	DMA support for Broadcom SiByte platforms.
> + *
> + *	Copyright (c) 2018  Maciej W. Rozycki
> + *
> + *	This program is free software; you can redistribute it and/or
> + *	modify it under the terms of the GNU General Public License
> + *	as published by the Free Software Foundation; either version
> + *	2 of the License, or (at your option) any later version.
> + */

Ok, so you have the SPDX tag, but also the duplicate license text,
which is what confused me earlier.

This should just be:

// SPDX-License-Identifier: GPL-2.0+
/*
 *	DMA support for Broadcom SiByte platforms.
 *
 *	Copyright (c) 2018  Maciej W. Rozycki
 */

With that looks good to me:

Reviewed-by: Christoph Hellwig <hch@lst.de>

[PATCH v4 3/3] MIPS: SiByte: Enable swiotlb for SWARM, LittleSur and BigSur

[Maciej W. Rozycki]

The Broadcom SiByte BCM1250, BCM1125, and BCM1125H SOCs have an onchip 
DRAM controller that supports memory amounts of up to 16GiB, and due to 
how the address decoder has been wired in the SOC any memory beyond 1GiB 
is actually mapped starting from 4GiB physical up, that is beyond the 
32-bit addressable limit[1].  Consequently if the maximum amount of 
memory has been installed, then it will span up to 19GiB.

Many of the evaluation boards we support that are based on one of these 
SOCs have their memory soldered and the amount present fits in the 
32-bit address range.  The BCM91250A SWARM board however has actual DIMM 
slots and accepts, depending on the peripherals revision of the SOC, up 
to 4GiB or 8GiB of memory in commercially available JEDEC modules[2].  
I believe this is also the case with the BCM91250C2 LittleSur board. 
This means that up to either 3GiB or 7GiB of memory requires 64-bit 
addressing to access.

I believe the BCM91480B BigSur board, which has the BCM1480 SOC instead, 
accepts at least as much memory, although I have no documentation or 
actual hardware available to verify that.

Both systems have PCI slots installed for use by any PCI option boards, 
including ones that only support 32-bit addressing (additionally the 
32-bit PCI host bridge of the BCM1250, BCM1125, and BCM1125H SOCs limits 
addressing to 32-bits), and there is no IOMMU available.  Therefore for 
PCI DMA to work in the presence of memory beyond enable swiotlb for the
affected systems.

All the other SOC onchip DMA devices use 40-bit addressing and therefore 
can address the whole memory, so only enable swiotlb if PCI support and 
support for DMA beyond 4GiB have been both enabled in the configuration 
of the kernel.

This shows up as follows:

Broadcom SiByte BCM1250 B2 @ 800 MHz (SB1 rev 2)
Board type: SiByte BCM91250A (SWARM)
Determined physical RAM map:
 memory: 000000000fe7fe00 @ 0000000000000000 (usable)
 memory: 000000001ffffe00 @ 0000000080000000 (usable)
 memory: 000000000ffffe00 @ 00000000c0000000 (usable)
 memory: 0000000087fffe00 @ 0000000100000000 (usable)
software IO TLB: mapped [mem 0xcbffc000-0xcfffc000] (64MB)

in the bootstrap log and removes failures like these:

defxx 0000:02:00.0: dma_direct_map_page: overflow 0x0000000185bc6080+4608 of device mask ffffffff bus mask 0
fddi0: Receive buffer allocation failed
fddi0: Adapter open failed!
IP-Config: Failed to open fddi0
defxx 0000:09:08.0: dma_direct_map_page: overflow 0x0000000185bc6080+4608 of device mask ffffffff bus mask 0
fddi1: Receive buffer allocation failed
fddi1: Adapter open failed!
IP-Config: Failed to open fddi1

when memory beyond 4GiB is handed out to devices that can only do 32-bit 
addressing.

This updates commit cce335ae47e2 ("[MIPS] 64-bit Sibyte kernels need 
DMA32.").

References:

[1] "BCM1250/BCM1125/BCM1125H User Manual", Revision 1250_1125-UM100-R, 
    Broadcom Corporation, 21 Oct 2002, Section 3: "System Overview", 
    "Memory Map", pp. 34-38

[2] "BCM91250A User Manual", Revision 91250A-UM100-R, Broadcom 
    Corporation, 18 May 2004, Section 3: "Physical Description", 
    "Supported DRAM", p. 23

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
References: cce335ae47e2 ("[MIPS] 64-bit Sibyte kernels need DMA32.")
---
Hi,

 Sending v4 of 3/3 only with the update requested, due to the minor scope 
of the change.  Hopefully patchwork gets it right.

  Maciej

Changes from v3:

- removed inline licence text.

Changes from v2:

- missing reference to [1] added.

Changes from v1:

- title updated to include LittleSur.
---
 arch/mips/Kconfig                |    3 +++
 arch/mips/sibyte/common/Makefile |    1 +
 arch/mips/sibyte/common/dma.c    |   14 ++++++++++++++
 3 files changed, 18 insertions(+)

linux-mips-sibyte-swiotlb.diff
Index: linux-20181104-swarm64-eb/arch/mips/Kconfig
===================================================================
--- linux-20181104-swarm64-eb.orig/arch/mips/Kconfig
+++ linux-20181104-swarm64-eb/arch/mips/Kconfig
@@ -794,6 +794,7 @@ config SIBYTE_SWARM
 	select SYS_SUPPORTS_HIGHMEM
 	select SYS_SUPPORTS_LITTLE_ENDIAN
 	select ZONE_DMA32 if 64BIT
+	select SWIOTLB if ARCH_DMA_ADDR_T_64BIT && PCI
 
 config SIBYTE_LITTLESUR
 	bool "Sibyte BCM91250C2-LittleSur"
@@ -806,6 +807,7 @@ config SIBYTE_LITTLESUR
 	select SYS_SUPPORTS_HIGHMEM
 	select SYS_SUPPORTS_LITTLE_ENDIAN
 	select ZONE_DMA32 if 64BIT
+	select SWIOTLB if ARCH_DMA_ADDR_T_64BIT && PCI
 
 config SIBYTE_SENTOSA
 	bool "Sibyte BCM91250E-Sentosa"
@@ -827,6 +829,7 @@ config SIBYTE_BIGSUR
 	select SYS_SUPPORTS_HIGHMEM
 	select SYS_SUPPORTS_LITTLE_ENDIAN
 	select ZONE_DMA32 if 64BIT
+	select SWIOTLB if ARCH_DMA_ADDR_T_64BIT && PCI
 
 config SNI_RM
 	bool "SNI RM200/300/400"
Index: linux-20181104-swarm64-eb/arch/mips/sibyte/common/Makefile
===================================================================
--- linux-20181104-swarm64-eb.orig/arch/mips/sibyte/common/Makefile
+++ linux-20181104-swarm64-eb/arch/mips/sibyte/common/Makefile
@@ -1,4 +1,5 @@
 obj-y := cfe.o
+obj-$(CONFIG_SWIOTLB)			+= dma.o
 obj-$(CONFIG_SIBYTE_BUS_WATCHER)	+= bus_watcher.o
 obj-$(CONFIG_SIBYTE_CFE_CONSOLE)	+= cfe_console.o
 obj-$(CONFIG_SIBYTE_TBPROF)		+= sb_tbprof.o
Index: linux-20181104-swarm64-eb/arch/mips/sibyte/common/dma.c
===================================================================
--- /dev/null
+++ linux-20181104-swarm64-eb/arch/mips/sibyte/common/dma.c
@@ -0,0 +1,14 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ *	DMA support for Broadcom SiByte platforms.
+ *
+ *	Copyright (c) 2018  Maciej W. Rozycki
+ */
+
+#include <linux/swiotlb.h>
+#include <asm/bootinfo.h>
+
+void __init plat_swiotlb_setup(void)
+{
+	swiotlb_init(1);
+}

Re: [PATCH v3 0/3] MIPS: SiByte: Handle PCI DMA with 64-bit memory addressing

[Paul Burton]

Hello,

Maciej W. Rozycki wrote:
> Hi,
> 
> This mini patch series enables correct support for DMA in the presence of
> memory outside the 32-bit address range with the Broadcom SiByte SOCs and
> the relevant development boards.
> 
> There is a quirk in the BCM1250, BCM1125 and BCM1125H SOCs in that their
> onchip 32-bit PCI host bridge does not support DAC, however the HT link
> (where available) does support 40-bit addressing as per the HT spec.
> Therefore the first patch sets the bus mask accordingly, and then the
> second patch enables swiotlb.  See individual change descriptions for
> additional details; there's also a further discussion alongside.
> 
> This has been verified with a Broadcom SWARM board equipped with 3200MiB
> of RAM (2176MiB of which the address decoder in the SOC maps above 4GiB),
> a pair of DEFPA FDDI adapters and an XHCI USB adapter.  There were also
> some other PCI and PCIe devices present in the system, though not actively
> used beyond being probed at boot, and none has shown any symptoms of
> breakage.
> 
> I have come across commit 9d7a224b463e ("dma-direct: always allow dma
> mask <= physiscal memory size") and realised we do need ZONE_DMA32 for
> LittleSur.  Hence this v3, adding a third (second in the series) change
> for LittleSur.
> 
> Also hopefully I'll have sorted out issues with threading in my MUA with
> this series update.
> 
> Please apply.
> 
> Maciej

Series applied to mips-next.

Thanks,
    Paul

[ This message was auto-generated; if you believe anything is incorrect
  then please email paul.burton@mips.com to report it. ]