[PATCH 13/14] net: axienet: Allow DMA to beyond 4GB

[PATCH 13/14] net: axienet: Allow DMA to beyond 4GB

[Andre Przywara]

With all DMA address accesses wrapped, we can actually support 64-bit
DMA if this option was chosen at IP integration time.
Since there is no way to autodetect the actual address bus width, we
make use of the existing "xlnx,addrwidth" DT property to let the driver
know about the width. The value in this property should match the
"Address Width" parameter used when synthesizing the IP.

This increases the DMA mask to let the kernel choose buffers from
memory at higher addresses.

Cc: Rob Herring <robh+dt@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: devicetree@vger.kernel.org
Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 .../net/ethernet/xilinx/xilinx_axienet_main.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
index f7f593df0c11..e036834549b3 100644
--- a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
+++ b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
@@ -1786,6 +1786,7 @@ static int axienet_probe(struct platform_device *pdev)
 	struct net_device *ndev;
 	const void *mac_addr;
 	struct resource *ethres;
+	int addr_width = 32;
 	u32 value;
 
 	ndev = alloc_etherdev(sizeof(*lp));
@@ -1915,6 +1916,8 @@ static int axienet_probe(struct platform_device *pdev)
 						     &dmares);
 		lp->rx_irq = irq_of_parse_and_map(np, 1);
 		lp->tx_irq = irq_of_parse_and_map(np, 0);
+		of_property_read_u32(np, "xlnx,addrwidth", &addr_width);
+
 		of_node_put(np);
 		lp->eth_irq = platform_get_irq(pdev, 0);
 	} else {
@@ -1944,6 +1947,9 @@ static int axienet_probe(struct platform_device *pdev)
 	 * We can detect this case by writing all 1's to one such register
 	 * and see if that sticks: when the IP is configured for 32 bits
 	 * only, those registers are RES0.
+	 * We can't autodetect the actual width this way, so we still use
+	 * a 32-bit DMA mask and rely on the xlnk,addrwidth DT property
+	 * to set this properly.
 	 * Those MSB registers were introduced in IP v7.1, which we check first.
 	 */
 	if ((axienet_ior(lp, XAE_ID_OFFSET) >> 24) >= 0x9) {
@@ -1961,6 +1967,19 @@ static int axienet_probe(struct platform_device *pdev)
 		}
 	}
 
+	if (!(lp->features & XAE_FEATURE_DMA_64BIT)) {
+		if (addr_width > 32)
+			dev_warn(&pdev->dev, "trimming DMA width from %d to 32 bits\n",
+				 addr_width);
+		addr_width = 32;
+	}
+
+	ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(addr_width));
+	if (ret) {
+		dev_err(&pdev->dev, "No suitable DMA available\n");
+		goto free_netdev;
+	}
+
 	/* Check for Ethernet core IRQ (optional) */
 	if (lp->eth_irq <= 0)
 		dev_info(&pdev->dev, "Ethernet core IRQ not defined\n");
-- 
2.17.1

[PATCH 14/14] net: axienet: Update devicetree binding documentation

[Andre Przywara]

This adds documentation about the newly introduced, optional
"xlnx,addrwidth" property to the binding documentation.

While at it, clarify the wording on some properties, replace obsolete
.txt file references with their new .yaml counterparts, and add a more
modern example, using the axistream-connected property.

Cc: Rob Herring <robh+dt@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: devicetree@vger.kernel.org
Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 .../bindings/net/xilinx_axienet.txt           | 57 ++++++++++++++++---
 1 file changed, 50 insertions(+), 7 deletions(-)

diff --git a/Documentation/devicetree/bindings/net/xilinx_axienet.txt b/Documentation/devicetree/bindings/net/xilinx_axienet.txt
index 7360617cdedb..78c278c5200f 100644
--- a/Documentation/devicetree/bindings/net/xilinx_axienet.txt
+++ b/Documentation/devicetree/bindings/net/xilinx_axienet.txt
@@ -12,7 +12,8 @@ sent and received through means of an AXI DMA controller. This driver
 includes the DMA driver code, so this driver is incompatible with AXI DMA
 driver.
 
-For more details about mdio please refer phy.txt file in the same directory.
+For more details about mdio please refer to the ethernet-phy.yaml file in
+the same directory.
 
 Required properties:
 - compatible	: Must be one of "xlnx,axi-ethernet-1.00.a",
@@ -27,14 +28,14 @@ Required properties:
 		  instead, and only the Ethernet core interrupt is optionally
 		  specified here.
 - phy-handle	: Should point to the external phy device.
-		  See ethernet.txt file in the same directory.
-- xlnx,rxmem	: Set to allocated memory buffer for Rx/Tx in the hardware
+		  See the ethernet-controller.yaml file in the same directory.
+- xlnx,rxmem	: Size of the RXMEM buffer in the hardware, in bytes.
 
 Optional properties:
-- phy-mode	: See ethernet.txt
+- phy-mode	: See ethernet-controller.yaml.
 - xlnx,phy-type	: Deprecated, do not use, but still accepted in preference
 		  to phy-mode.
-- xlnx,txcsum	: 0 or empty for disabling TX checksum offload,
+- xlnx,txcsum	: 0 for disabling TX checksum offload (default if omitted),
 		  1 to enable partial TX checksum offload,
 		  2 to enable full TX checksum offload
 - xlnx,rxcsum	: Same values as xlnx,txcsum but for RX checksum offload
@@ -48,10 +49,20 @@ Optional properties:
 		       If this is specified, the DMA-related resources from that
 		       device (DMA registers and DMA TX/RX interrupts) rather
 		       than this one will be used.
- - mdio		: Child node for MDIO bus. Must be defined if PHY access is
+- mdio		: Child node for MDIO bus. Must be defined if PHY access is
 		  required through the core's MDIO interface (i.e. always,
 		  unless the PHY is accessed through a different bus).
 
+Required properties for axistream-connected subnode:
+- reg		: Address and length of the AXI DMA controller MMIO space.
+- interrupts	: A list of 2 interrupts: TX DMA and RX DMA, in that order.
+
+Optional properties for axistream-connected subnode:
+- xlnx,addrwidth: Specifies the configured address width of the DMA. Newer
+		  versions of the IP allow setting this to a value between
+		  32 and 64. Defaults to 32 bits if not specified.
+
+
 Example:
 	axi_ethernet_eth: ethernet@40c00000 {
 		compatible = "xlnx,axi-ethernet-1.00.a";
@@ -60,7 +71,7 @@ Example:
 		interrupts = <2 0 1>;
 		clocks = <&axi_clk>;
 		phy-mode = "mii";
-		reg = <0x40c00000 0x40000 0x50c00000 0x40000>;
+		reg = <0x40c00000 0x40000>, <0x50c00000 0x40000>;
 		xlnx,rxcsum = <0x2>;
 		xlnx,rxmem = <0x800>;
 		xlnx,txcsum = <0x2>;
@@ -74,3 +85,35 @@ Example:
 			};
 		};
 	};
+    -----------------
+	axi_ethernet_eth: ethernet@7fe00000 {
+		compatible = "acme,fpga-ethernet", "xlnx,axi-ethernet-1.00.a";
+		reg = <0 0x7fe00000 0 0x40000>;
+		interrupts = <GIC_SPI 117 IRQ_TYPE_LEVEL_HIGH>;
+		clocks = <&soc_refclk100mhz>;
+
+		phy-mode = "sgmii";
+		phy-handle = <&phy0>;
+
+		xlnx,rxmem = <4096>;
+		axistream-connected = <&axi_dma_eth>;
+		#address-cells = <2>;
+		#size-cells = <2>;
+		ranges;
+
+		axi_dma_eth: axi_dma_ethernet@7fe40000 {
+			reg = <0 0x7fe40000 0 0x10000>;
+			interrupts = <GIC_SPI 118 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 120 IRQ_TYPE_LEVEL_HIGH>;
+			xlnx,addrwidth = <40>;
+		};
+
+		axi_ethernetlite_0_mdio: mdio {
+			#address-cells = <1>;
+			#size-cells = <0>;
+			phy0: phy@1 {
+				compatible = "ethernet-phy-ieee802.3-c22";
+				reg = <1>;
+			};
+		};
+	};
-- 
2.17.1

Re: [PATCH 14/14] net: axienet: Update devicetree binding documentation

[Rob Herring]

On Fri, Jan 10, 2020 at 11:54:15AM +0000, Andre Przywara wrote:
> This adds documentation about the newly introduced, optional
> "xlnx,addrwidth" property to the binding documentation.
> 
> While at it, clarify the wording on some properties, replace obsolete
> .txt file references with their new .yaml counterparts, and add a more
> modern example, using the axistream-connected property.
> 
> Cc: Rob Herring <robh+dt@kernel.org>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: devicetree@vger.kernel.org
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  .../bindings/net/xilinx_axienet.txt           | 57 ++++++++++++++++---
>  1 file changed, 50 insertions(+), 7 deletions(-)
> 
> diff --git a/Documentation/devicetree/bindings/net/xilinx_axienet.txt b/Documentation/devicetree/bindings/net/xilinx_axienet.txt
> index 7360617cdedb..78c278c5200f 100644
> --- a/Documentation/devicetree/bindings/net/xilinx_axienet.txt
> +++ b/Documentation/devicetree/bindings/net/xilinx_axienet.txt
> @@ -12,7 +12,8 @@ sent and received through means of an AXI DMA controller. This driver
>  includes the DMA driver code, so this driver is incompatible with AXI DMA
>  driver.
>  
> -For more details about mdio please refer phy.txt file in the same directory.
> +For more details about mdio please refer to the ethernet-phy.yaml file in
> +the same directory.
>  
>  Required properties:
>  - compatible	: Must be one of "xlnx,axi-ethernet-1.00.a",
> @@ -27,14 +28,14 @@ Required properties:
>  		  instead, and only the Ethernet core interrupt is optionally
>  		  specified here.
>  - phy-handle	: Should point to the external phy device.
> -		  See ethernet.txt file in the same directory.
> -- xlnx,rxmem	: Set to allocated memory buffer for Rx/Tx in the hardware
> +		  See the ethernet-controller.yaml file in the same directory.
> +- xlnx,rxmem	: Size of the RXMEM buffer in the hardware, in bytes.
>  
>  Optional properties:
> -- phy-mode	: See ethernet.txt
> +- phy-mode	: See ethernet-controller.yaml.
>  - xlnx,phy-type	: Deprecated, do not use, but still accepted in preference
>  		  to phy-mode.
> -- xlnx,txcsum	: 0 or empty for disabling TX checksum offload,
> +- xlnx,txcsum	: 0 for disabling TX checksum offload (default if omitted),
>  		  1 to enable partial TX checksum offload,
>  		  2 to enable full TX checksum offload
>  - xlnx,rxcsum	: Same values as xlnx,txcsum but for RX checksum offload
> @@ -48,10 +49,20 @@ Optional properties:
>  		       If this is specified, the DMA-related resources from that
>  		       device (DMA registers and DMA TX/RX interrupts) rather
>  		       than this one will be used.
> - - mdio		: Child node for MDIO bus. Must be defined if PHY access is
> +- mdio		: Child node for MDIO bus. Must be defined if PHY access is
>  		  required through the core's MDIO interface (i.e. always,
>  		  unless the PHY is accessed through a different bus).
>  
> +Required properties for axistream-connected subnode:
> +- reg		: Address and length of the AXI DMA controller MMIO space.
> +- interrupts	: A list of 2 interrupts: TX DMA and RX DMA, in that order.
> +
> +Optional properties for axistream-connected subnode:
> +- xlnx,addrwidth: Specifies the configured address width of the DMA. Newer
> +		  versions of the IP allow setting this to a value between
> +		  32 and 64. Defaults to 32 bits if not specified.

I think this should be expressed using dma-ranges. This is exactly the 
purpose of dma-ranges and we shouldn't need a device specific property 
for this sort of thing.

Rob

Re: [PATCH 14/14] net: axienet: Update devicetree binding documentation

[Andre Przywara]

On Tue, 21 Jan 2020 15:51:09 -0600
Rob Herring <robh@kernel.org> wrote:

Hi Rob,

thanks for having a look!

> On Fri, Jan 10, 2020 at 11:54:15AM +0000, Andre Przywara wrote:
> > This adds documentation about the newly introduced, optional
> > "xlnx,addrwidth" property to the binding documentation.
> > 
> > While at it, clarify the wording on some properties, replace obsolete
> > .txt file references with their new .yaml counterparts, and add a more
> > modern example, using the axistream-connected property.
> > 
> > Cc: Rob Herring <robh+dt@kernel.org>
> > Cc: Mark Rutland <mark.rutland@arm.com>
> > Cc: devicetree@vger.kernel.org
> > Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> > ---
> >  .../bindings/net/xilinx_axienet.txt           | 57 ++++++++++++++++---
> >  1 file changed, 50 insertions(+), 7 deletions(-)
> > 
> > diff --git a/Documentation/devicetree/bindings/net/xilinx_axienet.txt b/Documentation/devicetree/bindings/net/xilinx_axienet.txt
> > index 7360617cdedb..78c278c5200f 100644
> > --- a/Documentation/devicetree/bindings/net/xilinx_axienet.txt
> > +++ b/Documentation/devicetree/bindings/net/xilinx_axienet.txt
> > @@ -12,7 +12,8 @@ sent and received through means of an AXI DMA controller. This driver
> >  includes the DMA driver code, so this driver is incompatible with AXI DMA
> >  driver.
> >  
> > -For more details about mdio please refer phy.txt file in the same directory.
> > +For more details about mdio please refer to the ethernet-phy.yaml file in
> > +the same directory.
> >  
> >  Required properties:
> >  - compatible	: Must be one of "xlnx,axi-ethernet-1.00.a",
> > @@ -27,14 +28,14 @@ Required properties:
> >  		  instead, and only the Ethernet core interrupt is optionally
> >  		  specified here.
> >  - phy-handle	: Should point to the external phy device.
> > -		  See ethernet.txt file in the same directory.
> > -- xlnx,rxmem	: Set to allocated memory buffer for Rx/Tx in the hardware
> > +		  See the ethernet-controller.yaml file in the same directory.
> > +- xlnx,rxmem	: Size of the RXMEM buffer in the hardware, in bytes.
> >  
> >  Optional properties:
> > -- phy-mode	: See ethernet.txt
> > +- phy-mode	: See ethernet-controller.yaml.
> >  - xlnx,phy-type	: Deprecated, do not use, but still accepted in preference
> >  		  to phy-mode.
> > -- xlnx,txcsum	: 0 or empty for disabling TX checksum offload,
> > +- xlnx,txcsum	: 0 for disabling TX checksum offload (default if omitted),
> >  		  1 to enable partial TX checksum offload,
> >  		  2 to enable full TX checksum offload
> >  - xlnx,rxcsum	: Same values as xlnx,txcsum but for RX checksum offload
> > @@ -48,10 +49,20 @@ Optional properties:
> >  		       If this is specified, the DMA-related resources from that
> >  		       device (DMA registers and DMA TX/RX interrupts) rather
> >  		       than this one will be used.
> > - - mdio		: Child node for MDIO bus. Must be defined if PHY access is
> > +- mdio		: Child node for MDIO bus. Must be defined if PHY access is
> >  		  required through the core's MDIO interface (i.e. always,
> >  		  unless the PHY is accessed through a different bus).
> >  
> > +Required properties for axistream-connected subnode:
> > +- reg		: Address and length of the AXI DMA controller MMIO space.
> > +- interrupts	: A list of 2 interrupts: TX DMA and RX DMA, in that order.
> > +
> > +Optional properties for axistream-connected subnode:
> > +- xlnx,addrwidth: Specifies the configured address width of the DMA. Newer
> > +		  versions of the IP allow setting this to a value between
> > +		  32 and 64. Defaults to 32 bits if not specified.  
> 
> I think this should be expressed using dma-ranges. This is exactly the 
> purpose of dma-ranges and we shouldn't need a device specific property 
> for this sort of thing.

OK, after talking to Robin about it, I think I will indeed drop the whole usage of xlnx,addrwidth altogether.
Some thoughts:
- An integrator would choose the addrwidth value in the IP to be big enough for the whole bus. In our case it's actually 40 bits, because this is the max address size the interconnect supports. So any possible physical address the kernel could come up with would be valid for the DMA IP.
- Because of this we set the DMA mask to either 64-bit or 32-bit, depending on the auto detection of the MSB registers.
- If some integrator screws this up anyway, they can always set dma-ranges in the parent to limit the address range. IIUC, no further code would be needed in the Ethernet driver, as this would be handled by some (DMA?) framework?

Does that make sense?

Cheers,
Andre

RE: [PATCH 14/14] net: axienet: Update devicetree binding documentation

[Radhey Shyam Pandey]

> -----Original Message-----
> From: Andre Przywara <andre.przywara@arm.com>
> Sent: Friday, January 24, 2020 9:59 PM
> To: Rob Herring <robh@kernel.org>
> Cc: David S . Miller <davem@davemloft.net>; Radhey Shyam Pandey
> <radheys@xilinx.com>; Michal Simek <michals@xilinx.com>; Robert Hancock
> <hancock@sedsystems.ca>; netdev@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-kernel@vger.kernel.org; Mark Rutland
> <mark.rutland@arm.com>; devicetree@vger.kernel.org
> Subject: Re: [PATCH 14/14] net: axienet: Update devicetree binding
> documentation
> 
> On Tue, 21 Jan 2020 15:51:09 -0600
> Rob Herring <robh@kernel.org> wrote:
> 
> Hi Rob,
> 
> thanks for having a look!
> 
> > On Fri, Jan 10, 2020 at 11:54:15AM +0000, Andre Przywara wrote:
> > > This adds documentation about the newly introduced, optional
> > > "xlnx,addrwidth" property to the binding documentation.
> > >
> > > While at it, clarify the wording on some properties, replace obsolete
> > > .txt file references with their new .yaml counterparts, and add a more
> > > modern example, using the axistream-connected property.
> > >
> > > Cc: Rob Herring <robh+dt@kernel.org>
> > > Cc: Mark Rutland <mark.rutland@arm.com>
> > > Cc: devicetree@vger.kernel.org
> > > Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> > > ---
> > >  .../bindings/net/xilinx_axienet.txt           | 57 ++++++++++++++++---
> > >  1 file changed, 50 insertions(+), 7 deletions(-)
> > >
> > > diff --git a/Documentation/devicetree/bindings/net/xilinx_axienet.txt
> b/Documentation/devicetree/bindings/net/xilinx_axienet.txt
> > > index 7360617cdedb..78c278c5200f 100644
> > > --- a/Documentation/devicetree/bindings/net/xilinx_axienet.txt
> > > +++ b/Documentation/devicetree/bindings/net/xilinx_axienet.txt
> > > @@ -12,7 +12,8 @@ sent and received through means of an AXI DMA
> controller. This driver
> > >  includes the DMA driver code, so this driver is incompatible with AXI DMA
> > >  driver.
> > >
> > > -For more details about mdio please refer phy.txt file in the same directory.
> > > +For more details about mdio please refer to the ethernet-phy.yaml file in
> > > +the same directory.
> > >
> > >  Required properties:
> > >  - compatible	: Must be one of "xlnx,axi-ethernet-1.00.a",
> > > @@ -27,14 +28,14 @@ Required properties:
> > >  		  instead, and only the Ethernet core interrupt is optionally
> > >  		  specified here.
> > >  - phy-handle	: Should point to the external phy device.
> > > -		  See ethernet.txt file in the same directory.
> > > -- xlnx,rxmem	: Set to allocated memory buffer for Rx/Tx in the
> hardware
> > > +		  See the ethernet-controller.yaml file in the same directory.
> > > +- xlnx,rxmem	: Size of the RXMEM buffer in the hardware, in bytes.
> > >
> > >  Optional properties:
> > > -- phy-mode	: See ethernet.txt
> > > +- phy-mode	: See ethernet-controller.yaml.
> > >  - xlnx,phy-type	: Deprecated, do not use, but still accepted in
> preference
> > >  		  to phy-mode.
> > > -- xlnx,txcsum	: 0 or empty for disabling TX checksum offload,
> > > +- xlnx,txcsum	: 0 for disabling TX checksum offload (default if
> omitted),
> > >  		  1 to enable partial TX checksum offload,
> > >  		  2 to enable full TX checksum offload
> > >  - xlnx,rxcsum	: Same values as xlnx,txcsum but for RX checksum
> offload
> > > @@ -48,10 +49,20 @@ Optional properties:
> > >  		       If this is specified, the DMA-related resources from that
> > >  		       device (DMA registers and DMA TX/RX interrupts) rather
> > >  		       than this one will be used.
> > > - - mdio		: Child node for MDIO bus. Must be defined if PHY
> access is
> > > +- mdio		: Child node for MDIO bus. Must be defined if PHY
> access is
> > >  		  required through the core's MDIO interface (i.e. always,
> > >  		  unless the PHY is accessed through a different bus).
> > >
> > > +Required properties for axistream-connected subnode:
> > > +- reg		: Address and length of the AXI DMA controller MMIO
> space.
> > > +- interrupts	: A list of 2 interrupts: TX DMA and RX DMA, in that order.
> > > +
> > > +Optional properties for axistream-connected subnode:
> > > +- xlnx,addrwidth: Specifies the configured address width of the DMA. Newer
> > > +		  versions of the IP allow setting this to a value between
> > > +		  32 and 64. Defaults to 32 bits if not specified.
> >
> > I think this should be expressed using dma-ranges. This is exactly the
> > purpose of dma-ranges and we shouldn't need a device specific property
> > for this sort of thing.

dma-ranges define the relationship between the physical address spaces of the
parent and child nodes. In this case, ethernet and dma (parent-child) have
the same view of physical address space.   Do we mean to use the child-size
dma-range field and determine the address width?

> 
> OK, after talking to Robin about it, I think I will indeed drop the whole usage of
> xlnx,addrwidth altogether.
> Some thoughts:
> - An integrator would choose the addrwidth value in the IP to be big enough for
> the whole bus. In our case it's actually 40 bits, because this is the max address
> size the interconnect supports. So any possible physical address the kernel could
> come up with would be valid for the DMA IP.
> - Because of this we set the DMA mask to either 64-bit or 32-bit, depending on
> the auto detection of the MSB registers.
> - If some integrator screws this up anyway, they can always set dma-ranges in
> the parent to limit the address range. IIUC, no further code would be needed in
> the Ethernet driver, as this would be handled by some (DMA?) framework?

I think the current driver design will be simplified once we switch to the
dmaengine framework and use the xilinx dma(drivers/dma/xilinx_dma.c) driver.
The address width parsing is already handled by the dma driver. I am working
on an RFC to remove dma code from axiethernet and planning to post patchset.
Hopefully, that should address all concerns.

> 
> Does that make sense?
> 
> Cheers,
> Andre