[PATCH RFC net-next 00/19] net: mvpp2: Add TX Flow Control support

[PATCH RFC net-next 00/19] net: mvpp2: Add TX Flow Control support

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

Armada hardware has a pause generation mechanism in GOP (MAC).
GOP has to generate flow control frames based on an indication
programmed in Ports Control 0 Register. There is a bit per port.
However assertion of the PortX Pause bits in the ports control 0 register
only sends a one time pause. To complement the function the GOP has
a mechanism to periodically send pause control messages based on periodic counters.
This mechanism ensures that the pause is effective as long as the Appropriate PortX Pause
is asserted.

Problem is that Packet Processor witch actually can drop packets due to lack of resources
not connected to the GOP flow control generation mechanism.
To solve this issue Armada has firmware running on CM3 CPU dedectated for Flow Control
support. Firmware monitors Packet Processor resources and asserts XON/XOFF by writing
to Ports Control 0 Register.

MSS shared memory used to communicate between CM3 firmware and MVPP2 driver.
During init MVPP2 driver informs firmware about used BM pools, RXQs and congestion and
depletion thresholds.

The pause is generated whenever congestion or depletion in resources is detected.
The back pressure is stopped when the resource reaches a sufficient level.
So the congestion/depletion and sufficient implement a hysteresis mechanism that
reduces the toggle frequency.

For buffer pools which are a depletion means that a pause frame should be generated.
For this the SW needs to poll BPPINumberOfPointers and BPPENumberOfPoint. For queues
congestion means that a pause frame should be generated. For this the SW
needs to poll OccupiedDescriptorsCounter.

Packet Processor v23 has hardware support to monitor FIFO fill level.
patch "add PPv23 version definition" to differ between v23 and v22 hardware.
Patch "add TX FC firmware check" verifies that CM3 firmware support Flow Control
monitoring.

Konstantin Porotchkin (1):
  dts: marvell: add CM3 SRAM memory to cp115 ethernet device tree

Stefan Chulski (18):
  doc: marvell: add cm3-mem device tree bindings description
  net: mvpp2: add CM3 SRAM memory map
  net: mvpp2: add PPv23 version definition
  net: mvpp2: always compare hw-version vs MVPP21
  net: mvpp2: increase BM pool size to 2048 buffers
  net: mvpp2: increase RXQ size to 1024 descriptors
  net: mvpp2: add FCA periodic timer configurations
  net: mvpp2: add FCA RXQ non occupied descriptor threshold
  net: mvpp2: add spinlock for FW FCA configuration path
  net: mvpp2: add flow control RXQ and BM pool config callbacks
  net: mvpp2: enable global flow control
  net: mvpp2: add RXQ flow control configurations
  net: mvpp2: add ethtool flow control configuration support
  net: mvpp2: add BM protection underrun feature support
  net: mvpp2: add PPv23 RX FIFO flow control
  net: mvpp2: set 802.3x GoP Flow Control mode
  net: mvpp2: add ring size validation before enabling FC
  net: mvpp2: add TX FC firmware check

 Documentation/devicetree/bindings/net/marvell-pp2.txt |   1 +
 arch/arm64/boot/dts/marvell/armada-cp11x.dtsi         |  10 +
 drivers/net/ethernet/marvell/mvpp2/mvpp2.h            | 130 ++++-
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c       | 570 +++++++++++++++++++-
 4 files changed, 669 insertions(+), 42 deletions(-)

-- 
1.9.1

[PATCH RFC net-next 01/19] doc: marvell: add cm3-mem device tree bindings description

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 Documentation/devicetree/bindings/net/marvell-pp2.txt | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Documentation/devicetree/bindings/net/marvell-pp2.txt b/Documentation/devicetree/bindings/net/marvell-pp2.txt
index b783976..f9f8cc6 100644
--- a/Documentation/devicetree/bindings/net/marvell-pp2.txt
+++ b/Documentation/devicetree/bindings/net/marvell-pp2.txt
@@ -37,6 +37,7 @@ Required properties (port):
   GOP (Group Of Ports) point of view. This ID is used to index the
   per-port registers in the second register area.
 - phy-mode: See ethernet.txt file in the same directory
+- cm3-mem: phandle to CM3 SRAM definitions
 
 Optional properties (port):
 
-- 
1.9.1

[PATCH RFC net-next 02/19] dts: marvell: add CM3 SRAM memory to cp115 ethernet device tree

[stefanc]

From: Konstantin Porotchkin <kostap@marvell.com>

CM3 SRAM address space would be used for Flow Control configuration.

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 arch/arm64/boot/dts/marvell/armada-cp11x.dtsi | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/arm64/boot/dts/marvell/armada-cp11x.dtsi b/arch/arm64/boot/dts/marvell/armada-cp11x.dtsi
index 9dcf16b..359cf42 100644
--- a/arch/arm64/boot/dts/marvell/armada-cp11x.dtsi
+++ b/arch/arm64/boot/dts/marvell/armada-cp11x.dtsi
@@ -69,6 +69,8 @@
 			status = "disabled";
 			dma-coherent;
 
+			cm3-mem = <&CP11X_LABEL(cm3_sram)>;
+
 			CP11X_LABEL(eth0): eth0 {
 				interrupts = <39 IRQ_TYPE_LEVEL_HIGH>,
 					<43 IRQ_TYPE_LEVEL_HIGH>,
@@ -211,6 +213,14 @@
 			};
 		};
 
+		CP11X_LABEL(cm3_sram): cm3@220000 {
+			compatible = "mmio-sram";
+			reg = <0x220000 0x800>;
+			#address-cells = <1>;
+			#size-cells = <1>;
+			ranges = <0 0x220000 0x800>;
+		};
+
 		CP11X_LABEL(rtc): rtc@284000 {
 			compatible = "marvell,armada-8k-rtc";
 			reg = <0x284000 0x20>, <0x284080 0x24>;
-- 
1.9.1

[PATCH RFC net-next 03/19] net: mvpp2: add CM3 SRAM memory map

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

This patch adds CM3 memory map and CM3 read/write callbacks.
No functionality changes.

Change-Id: Ibae3f5e6695f3454f799568308d349addc730f01
Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2.h      |  7 +++
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 66 +++++++++++++++++++-
 2 files changed, 70 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
index 6bd7e40..aec9179 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
@@ -748,6 +748,9 @@
 #define MVPP2_TX_FIFO_THRESHOLD(kb)	\
 		((kb) * 1024 - MVPP2_TX_FIFO_THRESHOLD_MIN)
 
+/* MSS Flow control */
+#define MSS_SRAM_SIZE	0x800
+
 /* RX buffer constants */
 #define MVPP2_SKB_SHINFO_SIZE \
 	SKB_DATA_ALIGN(sizeof(struct skb_shared_info))
@@ -925,6 +928,7 @@ struct mvpp2 {
 	/* Shared registers' base addresses */
 	void __iomem *lms_base;
 	void __iomem *iface_base;
+	void __iomem *cm3_base;
 
 	/* On PPv2.2, each "software thread" can access the base
 	 * register through a separate address space, each 64 KB apart
@@ -996,6 +1000,9 @@ struct mvpp2 {
 
 	/* page_pool allocator */
 	struct page_pool *page_pool[MVPP2_PORT_MAX_RXQ];
+
+	/* CM3 SRAM pool */
+	struct gen_pool *sram_pool;
 };
 
 struct mvpp2_pcpu_stats {
diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index 3982956..5267746 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -25,6 +25,7 @@
 #include <linux/of_net.h>
 #include <linux/of_address.h>
 #include <linux/of_device.h>
+#include <linux/genalloc.h>
 #include <linux/phy.h>
 #include <linux/phylink.h>
 #include <linux/phy/phy.h>
@@ -91,6 +92,18 @@ static inline u32 mvpp2_cpu_to_thread(struct mvpp2 *priv, int cpu)
 	return cpu % priv->nthreads;
 }
 
+static inline
+void mvpp2_cm3_write(struct mvpp2 *priv, u32 offset, u32 data)
+{
+	writel(data, priv->cm3_base + offset);
+}
+
+static inline
+u32 mvpp2_cm3_read(struct mvpp2 *priv, u32 offset)
+{
+	return readl(priv->cm3_base + offset);
+}
+
 static struct page_pool *
 mvpp2_create_page_pool(struct device *dev, int num, int len,
 		       enum dma_data_direction dma_dir)
@@ -6848,6 +6861,35 @@ static int mvpp2_init(struct platform_device *pdev, struct mvpp2 *priv)
 	return 0;
 }
 
+static int mvpp2_get_sram(struct platform_device *pdev,
+			  struct mvpp2 *priv)
+{
+	struct device_node *dn = pdev->dev.of_node;
+	struct resource *res;
+
+	if (has_acpi_companion(&pdev->dev)) {
+		res = platform_get_resource(pdev, IORESOURCE_MEM, 2);
+		if (!res) {
+			dev_warn(&pdev->dev, "ACPI is too old, TX FC disabled\n");
+			return 0;
+		}
+		priv->cm3_base = devm_ioremap_resource(&pdev->dev, res);
+		if (IS_ERR(priv->cm3_base))
+			return PTR_ERR(priv->cm3_base);
+	} else {
+		priv->sram_pool = of_gen_pool_get(dn, "cm3-mem", 0);
+		if (!priv->sram_pool) {
+			dev_warn(&pdev->dev, "DT is too old, TX FC disabled\n");
+			return 0;
+		}
+		priv->cm3_base = (void __iomem *)gen_pool_alloc(priv->sram_pool,
+								MSS_SRAM_SIZE);
+		if (!priv->cm3_base)
+			return -ENOMEM;
+	}
+	return 0;
+}
+
 static int mvpp2_probe(struct platform_device *pdev)
 {
 	const struct acpi_device_id *acpi_id;
@@ -6904,6 +6946,11 @@ static int mvpp2_probe(struct platform_device *pdev)
 		priv->iface_base = devm_ioremap_resource(&pdev->dev, res);
 		if (IS_ERR(priv->iface_base))
 			return PTR_ERR(priv->iface_base);
+
+		/* Map CM3 SRAM */
+		err = mvpp2_get_sram(pdev, priv);
+		if (err)
+			dev_warn(&pdev->dev, "Fail to alloc CM3 SRAM\n");
 	}
 
 	if (priv->hw_version == MVPP22 && dev_of_node(&pdev->dev)) {
@@ -6949,11 +6996,13 @@ static int mvpp2_probe(struct platform_device *pdev)
 
 	if (dev_of_node(&pdev->dev)) {
 		priv->pp_clk = devm_clk_get(&pdev->dev, "pp_clk");
-		if (IS_ERR(priv->pp_clk))
-			return PTR_ERR(priv->pp_clk);
+		if (IS_ERR(priv->pp_clk)) {
+			err = PTR_ERR(priv->pp_clk);
+			goto err_cm3;
+		}
 		err = clk_prepare_enable(priv->pp_clk);
 		if (err < 0)
-			return err;
+			goto err_cm3;
 
 		priv->gop_clk = devm_clk_get(&pdev->dev, "gop_clk");
 		if (IS_ERR(priv->gop_clk)) {
@@ -7089,6 +7138,11 @@ static int mvpp2_probe(struct platform_device *pdev)
 	clk_disable_unprepare(priv->gop_clk);
 err_pp_clk:
 	clk_disable_unprepare(priv->pp_clk);
+err_cm3:
+	if (!has_acpi_companion(&pdev->dev) && priv->cm3_base)
+		gen_pool_free(priv->sram_pool, (unsigned long)priv->cm3_base,
+			      MSS_SRAM_SIZE);
+
 	return err;
 }
 
@@ -7129,6 +7183,12 @@ static int mvpp2_remove(struct platform_device *pdev)
 				  aggr_txq->descs_dma);
 	}
 
+	if (!has_acpi_companion(&pdev->dev) && priv->cm3_base) {
+		gen_pool_free(priv->sram_pool, (unsigned long)priv->cm3_base,
+			      MSS_SRAM_SIZE);
+		gen_pool_destroy(priv->sram_pool);
+	}
+
 	if (is_acpi_node(port_fwnode))
 		return 0;
 
-- 
1.9.1

[PATCH RFC net-next 04/19] net: mvpp2: add PPv23 version definition

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

This patch add PPv23 version definition.
PPv23 is new packet processor in CP115.
Everything that supported by PPv22, also supported by PPv23.
No functional changes in this stage.

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2.h      | 24 ++++++++++++--------
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 17 +++++++++-----
 2 files changed, 25 insertions(+), 16 deletions(-)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
index aec9179..89b3ede 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
@@ -60,6 +60,9 @@
 /* Top Registers */
 #define MVPP2_MH_REG(port)			(0x5040 + 4 * (port))
 #define MVPP2_DSA_EXTENDED			BIT(5)
+#define MVPP2_VER_ID_REG			0x50b0
+#define MVPP2_VER_PP22				0x10
+#define MVPP2_VER_PP23				0x11
 
 /* Parser Registers */
 #define MVPP2_PRS_INIT_LOOKUP_REG		0x1000
@@ -469,7 +472,7 @@
 #define     MVPP22_GMAC_INT_SUM_MASK_LINK_STAT	BIT(1)
 #define	    MVPP22_GMAC_INT_SUM_MASK_PTP	BIT(2)
 
-/* Per-port XGMAC registers. PPv2.2 only, only for GOP port 0,
+/* Per-port XGMAC registers. PPv2.2 and PPv2.3, only for GOP port 0,
  * relative to port->base.
  */
 #define MVPP22_XLG_CTRL0_REG			0x100
@@ -506,7 +509,7 @@
 #define     MVPP22_XLG_CTRL4_MACMODSELECT_GMAC	BIT(12)
 #define     MVPP22_XLG_CTRL4_EN_IDLE_CHECK	BIT(14)
 
-/* SMI registers. PPv2.2 only, relative to priv->iface_base. */
+/* SMI registers. PPv2.2 and PPv2.3, relative to priv->iface_base. */
 #define MVPP22_SMI_MISC_CFG_REG			0x1204
 #define     MVPP22_SMI_POLLING_EN		BIT(10)
 
@@ -582,7 +585,7 @@
 #define MVPP2_QUEUE_NEXT_DESC(q, index) \
 	(((index) < (q)->last_desc) ? ((index) + 1) : 0)
 
-/* XPCS registers. PPv2.2 only */
+/* XPCS registers.PPv2.2 and PPv2.3 */
 #define MVPP22_MPCS_BASE(port)			(0x7000 + (port) * 0x1000)
 #define MVPP22_MPCS_CTRL			0x14
 #define     MVPP22_MPCS_CTRL_FWD_ERR_CONN	BIT(10)
@@ -593,7 +596,7 @@
 #define     MVPP22_MPCS_CLK_RESET_DIV_RATIO(n)	((n) << 4)
 #define     MVPP22_MPCS_CLK_RESET_DIV_SET	BIT(11)
 
-/* XPCS registers. PPv2.2 only */
+/* XPCS registers. PPv2.2 and PPv2.3 */
 #define MVPP22_XPCS_BASE(port)			(0x7400 + (port) * 0x1000)
 #define MVPP22_XPCS_CFG0			0x0
 #define     MVPP22_XPCS_CFG0_RESET_DIS		BIT(0)
@@ -930,15 +933,16 @@ struct mvpp2 {
 	void __iomem *iface_base;
 	void __iomem *cm3_base;
 
-	/* On PPv2.2, each "software thread" can access the base
+	/* On PPv2.2 and PPv2.3, each "software thread" can access the base
 	 * register through a separate address space, each 64 KB apart
 	 * from each other. Typically, such address spaces will be
 	 * used per CPU.
 	 */
 	void __iomem *swth_base[MVPP2_MAX_THREADS];
 
-	/* On PPv2.2, some port control registers are located into the system
-	 * controller space. These registers are accessible through a regmap.
+	/* On PPv2.2 and PPv2.3, some port control registers are located into
+	 * the system controller space. These registers are accessible
+	 * through a regmap.
 	 */
 	struct regmap *sysctrl_base;
 
@@ -980,7 +984,7 @@ struct mvpp2 {
 	u32 tclk;
 
 	/* HW version */
-	enum { MVPP21, MVPP22 } hw_version;
+	enum { MVPP21, MVPP22, MVPP23 } hw_version;
 
 	/* Maximum number of RXQs per port */
 	unsigned int max_port_rxqs;
@@ -1227,7 +1231,7 @@ struct mvpp21_rx_desc {
 	__le32 reserved8;
 };
 
-/* HW TX descriptor for PPv2.2 */
+/* HW TX descriptor for PPv2.2 and PPv2.3 */
 struct mvpp22_tx_desc {
 	__le32 command;
 	u8  packet_offset;
@@ -1239,7 +1243,7 @@ struct mvpp22_tx_desc {
 	__le64 buf_cookie_misc;
 };
 
-/* HW RX descriptor for PPv2.2 */
+/* HW RX descriptor for PPv2.2 and PPv2.3 */
 struct mvpp22_rx_desc {
 	__le32 status;
 	__le16 reserved1;
diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index 5267746..6a08829 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -397,7 +397,7 @@ static int mvpp2_bm_pool_create(struct device *dev, struct mvpp2 *priv,
 	if (!IS_ALIGNED(size, 16))
 		return -EINVAL;
 
-	/* PPv2.1 needs 8 bytes per buffer pointer, PPv2.2 needs 16
+	/* PPv2.1 needs 8 bytes per buffer pointer, PPv2.2 and PPv2.3 needs 16
 	 * bytes per buffer pointer
 	 */
 	if (priv->hw_version == MVPP21)
@@ -1185,7 +1185,7 @@ static void mvpp2_interrupts_unmask(void *arg)
 	u32 val;
 	int i;
 
-	if (port->priv->hw_version != MVPP22)
+	if (port->priv->hw_version == MVPP21)
 		return;
 
 	if (mask)
@@ -5469,7 +5469,7 @@ static void mvpp2_rx_irqs_setup(struct mvpp2_port *port)
 		return;
 	}
 
-	/* Handle the more complicated PPv2.2 case */
+	/* Handle the more complicated PPv2.2 and PPv2.3 case */
 	for (i = 0; i < port->nqvecs; i++) {
 		struct mvpp2_queue_vector *qv = port->qvecs + i;
 
@@ -5646,7 +5646,7 @@ static bool mvpp22_port_has_legacy_tx_irqs(struct device_node *port_node,
 
 /* Checks if the port dt description has the required Tx interrupts:
  * - PPv2.1: there are no such interrupts.
- * - PPv2.2:
+ * - PPv2.2 and PPv2.3:
  *   - The old DTs have: "rx-shared", "tx-cpuX" with X in [0...3]
  *   - The new ones have: "hifX" with X in [0..8]
  *
@@ -6636,7 +6636,7 @@ static void mvpp22_rx_fifo_set_hw(struct mvpp2 *priv, int port, int data_size)
 	mvpp2_write(priv, MVPP2_RX_ATTR_FIFO_SIZE_REG(port), attr_size);
 }
 
-/* Initialize TX FIFO's: the total FIFO size is 48kB on PPv2.2.
+/* Initialize TX FIFO's: the total FIFO size is 48kB on PPv2.2 and PPv2.3.
  * 4kB fixed space must be assigned for the loopback port.
  * Redistribute remaining avialable 44kB space among all active ports.
  * Guarantee minimum 32kB for 10G port and 8kB for port 1, capable of 2.5G
@@ -6693,7 +6693,7 @@ static void mvpp22_tx_fifo_set_hw(struct mvpp2 *priv, int port, int size)
 	mvpp2_write(priv, MVPP22_TX_FIFO_THRESH_REG(port), threshold);
 }
 
-/* Initialize TX FIFO's: the total FIFO size is 19kB on PPv2.2.
+/* Initialize TX FIFO's: the total FIFO size is 19kB on PPv2.2 and PPv2.3.
  * 3kB fixed space must be assigned for the loopback port.
  * Redistribute remaining avialable 16kB space among all active ports.
  * The 10G interface should use 10kB (which is maximum possible size
@@ -7074,6 +7074,11 @@ static int mvpp2_probe(struct platform_device *pdev)
 			priv->port_map |= BIT(i);
 	}
 
+	if (priv->hw_version != MVPP21) {
+		if (mvpp2_read(priv, MVPP2_VER_ID_REG) == MVPP2_VER_PP23)
+			priv->hw_version = MVPP23;
+	}
+
 	/* Initialize network controller */
 	err = mvpp2_init(pdev, priv);
 	if (err < 0) {
-- 
1.9.1

[PATCH RFC net-next 05/19] net: mvpp2: always compare hw-version vs MVPP21

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

Currently we have PP2v1 and PP2v2 hw-versions, with some different
handlers depending upon condition hw_version = MVPP21/MVPP22.
In a future there will be also PP2v3. Let's use now the generic
"if equal/notEqual MVPP21" for all cases instead of "if MVPP22".

This patch does not change any functionality.
It is not intended to introduce PP2v3.
It just modifies MVPP21/MVPP22 check-condition
bringing it to generic and unified form correct for new-code
introducing and PP2v3 net-next generation.

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 36 ++++++++++----------
 1 file changed, 18 insertions(+), 18 deletions(-)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index 6a08829..aa1f6b4 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -332,7 +332,7 @@ static int mvpp2_get_nrxqs(struct mvpp2 *priv)
 {
 	unsigned int nrxqs;
 
-	if (priv->hw_version == MVPP22 && queue_mode == MVPP2_QDIST_SINGLE_MODE)
+	if (priv->hw_version != MVPP21 && queue_mode == MVPP2_QDIST_SINGLE_MODE)
 		return 1;
 
 	/* According to the PPv2.2 datasheet and our experiments on
@@ -459,7 +459,7 @@ static void mvpp2_bm_bufs_get_addrs(struct device *dev, struct mvpp2 *priv,
 				      MVPP2_BM_PHY_ALLOC_REG(bm_pool->id));
 	*phys_addr = mvpp2_thread_read(priv, thread, MVPP2_BM_VIRT_ALLOC_REG);
 
-	if (priv->hw_version == MVPP22) {
+	if (priv->hw_version != MVPP21) {
 		u32 val;
 		u32 dma_addr_highbits, phys_addr_highbits;
 
@@ -755,7 +755,7 @@ static inline void mvpp2_bm_pool_put(struct mvpp2_port *port, int pool,
 	if (test_bit(thread, &port->priv->lock_map))
 		spin_lock_irqsave(&port->bm_lock[thread], flags);
 
-	if (port->priv->hw_version == MVPP22) {
+	if (port->priv->hw_version != MVPP21) {
 		u32 val = 0;
 
 		if (sizeof(dma_addr_t) == 8)
@@ -1212,7 +1212,7 @@ static bool mvpp2_port_supports_xlg(struct mvpp2_port *port)
 
 static bool mvpp2_port_supports_rgmii(struct mvpp2_port *port)
 {
-	return !(port->priv->hw_version == MVPP22 && port->gop_id == 0);
+	return !(port->priv->hw_version != MVPP21 && port->gop_id == 0);
 }
 
 /* Port configuration routines */
@@ -1830,7 +1830,7 @@ static void mvpp2_mac_reset_assert(struct mvpp2_port *port)
 	      MVPP2_GMAC_PORT_RESET_MASK;
 	writel(val, port->base + MVPP2_GMAC_CTRL_2_REG);
 
-	if (port->priv->hw_version == MVPP22 && port->gop_id == 0) {
+	if (port->priv->hw_version != MVPP21 && port->gop_id == 0) {
 		val = readl(port->base + MVPP22_XLG_CTRL0_REG) &
 		      ~MVPP22_XLG_CTRL0_MAC_RESET_DIS;
 		writel(val, port->base + MVPP22_XLG_CTRL0_REG);
@@ -1843,7 +1843,7 @@ static void mvpp22_pcs_reset_assert(struct mvpp2_port *port)
 	void __iomem *mpcs, *xpcs;
 	u32 val;
 
-	if (port->priv->hw_version != MVPP22 || port->gop_id != 0)
+	if (port->priv->hw_version == MVPP21 || port->gop_id != 0)
 		return;
 
 	mpcs = priv->iface_base + MVPP22_MPCS_BASE(port->gop_id);
@@ -1864,7 +1864,7 @@ static void mvpp22_pcs_reset_deassert(struct mvpp2_port *port)
 	void __iomem *mpcs, *xpcs;
 	u32 val;
 
-	if (port->priv->hw_version != MVPP22 || port->gop_id != 0)
+	if (port->priv->hw_version == MVPP21 || port->gop_id != 0)
 		return;
 
 	mpcs = priv->iface_base + MVPP22_MPCS_BASE(port->gop_id);
@@ -4201,7 +4201,7 @@ static void mvpp2_start_dev(struct mvpp2_port *port)
 	/* Enable interrupts on all threads */
 	mvpp2_interrupts_enable(port);
 
-	if (port->priv->hw_version == MVPP22)
+	if (port->priv->hw_version != MVPP21)
 		mvpp22_mode_reconfigure(port);
 
 	if (port->phylink) {
@@ -4417,7 +4417,7 @@ static int mvpp2_open(struct net_device *dev)
 		valid = true;
 	}
 
-	if (priv->hw_version == MVPP22 && port->port_irq) {
+	if (priv->hw_version != MVPP21 && port->port_irq) {
 		err = request_irq(port->port_irq, mvpp2_port_isr, 0,
 				  dev->name, port);
 		if (err) {
@@ -6067,7 +6067,7 @@ static int mvpp2__mac_prepare(struct phylink_config *config, unsigned int mode,
 			     MVPP2_GMAC_PORT_RESET_MASK,
 			     MVPP2_GMAC_PORT_RESET_MASK);
 
-		if (port->priv->hw_version == MVPP22) {
+		if (port->priv->hw_version != MVPP21) {
 			mvpp22_gop_mask_irq(port);
 
 			phy_power_off(port->comphy);
@@ -6121,7 +6121,7 @@ static int mvpp2_mac_finish(struct phylink_config *config, unsigned int mode,
 {
 	struct mvpp2_port *port = mvpp2_phylink_to_port(config);
 
-	if (port->priv->hw_version == MVPP22 &&
+	if (port->priv->hw_version != MVPP21 &&
 	    port->phy_interface != interface) {
 		port->phy_interface = interface;
 
@@ -6801,7 +6801,7 @@ static int mvpp2_init(struct platform_device *pdev, struct mvpp2 *priv)
 	if (dram_target_info)
 		mvpp2_conf_mbus_windows(dram_target_info, priv);
 
-	if (priv->hw_version == MVPP22)
+	if (priv->hw_version != MVPP21)
 		mvpp2_axi_init(priv);
 
 	/* Disable HW PHY polling */
@@ -6953,7 +6953,7 @@ static int mvpp2_probe(struct platform_device *pdev)
 			dev_warn(&pdev->dev, "Fail to alloc CM3 SRAM\n");
 	}
 
-	if (priv->hw_version == MVPP22 && dev_of_node(&pdev->dev)) {
+	if (priv->hw_version != MVPP21 && dev_of_node(&pdev->dev)) {
 		priv->sysctrl_base =
 			syscon_regmap_lookup_by_phandle(pdev->dev.of_node,
 							"marvell,system-controller");
@@ -6966,7 +6966,7 @@ static int mvpp2_probe(struct platform_device *pdev)
 			priv->sysctrl_base = NULL;
 	}
 
-	if (priv->hw_version == MVPP22 &&
+	if (priv->hw_version != MVPP21 &&
 	    mvpp2_get_nrxqs(priv) * 2 <= MVPP2_BM_MAX_POOLS)
 		priv->percpu_pools = 1;
 
@@ -7013,7 +7013,7 @@ static int mvpp2_probe(struct platform_device *pdev)
 		if (err < 0)
 			goto err_pp_clk;
 
-		if (priv->hw_version == MVPP22) {
+		if (priv->hw_version != MVPP21) {
 			priv->mg_clk = devm_clk_get(&pdev->dev, "mg_clk");
 			if (IS_ERR(priv->mg_clk)) {
 				err = PTR_ERR(priv->mg_clk);
@@ -7054,7 +7054,7 @@ static int mvpp2_probe(struct platform_device *pdev)
 		return -EINVAL;
 	}
 
-	if (priv->hw_version == MVPP22) {
+	if (priv->hw_version != MVPP21) {
 		err = dma_set_mask(&pdev->dev, MVPP2_DESC_DMA_MASK);
 		if (err)
 			goto err_axi_clk;
@@ -7134,10 +7134,10 @@ static int mvpp2_probe(struct platform_device *pdev)
 	clk_disable_unprepare(priv->axi_clk);
 
 err_mg_core_clk:
-	if (priv->hw_version == MVPP22)
+	if (priv->hw_version != MVPP21)
 		clk_disable_unprepare(priv->mg_core_clk);
 err_mg_clk:
-	if (priv->hw_version == MVPP22)
+	if (priv->hw_version != MVPP21)
 		clk_disable_unprepare(priv->mg_clk);
 err_gop_clk:
 	clk_disable_unprepare(priv->gop_clk);
-- 
1.9.1

[PATCH RFC net-next 06/19] net: mvpp2: increase BM pool size to 2048 buffers

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

BM pool size increased to support Firmware Flow Control.
Minimum depletion thresholds to support FC is 1024 buffers.
BM pool size increased to 2048 to have some 1024 buffers
space between depletion thresholds and BM pool size.

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
index 89b3ede..8dc669d 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
@@ -851,8 +851,8 @@ enum mvpp22_ptp_packet_format {
 #define MVPP22_PTP_TIMESTAMPQUEUESELECT	BIT(18)
 
 /* BM constants */
-#define MVPP2_BM_JUMBO_BUF_NUM		512
-#define MVPP2_BM_LONG_BUF_NUM		1024
+#define MVPP2_BM_JUMBO_BUF_NUM		2048
+#define MVPP2_BM_LONG_BUF_NUM		2048
 #define MVPP2_BM_SHORT_BUF_NUM		2048
 #define MVPP2_BM_POOL_SIZE_MAX		(16*1024 - MVPP2_BM_POOL_PTR_ALIGN/4)
 #define MVPP2_BM_POOL_PTR_ALIGN		128
-- 
1.9.1

[PATCH RFC net-next 07/19] net: mvpp2: increase RXQ size to 1024 descriptors

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

RXQ size increased to support Firmware Flow Control.
Minimum depletion thresholds to support FC is 1024 buffers.
Default set to 1024 descriptors and maximum size to 2048.

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
index 8dc669d..cac9885 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
@@ -715,8 +715,8 @@
 #define MVPP2_PORT_MAX_RXQ		32
 
 /* Max number of Rx descriptors */
-#define MVPP2_MAX_RXD_MAX		1024
-#define MVPP2_MAX_RXD_DFLT		128
+#define MVPP2_MAX_RXD_MAX		2048
+#define MVPP2_MAX_RXD_DFLT		1024
 
 /* Max number of Tx descriptors */
 #define MVPP2_MAX_TXD_MAX		2048
-- 
1.9.1

[PATCH RFC net-next 08/19] net: mvpp2: add FCA periodic timer configurations

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

Flow Control periodic timer would be used if port in
XOFF to transmit periodic XOFF frames.

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2.h      | 13 +++++-
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 45 ++++++++++++++++++++
 2 files changed, 57 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
index cac9885..0861c0b 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
@@ -596,6 +596,15 @@
 #define     MVPP22_MPCS_CLK_RESET_DIV_RATIO(n)	((n) << 4)
 #define     MVPP22_MPCS_CLK_RESET_DIV_SET	BIT(11)
 
+/* FCA registers. PPv2.2 and PPv2.3 */
+#define MVPP22_FCA_BASE(port)			(0x7600 + (port) * 0x1000)
+#define MVPP22_FCA_REG_SIZE			16
+#define MVPP22_FCA_REG_MASK			0xFFFF
+#define MVPP22_FCA_CONTROL_REG			0x0
+#define MVPP22_FCA_ENABLE_PERIODIC		BIT(11)
+#define MVPP22_PERIODIC_COUNTER_LSB_REG		(0x110)
+#define MVPP22_PERIODIC_COUNTER_MSB_REG		(0x114)
+
 /* XPCS registers. PPv2.2 and PPv2.3 */
 #define MVPP22_XPCS_BASE(port)			(0x7400 + (port) * 0x1000)
 #define MVPP22_XPCS_CFG0			0x0
@@ -752,7 +761,9 @@
 		((kb) * 1024 - MVPP2_TX_FIFO_THRESHOLD_MIN)
 
 /* MSS Flow control */
-#define MSS_SRAM_SIZE	0x800
+#define MSS_SRAM_SIZE		0x800
+#define FC_QUANTA		0xFFFF
+#define FC_CLK_DIVIDER		0x140
 
 /* RX buffer constants */
 #define MVPP2_SKB_SHINFO_SIZE \
diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index aa1f6b4..b5b7902 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -1293,6 +1293,49 @@ static void mvpp22_gop_init_10gkr(struct mvpp2_port *port)
 	writel(val, mpcs + MVPP22_MPCS_CLK_RESET);
 }
 
+static void mvpp22_gop_fca_enable_periodic(struct mvpp2_port *port, bool en)
+{
+	struct mvpp2 *priv = port->priv;
+	void __iomem *fca = priv->iface_base + MVPP22_FCA_BASE(port->gop_id);
+	u32 val;
+
+	val = readl(fca + MVPP22_FCA_CONTROL_REG);
+	val &= ~MVPP22_FCA_ENABLE_PERIODIC;
+	if (en)
+		val |= MVPP22_FCA_ENABLE_PERIODIC;
+	writel(val, fca + MVPP22_FCA_CONTROL_REG);
+}
+
+static void mvpp22_gop_fca_set_timer(struct mvpp2_port *port, u32 timer)
+{
+	struct mvpp2 *priv = port->priv;
+	void __iomem *fca = priv->iface_base + MVPP22_FCA_BASE(port->gop_id);
+	u32 lsb, msb;
+
+	lsb = timer & MVPP22_FCA_REG_MASK;
+	msb = timer >> MVPP22_FCA_REG_SIZE;
+
+	writel(lsb, fca + MVPP22_PERIODIC_COUNTER_LSB_REG);
+	writel(msb, fca + MVPP22_PERIODIC_COUNTER_MSB_REG);
+}
+
+/* Set Flow Control timer x140 faster than pause quanta to ensure that link
+ * partner won't send taffic if port in XOFF mode.
+ */
+static void mvpp22_gop_fca_set_periodic_timer(struct mvpp2_port *port)
+{
+	u32 timer;
+
+	timer = (port->priv->tclk / (USEC_PER_SEC * FC_CLK_DIVIDER))
+		* FC_QUANTA;
+
+	mvpp22_gop_fca_enable_periodic(port, false);
+
+	mvpp22_gop_fca_set_timer(port, timer);
+
+	mvpp22_gop_fca_enable_periodic(port, true);
+}
+
 static int mvpp22_gop_init(struct mvpp2_port *port)
 {
 	struct mvpp2 *priv = port->priv;
@@ -1337,6 +1380,8 @@ static int mvpp22_gop_init(struct mvpp2_port *port)
 	val |= GENCONF_SOFT_RESET1_GOP;
 	regmap_write(priv->sysctrl_base, GENCONF_SOFT_RESET1, val);
 
+	mvpp22_gop_fca_set_periodic_timer(port);
+
 unsupported_conf:
 	return 0;
 
-- 
1.9.1

[PATCH RFC net-next 09/19] net: mvpp2: add FCA RXQ non occupied descriptor threshold

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

RXQ non occupied descriptor threshold would be used by
Flow Control Firmware feature to move to the XOFF mode.
RXQ non occupied threshold would change interrupt cause
that polled by CM3 Firmware.
Actual non occupied interrupt masked and won't trigger interrupt.

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2.h      |  3 ++
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 29 ++++++++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
index 0861c0b..3df8f60 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
@@ -295,6 +295,8 @@
 #define     MVPP2_PON_CAUSE_TXP_OCCUP_DESC_ALL_MASK	0x3fc00000
 #define     MVPP2_PON_CAUSE_MISC_SUM_MASK		BIT(31)
 #define MVPP2_ISR_MISC_CAUSE_REG		0x55b0
+#define MVPP2_ISR_RX_ERR_CAUSE_REG(port)	(0x5520 + 4 * (port))
+#define	    MVPP2_ISR_RX_ERR_CAUSE_NONOCC_MASK	0x00ff
 
 /* Buffer Manager registers */
 #define MVPP2_BM_POOL_BASE_REG(pool)		(0x6000 + ((pool) * 4))
@@ -764,6 +766,7 @@
 #define MSS_SRAM_SIZE		0x800
 #define FC_QUANTA		0xFFFF
 #define FC_CLK_DIVIDER		0x140
+#define MSS_THRESHOLD_STOP    768
 
 /* RX buffer constants */
 #define MVPP2_SKB_SHINFO_SIZE \
diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index b5b7902..4d4e886 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -1154,6 +1154,9 @@ static void mvpp2_interrupts_mask(void *arg)
 	mvpp2_thread_write(port->priv,
 			   mvpp2_cpu_to_thread(port->priv, smp_processor_id()),
 			   MVPP2_ISR_RX_TX_MASK_REG(port->id), 0);
+	mvpp2_thread_write(port->priv,
+			   mvpp2_cpu_to_thread(port->priv, smp_processor_id()),
+			   MVPP2_ISR_RX_ERR_CAUSE_REG(port->id), 0);
 }
 
 /* Unmask the current thread's Rx/Tx interrupts.
@@ -1177,6 +1180,10 @@ static void mvpp2_interrupts_unmask(void *arg)
 	mvpp2_thread_write(port->priv,
 			   mvpp2_cpu_to_thread(port->priv, smp_processor_id()),
 			   MVPP2_ISR_RX_TX_MASK_REG(port->id), val);
+	mvpp2_thread_write(port->priv,
+			   mvpp2_cpu_to_thread(port->priv, smp_processor_id()),
+			   MVPP2_ISR_RX_ERR_CAUSE_REG(port->id),
+			   MVPP2_ISR_RX_ERR_CAUSE_NONOCC_MASK);
 }
 
 static void
@@ -1201,6 +1208,9 @@ static void mvpp2_interrupts_unmask(void *arg)
 
 		mvpp2_thread_write(port->priv, v->sw_thread_id,
 				   MVPP2_ISR_RX_TX_MASK_REG(port->id), val);
+		mvpp2_thread_write(port->priv, v->sw_thread_id,
+				   MVPP2_ISR_RX_ERR_CAUSE_REG(port->id),
+				   MVPP2_ISR_RX_ERR_CAUSE_NONOCC_MASK);
 	}
 }
 
@@ -2406,6 +2416,22 @@ static void mvpp2_txp_max_tx_size_set(struct mvpp2_port *port)
 	}
 }
 
+/* Routine set the number of non-occupied descriptors threshold that change
+ * interrupt error cause polled by FW Flow Control
+ */
+void mvpp2_set_rxq_free_tresh(struct mvpp2_port *port,
+			      struct mvpp2_rx_queue *rxq)
+{
+	u32 val;
+
+	mvpp2_write(port->priv, MVPP2_RXQ_NUM_REG, rxq->id);
+
+	val = mvpp2_read(port->priv, MVPP2_RXQ_THRESH_REG);
+	val &= ~MVPP2_RXQ_NON_OCCUPIED_MASK;
+	val |= MSS_THRESHOLD_STOP << MVPP2_RXQ_NON_OCCUPIED_OFFSET;
+	mvpp2_write(port->priv, MVPP2_RXQ_THRESH_REG, val);
+}
+
 /* Set the number of packets that will be received before Rx interrupt
  * will be generated by HW.
  */
@@ -2661,6 +2687,9 @@ static int mvpp2_rxq_init(struct mvpp2_port *port,
 	mvpp2_rx_pkts_coal_set(port, rxq);
 	mvpp2_rx_time_coal_set(port, rxq);
 
+	/* Set the number of non occupied descriptors threshold */
+	mvpp2_set_rxq_free_tresh(port, rxq);
+
 	/* Add number of descriptors ready for receiving packets */
 	mvpp2_rxq_status_update(port, rxq->id, 0, rxq->size);
 
-- 
1.9.1

[PATCH RFC net-next 10/19] net: mvpp2: add spinlock for FW FCA configuration path

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

Spinlock added to MSS shared memory configuration space.

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2.h      | 5 +++++
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 3 +++
 2 files changed, 8 insertions(+)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
index 3df8f60..4d58af6 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
@@ -1021,6 +1021,11 @@ struct mvpp2 {
 
 	/* CM3 SRAM pool */
 	struct gen_pool *sram_pool;
+
+	bool custom_dma_mask;
+
+	/* Spinlocks for CM3 shared memory configuration */
+	spinlock_t mss_spinlock;
 };
 
 struct mvpp2_pcpu_stats {
diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index 4d4e886..bc4b8069 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -7153,6 +7153,9 @@ static int mvpp2_probe(struct platform_device *pdev)
 			priv->hw_version = MVPP23;
 	}
 
+	/* Init mss lock */
+	spin_lock_init(&priv->mss_spinlock);
+
 	/* Initialize network controller */
 	err = mvpp2_init(pdev, priv);
 	if (err < 0) {
-- 
1.9.1

[PATCH RFC net-next 11/19] net: mvpp2: add flow control RXQ and BM pool config callbacks

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

This patch did not change any functionality.
Added flow control RXQ and BM pool config callbacks that would be
used to configure RXQ and BM pool thresholds.
APIs also will disable/enable RXQ and pool Flow Control polling.

In this stage BM pool and RXQ has same stop/start thresholds
defined in code.
Also there are common thresholds for all RXQs.

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2.h      |  51 +++++-
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 169 ++++++++++++++++++++
 2 files changed, 216 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
index 4d58af6..0ba0598 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
@@ -763,10 +763,53 @@
 		((kb) * 1024 - MVPP2_TX_FIFO_THRESHOLD_MIN)
 
 /* MSS Flow control */
-#define MSS_SRAM_SIZE		0x800
-#define FC_QUANTA		0xFFFF
-#define FC_CLK_DIVIDER		0x140
-#define MSS_THRESHOLD_STOP    768
+#define MSS_SRAM_SIZE			0x800
+#define MSS_FC_COM_REG			0
+#define FLOW_CONTROL_ENABLE_BIT		BIT(0)
+#define FLOW_CONTROL_UPDATE_COMMAND_BIT	BIT(31)
+#define FC_QUANTA			0xFFFF
+#define FC_CLK_DIVIDER			0x140
+
+#define MSS_BUF_POOL_BASE		0x40
+#define MSS_BUF_POOL_OFFS		4
+#define MSS_BUF_POOL_REG(id)		(MSS_BUF_POOL_BASE		\
+					+ (id) * MSS_BUF_POOL_OFFS)
+
+#define MSS_BUF_POOL_STOP_MASK		0xFFF
+#define MSS_BUF_POOL_START_MASK		(0xFFF << MSS_BUF_POOL_START_OFFS)
+#define MSS_BUF_POOL_START_OFFS		12
+#define MSS_BUF_POOL_PORTS_MASK		(0xF << MSS_BUF_POOL_PORTS_OFFS)
+#define MSS_BUF_POOL_PORTS_OFFS		24
+#define MSS_BUF_POOL_PORT_OFFS(id)	(0x1 <<				\
+					((id) + MSS_BUF_POOL_PORTS_OFFS))
+
+#define MSS_RXQ_TRESH_BASE		0x200
+#define MSS_RXQ_TRESH_OFFS		4
+#define MSS_RXQ_TRESH_REG(q, fq)	(MSS_RXQ_TRESH_BASE + (((q) + (fq)) \
+					* MSS_RXQ_TRESH_OFFS))
+
+#define MSS_RXQ_TRESH_START_MASK	0xFFFF
+#define MSS_RXQ_TRESH_STOP_MASK		(0xFFFF << MSS_RXQ_TRESH_STOP_OFFS)
+#define MSS_RXQ_TRESH_STOP_OFFS		16
+
+#define MSS_RXQ_ASS_BASE	0x80
+#define MSS_RXQ_ASS_OFFS	4
+#define MSS_RXQ_ASS_PER_REG	4
+#define MSS_RXQ_ASS_PER_OFFS	8
+#define MSS_RXQ_ASS_PORTID_OFFS	0
+#define MSS_RXQ_ASS_PORTID_MASK	0x3
+#define MSS_RXQ_ASS_HOSTID_OFFS	2
+#define MSS_RXQ_ASS_HOSTID_MASK	0x3F
+
+#define MSS_RXQ_ASS_Q_BASE(q, fq) ((((q) + (fq)) % MSS_RXQ_ASS_PER_REG)	 \
+				  * MSS_RXQ_ASS_PER_OFFS)
+#define MSS_RXQ_ASS_PQ_BASE(q, fq) ((((q) + (fq)) / MSS_RXQ_ASS_PER_REG) \
+				   * MSS_RXQ_ASS_OFFS)
+#define MSS_RXQ_ASS_REG(q, fq) (MSS_RXQ_ASS_BASE + MSS_RXQ_ASS_PQ_BASE(q, fq))
+
+#define MSS_THRESHOLD_STOP	768
+#define MSS_THRESHOLD_START	1024
+
 
 /* RX buffer constants */
 #define MVPP2_SKB_SHINFO_SIZE \
diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index bc4b8069..19648c4 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -744,6 +744,175 @@ static void *mvpp2_buf_alloc(struct mvpp2_port *port,
 	return data;
 }
 
+/* Routine calculate single queue shares address space */
+static int mvpp22_calc_shared_addr_space(struct mvpp2_port *port)
+{
+	/* If number of CPU's greater than number of threads, return last
+	 * address space
+	 */
+	if (num_active_cpus() >= MVPP2_MAX_THREADS)
+		return MVPP2_MAX_THREADS - 1;
+
+	return num_active_cpus();
+}
+
+/* Routine enable flow control for RXQs conditon */
+void mvpp2_rxq_enable_fc(struct mvpp2_port *port)
+{
+	int val, cm3_state, host_id, q;
+	int fq = port->first_rxq;
+	unsigned long flags;
+
+	spin_lock_irqsave(&port->priv->mss_spinlock, flags);
+
+	/* Remove Flow control enable bit to prevent race between FW and Kernel
+	 * If Flow control were enabled, it would be re-enabled.
+	 */
+	val = mvpp2_cm3_read(port->priv, MSS_FC_COM_REG);
+	cm3_state = (val & FLOW_CONTROL_ENABLE_BIT);
+	val &= ~FLOW_CONTROL_ENABLE_BIT;
+	mvpp2_cm3_write(port->priv, MSS_FC_COM_REG, val);
+
+	/* Set same Flow control for all RXQs */
+	for (q = 0; q < port->nrxqs; q++) {
+		/* Set stop and start Flow control RXQ thresholds */
+		val = MSS_THRESHOLD_START;
+		val |= (MSS_THRESHOLD_STOP << MSS_RXQ_TRESH_STOP_OFFS);
+		mvpp2_cm3_write(port->priv, MSS_RXQ_TRESH_REG(q, fq), val);
+
+		val = mvpp2_cm3_read(port->priv, MSS_RXQ_ASS_REG(q, fq));
+		/* Set RXQ port ID */
+		val &= ~(MSS_RXQ_ASS_PORTID_MASK << MSS_RXQ_ASS_Q_BASE(q, fq));
+		val |= (port->id << MSS_RXQ_ASS_Q_BASE(q, fq));
+		val &= ~(MSS_RXQ_ASS_HOSTID_MASK << (MSS_RXQ_ASS_Q_BASE(q, fq)
+			+ MSS_RXQ_ASS_HOSTID_OFFS));
+
+		/* Calculate RXQ host ID:
+		 * In Single queue mode: Host ID equal to Host ID used for
+		 *			 shared RX interrupt
+		 * In Multi queue mode: Host ID equal to number of
+		 *			RXQ ID / number of CoS queues
+		 * In Single resource mode: Host ID always equal to 0
+		 */
+		if (queue_mode == MVPP2_QDIST_SINGLE_MODE)
+			host_id = mvpp22_calc_shared_addr_space(port);
+		else if (queue_mode == MVPP2_QDIST_MULTI_MODE)
+			host_id = q;
+		else
+			host_id = 0;
+
+		/* Set RXQ host ID */
+		val |= (host_id << (MSS_RXQ_ASS_Q_BASE(q, fq)
+			+ MSS_RXQ_ASS_HOSTID_OFFS));
+
+		mvpp2_cm3_write(port->priv, MSS_RXQ_ASS_REG(q, fq), val);
+	}
+
+	/* Notify Firmware that Flow control config space ready for update */
+	val = mvpp2_cm3_read(port->priv, MSS_FC_COM_REG);
+	val |= FLOW_CONTROL_UPDATE_COMMAND_BIT;
+	val |= cm3_state;
+	mvpp2_cm3_write(port->priv, MSS_FC_COM_REG, val);
+
+	spin_unlock_irqrestore(&port->priv->mss_spinlock, flags);
+}
+
+/* Routine disable flow control for RXQs conditon */
+void mvpp2_rxq_disable_fc(struct mvpp2_port *port)
+{
+	int val, cm3_state, q;
+	unsigned long flags;
+	int fq = port->first_rxq;
+
+	spin_lock_irqsave(&port->priv->mss_spinlock, flags);
+
+	/* Remove Flow control enable bit to prevent race between FW and Kernel
+	 * If Flow control were enabled, it would be re-enabled.
+	 */
+	val = mvpp2_cm3_read(port->priv, MSS_FC_COM_REG);
+	cm3_state = (val & FLOW_CONTROL_ENABLE_BIT);
+	val &= ~FLOW_CONTROL_ENABLE_BIT;
+	mvpp2_cm3_write(port->priv, MSS_FC_COM_REG, val);
+
+	/* Disable Flow control for all RXQs */
+	for (q = 0; q < port->nrxqs; q++) {
+		/* Set threshold 0 to disable Flow control */
+		val = 0;
+		val |= (0 << MSS_RXQ_TRESH_STOP_OFFS);
+		mvpp2_cm3_write(port->priv, MSS_RXQ_TRESH_REG(q, fq), val);
+
+		val = mvpp2_cm3_read(port->priv, MSS_RXQ_ASS_REG(q, fq));
+
+		val &= ~(MSS_RXQ_ASS_PORTID_MASK << MSS_RXQ_ASS_Q_BASE(q, fq));
+
+		val &= ~(MSS_RXQ_ASS_HOSTID_MASK << (MSS_RXQ_ASS_Q_BASE(q, fq)
+			+ MSS_RXQ_ASS_HOSTID_OFFS));
+
+		mvpp2_cm3_write(port->priv, MSS_RXQ_ASS_REG(q, fq), val);
+	}
+
+	/* Notify Firmware that Flow control config space ready for update */
+	val = mvpp2_cm3_read(port->priv, MSS_FC_COM_REG);
+	val |= FLOW_CONTROL_UPDATE_COMMAND_BIT;
+	val |= cm3_state;
+	mvpp2_cm3_write(port->priv, MSS_FC_COM_REG, val);
+
+	spin_unlock_irqrestore(&port->priv->mss_spinlock, flags);
+}
+
+/* Routine disable/enable flow control for BM pool conditon */
+void mvpp2_bm_pool_update_fc(struct mvpp2_port *port,
+			     struct mvpp2_bm_pool *pool,
+			     bool en)
+{
+	int val, cm3_state;
+	unsigned long flags;
+
+	spin_lock_irqsave(&port->priv->mss_spinlock, flags);
+
+	/* Remove Flow control enable bit to prevent race between FW and Kernel
+	 * If Flow control were enabled, it would be re-enabled.
+	 */
+	val = mvpp2_cm3_read(port->priv, MSS_FC_COM_REG);
+	cm3_state = (val & FLOW_CONTROL_ENABLE_BIT);
+	val &= ~FLOW_CONTROL_ENABLE_BIT;
+	mvpp2_cm3_write(port->priv, MSS_FC_COM_REG, val);
+
+	/* Check if BM pool should be enabled/disable */
+	if (en) {
+		/* Set BM pool start and stop thresholds per port */
+		val = mvpp2_cm3_read(port->priv, MSS_BUF_POOL_REG(pool->id));
+		val |= MSS_BUF_POOL_PORT_OFFS(port->id);
+		val &= ~MSS_BUF_POOL_START_MASK;
+		val |= (MSS_THRESHOLD_START << MSS_BUF_POOL_START_OFFS);
+		val &= ~MSS_BUF_POOL_STOP_MASK;
+		val |= MSS_THRESHOLD_STOP;
+		mvpp2_cm3_write(port->priv, MSS_BUF_POOL_REG(pool->id), val);
+	} else {
+		/* Remove BM pool from the port */
+		val = mvpp2_cm3_read(port->priv, MSS_BUF_POOL_REG(pool->id));
+		val &= ~MSS_BUF_POOL_PORT_OFFS(port->id);
+
+		/* Zero BM pool start and stop thresholds to disable pool
+		 * flow control if pool empty (not used by any port)
+		 */
+		if (!pool->buf_num) {
+			val &= ~MSS_BUF_POOL_START_MASK;
+			val &= ~MSS_BUF_POOL_STOP_MASK;
+		}
+
+		mvpp2_cm3_write(port->priv, MSS_BUF_POOL_REG(pool->id), val);
+	}
+
+	/* Notify Firmware that Flow control config space ready for update */
+	val = mvpp2_cm3_read(port->priv, MSS_FC_COM_REG);
+	val |= FLOW_CONTROL_UPDATE_COMMAND_BIT;
+	val |= cm3_state;
+	mvpp2_cm3_write(port->priv, MSS_FC_COM_REG, val);
+
+	spin_unlock_irqrestore(&port->priv->mss_spinlock, flags);
+}
+
 /* Release buffer to BM */
 static inline void mvpp2_bm_pool_put(struct mvpp2_port *port, int pool,
 				     dma_addr_t buf_dma_addr,
-- 
1.9.1

[PATCH RFC net-next 12/19] net: mvpp2: enable global flow control

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

This patch enable global flow control in FW.
Per port flow control is still disabled.

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2.h      |  3 +++
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 15 ++++++++++++++-
 2 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
index 0ba0598..e6bab52 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
@@ -1065,6 +1065,9 @@ struct mvpp2 {
 	/* CM3 SRAM pool */
 	struct gen_pool *sram_pool;
 
+	/* Global TX Flow Control config */
+	bool global_tx_fc;
+
 	bool custom_dma_mask;
 
 	/* Spinlocks for CM3 shared memory configuration */
diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index 19648c4..b7ea94f 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -7142,7 +7142,7 @@ static int mvpp2_probe(struct platform_device *pdev)
 	struct resource *res;
 	void __iomem *base;
 	int i, shared;
-	int err;
+	int err, val;
 
 	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
 	if (!priv)
@@ -7194,6 +7194,10 @@ static int mvpp2_probe(struct platform_device *pdev)
 		err = mvpp2_get_sram(pdev, priv);
 		if (err)
 			dev_warn(&pdev->dev, "Fail to alloc CM3 SRAM\n");
+
+		/* Enable global Flow Control only if hanler to SRAM not NULL */
+		if (priv->cm3_base)
+			priv->global_tx_fc = true;
 	}
 
 	if (priv->hw_version != MVPP21 && dev_of_node(&pdev->dev)) {
@@ -7364,6 +7368,15 @@ static int mvpp2_probe(struct platform_device *pdev)
 		goto err_port_probe;
 	}
 
+	/* Enable global flow control. In this stage global
+	 * flow control enabled, but still disabled per port.
+	 */
+	if (priv->global_tx_fc && priv->hw_version != MVPP21) {
+		val = mvpp2_cm3_read(priv, MSS_FC_COM_REG);
+		val |= FLOW_CONTROL_ENABLE_BIT;
+		mvpp2_cm3_write(priv, MSS_FC_COM_REG, val);
+	}
+
 	mvpp2_dbgfs_init(priv, pdev->name);
 
 	platform_set_drvdata(pdev, priv);
-- 
1.9.1

[PATCH RFC net-next 13/19] net: mvpp2: add RXQ flow control configurations

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

This patch add RXQ flow control configurations.
Patch do not enable flow control itself, flow control
disabled by default.

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2.h      | 3 +++
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 7 +++++++
 2 files changed, 10 insertions(+)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
index e6bab52..57f7bbc 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
@@ -1234,6 +1234,9 @@ struct mvpp2_port {
 	bool rx_hwtstamp;
 	enum hwtstamp_tx_types tx_hwtstamp_type;
 	struct mvpp2_hwtstamp_queue tx_hwtstamp_queue[2];
+
+	/* Firmware TX flow control */
+	bool tx_fc;
 };
 
 /* The mvpp2_tx_desc and mvpp2_rx_desc structures describe the
diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index b7ea94f..3b85aec 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -3176,6 +3176,9 @@ static void mvpp2_cleanup_rxqs(struct mvpp2_port *port)
 
 	for (queue = 0; queue < port->nrxqs; queue++)
 		mvpp2_rxq_deinit(port, port->rxqs[queue]);
+
+	if (port->tx_fc)
+		mvpp2_rxq_disable_fc(port);
 }
 
 /* Init all Rx queues for port */
@@ -3188,6 +3191,10 @@ static int mvpp2_setup_rxqs(struct mvpp2_port *port)
 		if (err)
 			goto err_cleanup;
 	}
+
+	if (port->tx_fc)
+		mvpp2_rxq_enable_fc(port);
+
 	return 0;
 
 err_cleanup:
-- 
1.9.1

[PATCH RFC net-next 14/19] net: mvpp2: add ethtool flow control configuration support

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

This patch add ethtool flow control configuration support.

Tx flow control retrieved correctly by ethtool get function.
FW per port ethtool configuration capability added.

Patch also takes care about mtu change procedure, if PPv2 switch
BM pools during mtu change.

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 53 ++++++++++++++++++++
 1 file changed, 53 insertions(+)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index 3b85aec..4869b14 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -1243,6 +1243,16 @@ static int mvpp2_bm_update_mtu(struct net_device *dev, int mtu)
 		new_long_pool = MVPP2_BM_LONG;
 
 	if (new_long_pool != port->pool_long->id) {
+		if (port->tx_fc) {
+			if (pkt_size > MVPP2_BM_LONG_PKT_SIZE)
+				mvpp2_bm_pool_update_fc(port,
+							port->pool_short,
+							false);
+			else
+				mvpp2_bm_pool_update_fc(port, port->pool_long,
+							false);
+		}
+
 		/* Remove port from old short & long pool */
 		port->pool_long = mvpp2_bm_pool_use(port, port->pool_long->id,
 						    port->pool_long->pkt_size);
@@ -1260,6 +1270,25 @@ static int mvpp2_bm_update_mtu(struct net_device *dev, int mtu)
 		mvpp2_swf_bm_pool_init(port);
 
 		mvpp2_set_hw_csum(port, new_long_pool);
+
+		if (port->tx_fc) {
+			if (pkt_size > MVPP2_BM_LONG_PKT_SIZE)
+				mvpp2_bm_pool_update_fc(port, port->pool_long,
+							true);
+			else
+				mvpp2_bm_pool_update_fc(port, port->pool_short,
+							true);
+		}
+
+		/* Update L4 checksum when jumbo enable/disable on port */
+		if (new_long_pool == MVPP2_BM_JUMBO && port->id != 0) {
+			dev->features &= ~(NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM);
+			dev->hw_features &= ~(NETIF_F_IP_CSUM |
+					      NETIF_F_IPV6_CSUM);
+		} else {
+			dev->features |= NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM;
+			dev->hw_features |= NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM;
+		}
 	}
 
 out_set:
@@ -5373,6 +5402,30 @@ static int mvpp2_ethtool_set_pause_param(struct net_device *dev,
 					 struct ethtool_pauseparam *pause)
 {
 	struct mvpp2_port *port = netdev_priv(dev);
+	int i;
+
+	if (pause->tx_pause && port->priv->global_tx_fc) {
+		port->tx_fc = true;
+		mvpp2_rxq_enable_fc(port);
+		if (port->priv->percpu_pools) {
+			for (i = 0; i < port->nrxqs; i++)
+				mvpp2_bm_pool_update_fc(port, &port->priv->bm_pools[i], true);
+		} else {
+			mvpp2_bm_pool_update_fc(port, port->pool_long, true);
+			mvpp2_bm_pool_update_fc(port, port->pool_short, true);
+		}
+
+	} else if (port->priv->global_tx_fc) {
+		port->tx_fc = false;
+		mvpp2_rxq_disable_fc(port);
+		if (port->priv->percpu_pools) {
+			for (i = 0; i < port->nrxqs; i++)
+				mvpp2_bm_pool_update_fc(port, &port->priv->bm_pools[i], false);
+		} else {
+			mvpp2_bm_pool_update_fc(port, port->pool_long, false);
+			mvpp2_bm_pool_update_fc(port, port->pool_short, false);
+		}
+	}
 
 	if (!port->phylink)
 		return -ENOTSUPP;
-- 
1.9.1

[PATCH RFC net-next 15/19] net: mvpp2: add BM protection underrun feature support

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

Feature double size of BPPI by decreasing number of pools from 16 to 8.
Increasing of BPPI size protect BM drop from BPPI underrun.
Underrun could occurred due to stress on DDR and as result slow buffer
transition from BPPE to BPPI.
New BPPI threshold recommended by spec is:
BPPI low threshold - 640 buffers
BPPI high threshold - 832 buffers
Supported only in PPv23.

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2.h      |  8 +++++
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 36 +++++++++++++++++++-
 2 files changed, 43 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
index 57f7bbc..27aa593 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
@@ -324,6 +324,10 @@
 #define     MVPP2_BM_HIGH_THRESH_MASK		0x7f0000
 #define     MVPP2_BM_HIGH_THRESH_VALUE(val)	((val) << \
 						MVPP2_BM_HIGH_THRESH_OFFS)
+#define     MVPP2_BM_BPPI_HIGH_THRESH		0x1E
+#define     MVPP2_BM_BPPI_LOW_THRESH		0x1C
+#define     MVPP23_BM_BPPI_HIGH_THRESH		0x34
+#define     MVPP23_BM_BPPI_LOW_THRESH		0x28
 #define MVPP2_BM_INTR_CAUSE_REG(pool)		(0x6240 + ((pool) * 4))
 #define     MVPP2_BM_RELEASED_DELAY_MASK	BIT(0)
 #define     MVPP2_BM_ALLOC_FAILED_MASK		BIT(1)
@@ -352,6 +356,10 @@
 #define MVPP2_OVERRUN_ETH_DROP			0x7000
 #define MVPP2_CLS_ETH_DROP			0x7020
 
+#define MVPP22_BM_POOL_BASE_ADDR_HIGH_REG	0x6310
+#define     MVPP22_BM_POOL_BASE_ADDR_HIGH_MASK	0xff
+#define     MVPP23_BM_8POOL_MODE		BIT(8)
+
 /* Hit counters registers */
 #define MVPP2_CTRS_IDX				0x7040
 #define     MVPP22_CTRS_TX_CTR(port, txq)	((txq) | ((port) << 3) | BIT(7))
diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index 4869b14..8827f52 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -70,6 +70,11 @@ enum mvpp2_bm_pool_log_num {
 module_param(queue_mode, int, 0444);
 MODULE_PARM_DESC(queue_mode, "Set queue_mode (single=0, multi=1)");
 
+static int bm_underrun_protect = 1;
+
+module_param(bm_underrun_protect, int, 0444);
+MODULE_PARM_DESC(bm_underrun_protect, "Set BM underrun protect feature (0-1), def=1");
+
 /* Utility/helper methods */
 
 void mvpp2_write(struct mvpp2 *priv, u32 offset, u32 data)
@@ -426,6 +431,21 @@ static int mvpp2_bm_pool_create(struct device *dev, struct mvpp2 *priv,
 
 	val = mvpp2_read(priv, MVPP2_BM_POOL_CTRL_REG(bm_pool->id));
 	val |= MVPP2_BM_START_MASK;
+
+	val &= ~MVPP2_BM_LOW_THRESH_MASK;
+	val &= ~MVPP2_BM_HIGH_THRESH_MASK;
+
+	/* Set 8 Pools BPPI threshold if BM underrun protection feature
+	 * were enabled
+	 */
+	if (priv->hw_version == MVPP23 && bm_underrun_protect) {
+		val |= MVPP2_BM_LOW_THRESH_VALUE(MVPP23_BM_BPPI_LOW_THRESH);
+		val |= MVPP2_BM_HIGH_THRESH_VALUE(MVPP23_BM_BPPI_HIGH_THRESH);
+	} else {
+		val |= MVPP2_BM_LOW_THRESH_VALUE(MVPP2_BM_BPPI_LOW_THRESH);
+		val |= MVPP2_BM_HIGH_THRESH_VALUE(MVPP2_BM_BPPI_HIGH_THRESH);
+	}
+
 	mvpp2_write(priv, MVPP2_BM_POOL_CTRL_REG(bm_pool->id), val);
 
 	bm_pool->size = size;
@@ -594,6 +614,16 @@ static int mvpp2_bm_pools_init(struct device *dev, struct mvpp2 *priv)
 	return err;
 }
 
+/* Routine enable PPv23 8 pool mode */
+static void mvpp23_bm_set_8pool_mode(struct mvpp2 *priv)
+{
+	int val;
+
+	val = mvpp2_read(priv, MVPP22_BM_POOL_BASE_ADDR_HIGH_REG);
+	val |= MVPP23_BM_8POOL_MODE;
+	mvpp2_write(priv, MVPP22_BM_POOL_BASE_ADDR_HIGH_REG, val);
+}
+
 static int mvpp2_bm_init(struct device *dev, struct mvpp2 *priv)
 {
 	enum dma_data_direction dma_dir = DMA_FROM_DEVICE;
@@ -647,6 +677,9 @@ static int mvpp2_bm_init(struct device *dev, struct mvpp2 *priv)
 	if (!priv->bm_pools)
 		return -ENOMEM;
 
+	if (priv->hw_version == MVPP23 && bm_underrun_protect)
+		mvpp23_bm_set_8pool_mode(priv);
+
 	err = mvpp2_bm_pools_init(dev, priv);
 	if (err < 0)
 		return err;
@@ -5404,7 +5437,8 @@ static int mvpp2_ethtool_set_pause_param(struct net_device *dev,
 	struct mvpp2_port *port = netdev_priv(dev);
 	int i;
 
-	if (pause->tx_pause && port->priv->global_tx_fc) {
+	if (pause->tx_pause && port->priv->global_tx_fc &&
+	    bm_underrun_protect) {
 		port->tx_fc = true;
 		mvpp2_rxq_enable_fc(port);
 		if (port->priv->percpu_pools) {
-- 
1.9.1

[PATCH RFC net-next 16/19] net: mvpp2: add PPv23 RX FIFO flow control

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

New FIFO flow control feature were added in PPv23.
PPv2 FIFO polled by HW and trigger pause frame if FIFO
fill level is below threshold.
FIFO HW flow control enabled with CM3 RXQ&BM flow
control with ethtool.
Current  FIFO thresholds is:
9KB for port with maximum speed 10Gb/s port
4KB for port with maximum speed 5Gb/s port
2KB for port with maximum speed 1Gb/s port

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2.h      | 16 +++++-
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 55 ++++++++++++++++++++
 2 files changed, 70 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
index 27aa593..3451618 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
@@ -770,6 +770,18 @@
 #define MVPP2_TX_FIFO_THRESHOLD(kb)	\
 		((kb) * 1024 - MVPP2_TX_FIFO_THRESHOLD_MIN)
 
+/* RX FIFO threshold in 1KB granularity */
+#define MVPP23_PORT0_FIFO_TRSH	(9 * 1024)
+#define MVPP23_PORT1_FIFO_TRSH	(4 * 1024)
+#define MVPP23_PORT2_FIFO_TRSH	(2 * 1024)
+
+/* RX Flow Control Registers */
+#define MVPP2_RX_FC_REG(port)		(0x150 + 4 * (port))
+#define     MVPP2_RX_FC_EN		BIT(24)
+#define     MVPP2_RX_FC_TRSH_OFFS	16
+#define     MVPP2_RX_FC_TRSH_MASK	(0xFF << MVPP2_RX_FC_TRSH_OFFS)
+#define     MVPP2_RX_FC_TRSH_UNIT	256
+
 /* MSS Flow control */
 #define MSS_SRAM_SIZE			0x800
 #define MSS_FC_COM_REG			0
@@ -818,7 +830,6 @@
 #define MSS_THRESHOLD_STOP	768
 #define MSS_THRESHOLD_START	1024
 
-
 /* RX buffer constants */
 #define MVPP2_SKB_SHINFO_SIZE \
 	SKB_DATA_ALIGN(sizeof(struct skb_shared_info))
@@ -1505,6 +1516,8 @@ struct mvpp2_bm_pool {
 
 void mvpp2_dbgfs_cleanup(struct mvpp2 *priv);
 
+void mvpp23_rx_fifo_fc_en(struct mvpp2 *priv, int port, bool en);
+
 #ifdef CONFIG_MVPP2_PTP
 int mvpp22_tai_probe(struct device *dev, struct mvpp2 *priv);
 void mvpp22_tai_tstamp(struct mvpp2_tai *tai, u32 tstamp,
@@ -1537,4 +1550,5 @@ static inline bool mvpp22_rx_hwtstamping(struct mvpp2_port *port)
 {
 	return IS_ENABLED(CONFIG_MVPP2_PTP) && port->rx_hwtstamp;
 }
+
 #endif
diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index 8827f52..757dfe0 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -5448,6 +5448,8 @@ static int mvpp2_ethtool_set_pause_param(struct net_device *dev,
 			mvpp2_bm_pool_update_fc(port, port->pool_long, true);
 			mvpp2_bm_pool_update_fc(port, port->pool_short, true);
 		}
+		if (port->priv->hw_version == MVPP23)
+			mvpp23_rx_fifo_fc_en(port->priv, port->id, true);
 
 	} else if (port->priv->global_tx_fc) {
 		port->tx_fc = false;
@@ -5459,6 +5461,8 @@ static int mvpp2_ethtool_set_pause_param(struct net_device *dev,
 			mvpp2_bm_pool_update_fc(port, port->pool_long, false);
 			mvpp2_bm_pool_update_fc(port, port->pool_short, false);
 		}
+		if (port->priv->hw_version == MVPP23)
+			mvpp23_rx_fifo_fc_en(port->priv, port->id, false);
 	}
 
 	if (!port->phylink)
@@ -7022,6 +7026,55 @@ static void mvpp22_rx_fifo_init(struct mvpp2 *priv)
 	mvpp2_write(priv, MVPP2_RX_FIFO_INIT_REG, 0x1);
 }
 
+/* Configure Rx FIFO Flow control thresholds */
+static void mvpp23_rx_fifo_fc_set_tresh(struct mvpp2 *priv)
+{
+	int port, val;
+
+	/* Port 0: maximum speed -10Gb/s port
+	 *	   required by spec RX FIFO threshold 9KB
+	 * Port 1: maximum speed -5Gb/s port
+	 *	   required by spec RX FIFO threshold 4KB
+	 * Port 2: maximum speed -1Gb/s port
+	 *	   required by spec RX FIFO threshold 2KB
+	 */
+
+	/* Without loopback port */
+	for (port = 0; port < (MVPP2_MAX_PORTS - 1); port++) {
+		if (port == 0) {
+			val = (MVPP23_PORT0_FIFO_TRSH / MVPP2_RX_FC_TRSH_UNIT)
+				<< MVPP2_RX_FC_TRSH_OFFS;
+			val &= MVPP2_RX_FC_TRSH_MASK;
+			mvpp2_write(priv, MVPP2_RX_FC_REG(port), val);
+		} else if (port == 1) {
+			val = (MVPP23_PORT1_FIFO_TRSH / MVPP2_RX_FC_TRSH_UNIT)
+				<< MVPP2_RX_FC_TRSH_OFFS;
+			val &= MVPP2_RX_FC_TRSH_MASK;
+			mvpp2_write(priv, MVPP2_RX_FC_REG(port), val);
+		} else {
+			val = (MVPP23_PORT2_FIFO_TRSH / MVPP2_RX_FC_TRSH_UNIT)
+				<< MVPP2_RX_FC_TRSH_OFFS;
+			val &= MVPP2_RX_FC_TRSH_MASK;
+			mvpp2_write(priv, MVPP2_RX_FC_REG(port), val);
+		}
+	}
+}
+
+/* Configure Rx FIFO Flow control thresholds */
+void mvpp23_rx_fifo_fc_en(struct mvpp2 *priv, int port, bool en)
+{
+	int val;
+
+	val = mvpp2_read(priv, MVPP2_RX_FC_REG(port));
+
+	if (en)
+		val |= MVPP2_RX_FC_EN;
+	else
+		val &= ~MVPP2_RX_FC_EN;
+
+	mvpp2_write(priv, MVPP2_RX_FC_REG(port), val);
+}
+
 static void mvpp22_tx_fifo_set_hw(struct mvpp2 *priv, int port, int size)
 {
 	int threshold = MVPP2_TX_FIFO_THRESHOLD(size);
@@ -7173,6 +7226,8 @@ static int mvpp2_init(struct platform_device *pdev, struct mvpp2 *priv)
 	} else {
 		mvpp22_rx_fifo_init(priv);
 		mvpp22_tx_fifo_init(priv);
+		if (priv->hw_version == MVPP23)
+			mvpp23_rx_fifo_fc_set_tresh(priv);
 	}
 
 	if (priv->hw_version == MVPP21)
-- 
1.9.1

[PATCH RFC net-next 17/19] net: mvpp2: set 802.3x GoP Flow Control mode

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

This patch fix GMAC TX flow control autoneg.
Flow control autoneg wrongly were disabled with enabled TX
flow control.

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index 757dfe0..06e1000 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -6321,7 +6321,7 @@ static void mvpp2_gmac_config(struct mvpp2_port *port, unsigned int mode,
 	old_ctrl4 = ctrl4 = readl(port->base + MVPP22_GMAC_CTRL_4_REG);
 
 	ctrl0 &= ~MVPP2_GMAC_PORT_TYPE_MASK;
-	ctrl2 &= ~(MVPP2_GMAC_INBAND_AN_MASK | MVPP2_GMAC_PCS_ENABLE_MASK);
+	ctrl2 &= ~(MVPP2_GMAC_INBAND_AN_MASK | MVPP2_GMAC_PCS_ENABLE_MASK | MVPP2_GMAC_FLOW_CTRL_MASK);
 
 	/* Configure port type */
 	if (phy_interface_mode_is_8023z(state->interface)) {
-- 
1.9.1

[PATCH RFC net-next 18/19] net: mvpp2: add ring size validation before enabling FC

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

This patch add ring size validation before enabling FC.
1. Flow control cannot be enabled if ring size is below start
threshold.
2. Flow control disabled if ring size set below start
threshold.

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index 06e1000..3607382 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -5372,6 +5372,15 @@ static int mvpp2_ethtool_set_ringparam(struct net_device *dev,
 	if (err)
 		return err;
 
+	if (ring->rx_pending < MSS_THRESHOLD_START && port->tx_fc) {
+		netdev_warn(dev, "TX FC disabled. Ring size is less than %d\n",
+			    MSS_THRESHOLD_START);
+		port->tx_fc = false;
+		mvpp2_rxq_disable_fc(port);
+		if (port->priv->hw_version == MVPP23)
+			mvpp23_rx_fifo_fc_en(port->priv, port->id, false);
+	}
+
 	if (!netif_running(dev)) {
 		port->rx_ring_size = ring->rx_pending;
 		port->tx_ring_size = ring->tx_pending;
@@ -5439,6 +5448,13 @@ static int mvpp2_ethtool_set_pause_param(struct net_device *dev,
 
 	if (pause->tx_pause && port->priv->global_tx_fc &&
 	    bm_underrun_protect) {
+		if (port->rx_ring_size < MSS_THRESHOLD_START) {
+			netdev_err(dev, "TX FC cannot be supported.");
+			netdev_err(dev, "Ring size is less than %d\n",
+				   MSS_THRESHOLD_START);
+			return -EINVAL;
+		}
+
 		port->tx_fc = true;
 		mvpp2_rxq_enable_fc(port);
 		if (port->priv->percpu_pools) {
-- 
1.9.1

[PATCH RFC net-next 19/19] net: mvpp2: add TX FC firmware check

[stefanc]

From: Stefan Chulski <stefanc@marvell.com>

Patch check that TX FC firmware is running in CM3.
If not, global TX FC would be disabled.

Signed-off-by: Stefan Chulski <stefanc@marvell.com>
---
 drivers/net/ethernet/marvell/mvpp2/mvpp2.h      |  1 +
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 39 ++++++++++++++++----
 2 files changed, 33 insertions(+), 7 deletions(-)

diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
index 3451618..1a65f2c 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
@@ -829,6 +829,7 @@
 
 #define MSS_THRESHOLD_STOP	768
 #define MSS_THRESHOLD_START	1024
+#define MSS_FC_MAX_TIMEOUT	5000
 
 /* RX buffer constants */
 #define MVPP2_SKB_SHINFO_SIZE \
diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index 3607382..1690142 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -946,6 +946,34 @@ void mvpp2_bm_pool_update_fc(struct mvpp2_port *port,
 	spin_unlock_irqrestore(&port->priv->mss_spinlock, flags);
 }
 
+static int mvpp2_enable_global_fc(struct mvpp2 *priv)
+{
+	int val, timeout = 0;
+
+	/* Enable global flow control. In this stage global
+	 * flow control enabled, but still disabled per port.
+	 */
+	val = mvpp2_cm3_read(priv, MSS_FC_COM_REG);
+	val |= FLOW_CONTROL_ENABLE_BIT;
+	mvpp2_cm3_write(priv, MSS_FC_COM_REG, val);
+
+	/* Check if Firmware running and disable FC if not*/
+	val |= FLOW_CONTROL_UPDATE_COMMAND_BIT;
+	mvpp2_cm3_write(priv, MSS_FC_COM_REG, val);
+
+	while (timeout < MSS_FC_MAX_TIMEOUT) {
+		val = mvpp2_cm3_read(priv, MSS_FC_COM_REG);
+
+		if (!(val & FLOW_CONTROL_UPDATE_COMMAND_BIT))
+			return 0;
+		usleep_range(10, 20);
+		timeout++;
+	}
+
+	priv->global_tx_fc = false;
+	return -EOPNOTSUPP;
+}
+
 /* Release buffer to BM */
 static inline void mvpp2_bm_pool_put(struct mvpp2_port *port, int pool,
 				     dma_addr_t buf_dma_addr,
@@ -7307,7 +7335,7 @@ static int mvpp2_probe(struct platform_device *pdev)
 	struct resource *res;
 	void __iomem *base;
 	int i, shared;
-	int err, val;
+	int err;
 
 	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
 	if (!priv)
@@ -7533,13 +7561,10 @@ static int mvpp2_probe(struct platform_device *pdev)
 		goto err_port_probe;
 	}
 
-	/* Enable global flow control. In this stage global
-	 * flow control enabled, but still disabled per port.
-	 */
 	if (priv->global_tx_fc && priv->hw_version != MVPP21) {
-		val = mvpp2_cm3_read(priv, MSS_FC_COM_REG);
-		val |= FLOW_CONTROL_ENABLE_BIT;
-		mvpp2_cm3_write(priv, MSS_FC_COM_REG, val);
+		err = mvpp2_enable_global_fc(priv);
+		if (err)
+			dev_warn(&pdev->dev, "CM3 firmware not running, TX FC disabled\n");
 	}
 
 	mvpp2_dbgfs_init(priv, pdev->name);
-- 
1.9.1

Re: [PATCH RFC net-next 03/19] net: mvpp2: add CM3 SRAM memory map

[Andrew Lunn]

> +static int mvpp2_get_sram(struct platform_device *pdev,
> +			  struct mvpp2 *priv)
> +{
> +	struct device_node *dn = pdev->dev.of_node;
> +	struct resource *res;
> +
> +	if (has_acpi_companion(&pdev->dev)) {
> +		res = platform_get_resource(pdev, IORESOURCE_MEM, 2);
> +		if (!res) {
> +			dev_warn(&pdev->dev, "ACPI is too old, TX FC disabled\n");
> +			return 0;
> +		}
> +		priv->cm3_base = devm_ioremap_resource(&pdev->dev, res);
> +		if (IS_ERR(priv->cm3_base))
> +			return PTR_ERR(priv->cm3_base);
> +	} else {
> +		priv->sram_pool = of_gen_pool_get(dn, "cm3-mem", 0);
> +		if (!priv->sram_pool) {
> +			dev_warn(&pdev->dev, "DT is too old, TX FC disabled\n");
> +			return 0;
> +		}
> +		priv->cm3_base = (void __iomem *)gen_pool_alloc(priv->sram_pool,
> +								MSS_SRAM_SIZE);
> +		if (!priv->cm3_base)
> +			return -ENOMEM;

Should there be -EPROBE_DEFER handling in here somewhere? The SRAM is
a device, so it might not of been probed yet?

  Andrew

RE: [EXT] Re: [PATCH RFC net-next 03/19] net: mvpp2: add CM3 SRAM memory map

[Stefan Chulski]

> -----Original Message-----
> From: Andrew Lunn <andrew@lunn.ch>
> Sent: Sunday, January 10, 2021 7:05 PM
> To: Stefan Chulski <stefanc@marvell.com>
> Cc: netdev@vger.kernel.org; thomas.petazzoni@bootlin.com;
> davem@davemloft.net; Nadav Haklai <nadavh@marvell.com>; Yan Markman
> <ymarkman@marvell.com>; linux-kernel@vger.kernel.org; kuba@kernel.org;
> linux@armlinux.org.uk; mw@semihalf.com; rmk+kernel@armlinux.org.uk;
> atenart@kernel.org
> Subject: [EXT] Re: [PATCH RFC net-next 03/19] net: mvpp2: add CM3 SRAM
> memory map
> 
> External Email
> 
> ----------------------------------------------------------------------
> > +static int mvpp2_get_sram(struct platform_device *pdev,
> > +			  struct mvpp2 *priv)
> > +{
> > +	struct device_node *dn = pdev->dev.of_node;
> > +	struct resource *res;
> > +
> > +	if (has_acpi_companion(&pdev->dev)) {
> > +		res = platform_get_resource(pdev, IORESOURCE_MEM, 2);
> > +		if (!res) {
> > +			dev_warn(&pdev->dev, "ACPI is too old, TX FC
> disabled\n");
> > +			return 0;
> > +		}
> > +		priv->cm3_base = devm_ioremap_resource(&pdev->dev,
> res);
> > +		if (IS_ERR(priv->cm3_base))
> > +			return PTR_ERR(priv->cm3_base);
> > +	} else {
> > +		priv->sram_pool = of_gen_pool_get(dn, "cm3-mem", 0);
> > +		if (!priv->sram_pool) {
> > +			dev_warn(&pdev->dev, "DT is too old, TX FC
> disabled\n");
> > +			return 0;
> > +		}
> > +		priv->cm3_base = (void __iomem *)gen_pool_alloc(priv-
> >sram_pool,
> > +
> 	MSS_SRAM_SIZE);
> > +		if (!priv->cm3_base)
> > +			return -ENOMEM;
> 
> Should there be -EPROBE_DEFER handling in here somewhere? The SRAM is a
> device, so it might not of been probed yet?

No, firmware probed during bootloader boot and we can use SRAM. SRAM memory can be safely used. 

Regards.

Re: [PATCH RFC net-next 06/19] net: mvpp2: increase BM pool size to 2048 buffers

[Andrew Lunn]

On Sun, Jan 10, 2021 at 05:30:10PM +0200, stefanc@marvell.com wrote:
> From: Stefan Chulski <stefanc@marvell.com>
> 
> BM pool size increased to support Firmware Flow Control.
> Minimum depletion thresholds to support FC is 1024 buffers.
> BM pool size increased to 2048 to have some 1024 buffers
> space between depletion thresholds and BM pool size.
> 
> Signed-off-by: Stefan Chulski <stefanc@marvell.com>
> ---
>  drivers/net/ethernet/marvell/mvpp2/mvpp2.h | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
> index 89b3ede..8dc669d 100644
> --- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
> +++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
> @@ -851,8 +851,8 @@ enum mvpp22_ptp_packet_format {
>  #define MVPP22_PTP_TIMESTAMPQUEUESELECT	BIT(18)
>  
>  /* BM constants */
> -#define MVPP2_BM_JUMBO_BUF_NUM		512
> -#define MVPP2_BM_LONG_BUF_NUM		1024
> +#define MVPP2_BM_JUMBO_BUF_NUM		2048
> +#define MVPP2_BM_LONG_BUF_NUM		2048

Hi Stefan

Jumbo used to be 1/2 of regular. Do you know why?

It would be nice to have a comment in the commit message about why it
is O.K. to change the ratio of jumbo to regular frames, and what if
anything this does for memory requirements.

	 Andrew
 

RE: [EXT] Re: [PATCH RFC net-next 06/19] net: mvpp2: increase BM pool size to 2048 buffers

[Stefan Chulski]

> External Email
> 
> ----------------------------------------------------------------------
> On Sun, Jan 10, 2021 at 05:30:10PM +0200, stefanc@marvell.com wrote:
> > From: Stefan Chulski <stefanc@marvell.com>
> >
> > BM pool size increased to support Firmware Flow Control.
> > Minimum depletion thresholds to support FC is 1024 buffers.
> > BM pool size increased to 2048 to have some 1024 buffers space between
> > depletion thresholds and BM pool size.
> >
> > Signed-off-by: Stefan Chulski <stefanc@marvell.com>
> > ---
> >  drivers/net/ethernet/marvell/mvpp2/mvpp2.h | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
> > b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
> > index 89b3ede..8dc669d 100644
> > --- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
> > +++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
> > @@ -851,8 +851,8 @@ enum mvpp22_ptp_packet_format {
> >  #define MVPP22_PTP_TIMESTAMPQUEUESELECT	BIT(18)
> >
> >  /* BM constants */
> > -#define MVPP2_BM_JUMBO_BUF_NUM		512
> > -#define MVPP2_BM_LONG_BUF_NUM		1024
> > +#define MVPP2_BM_JUMBO_BUF_NUM		2048
> > +#define MVPP2_BM_LONG_BUF_NUM		2048
> 
> Hi Stefan
> 
> Jumbo used to be 1/2 of regular. Do you know why?
> 
> It would be nice to have a comment in the commit message about why it is
> O.K. to change the ratio of jumbo to regular frames, and what if anything this
> does for memory requirements.
> 
> 	 Andrew

I don't know why it is half(no hardware restrictions for this). I would add to commit message new memory requirements for buffer allocations.

Thanks.

Re: [PATCH RFC net-next 14/19] net: mvpp2: add ethtool flow control configuration support

[Andrew Lunn]

> @@ -5373,6 +5402,30 @@ static int mvpp2_ethtool_set_pause_param(struct net_device *dev,
>  					 struct ethtool_pauseparam *pause)
>  {
>  	struct mvpp2_port *port = netdev_priv(dev);
> +	int i;
> +
> +	if (pause->tx_pause && port->priv->global_tx_fc) {
> +		port->tx_fc = true;
> +		mvpp2_rxq_enable_fc(port);
> +		if (port->priv->percpu_pools) {
> +			for (i = 0; i < port->nrxqs; i++)
> +				mvpp2_bm_pool_update_fc(port, &port->priv->bm_pools[i], true);
> +		} else {
> +			mvpp2_bm_pool_update_fc(port, port->pool_long, true);
> +			mvpp2_bm_pool_update_fc(port, port->pool_short, true);
> +		}
> +
> +	} else if (port->priv->global_tx_fc) {
> +		port->tx_fc = false;
> +		mvpp2_rxq_disable_fc(port);
> +		if (port->priv->percpu_pools) {
> +			for (i = 0; i < port->nrxqs; i++)
> +				mvpp2_bm_pool_update_fc(port, &port->priv->bm_pools[i], false);
> +		} else {
> +			mvpp2_bm_pool_update_fc(port, port->pool_long, false);
> +			mvpp2_bm_pool_update_fc(port, port->pool_short, false);
> +		}
> +	}


This looks wrong. Flow control is normally the result of auto
negotiation. Both ends need to agree to it. Which is why
mvpp2_ethtool_set_pause_param() passes the users request onto phylink.
phylink will handle the autoneg and then ask the MAC to setup flow
control depending on the result in mvpp2_mac_link_up().

	Andrew

Re: [PATCH RFC net-next 18/19] net: mvpp2: add ring size validation before enabling FC

[Andrew Lunn]

On Sun, Jan 10, 2021 at 05:30:22PM +0200, stefanc@marvell.com wrote:
> From: Stefan Chulski <stefanc@marvell.com>
> 
> This patch add ring size validation before enabling FC.
> 1. Flow control cannot be enabled if ring size is below start
> threshold.
> 2. Flow control disabled if ring size set below start
> threshold.

You should also tell phylink if pause is not supported, so it can
change what is auto-negotiated, letting the link partner know.

       Andrew

Re: [EXT] Re: [PATCH RFC net-next 03/19] net: mvpp2: add CM3 SRAM memory map

[Andrew Lunn]

> > Should there be -EPROBE_DEFER handling in here somewhere? The SRAM is a
> > device, so it might not of been probed yet?
> 

> No, firmware probed during bootloader boot and we can use SRAM. SRAM
> memory can be safely used.

A previous patch added:

+               CP11X_LABEL(cm3_sram): cm3@220000 {
+                       compatible = "mmio-sram";
+                       reg = <0x220000 0x800>;
+                       #address-cells = <1>;
+                       #size-cells = <1>;
+                       ranges = <0 0x220000 0x800>;
+               };
+

So it looks like the SRAM is a device, in the linux driver model. And
there is a driver for this, driver/misc/sram.c. How do you know this
device has been probed before the Ethernet driver?

       Andrew

RE: [EXT] Re: [PATCH RFC net-next 03/19] net: mvpp2: add CM3 SRAM memory map

[Stefan Chulski]

> 
> > > Should there be -EPROBE_DEFER handling in here somewhere? The SRAM
> > > is a device, so it might not of been probed yet?
> >
> 
> > No, firmware probed during bootloader boot and we can use SRAM. SRAM
> > memory can be safely used.
> 
> A previous patch added:
> 
> +               CP11X_LABEL(cm3_sram): cm3@220000 {
> +                       compatible = "mmio-sram";
> +                       reg = <0x220000 0x800>;
> +                       #address-cells = <1>;
> +                       #size-cells = <1>;
> +                       ranges = <0 0x220000 0x800>;
> +               };
> +
> 
> So it looks like the SRAM is a device, in the linux driver model. And there is a
> driver for this, driver/misc/sram.c. How do you know this device has been
> probed before the Ethernet driver?
> 
>        Andrew

You right, I would add EPROBE_DEFER if of_gen_pool_get return NULL.

Thanks.

RE: [EXT] Re: [PATCH RFC net-next 14/19] net: mvpp2: add ethtool flow control configuration support

[Stefan Chulski]

> > @@ -5373,6 +5402,30 @@ static int
> mvpp2_ethtool_set_pause_param(struct net_device *dev,
> >  					 struct ethtool_pauseparam *pause)  {
> >  	struct mvpp2_port *port = netdev_priv(dev);
> > +	int i;
> > +
> > +	if (pause->tx_pause && port->priv->global_tx_fc) {
> > +		port->tx_fc = true;
> > +		mvpp2_rxq_enable_fc(port);
> > +		if (port->priv->percpu_pools) {
> > +			for (i = 0; i < port->nrxqs; i++)
> > +				mvpp2_bm_pool_update_fc(port, &port-
> >priv->bm_pools[i], true);
> > +		} else {
> > +			mvpp2_bm_pool_update_fc(port, port->pool_long,
> true);
> > +			mvpp2_bm_pool_update_fc(port, port->pool_short,
> true);
> > +		}
> > +
> > +	} else if (port->priv->global_tx_fc) {
> > +		port->tx_fc = false;
> > +		mvpp2_rxq_disable_fc(port);
> > +		if (port->priv->percpu_pools) {
> > +			for (i = 0; i < port->nrxqs; i++)
> > +				mvpp2_bm_pool_update_fc(port, &port-
> >priv->bm_pools[i], false);
> > +		} else {
> > +			mvpp2_bm_pool_update_fc(port, port->pool_long,
> false);
> > +			mvpp2_bm_pool_update_fc(port, port->pool_short,
> false);
> > +		}
> > +	}
> 
> 
> This looks wrong. Flow control is normally the result of auto negotiation. Both
> ends need to agree to it. Which is why
> mvpp2_ethtool_set_pause_param() passes the users request onto phylink.
> phylink will handle the autoneg and then ask the MAC to setup flow control
> depending on the result in mvpp2_mac_link_up().
> 
> 	Andrew

Ok, I would move it to mvpp2_mac_link_up.

Stefan,
Thanks.

Re: [PATCH RFC net-next 03/19] net: mvpp2: add CM3 SRAM memory map

[Russell King - ARM Linux admin]

On Sun, Jan 10, 2021 at 05:30:07PM +0200, stefanc@marvell.com wrote:
> +	} else {
> +		priv->sram_pool = of_gen_pool_get(dn, "cm3-mem", 0);
> +		if (!priv->sram_pool) {
> +			dev_warn(&pdev->dev, "DT is too old, TX FC disabled\n");

I don't see anything in this patch that disables TX flow control, which
means this warning message is misleading.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

RE: [EXT] Re: [PATCH RFC net-next 03/19] net: mvpp2: add CM3 SRAM memory map

[Stefan Chulski]

> > +	} else {
> > +		priv->sram_pool = of_gen_pool_get(dn, "cm3-mem", 0);
> > +		if (!priv->sram_pool) {
> > +			dev_warn(&pdev->dev, "DT is too old, TX FC
> disabled\n");
> 
> I don't see anything in this patch that disables TX flow control, which means
> this warning message is misleading.

OK, I would change to TX FC not supported.

Stefan.

Re: [EXT] Re: [PATCH RFC net-next 03/19] net: mvpp2: add CM3 SRAM memory map

[Andrew Lunn]

On Sun, Jan 10, 2021 at 05:57:14PM +0000, Stefan Chulski wrote:
> > > +	} else {
> > > +		priv->sram_pool = of_gen_pool_get(dn, "cm3-mem", 0);
> > > +		if (!priv->sram_pool) {
> > > +			dev_warn(&pdev->dev, "DT is too old, TX FC
> > disabled\n");
> > 
> > I don't see anything in this patch that disables TX flow control, which means
> > this warning message is misleading.
> 
> OK, I would change to TX FC not supported.

And you should tell phlylink, so it knows to disable it in autoneg.

Which make me wonder, do we need a fix for stable? Has flow control
never been support in this device up until these patches get merged?
It should not be negotiated if it is not supported, which means
telling phylink.

   Andrew

Re: [PATCH RFC net-next 11/19] net: mvpp2: add flow control RXQ and BM pool config callbacks

[Russell King - ARM Linux admin]

On Sun, Jan 10, 2021 at 05:30:15PM +0200, stefanc@marvell.com wrote:
> From: Stefan Chulski <stefanc@marvell.com>
> 
> This patch did not change any functionality.
> Added flow control RXQ and BM pool config callbacks that would be
> used to configure RXQ and BM pool thresholds.
> APIs also will disable/enable RXQ and pool Flow Control polling.
> 
> In this stage BM pool and RXQ has same stop/start thresholds
> defined in code.
> Also there are common thresholds for all RXQs.
> 
> Signed-off-by: Stefan Chulski <stefanc@marvell.com>
> ---
>  drivers/net/ethernet/marvell/mvpp2/mvpp2.h      |  51 +++++-
>  drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 169 ++++++++++++++++++++
>  2 files changed, 216 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
> index 4d58af6..0ba0598 100644
> --- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
> +++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h
> @@ -763,10 +763,53 @@
>  		((kb) * 1024 - MVPP2_TX_FIFO_THRESHOLD_MIN)
>  
>  /* MSS Flow control */
> -#define MSS_SRAM_SIZE		0x800
> -#define FC_QUANTA		0xFFFF
> -#define FC_CLK_DIVIDER		0x140
> -#define MSS_THRESHOLD_STOP    768
> +#define MSS_SRAM_SIZE			0x800
> +#define MSS_FC_COM_REG			0
> +#define FLOW_CONTROL_ENABLE_BIT		BIT(0)
> +#define FLOW_CONTROL_UPDATE_COMMAND_BIT	BIT(31)
> +#define FC_QUANTA			0xFFFF
> +#define FC_CLK_DIVIDER			0x140
> +
> +#define MSS_BUF_POOL_BASE		0x40
> +#define MSS_BUF_POOL_OFFS		4
> +#define MSS_BUF_POOL_REG(id)		(MSS_BUF_POOL_BASE		\
> +					+ (id) * MSS_BUF_POOL_OFFS)
> +
> +#define MSS_BUF_POOL_STOP_MASK		0xFFF
> +#define MSS_BUF_POOL_START_MASK		(0xFFF << MSS_BUF_POOL_START_OFFS)
> +#define MSS_BUF_POOL_START_OFFS		12
> +#define MSS_BUF_POOL_PORTS_MASK		(0xF << MSS_BUF_POOL_PORTS_OFFS)
> +#define MSS_BUF_POOL_PORTS_OFFS		24
> +#define MSS_BUF_POOL_PORT_OFFS(id)	(0x1 <<				\
> +					((id) + MSS_BUF_POOL_PORTS_OFFS))
> +
> +#define MSS_RXQ_TRESH_BASE		0x200
> +#define MSS_RXQ_TRESH_OFFS		4
> +#define MSS_RXQ_TRESH_REG(q, fq)	(MSS_RXQ_TRESH_BASE + (((q) + (fq)) \
> +					* MSS_RXQ_TRESH_OFFS))
> +
> +#define MSS_RXQ_TRESH_START_MASK	0xFFFF
> +#define MSS_RXQ_TRESH_STOP_MASK		(0xFFFF << MSS_RXQ_TRESH_STOP_OFFS)
> +#define MSS_RXQ_TRESH_STOP_OFFS		16
> +
> +#define MSS_RXQ_ASS_BASE	0x80
> +#define MSS_RXQ_ASS_OFFS	4
> +#define MSS_RXQ_ASS_PER_REG	4
> +#define MSS_RXQ_ASS_PER_OFFS	8
> +#define MSS_RXQ_ASS_PORTID_OFFS	0
> +#define MSS_RXQ_ASS_PORTID_MASK	0x3
> +#define MSS_RXQ_ASS_HOSTID_OFFS	2
> +#define MSS_RXQ_ASS_HOSTID_MASK	0x3F
> +
> +#define MSS_RXQ_ASS_Q_BASE(q, fq) ((((q) + (fq)) % MSS_RXQ_ASS_PER_REG)	 \
> +				  * MSS_RXQ_ASS_PER_OFFS)
> +#define MSS_RXQ_ASS_PQ_BASE(q, fq) ((((q) + (fq)) / MSS_RXQ_ASS_PER_REG) \
> +				   * MSS_RXQ_ASS_OFFS)
> +#define MSS_RXQ_ASS_REG(q, fq) (MSS_RXQ_ASS_BASE + MSS_RXQ_ASS_PQ_BASE(q, fq))
> +
> +#define MSS_THRESHOLD_STOP	768
> +#define MSS_THRESHOLD_START	1024
> +
>  
>  /* RX buffer constants */
>  #define MVPP2_SKB_SHINFO_SIZE \
> diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
> index bc4b8069..19648c4 100644
> --- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
> +++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
> @@ -744,6 +744,175 @@ static void *mvpp2_buf_alloc(struct mvpp2_port *port,
>  	return data;
>  }
>  
> +/* Routine calculate single queue shares address space */
> +static int mvpp22_calc_shared_addr_space(struct mvpp2_port *port)
> +{
> +	/* If number of CPU's greater than number of threads, return last
> +	 * address space
> +	 */
> +	if (num_active_cpus() >= MVPP2_MAX_THREADS)
> +		return MVPP2_MAX_THREADS - 1;
> +
> +	return num_active_cpus();

Firstly - this can be written as:

	return min(num_active_cpus(), MVPP2_MAX_THREADS - 1);

Secondly - what if the number of active CPUs change, for example due
to hotplug activity. What if we boot with maxcpus=1 and then bring the
other CPUs online after networking has been started? The number of
active CPUs is dynamically managed via the scheduler as CPUs are
brought online or offline.

> +/* Routine enable flow control for RXQs conditon */
> +void mvpp2_rxq_enable_fc(struct mvpp2_port *port)
...
> +/* Routine disable flow control for RXQs conditon */
> +void mvpp2_rxq_disable_fc(struct mvpp2_port *port)

Nothing seems to call these in this patch, so on its own, it's not
obvious how these are being called, and therefore what remedy to
suggest for num_active_cpus().

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: [PATCH RFC net-next 12/19] net: mvpp2: enable global flow control

[Russell King - ARM Linux admin]

On Sun, Jan 10, 2021 at 05:30:16PM +0200, stefanc@marvell.com wrote:
> +		/* Enable global Flow Control only if hanler to SRAM not NULL */

I think this comment needs fixing. I'm not sure what a "hanler" is,
and "handler" doesn't make sense here.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

RE: [EXT] Re: [PATCH RFC net-next 03/19] net: mvpp2: add CM3 SRAM memory map

[Stefan Chulski]

> > > > +	} else {
> > > > +		priv->sram_pool = of_gen_pool_get(dn, "cm3-mem", 0);
> > > > +		if (!priv->sram_pool) {
> > > > +			dev_warn(&pdev->dev, "DT is too old, TX FC
> > > disabled\n");
> > >
> > > I don't see anything in this patch that disables TX flow control,
> > > which means this warning message is misleading.
> >
> > OK, I would change to TX FC not supported.
> 
> And you should tell phlylink, so it knows to disable it in autoneg.
> 
> Which make me wonder, do we need a fix for stable? Has flow control never
> been support in this device up until these patches get merged?
> It should not be negotiated if it is not supported, which means telling phylink.
> 
>    Andrew

TX FC never were really supported. MAC or PHY can negotiated flow control.
But MAC would never trigger FC frame.

Should I prepare separate patch that disable TX FC till we merge this patches?

Regards,
Stefan.

Re: [PATCH RFC net-next 14/19] net: mvpp2: add ethtool flow control configuration support

[Russell King - ARM Linux admin]

On Sun, Jan 10, 2021 at 05:30:18PM +0200, stefanc@marvell.com wrote:
> @@ -5373,6 +5402,30 @@ static int mvpp2_ethtool_set_pause_param(struct net_device *dev,
>  					 struct ethtool_pauseparam *pause)
>  {
>  	struct mvpp2_port *port = netdev_priv(dev);
> +	int i;
> +
> +	if (pause->tx_pause && port->priv->global_tx_fc) {
> +		port->tx_fc = true;
> +		mvpp2_rxq_enable_fc(port);
> +		if (port->priv->percpu_pools) {
> +			for (i = 0; i < port->nrxqs; i++)
> +				mvpp2_bm_pool_update_fc(port, &port->priv->bm_pools[i], true);
> +		} else {
> +			mvpp2_bm_pool_update_fc(port, port->pool_long, true);
> +			mvpp2_bm_pool_update_fc(port, port->pool_short, true);
> +		}
> +
> +	} else if (port->priv->global_tx_fc) {
> +		port->tx_fc = false;
> +		mvpp2_rxq_disable_fc(port);
> +		if (port->priv->percpu_pools) {
> +			for (i = 0; i < port->nrxqs; i++)
> +				mvpp2_bm_pool_update_fc(port, &port->priv->bm_pools[i], false);
> +		} else {
> +			mvpp2_bm_pool_update_fc(port, port->pool_long, false);
> +			mvpp2_bm_pool_update_fc(port, port->pool_short, false);
> +		}
> +	}

This doesn't look correct to me. This function is only called when
ethtool -A is used to change the flow control settings. This is not
the place to be configuring flow control, as flow control is
negotiated with the link partner.

The final resolved flow control settings are available in
mvpp2_mac_link_up() via the tx_pause and rx_pause parameters.

What also concerns me is whether flow control is supported in the
existing driver at all, given this patch set. If it isn't supported
without the firmware's help, then we should _not_ be negotiating flow
control with the link partner unless we actually support it, so the
Pause and Asym_Pause bits in mvpp2_phylink_validate() should be
cleared.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: [PATCH RFC net-next 00/19] net: mvpp2: Add TX Flow Control support

[Russell King - ARM Linux admin]

Hi,

On Sun, Jan 10, 2021 at 05:30:04PM +0200, stefanc@marvell.com wrote:
> Armada hardware has a pause generation mechanism in GOP (MAC).
> GOP has to generate flow control frames based on an indication
> programmed in Ports Control 0 Register. There is a bit per port.
> However assertion of the PortX Pause bits in the ports control 0 register
> only sends a one time pause. To complement the function the GOP has
> a mechanism to periodically send pause control messages based on periodic counters.
> This mechanism ensures that the pause is effective as long as the Appropriate PortX Pause
> is asserted.

Can you ensure that your commit messages are consistently wrapped?
Some lines are wrapped others aren't.

> Problem is that Packet Processor witch actually can drop packets due to lack of resources

Does the packet processor engage in magic or witchcraft? I suppose
some would argue that firmware does "magic"! However, I think you
mean "which". :)

> not connected to the GOP flow control generation mechanism.
> To solve this issue Armada has firmware running on CM3 CPU dedectated for Flow Control
> support. Firmware monitors Packet Processor resources and asserts XON/XOFF by writing
> to Ports Control 0 Register.

What is the minimum firmware version that supports this?

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

RE: [EXT] Re: [PATCH RFC net-next 11/19] net: mvpp2: add flow control RXQ and BM pool config callbacks

[Stefan Chulski]

> >
> > +/* Routine calculate single queue shares address space */ static int
> > +mvpp22_calc_shared_addr_space(struct mvpp2_port *port) {
> > +	/* If number of CPU's greater than number of threads, return last
> > +	 * address space
> > +	 */
> > +	if (num_active_cpus() >= MVPP2_MAX_THREADS)
> > +		return MVPP2_MAX_THREADS - 1;
> > +
> > +	return num_active_cpus();
> 
> Firstly - this can be written as:
> 
> 	return min(num_active_cpus(), MVPP2_MAX_THREADS - 1);

OK.

> Secondly - what if the number of active CPUs change, for example due to
> hotplug activity. What if we boot with maxcpus=1 and then bring the other
> CPUs online after networking has been started? The number of active CPUs is
> dynamically managed via the scheduler as CPUs are brought online or offline.
> 
> > +/* Routine enable flow control for RXQs conditon */ void
> > +mvpp2_rxq_enable_fc(struct mvpp2_port *port)
> ...
> > +/* Routine disable flow control for RXQs conditon */ void
> > +mvpp2_rxq_disable_fc(struct mvpp2_port *port)
> 
> Nothing seems to call these in this patch, so on its own, it's not obvious how
> these are being called, and therefore what remedy to suggest for
> num_active_cpus().

I don't think that current driver support CPU hotplug, anyway I can remove  num_active_cpus
and just use shared RX IRQ ID.

Thanks.
. 

Re: [EXT] Re: [PATCH RFC net-next 03/19] net: mvpp2: add CM3 SRAM memory map

[Russell King - ARM Linux admin]

On Sun, Jan 10, 2021 at 06:09:39PM +0000, Stefan Chulski wrote:
> > > > > +	} else {
> > > > > +		priv->sram_pool = of_gen_pool_get(dn, "cm3-mem", 0);
> > > > > +		if (!priv->sram_pool) {
> > > > > +			dev_warn(&pdev->dev, "DT is too old, TX FC
> > > > disabled\n");
> > > >
> > > > I don't see anything in this patch that disables TX flow control,
> > > > which means this warning message is misleading.
> > >
> > > OK, I would change to TX FC not supported.
> > 
> > And you should tell phlylink, so it knows to disable it in autoneg.
> > 
> > Which make me wonder, do we need a fix for stable? Has flow control never
> > been support in this device up until these patches get merged?
> > It should not be negotiated if it is not supported, which means telling phylink.
> > 
> >    Andrew
> 
> TX FC never were really supported. MAC or PHY can negotiated flow control.
> But MAC would never trigger FC frame.

That really sucks.

> Should I prepare separate patch that disable TX FC till we merge this patches?

From what I see in table 28B in 802.3, there is no way to advertise
that you only support RX flow control. If you advertise ASM_DIR=1
PAUSE=0, it basically means you support sending FC frames, but not
receiving them. Advertising anything with PAUSE=1 means you support
both sending and receiving FC frames, irrespective of the state of
ASM_DIR.

So, our only option would be to completely disable pause frames.
Yes, I think we need a separate patch for that for the net tree,
and it should be backported to stable kernels, IMHO.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

RE: [EXT] Re: [PATCH RFC net-next 14/19] net: mvpp2: add ethtool flow control configuration support

[Stefan Chulski]

> > @@ -5373,6 +5402,30 @@ static int
> mvpp2_ethtool_set_pause_param(struct net_device *dev,
> >  					 struct ethtool_pauseparam *pause)  {
> >  	struct mvpp2_port *port = netdev_priv(dev);
> > +	int i;
> > +
> > +	if (pause->tx_pause && port->priv->global_tx_fc) {
> > +		port->tx_fc = true;
> > +		mvpp2_rxq_enable_fc(port);
> > +		if (port->priv->percpu_pools) {
> > +			for (i = 0; i < port->nrxqs; i++)
> > +				mvpp2_bm_pool_update_fc(port, &port-
> >priv->bm_pools[i], true);
> > +		} else {
> > +			mvpp2_bm_pool_update_fc(port, port->pool_long,
> true);
> > +			mvpp2_bm_pool_update_fc(port, port->pool_short,
> true);
> > +		}
> > +
> > +	} else if (port->priv->global_tx_fc) {
> > +		port->tx_fc = false;
> > +		mvpp2_rxq_disable_fc(port);
> > +		if (port->priv->percpu_pools) {
> > +			for (i = 0; i < port->nrxqs; i++)
> > +				mvpp2_bm_pool_update_fc(port, &port-
> >priv->bm_pools[i], false);
> > +		} else {
> > +			mvpp2_bm_pool_update_fc(port, port->pool_long,
> false);
> > +			mvpp2_bm_pool_update_fc(port, port->pool_short,
> false);
> > +		}
> > +	}
> 
> This doesn't look correct to me. This function is only called when ethtool -A is
> used to change the flow control settings. This is not the place to be
> configuring flow control, as flow control is negotiated with the link partner.
> 
> The final resolved flow control settings are available in
> mvpp2_mac_link_up() via the tx_pause and rx_pause parameters.

I would move this to mvpp2_mac_link_up.
Thanks.
 
> What also concerns me is whether flow control is supported in the existing
> driver at all, given this patch set. If it isn't supported without the firmware's
> help, then we should _not_ be negotiating flow control with the link partner
> unless we actually support it, so the Pause and Asym_Pause bits in
> mvpp2_phylink_validate() should be cleared.

RX FC supported, issue only with TX FC.

Stefan,
Regards.

Re: [EXT] Re: [PATCH RFC net-next 11/19] net: mvpp2: add flow control RXQ and BM pool config callbacks

[Russell King - ARM Linux admin]

On Sun, Jan 10, 2021 at 06:24:30PM +0000, Stefan Chulski wrote:
> > >
> > > +/* Routine calculate single queue shares address space */ static int
> > > +mvpp22_calc_shared_addr_space(struct mvpp2_port *port) {
> > > +	/* If number of CPU's greater than number of threads, return last
> > > +	 * address space
> > > +	 */
> > > +	if (num_active_cpus() >= MVPP2_MAX_THREADS)
> > > +		return MVPP2_MAX_THREADS - 1;
> > > +
> > > +	return num_active_cpus();
> > 
> > Firstly - this can be written as:
> > 
> > 	return min(num_active_cpus(), MVPP2_MAX_THREADS - 1);
> 
> OK.
> 
> > Secondly - what if the number of active CPUs change, for example due to
> > hotplug activity. What if we boot with maxcpus=1 and then bring the other
> > CPUs online after networking has been started? The number of active CPUs is
> > dynamically managed via the scheduler as CPUs are brought online or offline.
> > 
> > > +/* Routine enable flow control for RXQs conditon */ void
> > > +mvpp2_rxq_enable_fc(struct mvpp2_port *port)
> > ...
> > > +/* Routine disable flow control for RXQs conditon */ void
> > > +mvpp2_rxq_disable_fc(struct mvpp2_port *port)
> > 
> > Nothing seems to call these in this patch, so on its own, it's not obvious how
> > these are being called, and therefore what remedy to suggest for
> > num_active_cpus().
> 
> I don't think that current driver support CPU hotplug, anyway I can
> remove  num_active_cpus and just use shared RX IRQ ID.

Sorry, but that is not really a decision the driver can make. It is
part of a kernel that _does_ support CPU hotplug, and the online
CPUs can be changed today.

It is likely that every distro out there builds the kernel with
CPU hotplug enabled.

If changing the online CPUs causes the driver to misbehave, that
is a(nother) bug with the driver.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: [EXT] Re: [PATCH RFC net-next 14/19] net: mvpp2: add ethtool flow control configuration support

[Russell King - ARM Linux admin]

On Sun, Jan 10, 2021 at 06:27:57PM +0000, Stefan Chulski wrote:
> > What also concerns me is whether flow control is supported in the existing
> > driver at all, given this patch set. If it isn't supported without the firmware's
> > help, then we should _not_ be negotiating flow control with the link partner
> > unless we actually support it, so the Pause and Asym_Pause bits in
> > mvpp2_phylink_validate() should be cleared.
> 
> RX FC supported, issue only with TX FC.

That doesn't seem relevant given table 28B in IEEE 802.3. There is
no advertisement combination that allows one to advertise an ability
to receive FC frames but not transmit FC frames.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

RE: [EXT] Re: [PATCH RFC net-next 11/19] net: mvpp2: add flow control RXQ and BM pool config callbacks

[Stefan Chulski]

> Sorry, but that is not really a decision the driver can make. It is part of a kernel
> that _does_ support CPU hotplug, and the online CPUs can be changed today.
> 
> It is likely that every distro out there builds the kernel with CPU hotplug
> enabled.
> 
> If changing the online CPUs causes the driver to misbehave, that is a(nother)
> bug with the driver.

This function doesn't really need to know num_active_cpus, only host ID used by
used by shared RX interrupt in single queue mode.
Host ID is just register address space used to access PPv2 register space.
So I can remove this use of  num_active_cpus.

Stefan,
Regards.

RE: [EXT] Re: [PATCH RFC net-next 00/19] net: mvpp2: Add TX Flow Control support

[Stefan Chulski]

> > not connected to the GOP flow control generation mechanism.
> > To solve this issue Armada has firmware running on CM3 CPU dedectated
> > for Flow Control support. Firmware monitors Packet Processor resources
> > and asserts XON/XOFF by writing to Ports Control 0 Register.
> 
> What is the minimum firmware version that supports this?
> 

Support were added to firmware about two years ago. 
All releases from 18.09 should has it.

Stefan,
Regards.

Re: [EXT] Re: [PATCH RFC net-next 00/19] net: mvpp2: Add TX Flow Control support

[Russell King - ARM Linux admin]

On Sun, Jan 10, 2021 at 06:55:11PM +0000, Stefan Chulski wrote:
> > > not connected to the GOP flow control generation mechanism.
> > > To solve this issue Armada has firmware running on CM3 CPU dedectated
> > > for Flow Control support. Firmware monitors Packet Processor resources
> > > and asserts XON/XOFF by writing to Ports Control 0 Register.
> > 
> > What is the minimum firmware version that supports this?
> > 
> 
> Support were added to firmware about two years ago. 
> All releases from 18.09 should has it.

Please add that vital bit of information somewhere appropriate.
I would not be surprised if people are still running e.g. 17.10
on some of their Armada 8040 boards. My Macchiatobin which is
acting as a server currently has 17.10, although I plan to upgrade
it to 18.12 in about three to six months time, once I've well and
truely proven that my ext4 problems are resolved.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: [EXT] Re: [PATCH RFC net-next 00/19] net: mvpp2: Add TX Flow Control support

[Andrew Lunn]

On Sun, Jan 10, 2021 at 06:55:11PM +0000, Stefan Chulski wrote:
> > > not connected to the GOP flow control generation mechanism.
> > > To solve this issue Armada has firmware running on CM3 CPU dedectated
> > > for Flow Control support. Firmware monitors Packet Processor resources
> > > and asserts XON/XOFF by writing to Ports Control 0 Register.
> > 
> > What is the minimum firmware version that supports this?
> > 
> 
> Support were added to firmware about two years ago. 
> All releases from 18.09 should has it.

Can you query the firmware and ask its version? We should keep all
this code disabled if the firmware it too old.

     Andrew

RE: [EXT] Re: [PATCH RFC net-next 00/19] net: mvpp2: Add TX Flow Control support

[Stefan Chulski]

> 
> On Sun, Jan 10, 2021 at 06:55:11PM +0000, Stefan Chulski wrote:
> > > > not connected to the GOP flow control generation mechanism.
> > > > To solve this issue Armada has firmware running on CM3 CPU
> > > > dedectated for Flow Control support. Firmware monitors Packet
> > > > Processor resources and asserts XON/XOFF by writing to Ports Control 0
> Register.
> > >
> > > What is the minimum firmware version that supports this?
> > >
> >
> > Support were added to firmware about two years ago.
> > All releases from 18.09 should has it.
> 
> Can you query the firmware and ask its version? We should keep all this code
> disabled if the firmware it too old.
> 
>      Andrew

This is exactly what " net: mvpp2: add TX FC firmware check " patch do. If handshake of flow control support fail, FC won't be supported.

Stefan. 

Re: [EXT] Re: [PATCH RFC net-next 00/19] net: mvpp2: Add TX Flow Control support

[Andrew Lunn]

On Sun, Jan 10, 2021 at 07:11:53PM +0000, Stefan Chulski wrote:
> > 
> > On Sun, Jan 10, 2021 at 06:55:11PM +0000, Stefan Chulski wrote:
> > > > > not connected to the GOP flow control generation mechanism.
> > > > > To solve this issue Armada has firmware running on CM3 CPU
> > > > > dedectated for Flow Control support. Firmware monitors Packet
> > > > > Processor resources and asserts XON/XOFF by writing to Ports Control 0
> > Register.
> > > >
> > > > What is the minimum firmware version that supports this?
> > > >
> > >
> > > Support were added to firmware about two years ago.
> > > All releases from 18.09 should has it.
> > 
> > Can you query the firmware and ask its version? We should keep all this code
> > disabled if the firmware it too old.
> > 
> >      Andrew
> 

> This is exactly what " net: mvpp2: add TX FC firmware check " patch
> do. If handshake of flow control support fail, FC won't be
> supported.

Ah, O.K. Please extend the kernel log message to include the minimum
firmware version.

	 Andrew