[PATCH 0/3] Some misc fixes and optimization for the mpipe driver

[PATCH 0/3] Some misc fixes and optimization for the mpipe driver

[Liming Sun]

This patch serie includes some fixes/enhancements to the mpipe driver.
1. Support native build on the TileGx platform;
   Previously only cross build was supported.

2. Optimize the mpipe buffer return mechanism by introducing a return
   pending counter and doing HW buffer return if pssible.

3. Move the mpipe link initialization to an earlier place to support
   link management APIs.

Liming Sun (3):
  driver/net/mpipe: support native build on tilegx platform.
  driver/net/mpipe: optimize mpipe buffer return mechanism.
  driver/net/mpipe: fix a mpipe link initialization ordering issue

 MAINTAINERS                               |    3 +-
 config/defconfig_tile-tilegx-linuxapp-gcc |    4 ++
 drivers/net/mpipe/mpipe_tilegx.c          |   64 +++++++++++++++++++++--------
 mk/arch/tile/rte.vars.mk                  |    6 +++
 4 files changed, 59 insertions(+), 18 deletions(-)

[PATCH 1/3] driver/net/mpipe: support native build on tilegx platform.

[Liming Sun]

This submit updates the CROSS setting to support native build on
TileGx platform. It also enable the combined library by default.

Signed-off-by: Liming Sun <lsun@ezchip.com>
---
 MAINTAINERS                               |    3 ++-
 config/defconfig_tile-tilegx-linuxapp-gcc |    4 ++++
 mk/arch/tile/rte.vars.mk                  |    6 ++++++
 3 files changed, 12 insertions(+), 1 deletions(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index 3292e84..8f7e9ca 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -138,8 +138,9 @@ M: Jianbo Liu <jianbo.liu@linaro.org>
 F: lib/librte_eal/common/include/arch/arm/*_64.h
 F: lib/librte_acl/acl_run_neon.*
 
-EZchip TILE-Gx
+EZchip TILE-Gx/Mx
 M: Zhigang Lu <zlu@ezchip.com>
+M: Liming Sun <lsun@ezchip.com>
 F: lib/librte_eal/common/include/arch/tile/
 F: drivers/net/mpipe/
 
diff --git a/config/defconfig_tile-tilegx-linuxapp-gcc b/config/defconfig_tile-tilegx-linuxapp-gcc
index 9df9d7f..fb61bcd 100644
--- a/config/defconfig_tile-tilegx-linuxapp-gcc
+++ b/config/defconfig_tile-tilegx-linuxapp-gcc
@@ -70,3 +70,7 @@ CONFIG_RTE_LIBRTE_SCHED=n
 CONFIG_RTE_LIBRTE_PORT=n
 CONFIG_RTE_LIBRTE_TABLE=n
 CONFIG_RTE_LIBRTE_PIPELINE=n
+CONFIG_RTE_LIBRTE_CXGBE_PMD=n
+
+# Compile combined lib by default.
+CONFIG_RTE_BUILD_COMBINE_LIBS=y
diff --git a/mk/arch/tile/rte.vars.mk b/mk/arch/tile/rte.vars.mk
index b518986..06dab18 100644
--- a/mk/arch/tile/rte.vars.mk
+++ b/mk/arch/tile/rte.vars.mk
@@ -30,7 +30,13 @@
 
 
 ARCH  ?= tile
+
+HOST_ARCH := ${shell uname -m}
+ifneq ($(filter tile%,${HOST_ARCH}),)
+CROSS =
+else
 CROSS ?= tile-
+endif
 
 CPU_CFLAGS  ?=
 CPU_LDFLAGS ?=
-- 
1.7.1

Re: [PATCH 1/3] driver/net/mpipe: support native build on tilegx platform.

[Tony Lu]

>-----Original Message-----
>From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Liming Sun
>Sent: Tuesday, December 15, 2015 11:37 PM
>To: dev@dpdk.org
>Subject: [dpdk-dev] [PATCH 1/3] driver/net/mpipe: support native build on
tilegx
>platform.
>
>This submit updates the CROSS setting to support native build on TileGx
>platform. It also enable the combined library by default.
>
>Signed-off-by: Liming Sun <lsun@ezchip.com>
>---
> MAINTAINERS                               |    3 ++-
> config/defconfig_tile-tilegx-linuxapp-gcc |    4 ++++
> mk/arch/tile/rte.vars.mk                  |    6 ++++++
> 3 files changed, 12 insertions(+), 1 deletions(-)
>
>diff --git a/MAINTAINERS b/MAINTAINERS
>index 3292e84..8f7e9ca 100644
>--- a/MAINTAINERS
>+++ b/MAINTAINERS
>@@ -138,8 +138,9 @@ M: Jianbo Liu <jianbo.liu@linaro.org>
> F: lib/librte_eal/common/include/arch/arm/*_64.h
> F: lib/librte_acl/acl_run_neon.*
>
>-EZchip TILE-Gx
>+EZchip TILE-Gx/Mx
> M: Zhigang Lu <zlu@ezchip.com>
>+M: Liming Sun <lsun@ezchip.com>
> F: lib/librte_eal/common/include/arch/tile/
> F: drivers/net/mpipe/
>
>diff --git a/config/defconfig_tile-tilegx-linuxapp-gcc
>b/config/defconfig_tile-tilegx-linuxapp-gcc
>index 9df9d7f..fb61bcd 100644
>--- a/config/defconfig_tile-tilegx-linuxapp-gcc
>+++ b/config/defconfig_tile-tilegx-linuxapp-gcc
>@@ -70,3 +70,7 @@ CONFIG_RTE_LIBRTE_SCHED=n
>CONFIG_RTE_LIBRTE_PORT=n  CONFIG_RTE_LIBRTE_TABLE=n
>CONFIG_RTE_LIBRTE_PIPELINE=n
>+CONFIG_RTE_LIBRTE_CXGBE_PMD=n
>+
>+# Compile combined lib by default.
>+CONFIG_RTE_BUILD_COMBINE_LIBS=y
>diff --git a/mk/arch/tile/rte.vars.mk b/mk/arch/tile/rte.vars.mk index
>b518986..06dab18 100644
>--- a/mk/arch/tile/rte.vars.mk
>+++ b/mk/arch/tile/rte.vars.mk
>@@ -30,7 +30,13 @@
>
>
> ARCH  ?= tile
>+
>+HOST_ARCH := ${shell uname -m}
>+ifneq ($(filter tile%,${HOST_ARCH}),)
>+CROSS =
>+else
> CROSS ?= tile-
>+endif
>
> CPU_CFLAGS  ?=
> CPU_LDFLAGS ?=
>--
>1.7.1

Acked-by: Zhigang Lu <zlu@ezchip.com>

[PATCH 2/3] driver/net/mpipe: optimize mpipe buffer return mechanism.

[Liming Sun]

This submit has changes to optimize the mpipe buffer return. When
a packet is received, instead of allocating and refilling the
buffer stack right away, it tracks the number of pending buffers,
and use HW buffer return as an optimization when the pending
number is below certain threshold, thus save two MMIO writes and
improves performance especially for bidirectional traffic case.

Signed-off-by: Liming Sun <lsun@ezchip.com>
---
 drivers/net/mpipe/mpipe_tilegx.c |   50 ++++++++++++++++++++++++++++++-------
 1 files changed, 40 insertions(+), 10 deletions(-)

diff --git a/drivers/net/mpipe/mpipe_tilegx.c b/drivers/net/mpipe/mpipe_tilegx.c
index 35134ba..be7b6f2 100644
--- a/drivers/net/mpipe/mpipe_tilegx.c
+++ b/drivers/net/mpipe/mpipe_tilegx.c
@@ -78,6 +78,13 @@ struct mpipe_context {
 	struct mpipe_channel_config channels[MPIPE_MAX_CHANNELS];
 };
 
+/* Per-core local data. */
+struct mpipe_local {
+	int mbuf_push_debt[RTE_MAX_ETHPORTS];	/* Buffer push debt. */
+} __rte_cache_aligned;
+
+#define MPIPE_BUF_DEBT_THRESHOLD	32
+static __thread struct mpipe_local mpipe_local;
 static struct mpipe_context mpipe_contexts[GXIO_MPIPE_INSTANCE_MAX];
 static int mpipe_instances;
 static const char *drivername = "MPIPE PMD";
@@ -137,7 +144,7 @@ struct mpipe_dev_priv {
 	int first_bucket;		/* mPIPE bucket start index. */
 	int first_ring;			/* mPIPE notif ring start index. */
 	int notif_group;		/* mPIPE notif group. */
-	rte_atomic32_t dp_count;	/* Active datapath thread count. */
+	rte_atomic32_t dp_count __rte_cache_aligned;	/* DP Entry count. */
 	int tx_stat_mapping[RTE_ETHDEV_QUEUE_STAT_CNTRS];
 	int rx_stat_mapping[RTE_ETHDEV_QUEUE_STAT_CNTRS];
 };
@@ -461,6 +468,14 @@ mpipe_dp_wait(struct mpipe_dev_priv *priv)
 	}
 }
 
+static inline int
+mpipe_mbuf_stack_index(struct mpipe_dev_priv *priv, struct rte_mbuf *mbuf)
+{
+	return (mbuf->port < RTE_MAX_ETHPORTS)?
+		mpipe_priv(&rte_eth_devices[mbuf->port])->stack :
+		priv->stack;
+}
+
 static inline struct rte_mbuf *
 mpipe_recv_mbuf(struct mpipe_dev_priv *priv, gxio_mpipe_idesc_t *idesc,
 		int in_port)
@@ -1267,6 +1282,7 @@ mpipe_do_xmit(struct mpipe_tx_queue *tx_queue, struct rte_mbuf **tx_pkts,
 	unsigned nb_bytes = 0;
 	unsigned nb_sent = 0;
 	int nb_slots, i;
+	uint8_t port_id;
 
 	PMD_DEBUG_TX("Trying to transmit %d packets on %s:%d.\n",
 		     nb_pkts, mpipe_name(tx_queue->q.priv),
@@ -1315,14 +1331,23 @@ mpipe_do_xmit(struct mpipe_tx_queue *tx_queue, struct rte_mbuf **tx_pkts,
 			if (priv->tx_comps[idx])
 				rte_pktmbuf_free_seg(priv->tx_comps[idx]);
 
+			port_id = (mbuf->port < RTE_MAX_ETHPORTS)?
+						mbuf->port : priv->port_id;
 			desc = (gxio_mpipe_edesc_t) { {
 				.va        = rte_pktmbuf_mtod(mbuf, uintptr_t),
 				.xfer_size = rte_pktmbuf_data_len(mbuf),
 				.bound     = next ? 0 : 1,
+				.stack_idx = mpipe_mbuf_stack_index(priv, mbuf),
 			} };
+			if (mpipe_local.mbuf_push_debt[port_id] > 0) {
+				mpipe_local.mbuf_push_debt[port_id]--;
+				desc.hwb = 1;
+				priv->tx_comps[idx] = NULL;
+			}
+			else
+				priv->tx_comps[idx] = mbuf;
 
 			nb_bytes += mbuf->data_len;
-			priv->tx_comps[idx] = mbuf;
 			gxio_mpipe_equeue_put_at(equeue, desc, slot + i);
 
 			PMD_DEBUG_TX("%s:%d: Sending packet %p, len %d\n",
@@ -1443,17 +1468,22 @@ mpipe_do_recv(struct mpipe_rx_queue *rx_queue, struct rte_mbuf **rx_pkts,
 				continue;
 			}
 
-			mbuf = __rte_mbuf_raw_alloc(priv->rx_mpool);
-			if (unlikely(!mbuf)) {
-				nb_nomem++;
-				gxio_mpipe_iqueue_drop(iqueue, idesc);
-				PMD_DEBUG_RX("%s:%d: RX alloc failure\n",
+			if (mpipe_local.mbuf_push_debt[in_port] <
+					MPIPE_BUF_DEBT_THRESHOLD)
+				mpipe_local.mbuf_push_debt[in_port]++;
+			else {
+				mbuf = __rte_mbuf_raw_alloc(priv->rx_mpool);
+				if (unlikely(!mbuf)) {
+					nb_nomem++;
+					gxio_mpipe_iqueue_drop(iqueue, idesc);
+					PMD_DEBUG_RX("%s:%d: alloc failure\n",
 					     mpipe_name(rx_queue->q.priv),
 					     rx_queue->q.queue_idx);
-				continue;
-			}
+					continue;
+				}
 
-			mpipe_recv_push(priv, mbuf);
+				mpipe_recv_push(priv, mbuf);
+			}
 
 			/* Get and setup the mbuf for the received packet. */
 			mbuf = mpipe_recv_mbuf(priv, idesc, in_port);
-- 
1.7.1

Re: [PATCH 2/3] driver/net/mpipe: optimize mpipe buffer return mechanism.

[Tony Lu]

>-----Original Message-----
>From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Liming Sun
>Sent: Tuesday, December 15, 2015 11:38 PM
>To: dev@dpdk.org
>Subject: [dpdk-dev] [PATCH 2/3] driver/net/mpipe: optimize mpipe buffer
return
>mechanism.
>
>This submit has changes to optimize the mpipe buffer return. When
>a packet is received, instead of allocating and refilling the
>buffer stack right away, it tracks the number of pending buffers,
>and use HW buffer return as an optimization when the pending
>number is below certain threshold, thus save two MMIO writes and
>improves performance especially for bidirectional traffic case.
>
>Signed-off-by: Liming Sun <lsun@ezchip.com>
>---
> drivers/net/mpipe/mpipe_tilegx.c |   50
>++++++++++++++++++++++++++++++-------
> 1 files changed, 40 insertions(+), 10 deletions(-)
>
>diff --git a/drivers/net/mpipe/mpipe_tilegx.c
>b/drivers/net/mpipe/mpipe_tilegx.c
>index 35134ba..be7b6f2 100644
>--- a/drivers/net/mpipe/mpipe_tilegx.c
>+++ b/drivers/net/mpipe/mpipe_tilegx.c
>@@ -78,6 +78,13 @@ struct mpipe_context {
> 	struct mpipe_channel_config channels[MPIPE_MAX_CHANNELS];
> };
>
>+/* Per-core local data. */
>+struct mpipe_local {
>+	int mbuf_push_debt[RTE_MAX_ETHPORTS];	/* Buffer push debt. */
>+} __rte_cache_aligned;
>+
>+#define MPIPE_BUF_DEBT_THRESHOLD	32
>+static __thread struct mpipe_local mpipe_local;
> static struct mpipe_context mpipe_contexts[GXIO_MPIPE_INSTANCE_MAX];
> static int mpipe_instances;
> static const char *drivername = "MPIPE PMD";
>@@ -137,7 +144,7 @@ struct mpipe_dev_priv {
> 	int first_bucket;		/* mPIPE bucket start index. */
> 	int first_ring;			/* mPIPE notif ring start index. */
> 	int notif_group;		/* mPIPE notif group. */
>-	rte_atomic32_t dp_count;	/* Active datapath thread count. */
>+	rte_atomic32_t dp_count __rte_cache_aligned;	/* DP Entry count.
*/
> 	int tx_stat_mapping[RTE_ETHDEV_QUEUE_STAT_CNTRS];
> 	int rx_stat_mapping[RTE_ETHDEV_QUEUE_STAT_CNTRS];
> };
>@@ -461,6 +468,14 @@ mpipe_dp_wait(struct mpipe_dev_priv *priv)
> 	}
> }
>
>+static inline int
>+mpipe_mbuf_stack_index(struct mpipe_dev_priv *priv, struct rte_mbuf *mbuf)
>+{
>+	return (mbuf->port < RTE_MAX_ETHPORTS)?
>+		mpipe_priv(&rte_eth_devices[mbuf->port])->stack :
>+		priv->stack;
>+}
>+
> static inline struct rte_mbuf *
> mpipe_recv_mbuf(struct mpipe_dev_priv *priv, gxio_mpipe_idesc_t *idesc,
> 		int in_port)
>@@ -1267,6 +1282,7 @@ mpipe_do_xmit(struct mpipe_tx_queue *tx_queue,
>struct rte_mbuf **tx_pkts,
> 	unsigned nb_bytes = 0;
> 	unsigned nb_sent = 0;
> 	int nb_slots, i;
>+	uint8_t port_id;
>
> 	PMD_DEBUG_TX("Trying to transmit %d packets on %s:%d.\n",
> 		     nb_pkts, mpipe_name(tx_queue->q.priv),
>@@ -1315,14 +1331,23 @@ mpipe_do_xmit(struct mpipe_tx_queue *tx_queue,
>struct rte_mbuf **tx_pkts,
> 			if (priv->tx_comps[idx])
> 				rte_pktmbuf_free_seg(priv->tx_comps[idx]);
>
>+			port_id = (mbuf->port < RTE_MAX_ETHPORTS)?
>+						mbuf->port : priv->port_id;
> 			desc = (gxio_mpipe_edesc_t) { {
> 				.va        = rte_pktmbuf_mtod(mbuf,
uintptr_t),
> 				.xfer_size = rte_pktmbuf_data_len(mbuf),
> 				.bound     = next ? 0 : 1,
>+				.stack_idx = mpipe_mbuf_stack_index(priv,
mbuf),
> 			} };
>+			if (mpipe_local.mbuf_push_debt[port_id] > 0) {
>+				mpipe_local.mbuf_push_debt[port_id]--;
>+				desc.hwb = 1;
>+				priv->tx_comps[idx] = NULL;
>+			}
>+			else
>+				priv->tx_comps[idx] = mbuf;
>
> 			nb_bytes += mbuf->data_len;
>-			priv->tx_comps[idx] = mbuf;
> 			gxio_mpipe_equeue_put_at(equeue, desc, slot + i);
>
> 			PMD_DEBUG_TX("%s:%d: Sending packet %p, len %d\n",
>@@ -1443,17 +1468,22 @@ mpipe_do_recv(struct mpipe_rx_queue *rx_queue,
>struct rte_mbuf **rx_pkts,
> 				continue;
> 			}
>
>-			mbuf = __rte_mbuf_raw_alloc(priv->rx_mpool);
>-			if (unlikely(!mbuf)) {
>-				nb_nomem++;
>-				gxio_mpipe_iqueue_drop(iqueue, idesc);
>-				PMD_DEBUG_RX("%s:%d: RX alloc failure\n",
>+			if (mpipe_local.mbuf_push_debt[in_port] <
>+					MPIPE_BUF_DEBT_THRESHOLD)
>+				mpipe_local.mbuf_push_debt[in_port]++;
>+			else {
>+				mbuf = __rte_mbuf_raw_alloc(priv->rx_mpool);
>+				if (unlikely(!mbuf)) {
>+					nb_nomem++;
>+					gxio_mpipe_iqueue_drop(iqueue,
idesc);
>+					PMD_DEBUG_RX("%s:%d: alloc
failure\n",
> 					     mpipe_name(rx_queue->q.priv),
> 					     rx_queue->q.queue_idx);
>-				continue;
>-			}
>+					continue;
>+				}
>
>-			mpipe_recv_push(priv, mbuf);
>+				mpipe_recv_push(priv, mbuf);
>+			}
>
> 			/* Get and setup the mbuf for the received packet.
*/
> 			mbuf = mpipe_recv_mbuf(priv, idesc, in_port);
>--
>1.7.1

Acked-by: Zhigang Lu <zlu@ezchip.com>

[PATCH 3/3] driver/net/mpipe: fix a mpipe link initialization ordering issue

[Liming Sun]

Mpipe link structure is initialized in function mpipe_link_init().
Currently it's only called from the eth_dev_ops.dev_start, which
caused crashes when link mgmt APIs (like promiscuous_enable)
was called before eth_dev_ops.dev_start(). This submit fixed it
by calling mpipe_link_init() in rte_pmd_mpipe_devinit().

Signed-off-by: Liming Sun <lsun@ezchip.com>
---
 drivers/net/mpipe/mpipe_tilegx.c |   14 +++++++-------
 1 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/net/mpipe/mpipe_tilegx.c b/drivers/net/mpipe/mpipe_tilegx.c
index be7b6f2..5845511 100644
--- a/drivers/net/mpipe/mpipe_tilegx.c
+++ b/drivers/net/mpipe/mpipe_tilegx.c
@@ -752,13 +752,6 @@ mpipe_init(struct mpipe_dev_priv *priv)
 	if (priv->initialized)
 		return 0;
 
-	rc = mpipe_link_init(priv);
-	if (rc < 0) {
-		RTE_LOG(ERR, PMD, "%s: Failed to init link.\n",
-			mpipe_name(priv));
-		return rc;
-	}
-
 	rc = mpipe_recv_init(priv);
 	if (rc < 0) {
 		RTE_LOG(ERR, PMD, "%s: Failed to init rx.\n",
@@ -1633,6 +1626,13 @@ rte_pmd_mpipe_devinit(const char *ifname,
 	eth_dev->rx_pkt_burst = &mpipe_recv_pkts;
 	eth_dev->tx_pkt_burst = &mpipe_xmit_pkts;
 
+	rc = mpipe_link_init(priv);
+	if (rc < 0) {
+		RTE_LOG(ERR, PMD, "%s: Failed to init link.\n",
+			mpipe_name(priv));
+		return rc;
+	}
+
 	return 0;
 }
 
-- 
1.7.1

Re: [PATCH 3/3] driver/net/mpipe: fix a mpipe link initialization ordering issue

[Tony Lu]

>-----Original Message-----
>From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Liming Sun
>Sent: Tuesday, December 15, 2015 11:38 PM
>To: dev@dpdk.org
>Subject: [dpdk-dev] [PATCH 3/3] driver/net/mpipe: fix a mpipe link
initialization
>ordering issue
>
>Mpipe link structure is initialized in function mpipe_link_init().
>Currently it's only called from the eth_dev_ops.dev_start, which
>caused crashes when link mgmt APIs (like promiscuous_enable)
>was called before eth_dev_ops.dev_start(). This submit fixed it
>by calling mpipe_link_init() in rte_pmd_mpipe_devinit().
>
>Signed-off-by: Liming Sun <lsun@ezchip.com>
>---
> drivers/net/mpipe/mpipe_tilegx.c |   14 +++++++-------
> 1 files changed, 7 insertions(+), 7 deletions(-)
>
>diff --git a/drivers/net/mpipe/mpipe_tilegx.c
>b/drivers/net/mpipe/mpipe_tilegx.c
>index be7b6f2..5845511 100644
>--- a/drivers/net/mpipe/mpipe_tilegx.c
>+++ b/drivers/net/mpipe/mpipe_tilegx.c
>@@ -752,13 +752,6 @@ mpipe_init(struct mpipe_dev_priv *priv)
> 	if (priv->initialized)
> 		return 0;
>
>-	rc = mpipe_link_init(priv);
>-	if (rc < 0) {
>-		RTE_LOG(ERR, PMD, "%s: Failed to init link.\n",
>-			mpipe_name(priv));
>-		return rc;
>-	}
>-
> 	rc = mpipe_recv_init(priv);
> 	if (rc < 0) {
> 		RTE_LOG(ERR, PMD, "%s: Failed to init rx.\n",
>@@ -1633,6 +1626,13 @@ rte_pmd_mpipe_devinit(const char *ifname,
> 	eth_dev->rx_pkt_burst = &mpipe_recv_pkts;
> 	eth_dev->tx_pkt_burst = &mpipe_xmit_pkts;
>
>+	rc = mpipe_link_init(priv);
>+	if (rc < 0) {
>+		RTE_LOG(ERR, PMD, "%s: Failed to init link.\n",
>+			mpipe_name(priv));
>+		return rc;
>+	}
>+
> 	return 0;
> }
>
>--
>1.7.1

Acked-by: Zhigang Lu <zlu@ezchip.com>

[PATCH v2 1/3] driver/net/mpipe: support native build on tilegx platform.

[Liming Sun]

This submit updates the CROSS setting to support native build on
TileGx platform. It also enable the combined library by default.

Signed-off-by: Liming Sun <lsun@ezchip.com>
Acked-by: Zhigang Lu <zlu@ezchip.com>
---
 MAINTAINERS                               | 3 ++-
 config/defconfig_tile-tilegx-linuxapp-gcc | 4 ++++
 mk/arch/tile/rte.vars.mk                  | 6 ++++++
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index 3292e84..8f7e9ca 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -138,8 +138,9 @@ M: Jianbo Liu <jianbo.liu@linaro.org>
 F: lib/librte_eal/common/include/arch/arm/*_64.h
 F: lib/librte_acl/acl_run_neon.*
 
-EZchip TILE-Gx
+EZchip TILE-Gx/Mx
 M: Zhigang Lu <zlu@ezchip.com>
+M: Liming Sun <lsun@ezchip.com>
 F: lib/librte_eal/common/include/arch/tile/
 F: drivers/net/mpipe/
 
diff --git a/config/defconfig_tile-tilegx-linuxapp-gcc b/config/defconfig_tile-tilegx-linuxapp-gcc
index 9df9d7f..fb61bcd 100644
--- a/config/defconfig_tile-tilegx-linuxapp-gcc
+++ b/config/defconfig_tile-tilegx-linuxapp-gcc
@@ -70,3 +70,7 @@ CONFIG_RTE_LIBRTE_SCHED=n
 CONFIG_RTE_LIBRTE_PORT=n
 CONFIG_RTE_LIBRTE_TABLE=n
 CONFIG_RTE_LIBRTE_PIPELINE=n
+CONFIG_RTE_LIBRTE_CXGBE_PMD=n
+
+# Compile combined lib by default.
+CONFIG_RTE_BUILD_COMBINE_LIBS=y
diff --git a/mk/arch/tile/rte.vars.mk b/mk/arch/tile/rte.vars.mk
index b518986..06dab18 100644
--- a/mk/arch/tile/rte.vars.mk
+++ b/mk/arch/tile/rte.vars.mk
@@ -30,7 +30,13 @@
 
 
 ARCH  ?= tile
+
+HOST_ARCH := ${shell uname -m}
+ifneq ($(filter tile%,${HOST_ARCH}),)
+CROSS =
+else
 CROSS ?= tile-
+endif
 
 CPU_CFLAGS  ?=
 CPU_LDFLAGS ?=
-- 
1.8.3.1

[PATCH v2 2/3] driver/net/mpipe: optimize mpipe buffer return mechanism.

[Liming Sun]

This submit has changes to optimize the mpipe buffer return. When
a packet is received, instead of allocating and refilling the
buffer stack right away, it tracks the number of pending buffers,
and use HW buffer return as an optimization when the pending
number is below certain threshold, thus save two MMIO writes and
improves performance especially for bidirectional traffic case.

Signed-off-by: Liming Sun <lsun@ezchip.com>
Acked-by: Zhigang Lu <zlu@ezchip.com>
---
 drivers/net/mpipe/mpipe_tilegx.c | 50 ++++++++++++++++++++++++++++++++--------
 1 file changed, 40 insertions(+), 10 deletions(-)

diff --git a/drivers/net/mpipe/mpipe_tilegx.c b/drivers/net/mpipe/mpipe_tilegx.c
index 35134ba..be7b6f2 100644
--- a/drivers/net/mpipe/mpipe_tilegx.c
+++ b/drivers/net/mpipe/mpipe_tilegx.c
@@ -78,6 +78,13 @@ struct mpipe_context {
 	struct mpipe_channel_config channels[MPIPE_MAX_CHANNELS];
 };
 
+/* Per-core local data. */
+struct mpipe_local {
+	int mbuf_push_debt[RTE_MAX_ETHPORTS];	/* Buffer push debt. */
+} __rte_cache_aligned;
+
+#define MPIPE_BUF_DEBT_THRESHOLD	32
+static __thread struct mpipe_local mpipe_local;
 static struct mpipe_context mpipe_contexts[GXIO_MPIPE_INSTANCE_MAX];
 static int mpipe_instances;
 static const char *drivername = "MPIPE PMD";
@@ -137,7 +144,7 @@ struct mpipe_dev_priv {
 	int first_bucket;		/* mPIPE bucket start index. */
 	int first_ring;			/* mPIPE notif ring start index. */
 	int notif_group;		/* mPIPE notif group. */
-	rte_atomic32_t dp_count;	/* Active datapath thread count. */
+	rte_atomic32_t dp_count __rte_cache_aligned;	/* DP Entry count. */
 	int tx_stat_mapping[RTE_ETHDEV_QUEUE_STAT_CNTRS];
 	int rx_stat_mapping[RTE_ETHDEV_QUEUE_STAT_CNTRS];
 };
@@ -461,6 +468,14 @@ mpipe_dp_wait(struct mpipe_dev_priv *priv)
 	}
 }
 
+static inline int
+mpipe_mbuf_stack_index(struct mpipe_dev_priv *priv, struct rte_mbuf *mbuf)
+{
+	return (mbuf->port < RTE_MAX_ETHPORTS)?
+		mpipe_priv(&rte_eth_devices[mbuf->port])->stack :
+		priv->stack;
+}
+
 static inline struct rte_mbuf *
 mpipe_recv_mbuf(struct mpipe_dev_priv *priv, gxio_mpipe_idesc_t *idesc,
 		int in_port)
@@ -1267,6 +1282,7 @@ mpipe_do_xmit(struct mpipe_tx_queue *tx_queue, struct rte_mbuf **tx_pkts,
 	unsigned nb_bytes = 0;
 	unsigned nb_sent = 0;
 	int nb_slots, i;
+	uint8_t port_id;
 
 	PMD_DEBUG_TX("Trying to transmit %d packets on %s:%d.\n",
 		     nb_pkts, mpipe_name(tx_queue->q.priv),
@@ -1315,14 +1331,23 @@ mpipe_do_xmit(struct mpipe_tx_queue *tx_queue, struct rte_mbuf **tx_pkts,
 			if (priv->tx_comps[idx])
 				rte_pktmbuf_free_seg(priv->tx_comps[idx]);
 
+			port_id = (mbuf->port < RTE_MAX_ETHPORTS)?
+						mbuf->port : priv->port_id;
 			desc = (gxio_mpipe_edesc_t) { {
 				.va        = rte_pktmbuf_mtod(mbuf, uintptr_t),
 				.xfer_size = rte_pktmbuf_data_len(mbuf),
 				.bound     = next ? 0 : 1,
+				.stack_idx = mpipe_mbuf_stack_index(priv, mbuf),
 			} };
+			if (mpipe_local.mbuf_push_debt[port_id] > 0) {
+				mpipe_local.mbuf_push_debt[port_id]--;
+				desc.hwb = 1;
+				priv->tx_comps[idx] = NULL;
+			}
+			else
+				priv->tx_comps[idx] = mbuf;
 
 			nb_bytes += mbuf->data_len;
-			priv->tx_comps[idx] = mbuf;
 			gxio_mpipe_equeue_put_at(equeue, desc, slot + i);
 
 			PMD_DEBUG_TX("%s:%d: Sending packet %p, len %d\n",
@@ -1443,17 +1468,22 @@ mpipe_do_recv(struct mpipe_rx_queue *rx_queue, struct rte_mbuf **rx_pkts,
 				continue;
 			}
 
-			mbuf = __rte_mbuf_raw_alloc(priv->rx_mpool);
-			if (unlikely(!mbuf)) {
-				nb_nomem++;
-				gxio_mpipe_iqueue_drop(iqueue, idesc);
-				PMD_DEBUG_RX("%s:%d: RX alloc failure\n",
+			if (mpipe_local.mbuf_push_debt[in_port] <
+					MPIPE_BUF_DEBT_THRESHOLD)
+				mpipe_local.mbuf_push_debt[in_port]++;
+			else {
+				mbuf = __rte_mbuf_raw_alloc(priv->rx_mpool);
+				if (unlikely(!mbuf)) {
+					nb_nomem++;
+					gxio_mpipe_iqueue_drop(iqueue, idesc);
+					PMD_DEBUG_RX("%s:%d: alloc failure\n",
 					     mpipe_name(rx_queue->q.priv),
 					     rx_queue->q.queue_idx);
-				continue;
-			}
+					continue;
+				}
 
-			mpipe_recv_push(priv, mbuf);
+				mpipe_recv_push(priv, mbuf);
+			}
 
 			/* Get and setup the mbuf for the received packet. */
 			mbuf = mpipe_recv_mbuf(priv, idesc, in_port);
-- 
1.8.3.1

[PATCH v2 3/3] driver/net/mpipe: fix a mpipe link initialization ordering issue

[Liming Sun]

Mpipe link structure is initialized in function mpipe_link_init().
Currently it's only called from the eth_dev_ops.dev_start, which
caused crashes when link mgmt APIs (like promiscuous_enable)
was called before eth_dev_ops.dev_start(). This submit fixed it
by calling mpipe_link_init() in rte_pmd_mpipe_devinit().

Signed-off-by: Liming Sun <lsun@ezchip.com>
Acked-by: Zhigang Lu <zlu@ezchip.com>
---
 drivers/net/mpipe/mpipe_tilegx.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/net/mpipe/mpipe_tilegx.c b/drivers/net/mpipe/mpipe_tilegx.c
index be7b6f2..5845511 100644
--- a/drivers/net/mpipe/mpipe_tilegx.c
+++ b/drivers/net/mpipe/mpipe_tilegx.c
@@ -752,13 +752,6 @@ mpipe_init(struct mpipe_dev_priv *priv)
 	if (priv->initialized)
 		return 0;
 
-	rc = mpipe_link_init(priv);
-	if (rc < 0) {
-		RTE_LOG(ERR, PMD, "%s: Failed to init link.\n",
-			mpipe_name(priv));
-		return rc;
-	}
-
 	rc = mpipe_recv_init(priv);
 	if (rc < 0) {
 		RTE_LOG(ERR, PMD, "%s: Failed to init rx.\n",
@@ -1633,6 +1626,13 @@ rte_pmd_mpipe_devinit(const char *ifname,
 	eth_dev->rx_pkt_burst = &mpipe_recv_pkts;
 	eth_dev->tx_pkt_burst = &mpipe_xmit_pkts;
 
+	rc = mpipe_link_init(priv);
+	if (rc < 0) {
+		RTE_LOG(ERR, PMD, "%s: Failed to init link.\n",
+			mpipe_name(priv));
+		return rc;
+	}
+
 	return 0;
 }
 
-- 
1.8.3.1

Re: [PATCH v2 1/3] driver/net/mpipe: support native build on tilegx platform.

[Bruce Richardson]

On Fri, Jan 08, 2016 at 09:30:36AM -0500, Liming Sun wrote:
> This submit updates the CROSS setting to support native build on
> TileGx platform. It also enable the combined library by default.
> 
> Signed-off-by: Liming Sun <lsun@ezchip.com>
> Acked-by: Zhigang Lu <zlu@ezchip.com>

Series applied to dpdk-next-net/rel_16_04.

/Bruce

Re: [PATCH v2 1/3] driver/net/mpipe: support native build on tilegx platform.

[Thomas Monjalon]

Hi,

Sorry for being late for commenting.

2016-01-08 09:30, Liming Sun:
> -EZchip TILE-Gx
> +EZchip TILE-Gx/Mx

A comment about the TILE-Mx would be welcome.
Is it supported currently?
Isn't it an ARM arch?

>  M: Zhigang Lu <zlu@ezchip.com>
> +M: Liming Sun <lsun@ezchip.com>
>  F: lib/librte_eal/common/include/arch/tile/
>  F: drivers/net/mpipe/

[...]
> +# Compile combined lib by default.
> +CONFIG_RTE_BUILD_COMBINE_LIBS=y

Why forcing this option in the defconfig file?

[...]
>  ARCH  ?= tile
> +
> +HOST_ARCH := ${shell uname -m}
> +ifneq ($(filter tile%,${HOST_ARCH}),)
> +CROSS =
> +else
>  CROSS ?= tile-
> +endif

I don't think the CROSS variable should have a default value.
It really depends on the toolchain.

Talking about the toolchain, is it possible to build DPDK with the
provided binary toolchain http://www.tilera.com/scm/tilegx-x86_64.tar.bz2 ?
This is the Tilera Open Source page: http://www.tilera.com/scm/

Re: [PATCH v2 1/3] driver/net/mpipe: support native build on tilegx platform.

[Liming Sun]

Thanks Thomas for the comments.
Please see the response inline.

Thanks,
Liming

-----Original Message-----
From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com] 
Sent: Tuesday, February 09, 2016 11:16 AM
To: Liming Sun
Cc: dev@dpdk.org; bruce.richardson@intel.com
Subject: Re: [dpdk-dev] [PATCH v2 1/3] driver/net/mpipe: support native build on tilegx platform.

Hi,

Sorry for being late for commenting.

2016-01-08 09:30, Liming Sun:
> -EZchip TILE-Gx
> +EZchip TILE-Gx/Mx

A comment about the TILE-Mx would be welcome.
Is it supported currently?
Isn't it an ARM arch?

[lsun] Yes, it's ARM arch, but with similar mpipe driver.
It's not fully supported yet. I'll remove this change and add it in a different serie when it's ready.

>  M: Zhigang Lu <zlu@ezchip.com>
> +M: Liming Sun <lsun@ezchip.com>
>  F: lib/librte_eal/common/include/arch/tile/
>  F: drivers/net/mpipe/

[...]
> +# Compile combined lib by default.
> +CONFIG_RTE_BUILD_COMBINE_LIBS=y

Why forcing this option in the defconfig file?

[lsun] It's just trying to make it handy for other applications like OVS or ODP on top of DPDK. However we could remove this change if it's not the recommended way.

[...]
>  ARCH  ?= tile
> +
> +HOST_ARCH := ${shell uname -m}
> +ifneq ($(filter tile%,${HOST_ARCH}),) CROSS = else
>  CROSS ?= tile-
> +endif

I don't think the CROSS variable should have a default value.
It really depends on the toolchain.

[lsun] Make sense. The current code (before the change) has default value 'CROSS ?= tile-' defined, which cause some issue when doing native build. Another way is to define it as "CROSS ?=" just like other platforms. So when doing cross-compile, we could pass " CROSS=tile-" .

Talking about the toolchain, is it possible to build DPDK with the provided binary toolchain http://www.tilera.com/scm/tilegx-x86_64.tar.bz2 ?
This is the Tilera Open Source page: http://www.tilera.com/scm/

[lsun] I tried it just now. The gcc appears ok. But this tarball lacks of some header files and libraries to compile DPDK.
We're looking into it to see whether it can be easily fixed.

Re: [PATCH v2 1/3] driver/net/mpipe: support native build on tilegx platform.

[Thomas Monjalon]

2016-02-09 18:37, Liming Sun:
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com] 
> A comment about the TILE-Mx would be welcome.
> Is it supported currently?
> Isn't it an ARM arch?
> 
> [lsun] Yes, it's ARM arch, but with similar mpipe driver.
> It's not fully supported yet. I'll remove this change and add it in a different serie when it's ready.

OK
So we'll discuss how to integrate it later.

> > +# Compile combined lib by default.
> > +CONFIG_RTE_BUILD_COMBINE_LIBS=y
> 
> Why forcing this option in the defconfig file?
> 
> [lsun] It's just trying to make it handy for other applications like OVS or ODP on top of DPDK. However we could remove this change if it's not the recommended way.

Yes please remove it.

> >  ARCH  ?= tile
> > +
> > +HOST_ARCH := ${shell uname -m}
> > +ifneq ($(filter tile%,${HOST_ARCH}),) CROSS = else
> >  CROSS ?= tile-
> > +endif
> 
> I don't think the CROSS variable should have a default value.
> It really depends on the toolchain.
> 
> [lsun] Make sense. The current code (before the change) has default value 'CROSS ?= tile-' defined, which cause some issue when doing native build. Another way is to define it as "CROSS ?=" just like other platforms. So when doing cross-compile, we could pass " CROSS=tile-" .

No need to keep a "CROSS ?=" line.
The variables defined in the command line get the priority.


> Talking about the toolchain, is it possible to build DPDK with the provided binary toolchain http://www.tilera.com/scm/tilegx-x86_64.tar.bz2 ?
> This is the Tilera Open Source page: http://www.tilera.com/scm/
> 
> [lsun] I tried it just now. The gcc appears ok. But this tarball lacks of some header files and libraries to compile DPDK.

Yes that's what I've experienced.

> We're looking into it to see whether it can be easily fixed.

Please keep us informed when the toolchain is ready. Thanks

Re: [PATCH v2 1/3] driver/net/mpipe: support native build on tilegx platform.

[Liming Sun]

Looks like this patch serie has been merged into dpdk-next-net/rel_16_04.
What would be the usual way to submit changes for new comments? Would it be incremental changes (new commit) based on previous one? Thanks.
________________________________________
From: Thomas Monjalon <thomas.monjalon@6wind.com>
Sent: Tuesday, February 9, 2016 3:33 PM
To: Liming Sun
Cc: dev@dpdk.org; bruce.richardson@intel.com
Subject: Re: [dpdk-dev] [PATCH v2 1/3] driver/net/mpipe: support native build on tilegx platform.

2016-02-09 18:37, Liming Sun:
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> A comment about the TILE-Mx would be welcome.
> Is it supported currently?
> Isn't it an ARM arch?
>
> [lsun] Yes, it's ARM arch, but with similar mpipe driver.
> It's not fully supported yet. I'll remove this change and add it in a different serie when it's ready.

OK
So we'll discuss how to integrate it later.

> > +# Compile combined lib by default.
> > +CONFIG_RTE_BUILD_COMBINE_LIBS=y
>
> Why forcing this option in the defconfig file?
>
> [lsun] It's just trying to make it handy for other applications like OVS or ODP on top of DPDK. However we could remove this change if it's not the recommended way.

Yes please remove it.

> >  ARCH  ?= tile
> > +
> > +HOST_ARCH := ${shell uname -m}
> > +ifneq ($(filter tile%,${HOST_ARCH}),) CROSS = else
> >  CROSS ?= tile-
> > +endif
>
> I don't think the CROSS variable should have a default value.
> It really depends on the toolchain.
>
> [lsun] Make sense. The current code (before the change) has default value 'CROSS ?= tile-' defined, which cause some issue when doing native build. Another way is to define it as "CROSS ?=" just like other platforms. So when doing cross-compile, we could pass " CROSS=tile-" .

No need to keep a "CROSS ?=" line.
The variables defined in the command line get the priority.


> Talking about the toolchain, is it possible to build DPDK with the provided binary toolchain http://www.tilera.com/scm/tilegx-x86_64.tar.bz2 ?
> This is the Tilera Open Source page: http://www.tilera.com/scm/
>
> [lsun] I tried it just now. The gcc appears ok. But this tarball lacks of some header files and libraries to compile DPDK.

Yes that's what I've experienced.

> We're looking into it to see whether it can be easily fixed.

Please keep us informed when the toolchain is ready. Thanks

Re: [PATCH v2 1/3] driver/net/mpipe: support native build on tilegx platform.

[Thomas Monjalon]

2016-02-09 21:15, Liming Sun:
> Looks like this patch serie has been merged into dpdk-next-net/rel_16_04.
> What would be the usual way to submit changes for new comments? Would it be incremental changes (new commit) based on previous one? Thanks.

Good question.
I think it's better if Bruce drops or reverts the commits from dpdk-next-net
to let you re-submit a better new version.
Bruce, do you agree?

Re: [PATCH v2 1/3] driver/net/mpipe: support native build on tilegx platform.

[Bruce Richardson]

On Tue, Feb 09, 2016 at 11:47:55PM +0100, Thomas Monjalon wrote:
> 2016-02-09 21:15, Liming Sun:
> > Looks like this patch serie has been merged into dpdk-next-net/rel_16_04.
> > What would be the usual way to submit changes for new comments? Would it be incremental changes (new commit) based on previous one? Thanks.
> 
> Good question.
> I think it's better if Bruce drops or reverts the commits from dpdk-next-net
> to let you re-submit a better new version.
> Bruce, do you agree?

Unless there is something actually broken - that was previously working - by 
this patchset I'd rather not revert it. This patch was sitting acked for a month
which is a reasonable time for comments before applying it. Allowing people to
step up post-apply and look for patches being reverted is not something we want
to encourage IMHO. There are already too many reviews being done at the last
minute, and allowing reverts may make that situation worse, while applying acked
patches within a reasonable time - irrespective of whether people subsequently
find issues with them - should encourage earlier reviews, and makes it easier on
contributors.

Therefore I'd rather see any additional enhancements or changes
done as incremental patches on top of this set.

Regards,
/Bruce

Re: [PATCH v2 1/3] driver/net/mpipe: support native build on tilegx platform.

[Thomas Monjalon]

2016-02-10 09:49, Bruce Richardson:
> On Tue, Feb 09, 2016 at 11:47:55PM +0100, Thomas Monjalon wrote:
> > 2016-02-09 21:15, Liming Sun:
> > > Looks like this patch serie has been merged into dpdk-next-net/rel_16_04.
> > > What would be the usual way to submit changes for new comments? Would it be incremental changes (new commit) based on previous one? Thanks.
> > 
> > Good question.
> > I think it's better if Bruce drops or reverts the commits from dpdk-next-net
> > to let you re-submit a better new version.
> > Bruce, do you agree?
> 
> Unless there is something actually broken - that was previously working - by 
> this patchset I'd rather not revert it. This patch was sitting acked for a month
> which is a reasonable time for comments before applying it. Allowing people to
> step up post-apply and look for patches being reverted is not something we want
> to encourage IMHO. There are already too many reviews being done at the last
> minute, and allowing reverts may make that situation worse, while applying acked
> patches within a reasonable time - irrespective of whether people subsequently
> find issues with them - should encourage earlier reviews, and makes it easier on
> contributors.

Yes you are right.

> Therefore I'd rather see any additional enhancements or changes
> done as incremental patches on top of this set.
> 
> Regards,
> /Bruce
> 

Re: [PATCH v2 1/3] driver/net/mpipe: support native build on tilegx platform.

[Thomas Monjalon]

Hi,

2016-02-09 21:33, Thomas Monjalon:
> 2016-02-09 18:37, Liming Sun:
> > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com] 
> > A comment about the TILE-Mx would be welcome.
> > Is it supported currently?
> > Isn't it an ARM arch?
> > 
> > [lsun] Yes, it's ARM arch, but with similar mpipe driver.
> > It's not fully supported yet. I'll remove this change and add it in a different serie when it's ready.
> 
> OK
> So we'll discuss how to integrate it later.
> 
> > > +# Compile combined lib by default.
> > > +CONFIG_RTE_BUILD_COMBINE_LIBS=y
> > 
> > Why forcing this option in the defconfig file?
> > 
> > [lsun] It's just trying to make it handy for other applications like OVS or ODP on top of DPDK. However we could remove this change if it's not the recommended way.
> 
> Yes please remove it.
> 
> > >  ARCH  ?= tile
> > > +
> > > +HOST_ARCH := ${shell uname -m}
> > > +ifneq ($(filter tile%,${HOST_ARCH}),) CROSS = else
> > >  CROSS ?= tile-
> > > +endif
> > 
> > I don't think the CROSS variable should have a default value.
> > It really depends on the toolchain.
> > 
> > [lsun] Make sense. The current code (before the change) has default value 'CROSS ?= tile-' defined, which cause some issue when doing native build. Another way is to define it as "CROSS ?=" just like other platforms. So when doing cross-compile, we could pass " CROSS=tile-" .
> 
> No need to keep a "CROSS ?=" line.
> The variables defined in the command line get the priority.

I have not seen patches to address these comments. Anything pending?


> > Talking about the toolchain, is it possible to build DPDK with the provided binary toolchain http://www.tilera.com/scm/tilegx-x86_64.tar.bz2 ?
> > This is the Tilera Open Source page: http://www.tilera.com/scm/
> > 
> > [lsun] I tried it just now. The gcc appears ok. But this tarball lacks of some header files and libraries to compile DPDK.
> 
> Yes that's what I've experienced.
> 
> > We're looking into it to see whether it can be easily fixed.
> 
> Please keep us informed when the toolchain is ready. Thanks

Any news about the ready-to-use toolchain?