[RFC PATCH 0/2] net: mvneta: Introduce RSS support

[RFC PATCH 0/2] net: mvneta: Introduce RSS support

[Gregory CLEMENT]

Hi,

this series is a first attempt to add RSS support on mvneta.

this series will allow associating an ethernet interface to a given
CPU through RSS by using "ethtool -X ethX weight". Indeed, currently I
only enable one entry in the RSS lookup table. Even if it is not
really RSS, it allows to get back the irq affinity feature we lost by
using the percpu interrupt.

The first patch really associates the RX queues with the CPUs instead
of masking the percpu interrupts for doing it. All the RX queues are
enabled and are statically associated with the CPUs by using a modulo
of the number of present CPUs. But at this stage only one RX queue
will receive the stream.

I also choose to associate all the TX queues on the same CPU that the
one associated to the RX queue. It allows to contain all the
interrupts on the same CPU. I think that an improvement on this side
would be the support of the XPS.

The second patch introduces a first level of RSS support through the
ethtool functions. As explained in the introduction there is only one
entry in the RSS lookup table which permits at the end to associate an
mvneta port to a CPU through the RX queues because the mapping is
static.

I really would like to have some feedback before going further and
then going in the wring direction.

Thanks,


Gregory CLEMENT (2):
  net: mvneta: Associate RX queues with each CPU
  net: mvneta: Add naive RSS support

 drivers/net/ethernet/marvell/mvneta.c | 264 +++++++++++++++++++++++++++++-----
 1 file changed, 229 insertions(+), 35 deletions(-)

-- 
2.5.0

[RFC PATCH 1/2] net: mvneta: Associate RX queues with each CPU

[Gregory CLEMENT]

We enable the percpu interrupt for all the CPU and we just associate a
CPU to a few queue at the neta level. The mapping between the CPUs and
the queues is static. The queues are associated to the CPU module the
number of CPUs. However currently we only use on RX queue for a given
Ethernet port.

Signed-off-by: Gregory CLEMENT <gregory.clement@free-electrons.com>
---
 drivers/net/ethernet/marvell/mvneta.c | 150 ++++++++++++++++++++++++++--------
 1 file changed, 115 insertions(+), 35 deletions(-)

diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
index 20a2363f9e40..c38326b848f9 100644
--- a/drivers/net/ethernet/marvell/mvneta.c
+++ b/drivers/net/ethernet/marvell/mvneta.c
@@ -109,9 +109,16 @@
 #define MVNETA_CPU_MAP(cpu)                      (0x2540 + ((cpu) << 2))
 #define      MVNETA_CPU_RXQ_ACCESS_ALL_MASK      0x000000ff
 #define      MVNETA_CPU_TXQ_ACCESS_ALL_MASK      0x0000ff00
+#define      MVNETA_CPU_RXQ_ACCESS(rxq)		 BIT(rxq)
 #define MVNETA_RXQ_TIME_COAL_REG(q)              (0x2580 + ((q) << 2))
 
-/* Exception Interrupt Port/Queue Cause register */
+/* Exception Interrupt Port/Queue Cause register
+ *
+ * Their behavior depend of the mapping done using the PCPX2Q
+ * registers. For a given CPU if the bit associated to a queue is not
+ * set, then for the register a read from this CPU will always return
+ * 0 and a write won't do anything
+ */
 
 #define MVNETA_INTR_NEW_CAUSE                    0x25a0
 #define MVNETA_INTR_NEW_MASK                     0x25a4
@@ -818,7 +825,13 @@ static void mvneta_port_up(struct mvneta_port *pp)
 	mvreg_write(pp, MVNETA_TXQ_CMD, q_map);
 
 	/* Enable all initialized RXQs. */
-	mvreg_write(pp, MVNETA_RXQ_CMD, BIT(pp->rxq_def));
+	for (queue = 0; queue < rxq_number; queue++) {
+		struct mvneta_rx_queue *rxq = &pp->rxqs[queue];
+
+		if (rxq->descs != NULL)
+			q_map |= (1 << queue);
+	}
+	mvreg_write(pp, MVNETA_RXQ_CMD, q_map);
 }
 
 /* Stop the Ethernet port activity */
@@ -986,6 +999,7 @@ static void mvneta_defaults_set(struct mvneta_port *pp)
 	int cpu;
 	int queue;
 	u32 val;
+	int max_cpu = num_present_cpus();
 
 	/* Clear all Cause registers */
 	mvreg_write(pp, MVNETA_INTR_NEW_CAUSE, 0);
@@ -1001,13 +1015,23 @@ static void mvneta_defaults_set(struct mvneta_port *pp)
 	/* Enable MBUS Retry bit16 */
 	mvreg_write(pp, MVNETA_MBUS_RETRY, 0x20);
 
-	/* Set CPU queue access map - all CPUs have access to all RX
-	 * queues and to all TX queues
+	/* Set CPU queue access map. CPUs are assigned to the RX
+	 * queues modulo their number and all the TX queues are
+	 * assigned to the CPU associated to the default RX queue.
 	 */
-	for_each_present_cpu(cpu)
-		mvreg_write(pp, MVNETA_CPU_MAP(cpu),
-			    (MVNETA_CPU_RXQ_ACCESS_ALL_MASK |
-			     MVNETA_CPU_TXQ_ACCESS_ALL_MASK));
+	for_each_present_cpu(cpu) {
+		int rxq_map = 0, txq_map = 0;
+		int rxq;
+
+		for (rxq = 0; rxq < rxq_number; rxq++)
+			if ((rxq % max_cpu) == cpu)
+				rxq_map |= MVNETA_CPU_RXQ_ACCESS(rxq);
+
+		if (cpu == rxq_def)
+			txq_map = MVNETA_CPU_TXQ_ACCESS_ALL_MASK;
+
+		mvreg_write(pp, MVNETA_CPU_MAP(cpu), rxq_map | txq_map);
+	}
 
 	/* Reset RX and TX DMAs */
 	mvreg_write(pp, MVNETA_PORT_RX_RESET, MVNETA_PORT_RX_DMA_RESET);
@@ -2149,6 +2173,7 @@ static int mvneta_poll(struct napi_struct *napi, int budget)
 {
 	int rx_done = 0;
 	u32 cause_rx_tx;
+	int rx_queue;
 	struct mvneta_port *pp = netdev_priv(napi->dev);
 	struct mvneta_pcpu_port *port = this_cpu_ptr(pp->ports);
 
@@ -2180,8 +2205,15 @@ static int mvneta_poll(struct napi_struct *napi, int budget)
 	/* For the case where the last mvneta_poll did not process all
 	 * RX packets
 	 */
+	rx_queue = fls(((cause_rx_tx >> 8) & 0xff));
+
 	cause_rx_tx |= port->cause_rx_tx;
-	rx_done = mvneta_rx(pp, budget, &pp->rxqs[pp->rxq_def]);
+
+	if (rx_queue) {
+		rx_queue = rx_queue - 1;
+		rx_done = mvneta_rx(pp, budget, &pp->rxqs[rx_queue]);
+	}
+
 	budget -= rx_done;
 
 	if (budget > 0) {
@@ -2394,19 +2426,27 @@ static void mvneta_cleanup_txqs(struct mvneta_port *pp)
 /* Cleanup all Rx queues */
 static void mvneta_cleanup_rxqs(struct mvneta_port *pp)
 {
-	mvneta_rxq_deinit(pp, &pp->rxqs[pp->rxq_def]);
+	int queue;
+
+	for (queue = 0; queue < txq_number; queue++)
+		mvneta_rxq_deinit(pp, &pp->rxqs[queue]);
 }
 
 
 /* Init all Rx queues */
 static int mvneta_setup_rxqs(struct mvneta_port *pp)
 {
-	int err = mvneta_rxq_init(pp, &pp->rxqs[pp->rxq_def]);
-	if (err) {
-		netdev_err(pp->dev, "%s: can't create rxq=%d\n",
-			   __func__, pp->rxq_def);
-		mvneta_cleanup_rxqs(pp);
-		return err;
+	int queue;
+
+	for (queue = 0; queue < rxq_number; queue++) {
+		int err = mvneta_rxq_init(pp, &pp->rxqs[queue]);
+
+		if (err) {
+			netdev_err(pp->dev, "%s: can't create rxq=%d\n",
+				   __func__, queue);
+			mvneta_cleanup_rxqs(pp);
+			return err;
+		}
 	}
 
 	return 0;
@@ -2430,6 +2470,19 @@ static int mvneta_setup_txqs(struct mvneta_port *pp)
 	return 0;
 }
 
+static void mvneta_percpu_unmask_interrupt(void *arg)
+{
+	struct mvneta_port *pp = arg;
+
+	/* All the queue are unmasked, but actually only the ones
+	 * maped to this CPU will be unmasked
+	 */
+	mvreg_write(pp, MVNETA_INTR_NEW_MASK,
+		    MVNETA_RX_INTR_MASK_ALL |
+		    MVNETA_TX_INTR_MASK_ALL |
+		    MVNETA_MISCINTR_INTR_MASK);
+}
+
 static void mvneta_start_dev(struct mvneta_port *pp)
 {
 	unsigned int cpu;
@@ -2447,11 +2500,10 @@ static void mvneta_start_dev(struct mvneta_port *pp)
 		napi_enable(&port->napi);
 	}
 
-	/* Unmask interrupts */
-	mvreg_write(pp, MVNETA_INTR_NEW_MASK,
-		    MVNETA_RX_INTR_MASK(rxq_number) |
-		    MVNETA_TX_INTR_MASK(txq_number) |
-		    MVNETA_MISCINTR_INTR_MASK);
+	/* Unmask interrupts. It has to be done from each CPU */
+	for_each_online_cpu(cpu)
+		smp_call_function_single(cpu, mvneta_percpu_unmask_interrupt,
+					 pp, true);
 	mvreg_write(pp, MVNETA_INTR_MISC_MASK,
 		    MVNETA_CAUSE_PHY_STATUS_CHANGE |
 		    MVNETA_CAUSE_LINK_CHANGE |
@@ -2727,22 +2779,35 @@ static void mvneta_percpu_disable(void *arg)
 
 static void mvneta_percpu_elect(struct mvneta_port *pp)
 {
-	int online_cpu_idx, cpu, i = 0;
+	int online_cpu_idx, max_cpu, cpu, i = 0;
 
 	online_cpu_idx = pp->rxq_def % num_online_cpus();
+	max_cpu = num_present_cpus();
 
 	for_each_online_cpu(cpu) {
-		if (i == online_cpu_idx)
-			/* Enable per-CPU interrupt on the one CPU we
-			 * just elected
+		int rxq_map = 0, txq_map = 0;
+		int rxq;
+
+		for (rxq = 0; rxq < rxq_number; rxq++)
+			if ((rxq % max_cpu) == cpu)
+				rxq_map |= MVNETA_CPU_RXQ_ACCESS(rxq);
+
+		if (i == online_cpu_idx) {
+			/* Map the default receive queue and transmit
+			 * queue to the elected CPU
 			 */
-			smp_call_function_single(cpu, mvneta_percpu_enable,
-						pp, true);
-		else
-			/* Disable per-CPU interrupt on all the other CPU */
-			smp_call_function_single(cpu, mvneta_percpu_disable,
-						pp, true);
+			rxq_map |= MVNETA_CPU_RXQ_ACCESS(pp->rxq_def);
+			txq_map = MVNETA_CPU_TXQ_ACCESS_ALL_MASK;
+		}
+		mvreg_write(pp, MVNETA_CPU_MAP(cpu), rxq_map | txq_map);
+
+		/* Update the interrupt mask on each CPU according the
+		 * new mapping
+		 */
+		smp_call_function_single(cpu, mvneta_percpu_unmask_interrupt,
+					 pp, true);
 		i++;
+
 	}
 };
 
@@ -2777,12 +2842,22 @@ static int mvneta_percpu_notifier(struct notifier_block *nfb,
 		mvreg_write(pp, MVNETA_INTR_MISC_MASK, 0);
 		napi_enable(&port->napi);
 
+
+		/* Enable per-CPU interrupts on the CPU that is
+		 * brought up.
+		 */
+		smp_call_function_single(cpu, mvneta_percpu_enable,
+					 pp, true);
+
 		/* Enable per-CPU interrupt on the one CPU we care
 		 * about.
 		 */
 		mvneta_percpu_elect(pp);
 
-		/* Unmask all ethernet port interrupts */
+		/* Unmask all ethernet port interrupts, as this
+		 * notifier is called for each CPU then the CPU to
+		 * Queue mapping is applied
+		 */
 		mvreg_write(pp, MVNETA_INTR_NEW_MASK,
 			MVNETA_RX_INTR_MASK(rxq_number) |
 			MVNETA_TX_INTR_MASK(txq_number) |
@@ -2833,7 +2908,7 @@ static int mvneta_percpu_notifier(struct notifier_block *nfb,
 static int mvneta_open(struct net_device *dev)
 {
 	struct mvneta_port *pp = netdev_priv(dev);
-	int ret;
+	int ret, cpu;
 
 	pp->pkt_size = MVNETA_RX_PKT_SIZE(pp->dev->mtu);
 	pp->frag_size = SKB_DATA_ALIGN(MVNETA_RX_BUF_SIZE(pp->pkt_size)) +
@@ -2863,8 +2938,13 @@ static int mvneta_open(struct net_device *dev)
 	 */
 	mvneta_percpu_disable(pp);
 
-	/* Elect a CPU to handle our RX queue interrupt */
-	mvneta_percpu_elect(pp);
+	/* Enable per-CPU interrupt on all the CPU to handle our RX
+	 * queue interrupts
+	 */
+	for_each_online_cpu(cpu)
+		smp_call_function_single(cpu, mvneta_percpu_enable,
+					 pp, true);
+
 
 	/* Register a CPU notifier to handle the case where our CPU
 	 * might be taken offline.
-- 
2.5.0

[RFC PATCH 2/2] net: mvneta: Add naive RSS support

[Gregory CLEMENT]

This patch add the support for the RSS related ethtool
function. Currently it only use one entry in the indirection table which
allows associating an mveneta interface to a given CPU.

Signed-off-by: Gregory CLEMENT <gregory.clement@free-electrons.com>
---
 drivers/net/ethernet/marvell/mvneta.c | 114 ++++++++++++++++++++++++++++++++++
 1 file changed, 114 insertions(+)

diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
index c38326b848f9..5f810a458443 100644
--- a/drivers/net/ethernet/marvell/mvneta.c
+++ b/drivers/net/ethernet/marvell/mvneta.c
@@ -259,6 +259,11 @@
 
 #define MVNETA_TX_MTU_MAX		0x3ffff
 
+/* The RSS lookup table actually has 256 entries but we do not use
+ * them yet
+ */
+#define MVNETA_RSS_LU_TABLE_SIZE	1
+
 /* TSO header size */
 #define TSO_HEADER_SIZE 128
 
@@ -380,6 +385,8 @@ struct mvneta_port {
 	int use_inband_status:1;
 
 	u64 ethtool_stats[ARRAY_SIZE(mvneta_statistics)];
+
+	u32 indir[MVNETA_RSS_LU_TABLE_SIZE];
 };
 
 /* The mvneta_tx_desc and mvneta_rx_desc structures describe the
@@ -3173,6 +3180,107 @@ static int mvneta_ethtool_get_sset_count(struct net_device *dev, int sset)
 	return -EOPNOTSUPP;
 }
 
+static u32 mvneta_ethtool_get_rxfh_indir_size(struct net_device *dev)
+{
+	return MVNETA_RSS_LU_TABLE_SIZE;
+}
+
+static int mvneta_ethtool_get_rxnfc(struct net_device *dev,
+				    struct ethtool_rxnfc *info,
+				    u32 *rules __always_unused)
+{
+	switch (info->cmd) {
+	case ETHTOOL_GRXRINGS:
+		info->data =  rxq_number;
+		return 0;
+	case ETHTOOL_GRXFH:
+		return -EOPNOTSUPP;
+	default:
+		return -EOPNOTSUPP;
+	}
+}
+
+static int  mvneta_config_rss(struct mvneta_port *pp)
+{
+	int cpu;
+	u32 val;
+
+	netif_tx_stop_all_queues(pp->dev);
+
+	/* Mask all ethernet port interrupts */
+	mvreg_write(pp, MVNETA_INTR_NEW_MASK, 0);
+	mvreg_write(pp, MVNETA_INTR_OLD_MASK, 0);
+	mvreg_write(pp, MVNETA_INTR_MISC_MASK, 0);
+
+	/* We have to synchronise on the napi of each CPU */
+	for_each_online_cpu(cpu) {
+		struct mvneta_pcpu_port *pcpu_port =
+			per_cpu_ptr(pp->ports, cpu);
+
+		napi_synchronize(&pcpu_port->napi);
+		napi_disable(&pcpu_port->napi);
+	}
+
+	pp->rxq_def = pp->indir[0];
+
+	/* update unicast mapping */
+	mvneta_set_rx_mode(pp->dev);
+
+	/* Update val of portCfg register accordingly with all RxQueue types */
+	val = MVNETA_PORT_CONFIG_DEFL_VALUE(pp->rxq_def);
+	mvreg_write(pp, MVNETA_PORT_CONFIG, val);
+
+	/* Update the elected CPU matching the new rxq_def */
+	mvneta_percpu_elect(pp);
+
+	/* We have to synchronise on the napi of each CPU */
+	for_each_online_cpu(cpu) {
+		struct mvneta_pcpu_port *pcpu_port =
+			per_cpu_ptr(pp->ports, cpu);
+
+		napi_enable(&pcpu_port->napi);
+	}
+
+	netif_tx_start_all_queues(pp->dev);
+
+	return 0;
+}
+
+static int mvneta_ethtool_set_rxfh(struct net_device *dev, const u32 *indir,
+				   const u8 *key, const u8 hfunc)
+{
+	struct mvneta_port *pp = netdev_priv(dev);
+	/* We require at least one supported parameter to be changed
+	* and no change in any of the unsupported parameters
+	*/
+	if (key ||
+	    (hfunc != ETH_RSS_HASH_NO_CHANGE && hfunc != ETH_RSS_HASH_TOP))
+		return -EOPNOTSUPP;
+
+	if (!indir)
+		return 0;
+
+	memcpy(pp->indir, indir, MVNETA_RSS_LU_TABLE_SIZE);
+
+	return mvneta_config_rss(pp);
+}
+
+static int mvneta_ethtool_get_rxfh(struct net_device *dev, u32 *indir, u8 *key,
+				   u8 *hfunc)
+{
+	struct mvneta_port *pp = netdev_priv(dev);
+
+	if (hfunc)
+		*hfunc = ETH_RSS_HASH_TOP;
+
+	if (!indir)
+		return 0;
+
+	memcpy(indir, pp->indir, MVNETA_RSS_LU_TABLE_SIZE);
+
+	return 0;
+}
+
 static const struct net_device_ops mvneta_netdev_ops = {
 	.ndo_open            = mvneta_open,
 	.ndo_stop            = mvneta_stop,
@@ -3197,6 +3305,10 @@ const struct ethtool_ops mvneta_eth_tool_ops = {
 	.get_strings	= mvneta_ethtool_get_strings,
 	.get_ethtool_stats = mvneta_ethtool_get_stats,
 	.get_sset_count	= mvneta_ethtool_get_sset_count,
+	.get_rxfh_indir_size = mvneta_ethtool_get_rxfh_indir_size,
+	.get_rxnfc	= mvneta_ethtool_get_rxnfc,
+	.get_rxfh	= mvneta_ethtool_get_rxfh,
+	.set_rxfh	= mvneta_ethtool_set_rxfh,
 };
 
 /* Initialize hw */
@@ -3389,6 +3501,8 @@ static int mvneta_probe(struct platform_device *pdev)
 
 	pp->rxq_def = rxq_def;
 
+	pp->indir[0] = rxq_def;
+
 	pp->clk = devm_clk_get(&pdev->dev, NULL);
 	if (IS_ERR(pp->clk)) {
 		err = PTR_ERR(pp->clk);
-- 
2.5.0

Re: [RFC PATCH 2/2] net: mvneta: Add naive RSS support

[Marcin Wojtas]

Hi Gregory,

2015-11-06 19:35 GMT+01:00 Gregory CLEMENT <gregory.clement@free-electrons.com>:
> This patch add the support for the RSS related ethtool
> function. Currently it only use one entry in the indirection table which
> allows associating an mveneta interface to a given CPU.
>
> Signed-off-by: Gregory CLEMENT <gregory.clement@free-electrons.com>
> ---
>  drivers/net/ethernet/marvell/mvneta.c | 114 ++++++++++++++++++++++++++++++++++
>  1 file changed, 114 insertions(+)
>
> diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
> index c38326b848f9..5f810a458443 100644
> --- a/drivers/net/ethernet/marvell/mvneta.c
> +++ b/drivers/net/ethernet/marvell/mvneta.c
> @@ -259,6 +259,11 @@
>
>  #define MVNETA_TX_MTU_MAX              0x3ffff
>
> +/* The RSS lookup table actually has 256 entries but we do not use
> + * them yet
> + */
> +#define MVNETA_RSS_LU_TABLE_SIZE       1
> +
>  /* TSO header size */
>  #define TSO_HEADER_SIZE 128
>
> @@ -380,6 +385,8 @@ struct mvneta_port {
>         int use_inband_status:1;
>
>         u64 ethtool_stats[ARRAY_SIZE(mvneta_statistics)];
> +
> +       u32 indir[MVNETA_RSS_LU_TABLE_SIZE];
>  };
>
>  /* The mvneta_tx_desc and mvneta_rx_desc structures describe the
> @@ -3173,6 +3180,107 @@ static int mvneta_ethtool_get_sset_count(struct net_device *dev, int sset)
>         return -EOPNOTSUPP;
>  }
>
> +static u32 mvneta_ethtool_get_rxfh_indir_size(struct net_device *dev)
> +{
> +       return MVNETA_RSS_LU_TABLE_SIZE;
> +}
> +
> +static int mvneta_ethtool_get_rxnfc(struct net_device *dev,
> +                                   struct ethtool_rxnfc *info,
> +                                   u32 *rules __always_unused)
> +{
> +       switch (info->cmd) {
> +       case ETHTOOL_GRXRINGS:
> +               info->data =  rxq_number;
> +               return 0;
> +       case ETHTOOL_GRXFH:
> +               return -EOPNOTSUPP;
> +       default:
> +               return -EOPNOTSUPP;
> +       }
> +}
> +
> +static int  mvneta_config_rss(struct mvneta_port *pp)
> +{
> +       int cpu;
> +       u32 val;
> +
> +       netif_tx_stop_all_queues(pp->dev);
> +
> +       /* Mask all ethernet port interrupts */
> +       mvreg_write(pp, MVNETA_INTR_NEW_MASK, 0);

Shouldn't the interrupts be masked on each online cpu? There is percpu
unmask function (mvneta_percpu_unmask_interrupt), so maybe ther should
be also mvneta_percpu_mask_interrupt. With this masking should look
like below:

     for_each_online_cpu(cpu)
               smp_call_function_single(cpu, mvneta_percpu_unmask_interrupt,
                                        pp, true);

> +       mvreg_write(pp, MVNETA_INTR_OLD_MASK, 0);
> +       mvreg_write(pp, MVNETA_INTR_MISC_MASK, 0);
> +
> +       /* We have to synchronise on the napi of each CPU */
> +       for_each_online_cpu(cpu) {
> +               struct mvneta_pcpu_port *pcpu_port =
> +                       per_cpu_ptr(pp->ports, cpu);
> +
> +               napi_synchronize(&pcpu_port->napi);
> +               napi_disable(&pcpu_port->napi);
> +       }
> +
> +       pp->rxq_def = pp->indir[0];
> +
> +       /* update unicast mapping */
> +       mvneta_set_rx_mode(pp->dev);
> +
> +       /* Update val of portCfg register accordingly with all RxQueue types */
> +       val = MVNETA_PORT_CONFIG_DEFL_VALUE(pp->rxq_def);
> +       mvreg_write(pp, MVNETA_PORT_CONFIG, val);
> +
> +       /* Update the elected CPU matching the new rxq_def */
> +       mvneta_percpu_elect(pp);
> +
> +       /* We have to synchronise on the napi of each CPU */
> +       for_each_online_cpu(cpu) {
> +               struct mvneta_pcpu_port *pcpu_port =
> +                       per_cpu_ptr(pp->ports, cpu);
> +
> +               napi_enable(&pcpu_port->napi);
> +       }
> +

rxq_def changed, but txq vs CPU mapping remained as in the beginning -
is it intentional?

Best regards,
Marcin

Re: [RFC PATCH 0/2] net: mvneta: Introduce RSS support

[Marcin Wojtas]

Hi Gregory,


> I also choose to associate all the TX queues on the same CPU that the
> one associated to the RX queue. It allows to contain all the
> interrupts on the same CPU. I think that an improvement on this side
> would be the support of the XPS.
>

Did you make some tries? E.g. after mapping certain txqs to CPU1, when
using them was mvneta_tx() executing only on this CPU? Or it rather
means that the irq will hit according to the mapping? As far as I know
the HW, the latter should be true, which would mean the real XPS with
this controller is impossible and the maximum we can control is the
irq.

I think it may be worth to unmask TX irqs on all cpus, so all percpu
napi's would be able to read tx_cause and process sent packets. I'm
looking forward to your opinion.

Best regards,
Marcin

Re: [RFC PATCH 2/2] net: mvneta: Add naive RSS support

[Gregory CLEMENT]

Hi Marcin,
 
[...]

>> +static int  mvneta_config_rss(struct mvneta_port *pp)
>> +{
>> +       int cpu;
>> +       u32 val;
>> +
>> +       netif_tx_stop_all_queues(pp->dev);
>> +
>> +       /* Mask all ethernet port interrupts */
>> +       mvreg_write(pp, MVNETA_INTR_NEW_MASK, 0);
>
> Shouldn't the interrupts be masked on each online cpu? There is percpu
> unmask function (mvneta_percpu_unmask_interrupt), so maybe ther should
> be also mvneta_percpu_mask_interrupt. With this masking should look
> like below:
>
>      for_each_online_cpu(cpu)
>                smp_call_function_single(cpu, mvneta_percpu_unmask_interrupt,
>                                         pp, true);

Indeed you are right, however I am a bit surprised to not had had issue
cause by this. I will fix it.

>
>> +       mvreg_write(pp, MVNETA_INTR_OLD_MASK, 0);
>> +       mvreg_write(pp, MVNETA_INTR_MISC_MASK, 0);
>> +
>> +       /* We have to synchronise on the napi of each CPU */
>> +       for_each_online_cpu(cpu) {
>> +               struct mvneta_pcpu_port *pcpu_port =
>> +                       per_cpu_ptr(pp->ports, cpu);
>> +
>> +               napi_synchronize(&pcpu_port->napi);
>> +               napi_disable(&pcpu_port->napi);
>> +       }
>> +
>> +       pp->rxq_def = pp->indir[0];
>> +
>> +       /* update unicast mapping */
>> +       mvneta_set_rx_mode(pp->dev);
>> +
>> +       /* Update val of portCfg register accordingly with all RxQueue types */
>> +       val = MVNETA_PORT_CONFIG_DEFL_VALUE(pp->rxq_def);
>> +       mvreg_write(pp, MVNETA_PORT_CONFIG, val);
>> +
>> +       /* Update the elected CPU matching the new rxq_def */
>> +       mvneta_percpu_elect(pp);
>> +
>> +       /* We have to synchronise on the napi of each CPU */
>> +       for_each_online_cpu(cpu) {
>> +               struct mvneta_pcpu_port *pcpu_port =
>> +                       per_cpu_ptr(pp->ports, cpu);
>> +
>> +               napi_enable(&pcpu_port->napi);
>> +       }
>> +
>
> rxq_def changed, but txq vs CPU mapping remained as in the beginning -
> is it intentional?

txq vs CPU mapping is change in the mvneta_percpu_elect() function.

Thanks for this prompt review

Gregory

-- 
Gregory Clement, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Re: [RFC PATCH 0/2] net: mvneta: Introduce RSS support

[Gregory CLEMENT]

Hi Marcin,
 
 On ven., nov. 06 2015, Marcin Wojtas <mw@semihalf.com> wrote:

> Hi Gregory,
>
>
>> I also choose to associate all the TX queues on the same CPU that the
>> one associated to the RX queue. It allows to contain all the
>> interrupts on the same CPU. I think that an improvement on this side
>> would be the support of the XPS.
>>
>
> Did you make some tries? E.g. after mapping certain txqs to CPU1, when
> using them was mvneta_tx() executing only on this CPU? Or it rather
> means that the irq will hit according to the mapping? As far as I know
> the HW, the latter should be true, which would mean the real XPS with
> this controller is impossible and the maximum we can control is the
> irq.

I hoped that with XPS we can associate CPUs to all the tx queues of a
ethernet port. For example choosing for eth2, that all the tx queues will
be handled by CPU0 and CPU3. And for this it would involve allowing
receiving the tx interrupts only on CPU0 and CPU3. This kind of feature
seems to be doable with the hardware.

But after reading more carefully what XPS is about and how it is used in
the kernel, it seems we can't configure the hardware from the sysfs
interface. From my understanding we can only indicate to the kernel
which tx queue use for a given CPU.

>
> I think it may be worth to unmask TX irqs on all cpus, so all percpu
> napi's would be able to read tx_cause and process sent packets. I'm
> looking forward to your opinion.

By doing, this my understanding is that when an TX interrupt occurs then
all the CPUs enter in the interrupt handler but only one can manage
it. So, for me it looks like a big waste of CPU load. I could understand
that it would be interesting in some situation but it doesn't seem to be
the good default behavior. That's why I was looking for a way to let the
use configure it according to his needs.

Please correct me if I am wrong somewhere, because currently I don't
find a good solution for it.


Thanks,

Gregory
-- 
Gregory Clement, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com