[PATCH] net/mlx5: setup RSS regardless of queue count

[PATCH] net/mlx5: setup RSS regardless of queue count

[Allain Legacy]

From: Dahir Osman <dahir.osman@windriver.com>

In some environments it is desirable to have the NIC perform RSS
normally on the packet regardless of the number of queues configured.
The RSS hash result that is stored in the mbuf can then be used by
the application to make decisions about how to distribute workloads
to threads, secondary processes, or even virtual machines if the
application is a virtual switch.  This change to the mlx5 driver
aligns with how other drivers in the Intel family work.

Signed-off-by: Allain Legacy <allain.legacy@windriver.com>
---
 drivers/net/mlx5/mlx5_rxq.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index ff58c4921..e6b05b0ad 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -1028,7 +1028,8 @@ mlx5_priv_rxq_new(struct priv *priv, uint16_t idx, uint16_t desc,
 	      tmpl->rxq.crc_present ? "disabled" : "enabled",
 	      tmpl->rxq.crc_present << 2);
 	/* Save port ID. */
-	tmpl->rxq.rss_hash = priv->rxqs_n > 1;
+	tmpl->rxq.rss_hash = (!!(dev->data->dev_conf.rxmode.mq_mode &
+				 ETH_MQ_RX_RSS));
 	tmpl->rxq.port_id = dev->data->port_id;
 	tmpl->priv = priv;
 	tmpl->rxq.mp = mp;
-- 
2.12.1

Re: [PATCH] net/mlx5: setup RSS regardless of queue count

[Nélio Laranjeiro]

On Mon, Mar 19, 2018 at 11:30:07AM -0500, Allain Legacy wrote:
> From: Dahir Osman <dahir.osman@windriver.com>
> 
> In some environments it is desirable to have the NIC perform RSS
> normally on the packet regardless of the number of queues configured.
> The RSS hash result that is stored in the mbuf can then be used by
> the application to make decisions about how to distribute workloads
> to threads, secondary processes, or even virtual machines if the
> application is a virtual switch.  This change to the mlx5 driver
> aligns with how other drivers in the Intel family work.
> 
> Signed-off-by: Allain Legacy <allain.legacy@windriver.com>
> ---
>  drivers/net/mlx5/mlx5_rxq.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
> index ff58c4921..e6b05b0ad 100644
> --- a/drivers/net/mlx5/mlx5_rxq.c
> +++ b/drivers/net/mlx5/mlx5_rxq.c
> @@ -1028,7 +1028,8 @@ mlx5_priv_rxq_new(struct priv *priv, uint16_t idx, uint16_t desc,
>  	      tmpl->rxq.crc_present ? "disabled" : "enabled",
>  	      tmpl->rxq.crc_present << 2);
>  	/* Save port ID. */
> -	tmpl->rxq.rss_hash = priv->rxqs_n > 1;
> +	tmpl->rxq.rss_hash = (!!(dev->data->dev_conf.rxmode.mq_mode &
> +				 ETH_MQ_RX_RSS));
>  	tmpl->rxq.port_id = dev->data->port_id;
>  	tmpl->priv = priv;
>  	tmpl->rxq.mp = mp;
> -- 
> 2.12.1

Unfortunately, is not enough to have a valid RSS hash result when the
PMD has a single Rx queue, a little more work needs to be handled in the
mlx5_flow.c engine to configure the hash field in Verbs Hash Rx queues
when a single queues is being used.

Thanks,

-- 
Nélio Laranjeiro
6WIND

Re: [PATCH] net/mlx5: setup RSS regardless of queue count

[Legacy, Allain]

> -----Original Message-----
> From: Nélio Laranjeiro [mailto:nelio.laranjeiro@6wind.com]
> Sent: Tuesday, March 20, 2018 8:26 AM
...
> 
> Unfortunately, is not enough to have a valid RSS hash result when the PMD
> has a single Rx queue, a little more work needs to be handled in the
> mlx5_flow.c engine to configure the hash field in Verbs Hash Rx queues
> when a single queues is being used.

Ok, thanks.  We will take another look at this to see why it seems to be working in our environment.

Allain

Re: [PATCH] net/mlx5: setup RSS regardless of queue count

[Yongseok Koh]

On Tue, Mar 20, 2018 at 01:26:08PM +0100, Nélio Laranjeiro wrote:
> On Mon, Mar 19, 2018 at 11:30:07AM -0500, Allain Legacy wrote:
> > From: Dahir Osman <dahir.osman@windriver.com>
> > 
> > In some environments it is desirable to have the NIC perform RSS
> > normally on the packet regardless of the number of queues configured.
> > The RSS hash result that is stored in the mbuf can then be used by
> > the application to make decisions about how to distribute workloads
> > to threads, secondary processes, or even virtual machines if the
> > application is a virtual switch.  This change to the mlx5 driver
> > aligns with how other drivers in the Intel family work.
> > 
> > Signed-off-by: Allain Legacy <allain.legacy@windriver.com>
> > ---
> >  drivers/net/mlx5/mlx5_rxq.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
> > index ff58c4921..e6b05b0ad 100644
> > --- a/drivers/net/mlx5/mlx5_rxq.c
> > +++ b/drivers/net/mlx5/mlx5_rxq.c
> > @@ -1028,7 +1028,8 @@ mlx5_priv_rxq_new(struct priv *priv, uint16_t idx, uint16_t desc,
> >  	      tmpl->rxq.crc_present ? "disabled" : "enabled",
> >  	      tmpl->rxq.crc_present << 2);
> >  	/* Save port ID. */
> > -	tmpl->rxq.rss_hash = priv->rxqs_n > 1;
> > +	tmpl->rxq.rss_hash = (!!(dev->data->dev_conf.rxmode.mq_mode &
> > +				 ETH_MQ_RX_RSS));
> >  	tmpl->rxq.port_id = dev->data->port_id;
> >  	tmpl->priv = priv;
> >  	tmpl->rxq.mp = mp;
> > -- 
> > 2.12.1
> 
> Unfortunately, is not enough to have a valid RSS hash result when the
> PMD has a single Rx queue, a little more work needs to be handled in the
> mlx5_flow.c engine to configure the hash field in Verbs Hash Rx queues
> when a single queues is being used.

It is good to have such feature like described in the commit message. And from
Allain's other email, it seems datapath of mlx5 ignores valid hash result from
HW for no reason if the number of queue is one. Allain, you wanted to fix that,
didn't you?

Nelio, can you please share a bit more idea of what should be done further in
mlx5_flow.c for this, so that Allain can come up with a right patch?


Thanks,
Yongseok

Re: [PATCH] net/mlx5: setup RSS regardless of queue count

[Nélio Laranjeiro]

On Tue, Mar 20, 2018 at 04:07:11PM -0700, Yongseok Koh wrote:
> On Tue, Mar 20, 2018 at 01:26:08PM +0100, Nélio Laranjeiro wrote:
> > On Mon, Mar 19, 2018 at 11:30:07AM -0500, Allain Legacy wrote:
> > > From: Dahir Osman <dahir.osman@windriver.com>
> > > 
> > > In some environments it is desirable to have the NIC perform RSS
> > > normally on the packet regardless of the number of queues configured.
> > > The RSS hash result that is stored in the mbuf can then be used by
> > > the application to make decisions about how to distribute workloads
> > > to threads, secondary processes, or even virtual machines if the
> > > application is a virtual switch.  This change to the mlx5 driver
> > > aligns with how other drivers in the Intel family work.
> > > 
> > > Signed-off-by: Allain Legacy <allain.legacy@windriver.com>
> > > ---
> > >  drivers/net/mlx5/mlx5_rxq.c | 3 ++-
> > >  1 file changed, 2 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
> > > index ff58c4921..e6b05b0ad 100644
> > > --- a/drivers/net/mlx5/mlx5_rxq.c
> > > +++ b/drivers/net/mlx5/mlx5_rxq.c
> > > @@ -1028,7 +1028,8 @@ mlx5_priv_rxq_new(struct priv *priv, uint16_t idx, uint16_t desc,
> > >  	      tmpl->rxq.crc_present ? "disabled" : "enabled",
> > >  	      tmpl->rxq.crc_present << 2);
> > >  	/* Save port ID. */
> > > -	tmpl->rxq.rss_hash = priv->rxqs_n > 1;
> > > +	tmpl->rxq.rss_hash = (!!(dev->data->dev_conf.rxmode.mq_mode &
> > > +				 ETH_MQ_RX_RSS));
> > >  	tmpl->rxq.port_id = dev->data->port_id;
> > >  	tmpl->priv = priv;
> > >  	tmpl->rxq.mp = mp;
> > > -- 
> > > 2.12.1
> > 
> > Unfortunately, is not enough to have a valid RSS hash result when the
> > PMD has a single Rx queue, a little more work needs to be handled in the
> > mlx5_flow.c engine to configure the hash field in Verbs Hash Rx queues
> > when a single queues is being used.
> 
> It is good to have such feature like described in the commit message. And from
> Allain's other email, it seems datapath of mlx5 ignores valid hash result from
> HW for no reason if the number of queue is one.

Single queue means no RSS, it is not the datapath which ignore it, it is
the control plane which does not configure it.

> Allain, you wanted to fix that, didn't you?

>From the patch above, it seems :)

> Nelio, can you please share a bit more idea of what should be done further in
> mlx5_flow.c for this, so that Allain can come up with a right patch?

The engine is not trivial as it needs to convert flow API to Verbs flows
which the restrictions from the MLX5 kernel driver.
Trying to give instructions ends by writing it down directly as I need
to test them to see if it does not break your requirements also.

Allain,  I will finalise this patch (today) and make some test, I'll
contact you once it is finish to see if the behavior you expect is
reached.

Thanks,

-- 
Nélio Laranjeiro
6WIND

Re: [PATCH] net/mlx5: setup RSS regardless of queue count

[Legacy, Allain]

> -----Original Message-----
> From: Nélio Laranjeiro [mailto:nelio.laranjeiro@6wind.com]
> Sent: Wednesday, March 21, 2018 4:34 AM
<...>
> The engine is not trivial as it needs to convert flow API to Verbs flows which
> the restrictions from the MLX5 kernel driver.
> Trying to give instructions ends by writing it down directly as I need to test
> them to see if it does not break your requirements also.
> 
> Allain,  I will finalise this patch (today) and make some test, I'll contact you
> once it is finish to see if the behavior you expect is reached.
> 
Great, thanks!

[PATCH v2] net/mlx5: setup RSS regardless of queue count

[Nelio Laranjeiro]

From: Dahir Osman <dahir.osman@windriver.com>

In some environments it is desirable to have the NIC perform RSS
normally on the packet regardless of the number of queues configured.
The RSS hash result that is stored in the mbuf can then be used by
the application to make decisions about how to distribute workloads
to threads, secondary processes, or even virtual machines if the
application is a virtual switch.  This change to the mlx5 driver
aligns with how other drivers in the Intel family work.

Signed-off-by: Allain Legacy <allain.legacy@windriver.com>
Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
---
 drivers/net/mlx5/mlx5_flow.c |  2 +-
 drivers/net/mlx5/mlx5_rss.c  | 10 ++++++++++
 drivers/net/mlx5/mlx5_rxq.c  |  3 ++-
 3 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index 3eecdc604..49600e174 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -909,7 +909,7 @@ mlx5_flow_convert_finalise(struct mlx5_flow_parse *parser)
 	unsigned int i;
 
 	/* Remove any other flow not matching the pattern. */
-	if (parser->queues_n == 1) {
+	if (parser->queues_n == 1 && !parser->rss_conf.rss_hf) {
 		for (i = 0; i != hash_rxq_init_n; ++i) {
 			if (i == HASH_RXQ_ETH)
 				continue;
diff --git a/drivers/net/mlx5/mlx5_rss.c b/drivers/net/mlx5/mlx5_rss.c
index 5ac650163..c490854e1 100644
--- a/drivers/net/mlx5/mlx5_rss.c
+++ b/drivers/net/mlx5/mlx5_rss.c
@@ -42,6 +42,8 @@ mlx5_rss_hash_update(struct rte_eth_dev *dev,
 		     struct rte_eth_rss_conf *rss_conf)
 {
 	struct priv *priv = dev->data->dev_private;
+	unsigned int i;
+	unsigned int idx;
 
 	if (rss_conf->rss_hf & MLX5_RSS_HF_MASK) {
 		rte_errno = EINVAL;
@@ -59,6 +61,14 @@ mlx5_rss_hash_update(struct rte_eth_dev *dev,
 		priv->rss_conf.rss_key_len = rss_conf->rss_key_len;
 	}
 	priv->rss_conf.rss_hf = rss_conf->rss_hf;
+	/* Enable the RSS hash in all Rx queues. */
+	for (i = 0, idx = 0; idx != priv->rxqs_n; ++i) {
+		if (!(*priv->rxqs)[i])
+			continue;
+		(*priv->rxqs)[i]->rss_hash = !!rss_conf->rss_hf &&
+			!!(dev->data->dev_conf.rxmode.mq_mode & ETH_MQ_RX_RSS);
+		++idx;
+	}
 	return 0;
 }
 
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 56b2d19f2..1b4570586 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -1070,7 +1070,8 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
 		tmpl->rxq.crc_present ? "disabled" : "enabled",
 		tmpl->rxq.crc_present << 2);
 	/* Save port ID. */
-	tmpl->rxq.rss_hash = priv->rxqs_n > 1;
+	tmpl->rxq.rss_hash = !!priv->rss_conf.rss_hf &&
+		(!!(dev->data->dev_conf.rxmode.mq_mode & ETH_MQ_RX_RSS));
 	tmpl->rxq.port_id = dev->data->port_id;
 	tmpl->priv = priv;
 	tmpl->rxq.mp = mp;
-- 
2.11.0

Re: [PATCH v2] net/mlx5: setup RSS regardless of queue count

[Nélio Laranjeiro]

On Wed, Mar 21, 2018 at 01:47:51PM +0100, Nelio Laranjeiro wrote:
> From: Dahir Osman <dahir.osman@windriver.com>
> 
> In some environments it is desirable to have the NIC perform RSS
> normally on the packet regardless of the number of queues configured.
> The RSS hash result that is stored in the mbuf can then be used by
> the application to make decisions about how to distribute workloads
> to threads, secondary processes, or even virtual machines if the
> application is a virtual switch.  This change to the mlx5 driver
> aligns with how other drivers in the Intel family work.
> 
> Signed-off-by: Allain Legacy <allain.legacy@windriver.com>
> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
> ---
>  drivers/net/mlx5/mlx5_flow.c |  2 +-
>  drivers/net/mlx5/mlx5_rss.c  | 10 ++++++++++
>  drivers/net/mlx5/mlx5_rxq.c  |  3 ++-
>  3 files changed, 13 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
> index 3eecdc604..49600e174 100644
> --- a/drivers/net/mlx5/mlx5_flow.c
> +++ b/drivers/net/mlx5/mlx5_flow.c
> @@ -909,7 +909,7 @@ mlx5_flow_convert_finalise(struct mlx5_flow_parse *parser)
>  	unsigned int i;
>  
>  	/* Remove any other flow not matching the pattern. */
> -	if (parser->queues_n == 1) {
> +	if (parser->queues_n == 1 && !parser->rss_conf.rss_hf) {
>  		for (i = 0; i != hash_rxq_init_n; ++i) {
>  			if (i == HASH_RXQ_ETH)
>  				continue;
> diff --git a/drivers/net/mlx5/mlx5_rss.c b/drivers/net/mlx5/mlx5_rss.c
> index 5ac650163..c490854e1 100644
> --- a/drivers/net/mlx5/mlx5_rss.c
> +++ b/drivers/net/mlx5/mlx5_rss.c
> @@ -42,6 +42,8 @@ mlx5_rss_hash_update(struct rte_eth_dev *dev,
>  		     struct rte_eth_rss_conf *rss_conf)
>  {
>  	struct priv *priv = dev->data->dev_private;
> +	unsigned int i;
> +	unsigned int idx;
>  
>  	if (rss_conf->rss_hf & MLX5_RSS_HF_MASK) {
>  		rte_errno = EINVAL;
> @@ -59,6 +61,14 @@ mlx5_rss_hash_update(struct rte_eth_dev *dev,
>  		priv->rss_conf.rss_key_len = rss_conf->rss_key_len;
>  	}
>  	priv->rss_conf.rss_hf = rss_conf->rss_hf;
> +	/* Enable the RSS hash in all Rx queues. */
> +	for (i = 0, idx = 0; idx != priv->rxqs_n; ++i) {
> +		if (!(*priv->rxqs)[i])
> +			continue;
> +		(*priv->rxqs)[i]->rss_hash = !!rss_conf->rss_hf &&
> +			!!(dev->data->dev_conf.rxmode.mq_mode & ETH_MQ_RX_RSS);
> +		++idx;
> +	}
>  	return 0;
>  }
>  
> diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
> index 56b2d19f2..1b4570586 100644
> --- a/drivers/net/mlx5/mlx5_rxq.c
> +++ b/drivers/net/mlx5/mlx5_rxq.c
> @@ -1070,7 +1070,8 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
>  		tmpl->rxq.crc_present ? "disabled" : "enabled",
>  		tmpl->rxq.crc_present << 2);
>  	/* Save port ID. */
> -	tmpl->rxq.rss_hash = priv->rxqs_n > 1;
> +	tmpl->rxq.rss_hash = !!priv->rss_conf.rss_hf &&
> +		(!!(dev->data->dev_conf.rxmode.mq_mode & ETH_MQ_RX_RSS));
>  	tmpl->rxq.port_id = dev->data->port_id;
>  	tmpl->priv = priv;
>  	tmpl->rxq.mp = mp;
> -- 
> 2.11.0

I forgot to say that this patch applies on top of [1] and [2].

Regards,

[1] https://dpdk.org/browse/next/dpdk-next-net-mlx
[2] https://dpdk.org/dev/patchwork/patch/36066/

-- 
Nélio Laranjeiro
6WIND

Re: [PATCH v2] net/mlx5: setup RSS regardless of queue count

[Yongseok Koh]

> On Mar 21, 2018, at 5:47 AM, Nelio Laranjeiro <nelio.laranjeiro@6wind.com> wrote:
> 
> From: Dahir Osman <dahir.osman@windriver.com>
> 
> In some environments it is desirable to have the NIC perform RSS
> normally on the packet regardless of the number of queues configured.
> The RSS hash result that is stored in the mbuf can then be used by
> the application to make decisions about how to distribute workloads
> to threads, secondary processes, or even virtual machines if the
> application is a virtual switch.  This change to the mlx5 driver
> aligns with how other drivers in the Intel family work.
> 
> Signed-off-by: Allain Legacy <allain.legacy@windriver.com>
> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
> ---

Dahir, Allain

Did you get a chance to test this patch? It would be good to have 'tested-by' tag
from you.


Acked-by: Yongseok Koh <yskoh@mellanox.com> 

Thanks

Re: [PATCH v2] net/mlx5: setup RSS regardless of queue count

[Legacy, Allain]

> -----Original Message-----
> From: Yongseok Koh [mailto:yskoh@mellanox.com]
> Sent: Thursday, March 22, 2018 7:38 PM
<..>
> 
> Dahir, Allain
> 
> Did you get a chance to test this patch? It would be good to have 'tested-by'
> tag from you.

I will be able to test it early next week.

Allain

Re: [PATCH v2] net/mlx5: setup RSS regardless of queue count

[Legacy, Allain]

> -----Original Message-----
> From: Yongseok Koh [mailto:yskoh@mellanox.com]
> Sent: Thursday, March 22, 2018 7:38 PM
<..>
> >
> > Signed-off-by: Allain Legacy <allain.legacy@windriver.com>
> > Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
> > ---
> 
> Dahir, Allain
> 
> Did you get a chance to test this patch? It would be good to have 'tested-by'
> tag from you.
> 
> 
> Acked-by: Yongseok Koh <yskoh@mellanox.com>

Tested-by: Allain Legacy <allain.legacy@windriver.com>

Re: [PATCH v2] net/mlx5: setup RSS regardless of queue count

[Shahaf Shuler]

Monday, March 26, 2018 9:09 PM, Legacy, Allain
> > -----Original Message-----
> > From: Yongseok Koh [mailto:yskoh@mellanox.com]
> > Sent: Thursday, March 22, 2018 7:38 PM
> <..>
> > >
> > > Signed-off-by: Allain Legacy <allain.legacy@windriver.com>
> > > Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
> > > ---
> >
> > Dahir, Allain
> >
> > Did you get a chance to test this patch? It would be good to have 'tested-
> by'
> > tag from you.
> >
> >
> > Acked-by: Yongseok Koh <yskoh@mellanox.com>
> 
> Tested-by: Allain Legacy <allain.legacy@windriver.com>


Applied to next-net-mlx, thanks.