[PATCH] net/bonding: update link status on slave add

[PATCH] net/bonding: update link status on slave add

[Radu Nicolau]

Add a call to rte_eth_link_get_nowait on every slave to update
the internal link status struct. Otherwise slave add will fail
for mode 4 if the ports are all stopped but only one of them checked.

Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
---
 drivers/net/bonding/rte_eth_bond_api.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/net/bonding/rte_eth_bond_api.c b/drivers/net/bonding/rte_eth_bond_api.c
index d558df8..cad08b9 100644
--- a/drivers/net/bonding/rte_eth_bond_api.c
+++ b/drivers/net/bonding/rte_eth_bond_api.c
@@ -296,6 +296,8 @@ __eth_bond_slave_add_lock_free(uint16_t bonded_port_id, uint16_t slave_port_id)
 		return -1;
 	}
 
+	rte_eth_link_get_nowait(slave_port_id, &link_props);
+
 	slave_add(internals, slave_eth_dev);
 
 	/* We need to store slaves reta_size to be able to synchronize RETA for all
-- 
2.7.5

Re: [PATCH] net/bonding: update link status on slave add

[Ferruh Yigit]

On 5/9/2018 1:06 PM, Radu Nicolau wrote:
> Add a call to rte_eth_link_get_nowait on every slave to update
> the internal link status struct. Otherwise slave add will fail
> for mode 4 if the ports are all stopped but only one of them checked.
> 
> Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>

Hi Radu,

Can you please send a new version with updated commit log, with a fix title,
Fixes commit info, and the bugzilla id it is fixing?

Thanks,
ferruh

[PATCH v3] net/bonding: fix slave add for mode 4

[Radu Nicolau]

Add a call to rte_eth_link_get_nowait on every slave to update
the internal link status struct. Otherwise slave add will fail
for mode 4 if the ports are all stopped but only one of them checked.

Fixes: b77d21cc2364 ("ethdev: add link status get/set helper functions")
Bugzilla ID: 52

Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
---
v3: updated commit msg
v2: add fix and Bugzilla references

 drivers/net/bonding/rte_eth_bond_api.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/net/bonding/rte_eth_bond_api.c b/drivers/net/bonding/rte_eth_bond_api.c
index d558df8..cad08b9 100644
--- a/drivers/net/bonding/rte_eth_bond_api.c
+++ b/drivers/net/bonding/rte_eth_bond_api.c
@@ -296,6 +296,8 @@ __eth_bond_slave_add_lock_free(uint16_t bonded_port_id, uint16_t slave_port_id)
 		return -1;
 	}
 
+	rte_eth_link_get_nowait(slave_port_id, &link_props);
+
 	slave_add(internals, slave_eth_dev);
 
 	/* We need to store slaves reta_size to be able to synchronize RETA for all
-- 
2.7.5

Re: [PATCH v3] net/bonding: fix slave add for mode 4

[Chas Williams]

It's not clear to me that the issue here is the bonding slave add.
You can only add started PMDs.  When a PMD dev start is complete,
the PMD should have a valid link state and the link properties should be
valid.  A few of the PMDs are very good about this, particularly the
ones with LSC interrupts.  Those drivers often wait for the first
link interrupt before setting their link status.  So there is a
race where the link state isn't well defined.

And lastly, why do we care what the link state is when adding a
slave?  If the link state changes to down, do we remove the slave?
If the link speed of the slave changes, do we remove the slave?
So this test doesn't make much sense.  For mode 4, you should be
able to add a slave, but if the link state doesn't match what
has been negotiated, then the slave should fail to activate.

On Thu, May 31, 2018 at 12:10 PM, Radu Nicolau <radu.nicolau@intel.com>
wrote:
>
> Add a call to rte_eth_link_get_nowait on every slave to update
> the internal link status struct. Otherwise slave add will fail
> for mode 4 if the ports are all stopped but only one of them checked.
>
> Fixes: b77d21cc2364 ("ethdev: add link status get/set helper functions")
> Bugzilla ID: 52
>
> Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
> ---
> v3: updated commit msg
> v2: add fix and Bugzilla references
>
>  drivers/net/bonding/rte_eth_bond_api.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/drivers/net/bonding/rte_eth_bond_api.c
b/drivers/net/bonding/rte_eth_bond_api.c
> index d558df8..cad08b9 100644
> --- a/drivers/net/bonding/rte_eth_bond_api.c
> +++ b/drivers/net/bonding/rte_eth_bond_api.c
> @@ -296,6 +296,8 @@ __eth_bond_slave_add_lock_free(uint16_t
bonded_port_id, uint16_t slave_port_id)
>                 return -1;
>         }
>
> +       rte_eth_link_get_nowait(slave_port_id, &link_props);
> +
>         slave_add(internals, slave_eth_dev);
>
>         /* We need to store slaves reta_size to be able to synchronize
RETA for all
> --
> 2.7.5
>

Re: [PATCH v3] net/bonding: fix slave add for mode 4

[Radu Nicolau]

On 6/1/2018 1:05 AM, Chas Williams wrote:
> It's not clear to me that the issue here is the bonding slave add.
> You can only add started PMDs.  When a PMD dev start is complete,
> the PMD should have a valid link state and the link properties should be
> valid.  A few of the PMDs are very good about this, particularly the
> ones with LSC interrupts.  Those drivers often wait for the first
> link interrupt before setting their link status.  So there is a
> race where the link state isn't well defined.
Indeed, the source of the problem is that the link state is not properly 
reflected across all the ports. So, the issue steps are the following:

1. user issues "port stop all" in testpmd; when a port is stopped the 
internal link state bits are cleared (and LSC interrupt will not run 
from what I can see)
2. testpmd tries to update link status on all the ports, reads the state 
of the first port updating the bits that were cleared; with a stopped 
port ixgbe PMD and probably others sets the link_autoneg bit, but all 
other bits remain cleared
3. seeing a link down, tespmd stops checking; now first port has the 
link state link_autoneg bit set, but all other ports have it cleared
4. trying to create a bonded port in mode 4 fails because of the 
link_autoneg bit

To reiterate, the issue is creating a bonding port with stopped ports 
that have the link_status bits cleared, but not updated except the first 
port. My fix updated the bits on all the ports.

>
> And lastly, why do we care what the link state is when adding a
> slave?  If the link state changes to down, do we remove the slave?
> If the link speed of the slave changes, do we remove the slave?
> So this test doesn't make much sense.  For mode 4, you should be
> able to add a slave, but if the link state doesn't match what
> has been negotiated, then the slave should fail to activate.
You are right, I will send an updated patch that checks the slave link 
status before activation. This should also solve the initial issue, as 
the link state will be already updated.

>
> On Thu, May 31, 2018 at 12:10 PM, Radu Nicolau <radu.nicolau@intel.com 
> <mailto:radu.nicolau@intel.com>> wrote:
> >
> > Add a call to rte_eth_link_get_nowait on every slave to update
> > the internal link status struct. Otherwise slave add will fail
> > for mode 4 if the ports are all stopped but only one of them checked.
> >
> > Fixes: b77d21cc2364 ("ethdev: add link status get/set helper functions")
> > Bugzilla ID: 52
> >
> > Signed-off-by: Radu Nicolau <radu.nicolau@intel.com 
> <mailto:radu.nicolau@intel.com>>
> > ---
> > v3: updated commit msg
> > v2: add fix and Bugzilla references
> >
> >  drivers/net/bonding/rte_eth_bond_api.c | 2 ++
> >  1 file changed, 2 insertions(+)
> >
> > diff --git a/drivers/net/bonding/rte_eth_bond_api.c 
> b/drivers/net/bonding/rte_eth_bond_api.c
> > index d558df8..cad08b9 100644
> > --- a/drivers/net/bonding/rte_eth_bond_api.c
> > +++ b/drivers/net/bonding/rte_eth_bond_api.c
> > @@ -296,6 +296,8 @@ __eth_bond_slave_add_lock_free(uint16_t 
> bonded_port_id, uint16_t slave_port_id)
> >                 return -1;
> >         }
> >
> > +       rte_eth_link_get_nowait(slave_port_id, &link_props);
> > +
> >         slave_add(internals, slave_eth_dev);
> >
> >         /* We need to store slaves reta_size to be able to 
> synchronize RETA for all
> > --
> > 2.7.5
> >

[PATCH v4] net/bonding: fix slave add for mode 4

[Radu Nicolau]

Moved the link status validity check from the slave add to the slave
activation step. Otherwise slave add will fail for mode 4 if
the ports are all stopped but only one of them checked.

Fixes: b77d21cc2364 ("ethdev: add link status get/set helper functions")
Bugzilla ID: 52

Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
---
v4: reworked patch
v3: updated commit msg
v2: add fix and Bugzilla references

 drivers/net/bonding/rte_eth_bond_api.c | 8 --------
 drivers/net/bonding/rte_eth_bond_pmd.c | 9 ++++++++-
 2 files changed, 8 insertions(+), 9 deletions(-)

diff --git a/drivers/net/bonding/rte_eth_bond_api.c b/drivers/net/bonding/rte_eth_bond_api.c
index d558df8..853b23b 100644
--- a/drivers/net/bonding/rte_eth_bond_api.c
+++ b/drivers/net/bonding/rte_eth_bond_api.c
@@ -345,14 +345,6 @@ __eth_bond_slave_add_lock_free(uint16_t bonded_port_id, uint16_t slave_port_id)
 		internals->tx_queue_offload_capa &= dev_info.tx_queue_offload_capa;
 		internals->flow_type_rss_offloads &= dev_info.flow_type_rss_offloads;
 
-		if (link_properties_valid(bonded_eth_dev,
-				&slave_eth_dev->data->dev_link) != 0) {
-			RTE_BOND_LOG(ERR, "Invalid link properties for slave %d"
-					" in bonding mode %d", slave_port_id,
-					internals->mode);
-			return -1;
-		}
-
 		/* RETA size is GCD of all slaves RETA sizes, so, if all sizes will be
 		 * the power of 2, the lower one is GCD
 		 */
diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index 02d94b1..b84af43 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -2679,7 +2679,14 @@ bond_ethdev_lsc_event_callback(uint16_t port_id, enum rte_eth_event_type type,
 			mac_address_slaves_update(bonded_eth_dev);
 		}
 
-		activate_slave(bonded_eth_dev, port_id);
+		/* check link state properties */
+		if (link_properties_valid(bonded_eth_dev, &link)) {
+			activate_slave(bonded_eth_dev, port_id);
+		} else {
+			RTE_BOND_LOG(ERR, "Invalid link properties for slave %d"
+				     " in bonding mode %d", port_id,
+				     internals->mode);
+		}
 
 		/* If user has defined the primary port then default to using it */
 		if (internals->user_defined_primary_port &&
-- 
2.7.5

Re: [PATCH v4] net/bonding: fix slave add for mode 4

[Chas Williams]

This certainly seems more correct to me.  I can't think of a reason that
you shouldn't be able to enslave whatever ports you like.  Whether those
ports can become active is a different question.  I need to think about
this a bit more before approval but I don't think it makes anything worse.

The current problems I see:

- There's still an activate_slave() at the bottom of
 __eth_bond_slave_add_lock_free() -- We should be able to avoid this if
we fix LSC change to not miss slaves being added to the configuration (make
sure we get last_link_status right).  That section is a little racy with
respect to stop/start and timer.  If we don't do that, that activate still
need the guard against mis-matched link properties.

- What happens if you enslave multiple link down ports?  The link speed is
taken from the first slave.  What's speed of down link?  How do other
interfaces match this property when they come up?

- We need to make sure deactivate/active happens on the LSC for the enslaved
interfaces all the time.  A link may fail to activate, but if the down goes
down and comes back the "right" link speed, we need it to then activate.

- If all the links go down, should the "learned" default link properties be
cleared?  We are a blank slate at this point, there's no reason not to
learn a new default from the first interface that comes up.

On Fri, Jun 1, 2018 at 6:05 AM, Radu Nicolau <radu.nicolau@intel.com> wrote:

> Moved the link status validity check from the slave add to the slave
> activation step. Otherwise slave add will fail for mode 4 if
> the ports are all stopped but only one of them checked.
>
> Fixes: b77d21cc2364 ("ethdev: add link status get/set helper functions")
> Bugzilla ID: 52
>
> Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
> ---
> v4: reworked patch
> v3: updated commit msg
> v2: add fix and Bugzilla references
>
>  drivers/net/bonding/rte_eth_bond_api.c | 8 --------
>  drivers/net/bonding/rte_eth_bond_pmd.c | 9 ++++++++-
>  2 files changed, 8 insertions(+), 9 deletions(-)
>
> diff --git a/drivers/net/bonding/rte_eth_bond_api.c
> b/drivers/net/bonding/rte_eth_bond_api.c
> index d558df8..853b23b 100644
> --- a/drivers/net/bonding/rte_eth_bond_api.c
> +++ b/drivers/net/bonding/rte_eth_bond_api.c
> @@ -345,14 +345,6 @@ __eth_bond_slave_add_lock_free(uint16_t
> bonded_port_id, uint16_t slave_port_id)
>                 internals->tx_queue_offload_capa &=
> dev_info.tx_queue_offload_capa;
>                 internals->flow_type_rss_offloads &=
> dev_info.flow_type_rss_offloads;
>
> -               if (link_properties_valid(bonded_eth_dev,
> -                               &slave_eth_dev->data->dev_link) != 0) {
> -                       RTE_BOND_LOG(ERR, "Invalid link properties for
> slave %d"
> -                                       " in bonding mode %d",
> slave_port_id,
> -                                       internals->mode);
> -                       return -1;
> -               }
> -
>                 /* RETA size is GCD of all slaves RETA sizes, so, if all
> sizes will be
>                  * the power of 2, the lower one is GCD
>                  */
> diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c
> b/drivers/net/bonding/rte_eth_bond_pmd.c
> index 02d94b1..b84af43 100644
> --- a/drivers/net/bonding/rte_eth_bond_pmd.c
> +++ b/drivers/net/bonding/rte_eth_bond_pmd.c
> @@ -2679,7 +2679,14 @@ bond_ethdev_lsc_event_callback(uint16_t port_id,
> enum rte_eth_event_type type,
>                         mac_address_slaves_update(bonded_eth_dev);
>                 }
>
> -               activate_slave(bonded_eth_dev, port_id);
> +               /* check link state properties */
> +               if (link_properties_valid(bonded_eth_dev, &link)) {
> +                       activate_slave(bonded_eth_dev, port_id);
> +               } else {
> +                       RTE_BOND_LOG(ERR, "Invalid link properties for
> slave %d"
> +                                    " in bonding mode %d", port_id,
> +                                    internals->mode);
> +               }
>
>                 /* If user has defined the primary port then default to
> using it */
>                 if (internals->user_defined_primary_port &&
> --
> 2.7.5
>
>

[PATCH v5] net/bonding: fix slave add for mode 4

[Radu Nicolau]

Moved the link status validity check from the slave add to the slave
activation step. Otherwise slave add will fail for mode 4 if
the ports are all stopped but only one of them checked.

Removed activate slave call from slave add function.

Fixes: b77d21cc2364 ("ethdev: add link status get/set helper functions")
Bugzilla ID: 52
Cc: stable@dpdk.org

Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
---
v5: removed activate_slave call from slave add
v4: reworked patch
v3: updated commit msg
v2: add fix and Bugzilla references

 drivers/net/bonding/rte_eth_bond_api.c | 13 -------------
 drivers/net/bonding/rte_eth_bond_pmd.c | 11 +++++++++++
 2 files changed, 11 insertions(+), 13 deletions(-)

diff --git a/drivers/net/bonding/rte_eth_bond_api.c b/drivers/net/bonding/rte_eth_bond_api.c
index 49fa2d7..8bc04cf 100644
--- a/drivers/net/bonding/rte_eth_bond_api.c
+++ b/drivers/net/bonding/rte_eth_bond_api.c
@@ -345,14 +345,6 @@ __eth_bond_slave_add_lock_free(uint16_t bonded_port_id, uint16_t slave_port_id)
 		internals->tx_queue_offload_capa &= dev_info.tx_queue_offload_capa;
 		internals->flow_type_rss_offloads &= dev_info.flow_type_rss_offloads;
 
-		if (link_properties_valid(bonded_eth_dev,
-				&slave_eth_dev->data->dev_link) != 0) {
-			RTE_BOND_LOG(ERR, "Invalid link properties for slave %d"
-					" in bonding mode %d", slave_port_id,
-					internals->mode);
-			return -1;
-		}
-
 		/* RETA size is GCD of all slaves RETA sizes, so, if all sizes will be
 		 * the power of 2, the lower one is GCD
 		 */
@@ -412,11 +404,6 @@ __eth_bond_slave_add_lock_free(uint16_t bonded_port_id, uint16_t slave_port_id)
 			    !internals->user_defined_primary_port)
 				bond_ethdev_primary_set(internals,
 							slave_port_id);
-
-			if (find_slave_by_id(internals->active_slaves,
-					     internals->active_slave_count,
-					     slave_port_id) == internals->active_slave_count)
-				activate_slave(bonded_eth_dev, slave_port_id);
 		}
 	}
 
diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index fc4d4fd..c156888 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -2732,6 +2732,17 @@ bond_ethdev_lsc_event_callback(uint16_t port_id, enum rte_eth_event_type type,
 			mac_address_slaves_update(bonded_eth_dev);
 		}
 
+		/* check link state properties if bonded link is up*/
+		if (bonded_eth_dev->data->dev_link.link_status == ETH_LINK_UP) {
+			if (link_properties_valid(bonded_eth_dev, &link) != 0)
+				RTE_BOND_LOG(ERR, "Invalid link properties "
+					     "for slave %d in bonding mode %d",
+					     port_id, internals->mode);
+		} else {
+			/* inherit slave link properties */
+			link_properties_set(bonded_eth_dev, &link);
+		}
+
 		activate_slave(bonded_eth_dev, port_id);
 
 		/* If user has defined the primary port then default to using it */
-- 
2.7.5

Re: [PATCH v5] net/bonding: fix slave add for mode 4

[Doherty, Declan]

On 18/07/2018 2:21 PM, Radu Nicolau wrote:
> Moved the link status validity check from the slave add to the slave
> activation step. Otherwise slave add will fail for mode 4 if
> the ports are all stopped but only one of them checked.
> 
> Removed activate slave call from slave add function.
> 
> Fixes: b77d21cc2364 ("ethdev: add link status get/set helper functions")
> Bugzilla ID: 52
> Cc: stable@dpdk.org
> 
> Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
> ---
> v5: removed activate_slave call from slave add
> v4: reworked patch
> v3: updated commit msg
> v2: add fix and Bugzilla references
> 
...
> 

Acked-by: Declan Doherty <declan.doherty@intel.com>

Re: [PATCH v5] net/bonding: fix slave add for mode 4

[Thomas Monjalon]

02/08/2018 15:42, Doherty, Declan:
> On 18/07/2018 2:21 PM, Radu Nicolau wrote:
> > Moved the link status validity check from the slave add to the slave
> > activation step. Otherwise slave add will fail for mode 4 if
> > the ports are all stopped but only one of them checked.
> > 
> > Removed activate slave call from slave add function.
> > 
> > Fixes: b77d21cc2364 ("ethdev: add link status get/set helper functions")
> > Bugzilla ID: 52
> > Cc: stable@dpdk.org
> > 
> > Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
> > ---
> > v5: removed activate_slave call from slave add
> > v4: reworked patch
> > v3: updated commit msg
> > v2: add fix and Bugzilla references
> > 
> ...
> > 
> 
> Acked-by: Declan Doherty <declan.doherty@intel.com>

Applied, thanks