* Re: [PATCH] net/bonding: update link status on slave add
2018-05-09 12:06 [PATCH] net/bonding: update link status on slave add Radu Nicolau
@ 2018-05-31 14:07 ` Ferruh Yigit
2018-05-31 16:10 ` [PATCH v3] net/bonding: fix slave add for mode 4 Radu Nicolau
` (2 subsequent siblings)
3 siblings, 0 replies; 10+ messages in thread
From: Ferruh Yigit @ 2018-05-31 14:07 UTC (permalink / raw)
To: Radu Nicolau, dev; +Cc: declan.doherty
On 5/9/2018 1:06 PM, Radu Nicolau wrote:
> Add a call to rte_eth_link_get_nowait on every slave to update
> the internal link status struct. Otherwise slave add will fail
> for mode 4 if the ports are all stopped but only one of them checked.
>
> Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
Hi Radu,
Can you please send a new version with updated commit log, with a fix title,
Fixes commit info, and the bugzilla id it is fixing?
Thanks,
ferruh
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH v3] net/bonding: fix slave add for mode 4
2018-05-09 12:06 [PATCH] net/bonding: update link status on slave add Radu Nicolau
2018-05-31 14:07 ` Ferruh Yigit
@ 2018-05-31 16:10 ` Radu Nicolau
2018-06-01 0:05 ` Chas Williams
2018-06-01 10:05 ` [PATCH v4] " Radu Nicolau
2018-07-18 13:21 ` [PATCH v5] " Radu Nicolau
3 siblings, 1 reply; 10+ messages in thread
From: Radu Nicolau @ 2018-05-31 16:10 UTC (permalink / raw)
To: dev; +Cc: declan.doherty, ferruh.yigit, Radu Nicolau
Add a call to rte_eth_link_get_nowait on every slave to update
the internal link status struct. Otherwise slave add will fail
for mode 4 if the ports are all stopped but only one of them checked.
Fixes: b77d21cc2364 ("ethdev: add link status get/set helper functions")
Bugzilla ID: 52
Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
---
v3: updated commit msg
v2: add fix and Bugzilla references
drivers/net/bonding/rte_eth_bond_api.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/net/bonding/rte_eth_bond_api.c b/drivers/net/bonding/rte_eth_bond_api.c
index d558df8..cad08b9 100644
--- a/drivers/net/bonding/rte_eth_bond_api.c
+++ b/drivers/net/bonding/rte_eth_bond_api.c
@@ -296,6 +296,8 @@ __eth_bond_slave_add_lock_free(uint16_t bonded_port_id, uint16_t slave_port_id)
return -1;
}
+ rte_eth_link_get_nowait(slave_port_id, &link_props);
+
slave_add(internals, slave_eth_dev);
/* We need to store slaves reta_size to be able to synchronize RETA for all
--
2.7.5
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [PATCH v3] net/bonding: fix slave add for mode 4
2018-05-31 16:10 ` [PATCH v3] net/bonding: fix slave add for mode 4 Radu Nicolau
@ 2018-06-01 0:05 ` Chas Williams
2018-06-01 9:59 ` Radu Nicolau
0 siblings, 1 reply; 10+ messages in thread
From: Chas Williams @ 2018-06-01 0:05 UTC (permalink / raw)
To: Radu Nicolau; +Cc: dev, Declan Doherty, Ferruh Yigit
It's not clear to me that the issue here is the bonding slave add.
You can only add started PMDs. When a PMD dev start is complete,
the PMD should have a valid link state and the link properties should be
valid. A few of the PMDs are very good about this, particularly the
ones with LSC interrupts. Those drivers often wait for the first
link interrupt before setting their link status. So there is a
race where the link state isn't well defined.
And lastly, why do we care what the link state is when adding a
slave? If the link state changes to down, do we remove the slave?
If the link speed of the slave changes, do we remove the slave?
So this test doesn't make much sense. For mode 4, you should be
able to add a slave, but if the link state doesn't match what
has been negotiated, then the slave should fail to activate.
On Thu, May 31, 2018 at 12:10 PM, Radu Nicolau <radu.nicolau@intel.com>
wrote:
>
> Add a call to rte_eth_link_get_nowait on every slave to update
> the internal link status struct. Otherwise slave add will fail
> for mode 4 if the ports are all stopped but only one of them checked.
>
> Fixes: b77d21cc2364 ("ethdev: add link status get/set helper functions")
> Bugzilla ID: 52
>
> Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
> ---
> v3: updated commit msg
> v2: add fix and Bugzilla references
>
> drivers/net/bonding/rte_eth_bond_api.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/drivers/net/bonding/rte_eth_bond_api.c
b/drivers/net/bonding/rte_eth_bond_api.c
> index d558df8..cad08b9 100644
> --- a/drivers/net/bonding/rte_eth_bond_api.c
> +++ b/drivers/net/bonding/rte_eth_bond_api.c
> @@ -296,6 +296,8 @@ __eth_bond_slave_add_lock_free(uint16_t
bonded_port_id, uint16_t slave_port_id)
> return -1;
> }
>
> + rte_eth_link_get_nowait(slave_port_id, &link_props);
> +
> slave_add(internals, slave_eth_dev);
>
> /* We need to store slaves reta_size to be able to synchronize
RETA for all
> --
> 2.7.5
>
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH v3] net/bonding: fix slave add for mode 4
2018-06-01 0:05 ` Chas Williams
@ 2018-06-01 9:59 ` Radu Nicolau
0 siblings, 0 replies; 10+ messages in thread
From: Radu Nicolau @ 2018-06-01 9:59 UTC (permalink / raw)
To: Chas Williams; +Cc: dev, Declan Doherty, Ferruh Yigit
On 6/1/2018 1:05 AM, Chas Williams wrote:
> It's not clear to me that the issue here is the bonding slave add.
> You can only add started PMDs. When a PMD dev start is complete,
> the PMD should have a valid link state and the link properties should be
> valid. A few of the PMDs are very good about this, particularly the
> ones with LSC interrupts. Those drivers often wait for the first
> link interrupt before setting their link status. So there is a
> race where the link state isn't well defined.
Indeed, the source of the problem is that the link state is not properly
reflected across all the ports. So, the issue steps are the following:
1. user issues "port stop all" in testpmd; when a port is stopped the
internal link state bits are cleared (and LSC interrupt will not run
from what I can see)
2. testpmd tries to update link status on all the ports, reads the state
of the first port updating the bits that were cleared; with a stopped
port ixgbe PMD and probably others sets the link_autoneg bit, but all
other bits remain cleared
3. seeing a link down, tespmd stops checking; now first port has the
link state link_autoneg bit set, but all other ports have it cleared
4. trying to create a bonded port in mode 4 fails because of the
link_autoneg bit
To reiterate, the issue is creating a bonding port with stopped ports
that have the link_status bits cleared, but not updated except the first
port. My fix updated the bits on all the ports.
>
> And lastly, why do we care what the link state is when adding a
> slave? If the link state changes to down, do we remove the slave?
> If the link speed of the slave changes, do we remove the slave?
> So this test doesn't make much sense. For mode 4, you should be
> able to add a slave, but if the link state doesn't match what
> has been negotiated, then the slave should fail to activate.
You are right, I will send an updated patch that checks the slave link
status before activation. This should also solve the initial issue, as
the link state will be already updated.
>
> On Thu, May 31, 2018 at 12:10 PM, Radu Nicolau <radu.nicolau@intel.com
> <mailto:radu.nicolau@intel.com>> wrote:
> >
> > Add a call to rte_eth_link_get_nowait on every slave to update
> > the internal link status struct. Otherwise slave add will fail
> > for mode 4 if the ports are all stopped but only one of them checked.
> >
> > Fixes: b77d21cc2364 ("ethdev: add link status get/set helper functions")
> > Bugzilla ID: 52
> >
> > Signed-off-by: Radu Nicolau <radu.nicolau@intel.com
> <mailto:radu.nicolau@intel.com>>
> > ---
> > v3: updated commit msg
> > v2: add fix and Bugzilla references
> >
> > drivers/net/bonding/rte_eth_bond_api.c | 2 ++
> > 1 file changed, 2 insertions(+)
> >
> > diff --git a/drivers/net/bonding/rte_eth_bond_api.c
> b/drivers/net/bonding/rte_eth_bond_api.c
> > index d558df8..cad08b9 100644
> > --- a/drivers/net/bonding/rte_eth_bond_api.c
> > +++ b/drivers/net/bonding/rte_eth_bond_api.c
> > @@ -296,6 +296,8 @@ __eth_bond_slave_add_lock_free(uint16_t
> bonded_port_id, uint16_t slave_port_id)
> > return -1;
> > }
> >
> > + rte_eth_link_get_nowait(slave_port_id, &link_props);
> > +
> > slave_add(internals, slave_eth_dev);
> >
> > /* We need to store slaves reta_size to be able to
> synchronize RETA for all
> > --
> > 2.7.5
> >
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v4] net/bonding: fix slave add for mode 4
2018-05-09 12:06 [PATCH] net/bonding: update link status on slave add Radu Nicolau
2018-05-31 14:07 ` Ferruh Yigit
2018-05-31 16:10 ` [PATCH v3] net/bonding: fix slave add for mode 4 Radu Nicolau
@ 2018-06-01 10:05 ` Radu Nicolau
2018-06-02 21:23 ` Chas Williams
2018-07-18 13:21 ` [PATCH v5] " Radu Nicolau
3 siblings, 1 reply; 10+ messages in thread
From: Radu Nicolau @ 2018-06-01 10:05 UTC (permalink / raw)
To: dev; +Cc: declan.doherty, ferruh.yigit, 3chas3, Radu Nicolau
Moved the link status validity check from the slave add to the slave
activation step. Otherwise slave add will fail for mode 4 if
the ports are all stopped but only one of them checked.
Fixes: b77d21cc2364 ("ethdev: add link status get/set helper functions")
Bugzilla ID: 52
Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
---
v4: reworked patch
v3: updated commit msg
v2: add fix and Bugzilla references
drivers/net/bonding/rte_eth_bond_api.c | 8 --------
drivers/net/bonding/rte_eth_bond_pmd.c | 9 ++++++++-
2 files changed, 8 insertions(+), 9 deletions(-)
diff --git a/drivers/net/bonding/rte_eth_bond_api.c b/drivers/net/bonding/rte_eth_bond_api.c
index d558df8..853b23b 100644
--- a/drivers/net/bonding/rte_eth_bond_api.c
+++ b/drivers/net/bonding/rte_eth_bond_api.c
@@ -345,14 +345,6 @@ __eth_bond_slave_add_lock_free(uint16_t bonded_port_id, uint16_t slave_port_id)
internals->tx_queue_offload_capa &= dev_info.tx_queue_offload_capa;
internals->flow_type_rss_offloads &= dev_info.flow_type_rss_offloads;
- if (link_properties_valid(bonded_eth_dev,
- &slave_eth_dev->data->dev_link) != 0) {
- RTE_BOND_LOG(ERR, "Invalid link properties for slave %d"
- " in bonding mode %d", slave_port_id,
- internals->mode);
- return -1;
- }
-
/* RETA size is GCD of all slaves RETA sizes, so, if all sizes will be
* the power of 2, the lower one is GCD
*/
diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index 02d94b1..b84af43 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -2679,7 +2679,14 @@ bond_ethdev_lsc_event_callback(uint16_t port_id, enum rte_eth_event_type type,
mac_address_slaves_update(bonded_eth_dev);
}
- activate_slave(bonded_eth_dev, port_id);
+ /* check link state properties */
+ if (link_properties_valid(bonded_eth_dev, &link)) {
+ activate_slave(bonded_eth_dev, port_id);
+ } else {
+ RTE_BOND_LOG(ERR, "Invalid link properties for slave %d"
+ " in bonding mode %d", port_id,
+ internals->mode);
+ }
/* If user has defined the primary port then default to using it */
if (internals->user_defined_primary_port &&
--
2.7.5
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [PATCH v4] net/bonding: fix slave add for mode 4
2018-06-01 10:05 ` [PATCH v4] " Radu Nicolau
@ 2018-06-02 21:23 ` Chas Williams
0 siblings, 0 replies; 10+ messages in thread
From: Chas Williams @ 2018-06-02 21:23 UTC (permalink / raw)
To: Radu Nicolau; +Cc: dev, Declan Doherty, Ferruh Yigit
This certainly seems more correct to me. I can't think of a reason that
you shouldn't be able to enslave whatever ports you like. Whether those
ports can become active is a different question. I need to think about
this a bit more before approval but I don't think it makes anything worse.
The current problems I see:
- There's still an activate_slave() at the bottom of
__eth_bond_slave_add_lock_free() -- We should be able to avoid this if
we fix LSC change to not miss slaves being added to the configuration (make
sure we get last_link_status right). That section is a little racy with
respect to stop/start and timer. If we don't do that, that activate still
need the guard against mis-matched link properties.
- What happens if you enslave multiple link down ports? The link speed is
taken from the first slave. What's speed of down link? How do other
interfaces match this property when they come up?
- We need to make sure deactivate/active happens on the LSC for the enslaved
interfaces all the time. A link may fail to activate, but if the down goes
down and comes back the "right" link speed, we need it to then activate.
- If all the links go down, should the "learned" default link properties be
cleared? We are a blank slate at this point, there's no reason not to
learn a new default from the first interface that comes up.
On Fri, Jun 1, 2018 at 6:05 AM, Radu Nicolau <radu.nicolau@intel.com> wrote:
> Moved the link status validity check from the slave add to the slave
> activation step. Otherwise slave add will fail for mode 4 if
> the ports are all stopped but only one of them checked.
>
> Fixes: b77d21cc2364 ("ethdev: add link status get/set helper functions")
> Bugzilla ID: 52
>
> Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
> ---
> v4: reworked patch
> v3: updated commit msg
> v2: add fix and Bugzilla references
>
> drivers/net/bonding/rte_eth_bond_api.c | 8 --------
> drivers/net/bonding/rte_eth_bond_pmd.c | 9 ++++++++-
> 2 files changed, 8 insertions(+), 9 deletions(-)
>
> diff --git a/drivers/net/bonding/rte_eth_bond_api.c
> b/drivers/net/bonding/rte_eth_bond_api.c
> index d558df8..853b23b 100644
> --- a/drivers/net/bonding/rte_eth_bond_api.c
> +++ b/drivers/net/bonding/rte_eth_bond_api.c
> @@ -345,14 +345,6 @@ __eth_bond_slave_add_lock_free(uint16_t
> bonded_port_id, uint16_t slave_port_id)
> internals->tx_queue_offload_capa &=
> dev_info.tx_queue_offload_capa;
> internals->flow_type_rss_offloads &=
> dev_info.flow_type_rss_offloads;
>
> - if (link_properties_valid(bonded_eth_dev,
> - &slave_eth_dev->data->dev_link) != 0) {
> - RTE_BOND_LOG(ERR, "Invalid link properties for
> slave %d"
> - " in bonding mode %d",
> slave_port_id,
> - internals->mode);
> - return -1;
> - }
> -
> /* RETA size is GCD of all slaves RETA sizes, so, if all
> sizes will be
> * the power of 2, the lower one is GCD
> */
> diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c
> b/drivers/net/bonding/rte_eth_bond_pmd.c
> index 02d94b1..b84af43 100644
> --- a/drivers/net/bonding/rte_eth_bond_pmd.c
> +++ b/drivers/net/bonding/rte_eth_bond_pmd.c
> @@ -2679,7 +2679,14 @@ bond_ethdev_lsc_event_callback(uint16_t port_id,
> enum rte_eth_event_type type,
> mac_address_slaves_update(bonded_eth_dev);
> }
>
> - activate_slave(bonded_eth_dev, port_id);
> + /* check link state properties */
> + if (link_properties_valid(bonded_eth_dev, &link)) {
> + activate_slave(bonded_eth_dev, port_id);
> + } else {
> + RTE_BOND_LOG(ERR, "Invalid link properties for
> slave %d"
> + " in bonding mode %d", port_id,
> + internals->mode);
> + }
>
> /* If user has defined the primary port then default to
> using it */
> if (internals->user_defined_primary_port &&
> --
> 2.7.5
>
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v5] net/bonding: fix slave add for mode 4
2018-05-09 12:06 [PATCH] net/bonding: update link status on slave add Radu Nicolau
` (2 preceding siblings ...)
2018-06-01 10:05 ` [PATCH v4] " Radu Nicolau
@ 2018-07-18 13:21 ` Radu Nicolau
2018-08-02 13:42 ` Doherty, Declan
3 siblings, 1 reply; 10+ messages in thread
From: Radu Nicolau @ 2018-07-18 13:21 UTC (permalink / raw)
To: dev; +Cc: declan.doherty, chas3, ferruh.yigit, Radu Nicolau, stable
Moved the link status validity check from the slave add to the slave
activation step. Otherwise slave add will fail for mode 4 if
the ports are all stopped but only one of them checked.
Removed activate slave call from slave add function.
Fixes: b77d21cc2364 ("ethdev: add link status get/set helper functions")
Bugzilla ID: 52
Cc: stable@dpdk.org
Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
---
v5: removed activate_slave call from slave add
v4: reworked patch
v3: updated commit msg
v2: add fix and Bugzilla references
drivers/net/bonding/rte_eth_bond_api.c | 13 -------------
drivers/net/bonding/rte_eth_bond_pmd.c | 11 +++++++++++
2 files changed, 11 insertions(+), 13 deletions(-)
diff --git a/drivers/net/bonding/rte_eth_bond_api.c b/drivers/net/bonding/rte_eth_bond_api.c
index 49fa2d7..8bc04cf 100644
--- a/drivers/net/bonding/rte_eth_bond_api.c
+++ b/drivers/net/bonding/rte_eth_bond_api.c
@@ -345,14 +345,6 @@ __eth_bond_slave_add_lock_free(uint16_t bonded_port_id, uint16_t slave_port_id)
internals->tx_queue_offload_capa &= dev_info.tx_queue_offload_capa;
internals->flow_type_rss_offloads &= dev_info.flow_type_rss_offloads;
- if (link_properties_valid(bonded_eth_dev,
- &slave_eth_dev->data->dev_link) != 0) {
- RTE_BOND_LOG(ERR, "Invalid link properties for slave %d"
- " in bonding mode %d", slave_port_id,
- internals->mode);
- return -1;
- }
-
/* RETA size is GCD of all slaves RETA sizes, so, if all sizes will be
* the power of 2, the lower one is GCD
*/
@@ -412,11 +404,6 @@ __eth_bond_slave_add_lock_free(uint16_t bonded_port_id, uint16_t slave_port_id)
!internals->user_defined_primary_port)
bond_ethdev_primary_set(internals,
slave_port_id);
-
- if (find_slave_by_id(internals->active_slaves,
- internals->active_slave_count,
- slave_port_id) == internals->active_slave_count)
- activate_slave(bonded_eth_dev, slave_port_id);
}
}
diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index fc4d4fd..c156888 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -2732,6 +2732,17 @@ bond_ethdev_lsc_event_callback(uint16_t port_id, enum rte_eth_event_type type,
mac_address_slaves_update(bonded_eth_dev);
}
+ /* check link state properties if bonded link is up*/
+ if (bonded_eth_dev->data->dev_link.link_status == ETH_LINK_UP) {
+ if (link_properties_valid(bonded_eth_dev, &link) != 0)
+ RTE_BOND_LOG(ERR, "Invalid link properties "
+ "for slave %d in bonding mode %d",
+ port_id, internals->mode);
+ } else {
+ /* inherit slave link properties */
+ link_properties_set(bonded_eth_dev, &link);
+ }
+
activate_slave(bonded_eth_dev, port_id);
/* If user has defined the primary port then default to using it */
--
2.7.5
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [PATCH v5] net/bonding: fix slave add for mode 4
2018-07-18 13:21 ` [PATCH v5] " Radu Nicolau
@ 2018-08-02 13:42 ` Doherty, Declan
2018-08-05 0:15 ` Thomas Monjalon
0 siblings, 1 reply; 10+ messages in thread
From: Doherty, Declan @ 2018-08-02 13:42 UTC (permalink / raw)
To: Radu Nicolau, dev; +Cc: chas3, ferruh.yigit, stable
On 18/07/2018 2:21 PM, Radu Nicolau wrote:
> Moved the link status validity check from the slave add to the slave
> activation step. Otherwise slave add will fail for mode 4 if
> the ports are all stopped but only one of them checked.
>
> Removed activate slave call from slave add function.
>
> Fixes: b77d21cc2364 ("ethdev: add link status get/set helper functions")
> Bugzilla ID: 52
> Cc: stable@dpdk.org
>
> Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
> ---
> v5: removed activate_slave call from slave add
> v4: reworked patch
> v3: updated commit msg
> v2: add fix and Bugzilla references
>
...
>
Acked-by: Declan Doherty <declan.doherty@intel.com>
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH v5] net/bonding: fix slave add for mode 4
2018-08-02 13:42 ` Doherty, Declan
@ 2018-08-05 0:15 ` Thomas Monjalon
0 siblings, 0 replies; 10+ messages in thread
From: Thomas Monjalon @ 2018-08-05 0:15 UTC (permalink / raw)
To: Radu Nicolau; +Cc: dev, Doherty, Declan, chas3, ferruh.yigit, stable
02/08/2018 15:42, Doherty, Declan:
> On 18/07/2018 2:21 PM, Radu Nicolau wrote:
> > Moved the link status validity check from the slave add to the slave
> > activation step. Otherwise slave add will fail for mode 4 if
> > the ports are all stopped but only one of them checked.
> >
> > Removed activate slave call from slave add function.
> >
> > Fixes: b77d21cc2364 ("ethdev: add link status get/set helper functions")
> > Bugzilla ID: 52
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
> > ---
> > v5: removed activate_slave call from slave add
> > v4: reworked patch
> > v3: updated commit msg
> > v2: add fix and Bugzilla references
> >
> ...
> >
>
> Acked-by: Declan Doherty <declan.doherty@intel.com>
Applied, thanks
^ permalink raw reply [flat|nested] 10+ messages in thread