* [PATCH net] team: fix deadlock during setting of MTU
@ 2021-01-14 11:55 Ivan Vecera
2021-01-15 0:34 ` Saeed Mahameed
0 siblings, 1 reply; 4+ messages in thread
From: Ivan Vecera @ 2021-01-14 11:55 UTC (permalink / raw)
To: netdev; +Cc: Jiri Pirko
Team driver protects port list traversal in team_change_mtu()
by its team->lock mutex. This causes a deadlock with certain
devices that calls netdev_update_features() during their
.ndo_change_mtu() callback. In this case netdev_update_features()
calls team's netdevice notifier team_device_event() that in case
of NETDEV_FEAT_CHANGE tries lock team->lock mutex again.
Example (r8169 case):
...
[ 6391.348202] __mutex_lock.isra.6+0x2d0/0x4a0
[ 6391.358602] team_device_event+0x9d/0x160 [team]
[ 6391.363756] notifier_call_chain+0x47/0x70
[ 6391.368329] netdev_update_features+0x56/0x60
[ 6391.373207] rtl8169_change_mtu+0x14/0x50 [r8169]
[ 6391.378457] dev_set_mtu_ext+0xe1/0x1d0
[ 6391.387022] dev_set_mtu+0x52/0x90
[ 6391.390820] team_change_mtu+0x64/0xf0 [team]
[ 6391.395683] dev_set_mtu_ext+0xe1/0x1d0
[ 6391.399963] do_setlink+0x231/0xf50
...
To fix the problem the port list traversal in team_change_mtu() can
be protected by RCU read lock. In case of failure the failing port
is marked and unwind code-path is done also under RCU read lock
protection (but not in reverse order).
Fixes: 3d249d4ca7d0 ("net: introduce ethernet teaming device")
Cc: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Ivan Vecera <ivecera@redhat.com>
---
drivers/net/team/team.c | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/drivers/net/team/team.c b/drivers/net/team/team.c
index c19dac21c468..69d4b28beb17 100644
--- a/drivers/net/team/team.c
+++ b/drivers/net/team/team.c
@@ -1802,35 +1802,35 @@ static int team_set_mac_address(struct net_device *dev, void *p)
static int team_change_mtu(struct net_device *dev, int new_mtu)
{
struct team *team = netdev_priv(dev);
- struct team_port *port;
+ struct team_port *port, *fail_port;
int err;
- /*
- * Alhough this is reader, it's guarded by team lock. It's not possible
- * to traverse list in reverse under rcu_read_lock
- */
- mutex_lock(&team->lock);
+ rcu_read_lock();
team->port_mtu_change_allowed = true;
- list_for_each_entry(port, &team->port_list, list) {
+ list_for_each_entry_rcu(port, &team->port_list, list) {
err = dev_set_mtu(port->dev, new_mtu);
if (err) {
netdev_err(dev, "Device %s failed to change mtu",
port->dev->name);
+ fail_port = port;
goto unwind;
}
}
team->port_mtu_change_allowed = false;
- mutex_unlock(&team->lock);
+ rcu_read_unlock();
dev->mtu = new_mtu;
return 0;
unwind:
- list_for_each_entry_continue_reverse(port, &team->port_list, list)
+ list_for_each_entry_rcu(port, &team->port_list, list) {
+ if (port == fail_port)
+ break;
dev_set_mtu(port->dev, dev->mtu);
+ }
team->port_mtu_change_allowed = false;
- mutex_unlock(&team->lock);
+ rcu_read_unlock();
return err;
}
--
2.26.2
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH net] team: fix deadlock during setting of MTU
2021-01-14 11:55 [PATCH net] team: fix deadlock during setting of MTU Ivan Vecera
@ 2021-01-15 0:34 ` Saeed Mahameed
2021-01-15 8:24 ` Ivan Vecera
0 siblings, 1 reply; 4+ messages in thread
From: Saeed Mahameed @ 2021-01-15 0:34 UTC (permalink / raw)
To: Ivan Vecera, netdev; +Cc: Jiri Pirko
On Thu, 2021-01-14 at 12:55 +0100, Ivan Vecera wrote:
> Team driver protects port list traversal in team_change_mtu()
> by its team->lock mutex. This causes a deadlock with certain
> devices that calls netdev_update_features() during their
> .ndo_change_mtu() callback. In this case netdev_update_features()
> calls team's netdevice notifier team_device_event() that in case
> of NETDEV_FEAT_CHANGE tries lock team->lock mutex again.
>
> Example (r8169 case):
> ...
> [ 6391.348202] __mutex_lock.isra.6+0x2d0/0x4a0
> [ 6391.358602] team_device_event+0x9d/0x160 [team]
> [ 6391.363756] notifier_call_chain+0x47/0x70
> [ 6391.368329] netdev_update_features+0x56/0x60
> [ 6391.373207] rtl8169_change_mtu+0x14/0x50 [r8169]
> [ 6391.378457] dev_set_mtu_ext+0xe1/0x1d0
> [ 6391.387022] dev_set_mtu+0x52/0x90
> [ 6391.390820] team_change_mtu+0x64/0xf0 [team]
> [ 6391.395683] dev_set_mtu_ext+0xe1/0x1d0
> [ 6391.399963] do_setlink+0x231/0xf50
> ...
>
> To fix the problem the port list traversal in team_change_mtu() can
> be protected by RCU read lock. In case of failure the failing port
> is marked and unwind code-path is done also under RCU read lock
> protection (but not in reverse order).
>
> Fixes: 3d249d4ca7d0 ("net: introduce ethernet teaming device")
> Cc: Jiri Pirko <jiri@resnulli.us>
> Signed-off-by: Ivan Vecera <ivecera@redhat.com>
> ---
> drivers/net/team/team.c | 20 ++++++++++----------
> 1 file changed, 10 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/net/team/team.c b/drivers/net/team/team.c
> index c19dac21c468..69d4b28beb17 100644
> --- a/drivers/net/team/team.c
> +++ b/drivers/net/team/team.c
> @@ -1802,35 +1802,35 @@ static int team_set_mac_address(struct
> net_device *dev, void *p)
> static int team_change_mtu(struct net_device *dev, int new_mtu)
> {
> struct team *team = netdev_priv(dev);
> - struct team_port *port;
> + struct team_port *port, *fail_port;
> int err;
>
> - /*
> - * Alhough this is reader, it's guarded by team lock. It's not
> possible
> - * to traverse list in reverse under rcu_read_lock
> - */
> - mutex_lock(&team->lock);
> + rcu_read_lock();
> team->port_mtu_change_allowed = true;
> - list_for_each_entry(port, &team->port_list, list) {
> + list_for_each_entry_rcu(port, &team->port_list, list) {
> err = dev_set_mtu(port->dev, new_mtu);
some netdevs will need to sleep in their set_mtu, and you can't sleep
under rcu!
according to your explanation in the commit message the team->lock
mutex will be also taken under this rcu lock, so this is bad even
if dev_set_mtu does not sleep.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH net] team: fix deadlock during setting of MTU
2021-01-15 0:34 ` Saeed Mahameed
@ 2021-01-15 8:24 ` Ivan Vecera
2021-01-20 12:23 ` Ivan Vecera
0 siblings, 1 reply; 4+ messages in thread
From: Ivan Vecera @ 2021-01-15 8:24 UTC (permalink / raw)
To: Saeed Mahameed; +Cc: netdev, Jiri Pirko
On Thu, 14 Jan 2021 16:34:24 -0800
Saeed Mahameed <saeed@kernel.org> wrote:
> On Thu, 2021-01-14 at 12:55 +0100, Ivan Vecera wrote:
> > Team driver protects port list traversal in team_change_mtu()
> > by its team->lock mutex. This causes a deadlock with certain
> > devices that calls netdev_update_features() during their
> > .ndo_change_mtu() callback. In this case netdev_update_features()
> > calls team's netdevice notifier team_device_event() that in case
> > of NETDEV_FEAT_CHANGE tries lock team->lock mutex again.
> >
> > Example (r8169 case):
> > ...
> > [ 6391.348202] __mutex_lock.isra.6+0x2d0/0x4a0
> > [ 6391.358602] team_device_event+0x9d/0x160 [team]
> > [ 6391.363756] notifier_call_chain+0x47/0x70
> > [ 6391.368329] netdev_update_features+0x56/0x60
> > [ 6391.373207] rtl8169_change_mtu+0x14/0x50 [r8169]
> > [ 6391.378457] dev_set_mtu_ext+0xe1/0x1d0
> > [ 6391.387022] dev_set_mtu+0x52/0x90
> > [ 6391.390820] team_change_mtu+0x64/0xf0 [team]
> > [ 6391.395683] dev_set_mtu_ext+0xe1/0x1d0
> > [ 6391.399963] do_setlink+0x231/0xf50
> > ...
> >
> > To fix the problem the port list traversal in team_change_mtu() can
> > be protected by RCU read lock. In case of failure the failing port
> > is marked and unwind code-path is done also under RCU read lock
> > protection (but not in reverse order).
> >
> > Fixes: 3d249d4ca7d0 ("net: introduce ethernet teaming device")
> > Cc: Jiri Pirko <jiri@resnulli.us>
> > Signed-off-by: Ivan Vecera <ivecera@redhat.com>
> > ---
> > drivers/net/team/team.c | 20 ++++++++++----------
> > 1 file changed, 10 insertions(+), 10 deletions(-)
> >
> > diff --git a/drivers/net/team/team.c b/drivers/net/team/team.c
> > index c19dac21c468..69d4b28beb17 100644
> > --- a/drivers/net/team/team.c
> > +++ b/drivers/net/team/team.c
> > @@ -1802,35 +1802,35 @@ static int team_set_mac_address(struct
> > net_device *dev, void *p)
> > static int team_change_mtu(struct net_device *dev, int new_mtu)
> > {
> > struct team *team = netdev_priv(dev);
> > - struct team_port *port;
> > + struct team_port *port, *fail_port;
> > int err;
> >
> > - /*
> > - * Alhough this is reader, it's guarded by team lock. It's not
> > possible
> > - * to traverse list in reverse under rcu_read_lock
> > - */
> > - mutex_lock(&team->lock);
> > + rcu_read_lock();
> > team->port_mtu_change_allowed = true;
> > - list_for_each_entry(port, &team->port_list, list) {
> > + list_for_each_entry_rcu(port, &team->port_list, list) {
> > err = dev_set_mtu(port->dev, new_mtu);
>
> some netdevs will need to sleep in their set_mtu, and you can't sleep
> under rcu!
>
> according to your explanation in the commit message the team->lock
> mutex will be also taken under this rcu lock, so this is bad even
> if dev_set_mtu does not sleep.
>
Hmm, you are right... btw do we need to take this mutex at this place?
team_change_mtu() is protected by RTNL, team_device_event() as a netdevice
notifier as well... and team_{add,del}_slave() that modify port list
also.
Thoughts?
Thanks,
Ivan
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH net] team: fix deadlock during setting of MTU
2021-01-15 8:24 ` Ivan Vecera
@ 2021-01-20 12:23 ` Ivan Vecera
0 siblings, 0 replies; 4+ messages in thread
From: Ivan Vecera @ 2021-01-20 12:23 UTC (permalink / raw)
To: Saeed Mahameed; +Cc: netdev, Jiri Pirko
On Fri, 15 Jan 2021 09:24:05 +0100
Ivan Vecera <ivecera@redhat.com> wrote:
> > according to your explanation in the commit message the team->lock
> > mutex will be also taken under this rcu lock, so this is bad even
> > if dev_set_mtu does not sleep.
> >
> Hmm, you are right... btw do we need to take this mutex at this place?
>
> team_change_mtu() is protected by RTNL, team_device_event() as a netdevice
> notifier as well... and team_{add,del}_slave() that modify port list
> also.
>
> Thoughts?
After private discussion with Jiri, this is no-go as he would not like to
introduce a dependency on RTNL into team driver. The other solution is
to postpone features calculation in certain cases... the patch follows.
Ivan
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-01-20 12:53 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-14 11:55 [PATCH net] team: fix deadlock during setting of MTU Ivan Vecera
2021-01-15 0:34 ` Saeed Mahameed
2021-01-15 8:24 ` Ivan Vecera
2021-01-20 12:23 ` Ivan Vecera
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.