From: Nikolay Aleksandrov <nikolay@cumulusnetworks.com> To: Ido Schimmel <idosch@idosch.org>, Stefan Priebe - Profihost AG <s.priebe@profihost.ag> Cc: roopa@cumulusnetworks.com, davem@davemloft.net, "bridge@lists.linux-foundation.org" <bridge@lists.linux-foundation.org>, netdev@vger.kernel.org Subject: Re: BUG: soft lockup while deleting tap interface from vlan aware bridge Date: Thu, 30 Apr 2020 14:20:23 +0300 [thread overview] Message-ID: <4b3a6079-d8d4-24c5-8fc9-15bcb96bca80@cumulusnetworks.com> (raw) In-Reply-To: <20200430105551.GA4068275@splinter> On 30/04/2020 13:55, Ido Schimmel wrote: > On Wed, Apr 29, 2020 at 10:52:35PM +0200, Stefan Priebe - Profihost AG wrote: >> Hello, >> >> while running a stable vanilla kernel 4.19.115 i'm reproducably get this >> one: >> >> watchdog: BUG: soft lockup - CPU#38 stuck for 22s! [bridge:3570653] >> >> ... >> >> Call >> Trace:nbp_vlan_delete+0x59/0xa0br_vlan_info+0x66/0xd0br_afspec+0x18c/0x1d0br_dellink+0x74/0xd0rtnl_bridge_dellink+0x110/0x220rtnetlink_rcv_msg+0x283/0x360 > > Nik, Stefan, > > My theory is that 4K VLANs are deleted in a batch and preemption is > disabled (please confirm). For each VLAN the kernel needs to go over the Right, that's what I was expecting. :-) > entire FDB and delete affected entries. If the FDB is very large or the > FDB lock is contended this can cause the kernel to loop for more than 20 > seconds without calling schedule(). Indeed, we already have that issue also with expire which goes over all entries. I have rough patches that improve the situation from way back, will have to go over and polish them to submit when I got more time. Long ago I've tested it with expiring 10 million entries but on a rather powerful CPU. > > To reproduce I added mdelay(100) in br_fdb_delete_by_port() and ran > this: > > ip link add name br10 up type bridge vlan_filtering 1 > ip link add name dummy10 up type dummy > ip link set dev dummy10 master br10 > bridge vlan add vid 1-4094 dev dummy10 master > bridge vlan del vid 1-4094 dev dummy10 master > > Got a similar trace to Stefan's. Seems to be fixed by attached: > > diff --git a/net/bridge/br_netlink.c b/net/bridge/br_netlink.c > index a774e19c41bb..240e260e3461 100644 > --- a/net/bridge/br_netlink.c > +++ b/net/bridge/br_netlink.c > @@ -615,6 +615,7 @@ int br_process_vlan_info(struct net_bridge *br, > v - 1, rtm_cmd); > v_change_start = 0; > } > + cond_resched(); > } > /* v_change_start is set only if the last/whole range changed */ > if (v_change_start) > > WDYT? > Maybe we can batch the deletes at say 32 at a time? Otherwise looks good to me, thanks!
WARNING: multiple messages have this Message-ID (diff)
From: Nikolay Aleksandrov <nikolay@cumulusnetworks.com> To: Ido Schimmel <idosch@idosch.org>, Stefan Priebe - Profihost AG <s.priebe@profihost.ag> Cc: netdev@vger.kernel.org, roopa@cumulusnetworks.com, "bridge@lists.linux-foundation.org" <bridge@lists.linux-foundation.org>, davem@davemloft.net Subject: Re: [Bridge] BUG: soft lockup while deleting tap interface from vlan aware bridge Date: Thu, 30 Apr 2020 14:20:23 +0300 [thread overview] Message-ID: <4b3a6079-d8d4-24c5-8fc9-15bcb96bca80@cumulusnetworks.com> (raw) In-Reply-To: <20200430105551.GA4068275@splinter> On 30/04/2020 13:55, Ido Schimmel wrote: > On Wed, Apr 29, 2020 at 10:52:35PM +0200, Stefan Priebe - Profihost AG wrote: >> Hello, >> >> while running a stable vanilla kernel 4.19.115 i'm reproducably get this >> one: >> >> watchdog: BUG: soft lockup - CPU#38 stuck for 22s! [bridge:3570653] >> >> ... >> >> Call >> Trace:nbp_vlan_delete+0x59/0xa0br_vlan_info+0x66/0xd0br_afspec+0x18c/0x1d0br_dellink+0x74/0xd0rtnl_bridge_dellink+0x110/0x220rtnetlink_rcv_msg+0x283/0x360 > > Nik, Stefan, > > My theory is that 4K VLANs are deleted in a batch and preemption is > disabled (please confirm). For each VLAN the kernel needs to go over the Right, that's what I was expecting. :-) > entire FDB and delete affected entries. If the FDB is very large or the > FDB lock is contended this can cause the kernel to loop for more than 20 > seconds without calling schedule(). Indeed, we already have that issue also with expire which goes over all entries. I have rough patches that improve the situation from way back, will have to go over and polish them to submit when I got more time. Long ago I've tested it with expiring 10 million entries but on a rather powerful CPU. > > To reproduce I added mdelay(100) in br_fdb_delete_by_port() and ran > this: > > ip link add name br10 up type bridge vlan_filtering 1 > ip link add name dummy10 up type dummy > ip link set dev dummy10 master br10 > bridge vlan add vid 1-4094 dev dummy10 master > bridge vlan del vid 1-4094 dev dummy10 master > > Got a similar trace to Stefan's. Seems to be fixed by attached: > > diff --git a/net/bridge/br_netlink.c b/net/bridge/br_netlink.c > index a774e19c41bb..240e260e3461 100644 > --- a/net/bridge/br_netlink.c > +++ b/net/bridge/br_netlink.c > @@ -615,6 +615,7 @@ int br_process_vlan_info(struct net_bridge *br, > v - 1, rtm_cmd); > v_change_start = 0; > } > + cond_resched(); > } > /* v_change_start is set only if the last/whole range changed */ > if (v_change_start) > > WDYT? > Maybe we can batch the deletes at say 32 at a time? Otherwise looks good to me, thanks!
next prev parent reply other threads:[~2020-04-30 11:20 UTC|newest] Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-04-29 20:52 BUG: soft lockup while deleting tap interface from vlan aware bridge Stefan Priebe - Profihost AG 2020-04-29 21:23 ` Nikolay Aleksandrov 2020-04-29 21:23 ` [Bridge] " Nikolay Aleksandrov 2020-04-30 6:47 ` Stefan Priebe - Profihost AG 2020-04-30 10:55 ` Ido Schimmel 2020-04-30 10:55 ` [Bridge] " Ido Schimmel 2020-04-30 11:20 ` Nikolay Aleksandrov [this message] 2020-04-30 11:20 ` Nikolay Aleksandrov 2020-04-30 15:56 ` Ido Schimmel 2020-04-30 15:56 ` [Bridge] " Ido Schimmel 2020-04-30 15:57 ` Nikolay Aleksandrov 2020-04-30 15:57 ` [Bridge] " Nikolay Aleksandrov 2020-04-30 13:18 ` Stefan Priebe - Profihost AG 2020-04-30 15:57 ` Ido Schimmel 2020-04-30 15:57 ` [Bridge] " Ido Schimmel
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=4b3a6079-d8d4-24c5-8fc9-15bcb96bca80@cumulusnetworks.com \ --to=nikolay@cumulusnetworks.com \ --cc=bridge@lists.linux-foundation.org \ --cc=davem@davemloft.net \ --cc=idosch@idosch.org \ --cc=netdev@vger.kernel.org \ --cc=roopa@cumulusnetworks.com \ --cc=s.priebe@profihost.ag \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.