From: Thomas Gleixner <tglx@linutronix.de>
To: LKML <linux-kernel@vger.kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>,
Linus Torvalds <torvalds@linuxfoundation.org>,
Paul McKenney <paulmck@kernel.org>,
Matthew Wilcox <willy@infradead.org>,
Christian Benvenuti <benve@cisco.com>,
Govindarajulu Varadarajan <_govind@gmx.com>,
Dave Miller <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>,
netdev@vger.kernel.org, "David S. Miller" <davem@davemloft.net>,
Jonathan Corbet <corbet@lwn.net>,
Mauro Carvalho Chehab <mchehab+huawei@kernel.org>,
linux-doc@vger.kernel.org,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
Luc Van Oostenryck <luc.vanoostenryck@gmail.com>,
Jay Cliburn <jcliburn@gmail.com>,
Chris Snook <chris.snook@gmail.com>,
Vishal Kulkarni <vishal@chelsio.com>,
Jeff Kirsher <jeffrey.t.kirsher@intel.com>,
intel-wired-lan@lists.osuosl.org,
Shannon Nelson <snelson@pensando.io>,
Pensando Drivers <drivers@pensando.io>,
Andrew Lunn <andrew@lunn.ch>,
Heiner Kallweit <hkallweit1@gmail.com>,
Russell King <linux@armlinux.org.uk>,
Thomas Bogendoerfer <tsbogend@alpha.franken.de>,
Solarflare linux maintainers <linux-net-drivers@solarflare.com>,
Edward Cree <ecree@solarflare.com>,
Martin Habets <mhabets@solarflare.com>,
Jon Mason <jdmason@kudzu.us>, Daniel Drake <dsd@gentoo.org>,
Ulrich Kunitz <kune@deine-taler.de>,
Kalle Valo <kvalo@codeaurora.org>,
linux-wireless@vger.kernel.org, linux-usb@vger.kernel.org,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Arend van Spriel <arend.vanspriel@broadcom.com>,
Franky Lin <franky.lin@broadcom.com>,
Hante Meuleman <hante.meuleman@broadcom.com>,
Chi-Hsien Lin <chi-hsien.lin@cypress.com>,
Wright Feng <wright.feng@cypress.com>,
brcm80211-dev-list.pdl@broadcom.com,
brcm80211-dev-list@cypress.com,
Stanislav Yakovlev <stas.yakovlev@gmail.com>,
Stanislaw Gruszka <stf_xl@wp.pl>,
Johannes Berg <johannes.berg@intel.com>,
Emmanuel Grumbach <emmanuel.grumbach@intel.com>,
Luca Coelho <luciano.coelho@intel.com>,
Intel Linux Wireless <linuxwifi@intel.com>,
Jouni Malinen <j@w1.fi>, Amitkumar Karwar <amitkarwar@gmail.com>,
Ganapathi Bhat <ganapathi.bhat@nxp.com>,
Xinming Hu <huxinming820@gmail.com>,
libertas-dev@lists.infradead.org,
Pascal Terjan <pterjan@google.com>,
Ping-Ke Shih <pkshih@realtek.com>
Subject: [patch 01/35] net: enic: Cure the enic api locking trainwreck
Date: Sun, 27 Sep 2020 21:48:47 +0200 [thread overview]
Message-ID: <20200927194919.806110712@linutronix.de> (raw)
In-Reply-To: 20200927194846.045411263@linutronix.de
From: Thomas Gleixner <tglx@linutronix.de>
enic_dev_wait() has a BUG_ON(in_interrupt()).
Chasing the callers of enic_dev_wait() revealed the gems of enic_reset()
and enic_tx_hang_reset() which are both invoked through work queues in
order to be able to call rtnl_lock(). So far so good.
After locking rtnl both functions acquire enic::enic_api_lock which
serializes against the (ab)use from infiniband. This is where the
trainwreck starts.
enic::enic_api_lock is a spin_lock() which implicitly disables preemption,
but both functions invoke a ton of functions under that lock which can
sleep. The BUG_ON(in_interrupt()) does not trigger in that case because it
can't detect the preempt disabled condition.
This clearly has never been tested with any of the mandatory debug options
for 7+ years, which would have caught that for sure.
Cure it by adding a enic_api_busy member to struct enic, which is modified
and evaluated with enic::enic_api_lock held.
If enic_api_devcmd_proxy_by_index() observes enic::enic_api_busy as true,
it drops enic::enic_api_lock and busy waits for enic::enic_api_busy to
become false.
It would be smarter to wait for a completion of that busy period, but
enic_api_devcmd_proxy_by_index() is called with other spin locks held which
obviously can't sleep.
Remove the BUG_ON(in_interrupt()) check as well because it's incomplete and
with proper debugging enabled the problem would have been caught from the
debug checks in schedule_timeout().
Fixes: 0b038566c0ea ("drivers/net: enic: Add an interface for USNIC to interact with firmware")
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Christian Benvenuti <benve@cisco.com>
Cc: Govindarajulu Varadarajan <_govind@gmx.com>
Cc: Dave Miller <davem@davemloft.net>
Cc: Jakub Kicinski <kuba@kernel.org>
Cc: netdev@vger.kernel.org
---
drivers/net/ethernet/cisco/enic/enic.h | 1 +
drivers/net/ethernet/cisco/enic/enic_api.c | 6 ++++++
drivers/net/ethernet/cisco/enic/enic_main.c | 27 +++++++++++++++++++++------
3 files changed, 28 insertions(+), 6 deletions(-)
--- a/drivers/net/ethernet/cisco/enic/enic.h
+++ b/drivers/net/ethernet/cisco/enic/enic.h
@@ -169,6 +169,7 @@ struct enic {
u16 num_vfs;
#endif
spinlock_t enic_api_lock;
+ bool enic_api_busy;
struct enic_port_profile *pp;
/* work queue cache line section */
--- a/drivers/net/ethernet/cisco/enic/enic_api.c
+++ b/drivers/net/ethernet/cisco/enic/enic_api.c
@@ -34,6 +34,12 @@ int enic_api_devcmd_proxy_by_index(struc
struct vnic_dev *vdev = enic->vdev;
spin_lock(&enic->enic_api_lock);
+ while (enic->enic_api_busy) {
+ spin_unlock(&enic->enic_api_lock);
+ cpu_relax();
+ spin_lock(&enic->enic_api_lock);
+ }
+
spin_lock_bh(&enic->devcmd_lock);
vnic_dev_cmd_proxy_by_index_start(vdev, vf);
--- a/drivers/net/ethernet/cisco/enic/enic_main.c
+++ b/drivers/net/ethernet/cisco/enic/enic_main.c
@@ -2106,8 +2106,6 @@ static int enic_dev_wait(struct vnic_dev
int done;
int err;
- BUG_ON(in_interrupt());
-
err = start(vdev, arg);
if (err)
return err;
@@ -2295,6 +2293,13 @@ static int enic_set_rss_nic_cfg(struct e
rss_hash_bits, rss_base_cpu, rss_enable);
}
+static void enic_set_api_busy(struct enic *enic, bool busy)
+{
+ spin_lock(&enic->enic_api_lock);
+ enic->enic_api_busy = busy;
+ spin_unlock(&enic->enic_api_lock);
+}
+
static void enic_reset(struct work_struct *work)
{
struct enic *enic = container_of(work, struct enic, reset);
@@ -2304,7 +2309,9 @@ static void enic_reset(struct work_struc
rtnl_lock();
- spin_lock(&enic->enic_api_lock);
+ /* Stop any activity from infiniband */
+ enic_set_api_state(enic, true);
+
enic_stop(enic->netdev);
enic_dev_soft_reset(enic);
enic_reset_addr_lists(enic);
@@ -2312,7 +2319,10 @@ static void enic_reset(struct work_struc
enic_set_rss_nic_cfg(enic);
enic_dev_set_ig_vlan_rewrite_mode(enic);
enic_open(enic->netdev);
- spin_unlock(&enic->enic_api_lock);
+
+ /* Allow infiniband to fiddle with the device again */
+ enic_set_api_state(enic, false);
+
call_netdevice_notifiers(NETDEV_REBOOT, enic->netdev);
rtnl_unlock();
@@ -2324,7 +2334,9 @@ static void enic_tx_hang_reset(struct wo
rtnl_lock();
- spin_lock(&enic->enic_api_lock);
+ /* Stop any activity from infiniband */
+ enic_set_api_state(enic, true);
+
enic_dev_hang_notify(enic);
enic_stop(enic->netdev);
enic_dev_hang_reset(enic);
@@ -2333,7 +2345,10 @@ static void enic_tx_hang_reset(struct wo
enic_set_rss_nic_cfg(enic);
enic_dev_set_ig_vlan_rewrite_mode(enic);
enic_open(enic->netdev);
- spin_unlock(&enic->enic_api_lock);
+
+ /* Allow infiniband to fiddle with the device again */
+ enic_set_api_state(enic, false);
+
call_netdevice_notifiers(NETDEV_REBOOT, enic->netdev);
rtnl_unlock();
next prev parent reply other threads:[~2020-09-27 20:01 UTC|newest]
Thread overview: 59+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-27 19:48 [patch 00/35] net: in_interrupt() cleanup and fixes Thomas Gleixner
2020-09-27 19:48 ` Thomas Gleixner [this message]
2020-09-27 19:48 ` [patch 02/35] net: caif: Remove unused caif SPI driver Thomas Gleixner
2020-09-27 19:48 ` [patch 03/35] net: Add netif_rx_any_context() Thomas Gleixner
2020-09-27 19:48 ` [patch 04/35] net: caif: Use netif_rx_any_context() Thomas Gleixner
2020-09-27 19:48 ` [patch 05/35] net: atheros: Remove WARN_ON(in_interrupt()) Thomas Gleixner
2020-09-27 19:48 ` [patch 06/35] net: cxgb3: Cleanup in_interrupt() usage Thomas Gleixner
2020-09-27 19:48 ` [patch 07/35] net: cxbg4: Remove pointless in_interrupt() check Thomas Gleixner
2020-09-27 19:48 ` [patch 08/35] net: e100: Remove in_interrupt() usage and pointless GFP_ATOMIC allocation Thomas Gleixner
2020-09-27 19:48 ` [patch 09/35] net: fec_mpc52xx: Replace in_interrupt() usage Thomas Gleixner
2020-09-27 19:48 ` [patch 10/35] net: intel: Remove in_interrupt() warnings Thomas Gleixner
2020-09-28 23:04 ` Alexander Duyck
2020-09-27 19:48 ` [patch 11/35] net: ionic: Replace in_interrupt() usage Thomas Gleixner
2020-09-28 17:24 ` Shannon Nelson
2020-09-28 19:51 ` Shannon Nelson
2020-09-29 14:37 ` Thomas Gleixner
2020-09-27 19:48 ` [patch 12/35] net: ionic: Remove WARN_ON(in_interrupt()) Thomas Gleixner
2020-09-28 17:26 ` Shannon Nelson
2020-09-27 19:48 ` [patch 13/35] net: mdiobus: Remove WARN_ON_ONCE(in_interrupt()) Thomas Gleixner
2020-09-27 23:00 ` Andrew Lunn
2020-09-27 19:49 ` [patch 14/35] net: natsemi: Replace in_interrupt() usage Thomas Gleixner
2020-09-27 19:49 ` [patch 15/35] net: sfc: " Thomas Gleixner
2020-09-28 19:03 ` Edward Cree
2020-09-28 20:05 ` [RFC PATCH net-next] sfc: replace " Edward Cree
2020-09-28 20:24 ` Thomas Gleixner
2020-09-29 15:15 ` Edward Cree
2020-09-29 19:27 ` Thomas Gleixner
2020-09-29 9:39 ` Martin Habets
2020-09-27 19:49 ` [patch 16/35] net: sunbmac: Replace " Thomas Gleixner
2020-09-27 19:49 ` [patch 17/35] net: sun3lance: Remove redundant checks in interrupt handler Thomas Gleixner
2020-09-27 19:49 ` [patch 18/35] net: vxge: Remove in_interrupt() conditionals Thomas Gleixner
2020-09-27 19:49 ` [patch 19/35] net: zd1211rw: Remove ZD_ASSERT(in_interrupt()) Thomas Gleixner
2020-09-27 19:49 ` [patch 20/35] net: usb: kaweth: Replace kaweth_control() with usb_control_msg() Thomas Gleixner
2020-09-27 19:49 ` [patch 21/35] net: usb: kaweth: Remove last user of kaweth_control() Thomas Gleixner
2020-09-28 6:50 ` Greg Kroah-Hartman
2020-09-27 19:49 ` [patch 22/35] net: usb: net1080: Remove in_interrupt() comment Thomas Gleixner
2020-09-27 19:49 ` [patch 23/35] net: wan/lmc: Remove lmc_trace() Thomas Gleixner
2020-09-27 19:49 ` [patch 24/35] net: brcmfmac: Replace in_interrupt() Thomas Gleixner
2020-09-28 7:35 ` Arend Van Spriel
2020-09-28 9:19 ` Ulf Hansson
2020-09-28 9:37 ` Arend Van Spriel
2020-09-28 9:40 ` Arend Van Spriel
2020-09-27 19:49 ` [patch 25/35] net: brcmfmac: Use netif_rx_any_context() Thomas Gleixner
2020-09-28 8:03 ` Arend Van Spriel
2020-09-27 19:49 ` [patch 26/35] net: brcmfmac: Convey allocation mode as argument Thomas Gleixner
2020-09-28 9:34 ` Arend Van Spriel
2020-09-27 19:49 ` [patch 27/35] net: ipw2x00,iwlegacy,iwlwifi: Remove in_interrupt() from debug macros Thomas Gleixner
2020-09-27 19:49 ` [patch 28/35] net: iwlwifi: Remove in_interrupt() from tracing macro Thomas Gleixner
2020-09-28 6:13 ` Coelho, Luciano
2020-09-27 19:49 ` [patch 29/35] net: hostap: Remove in_interrupt() usage Thomas Gleixner
2020-09-27 19:49 ` [patch 30/35] net: mwifiex: Use netif_rx_any_context() Thomas Gleixner
2020-09-27 19:49 ` [patch 31/35] net: libertas libertas_tf: Remove in_interrupt() from debug macro Thomas Gleixner
2020-09-27 19:49 ` [patch 32/35] net: libertas: Use netif_rx_any_context() Thomas Gleixner
2020-09-27 19:49 ` [patch 33/35] net: rtlwifi: Remove void* casts related to delayed work Thomas Gleixner
2020-09-27 19:49 ` [patch 34/35] net: rtlwifi: Remove in_interrupt() from debug macro Thomas Gleixner
2020-09-27 19:49 ` [patch 35/35] net: rtlwifi: Replace in_interrupt() for context detection Thomas Gleixner
2020-09-27 20:22 ` [patch 00/35] net: in_interrupt() cleanup and fixes Joe Perches
2020-09-27 20:57 ` David Miller
2020-09-28 10:25 ` Thomas Gleixner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200927194919.806110712@linutronix.de \
--to=tglx@linutronix.de \
--cc=_govind@gmx.com \
--cc=amitkarwar@gmail.com \
--cc=andrew@lunn.ch \
--cc=arend.vanspriel@broadcom.com \
--cc=benve@cisco.com \
--cc=bigeasy@linutronix.de \
--cc=brcm80211-dev-list.pdl@broadcom.com \
--cc=brcm80211-dev-list@cypress.com \
--cc=chi-hsien.lin@cypress.com \
--cc=chris.snook@gmail.com \
--cc=corbet@lwn.net \
--cc=davem@davemloft.net \
--cc=drivers@pensando.io \
--cc=dsd@gentoo.org \
--cc=ecree@solarflare.com \
--cc=emmanuel.grumbach@intel.com \
--cc=franky.lin@broadcom.com \
--cc=ganapathi.bhat@nxp.com \
--cc=gregkh@linuxfoundation.org \
--cc=hante.meuleman@broadcom.com \
--cc=hkallweit1@gmail.com \
--cc=huxinming820@gmail.com \
--cc=intel-wired-lan@lists.osuosl.org \
--cc=j@w1.fi \
--cc=jcliburn@gmail.com \
--cc=jdmason@kudzu.us \
--cc=jeffrey.t.kirsher@intel.com \
--cc=johannes.berg@intel.com \
--cc=kuba@kernel.org \
--cc=kune@deine-taler.de \
--cc=kvalo@codeaurora.org \
--cc=libertas-dev@lists.infradead.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-net-drivers@solarflare.com \
--cc=linux-usb@vger.kernel.org \
--cc=linux-wireless@vger.kernel.org \
--cc=linux@armlinux.org.uk \
--cc=linuxwifi@intel.com \
--cc=luc.vanoostenryck@gmail.com \
--cc=luciano.coelho@intel.com \
--cc=mchehab+huawei@kernel.org \
--cc=mhabets@solarflare.com \
--cc=netdev@vger.kernel.org \
--cc=paulmck@kernel.org \
--cc=peterz@infradead.org \
--cc=pkshih@realtek.com \
--cc=pterjan@google.com \
--cc=snelson@pensando.io \
--cc=stas.yakovlev@gmail.com \
--cc=stf_xl@wp.pl \
--cc=torvalds@linuxfoundation.org \
--cc=tsbogend@alpha.franken.de \
--cc=vishal@chelsio.com \
--cc=willy@infradead.org \
--cc=wright.feng@cypress.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).