From: Sasha Levin <sashal@kernel.org>
To: stable@vger.kernel.org, linux-kernel@vger.kernel.org
Cc: Alex Vesker <valex@mellanox.com>,
Leon Romanovsky <leon@kernel.org>,
Jason Gunthorpe <jgg@mellanox.com>,
Sasha Levin <sashal@kernel.org>
Subject: [PATCH AUTOSEL 4.14 08/46] IB/ipoib: Fix lockdep issue found on ipoib_ib_dev_heavy_flush
Date: Thu, 25 Oct 2018 10:10:15 -0400 [thread overview]
Message-ID: <20181025141053.213330-8-sashal@kernel.org> (raw)
In-Reply-To: <20181025141053.213330-1-sashal@kernel.org>
From: Alex Vesker <valex@mellanox.com>
[ Upstream commit 1f80bd6a6cc8358b81194e1f5fc16449947396ec ]
The locking order of vlan_rwsem (LOCK A) and then rtnl (LOCK B),
contradicts other flows such as ipoib_open possibly causing a deadlock.
To prevent this deadlock heavy flush is called with RTNL locked and
only then tries to acquire vlan_rwsem.
This deadlock is possible only when there are child interfaces.
[ 140.941758] ======================================================
[ 140.946276] WARNING: possible circular locking dependency detected
[ 140.950950] 4.15.0-rc1+ #9 Tainted: G O
[ 140.954797] ------------------------------------------------------
[ 140.959424] kworker/u32:1/146 is trying to acquire lock:
[ 140.963450] (rtnl_mutex){+.+.}, at: [<ffffffffc083516a>] __ipoib_ib_dev_flush+0x2da/0x4e0 [ib_ipoib]
[ 140.970006]
but task is already holding lock:
[ 140.975141] (&priv->vlan_rwsem){++++}, at: [<ffffffffc0834ee1>] __ipoib_ib_dev_flush+0x51/0x4e0 [ib_ipoib]
[ 140.982105]
which lock already depends on the new lock.
[ 140.990023]
the existing dependency chain (in reverse order) is:
[ 140.998650]
-> #1 (&priv->vlan_rwsem){++++}:
[ 141.005276] down_read+0x4d/0xb0
[ 141.009560] ipoib_open+0xad/0x120 [ib_ipoib]
[ 141.014400] __dev_open+0xcb/0x140
[ 141.017919] __dev_change_flags+0x1a4/0x1e0
[ 141.022133] dev_change_flags+0x23/0x60
[ 141.025695] devinet_ioctl+0x704/0x7d0
[ 141.029156] sock_do_ioctl+0x20/0x50
[ 141.032526] sock_ioctl+0x221/0x300
[ 141.036079] do_vfs_ioctl+0xa6/0x6d0
[ 141.039656] SyS_ioctl+0x74/0x80
[ 141.042811] entry_SYSCALL_64_fastpath+0x1f/0x96
[ 141.046891]
-> #0 (rtnl_mutex){+.+.}:
[ 141.051701] lock_acquire+0xd4/0x220
[ 141.055212] __mutex_lock+0x88/0x970
[ 141.058631] __ipoib_ib_dev_flush+0x2da/0x4e0 [ib_ipoib]
[ 141.063160] __ipoib_ib_dev_flush+0x71/0x4e0 [ib_ipoib]
[ 141.067648] process_one_work+0x1f5/0x610
[ 141.071429] worker_thread+0x4a/0x3f0
[ 141.074890] kthread+0x141/0x180
[ 141.078085] ret_from_fork+0x24/0x30
[ 141.081559]
other info that might help us debug this:
[ 141.088967] Possible unsafe locking scenario:
[ 141.094280] CPU0 CPU1
[ 141.097953] ---- ----
[ 141.101640] lock(&priv->vlan_rwsem);
[ 141.104771] lock(rtnl_mutex);
[ 141.109207] lock(&priv->vlan_rwsem);
[ 141.114032] lock(rtnl_mutex);
[ 141.116800]
*** DEADLOCK ***
Fixes: b4b678b06f6e ("IB/ipoib: Grab rtnl lock on heavy flush when calling ndo_open/stop")
Signed-off-by: Alex Vesker <valex@mellanox.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
drivers/infiniband/ulp/ipoib/ipoib_ib.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
index c97384c914a4..d77e8e2ae05f 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
@@ -1203,13 +1203,10 @@ static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv,
ipoib_ib_dev_down(dev);
if (level == IPOIB_FLUSH_HEAVY) {
- rtnl_lock();
if (test_bit(IPOIB_FLAG_INITIALIZED, &priv->flags))
ipoib_ib_dev_stop(dev);
- result = ipoib_ib_dev_open(dev);
- rtnl_unlock();
- if (result)
+ if (ipoib_ib_dev_open(dev))
return;
if (netif_queue_stopped(dev))
@@ -1249,7 +1246,9 @@ void ipoib_ib_dev_flush_heavy(struct work_struct *work)
struct ipoib_dev_priv *priv =
container_of(work, struct ipoib_dev_priv, flush_heavy);
+ rtnl_lock();
__ipoib_ib_dev_flush(priv, IPOIB_FLUSH_HEAVY, 0);
+ rtnl_unlock();
}
void ipoib_ib_dev_cleanup(struct net_device *dev)
--
2.17.1
next prev parent reply other threads:[~2018-10-25 14:11 UTC|newest]
Thread overview: 49+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-10-25 14:10 [PATCH AUTOSEL 4.14 01/46] iwlwifi: mvm: check for short GI only for OFDM Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 02/46] iwlwifi: dbg: allow wrt collection before ALIVE Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 03/46] iwlwifi: fix the ALIVE notification layout Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 04/46] x86/power: Fix some ordering bugs in __restore_processor_context() Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 05/46] tools/testing/nvdimm: unit test clear-error commands Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 06/46] usbip: vhci_hcd: update 'status' file header and format Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 07/46] scsi: aacraid: address UBSAN warning regression Sasha Levin
2018-10-25 14:10 ` Sasha Levin [this message]
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 09/46] IB/rxe: put the pool on allocation failure Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 10/46] s390/qeth: fix error handling in adapter command callbacks Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 11/46] net/mlx5: Fix mlx5_get_vector_affinity function Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 12/46] powerpc/pseries: Add empty update_numa_cpu_lookup_table() for NUMA=n Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 13/46] dm integrity: fail early if required HMAC key is not available Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 14/46] net: phy: realtek: Use the dummy stubs for MMD register access for rtl8211b Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 15/46] net: phy: Add general dummy stubs for MMD register access Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 16/46] net/mlx5e: Refine ets validation function Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 17/46] scsi: qla2xxx: Avoid double completion of abort command Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 18/46] kbuild: set no-integrated-as before incl. arch Makefile Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 19/46] IB/mlx5: Avoid passing an invalid QP type to firmware Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 20/46] ARM: tegra: Fix ULPI regression on Tegra20 Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 21/46] l2tp: remove configurable payload offset Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 22/46] cifs: Use ULL suffix for 64-bit constant Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 23/46] test_bpf: Fix testing with CONFIG_BPF_JIT_ALWAYS_ON=y on other arches Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 24/46] KVM: x86: Update the exit_qualification access bits while walking an address Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 25/46] sparc64: Fix regression in pmdp_invalidate() Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 26/46] tpm: move the delay_msec increment after sleep in tpm_transmit() Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 27/46] bpf: sockmap, map_release does not hold refcnt for pinned maps Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 28/46] tpm: tpm_crb: relinquish locality on error path Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 29/46] xen-netfront: Update features after registering netdev Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 30/46] xen-netfront: Fix mismatched rtnl_unlock Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 31/46] IB/usnic: Update with bug fixes from core code Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 32/46] mmc: dw_mmc-rockchip: correct property names in debug Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 33/46] MIPS: Workaround GCC __builtin_unreachable reordering bug Sasha Levin
2018-10-25 19:52 ` Paul Burton
2018-10-26 7:36 ` Arnd Bergmann
2018-10-29 13:36 ` Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 34/46] lan78xx: Don't reset the interface on open Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 35/46] enic: do not overwrite error code Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 36/46] iio: buffer: fix the function signature to match implementation Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 37/46] selftests/powerpc: Add ptrace hw breakpoint test Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 38/46] scsi: ibmvfc: Avoid unnecessary port relogin Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 39/46] scsi: sd: Remember that READ CAPACITY(16) succeeded Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 40/46] btrfs: quota: Set rescan progress to (u64)-1 if we hit last leaf Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 41/46] net: phy: phylink: Don't release NULL GPIO Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 42/46] x86/paravirt: Fix some warning messages Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 43/46] net: stmmac: mark PM functions as __maybe_unused Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 44/46] kconfig: fix the rule of mainmenu_stmt symbol Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 45/46] libertas: call into generic suspend code before turning off power Sasha Levin
2018-10-25 14:10 ` [PATCH AUTOSEL 4.14 46/46] perf tests: Fix indexing when invoking subtests Sasha Levin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181025141053.213330-8-sashal@kernel.org \
--to=sashal@kernel.org \
--cc=jgg@mellanox.com \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=stable@vger.kernel.org \
--cc=valex@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).