From: Dexuan Cui <decui@microsoft.com>
To: KY Srinivasan <kys@microsoft.com>,
Haiyang Zhang <haiyangz@microsoft.com>,
Stephen Hemminger <sthemmin@microsoft.com>,
"'David S. Miller'" <davem@davemloft.net>,
"'netdev@vger.kernel.org'" <netdev@vger.kernel.org>
Cc: "'devel@linuxdriverproject.org'" <devel@linuxdriverproject.org>,
"'linux-kernel@vger.kernel.org'" <linux-kernel@vger.kernel.org>,
"'olaf@aepfle.de'" <olaf@aepfle.de>,
"'apw@canonical.com'" <apw@canonical.com>,
"'jasowang@redhat.com'" <jasowang@redhat.com>,
vkuznets <vkuznets@redhat.com>,
"'marcelo.cerri@canonical.com'" <marcelo.cerri@canonical.com>,
Josh Poulson <jopoulso@microsoft.com>
Subject: [PATCH] hv_netvsc: Fix a deadlock by getting rtnl_lock earlier in netvsc_probe()
Date: Wed, 22 Aug 2018 21:20:03 +0000 [thread overview]
Message-ID: <PU1P153MB01692432D92B3A726FBDDC68BF300@PU1P153MB0169.APCP153.PROD.OUTLOOK.COM> (raw)
This patch fixes the race between netvsc_probe() and
rndis_set_subchannel(), which can cause a deadlock.
Fixes: 8195b1396ec8 ("hv_netvsc: fix deadlock on hotplug")
Signed-off-by: Dexuan Cui <decui@microsoft.com>
Cc: Stephen Hemminger <sthemmin@microsoft.com>
Cc: K. Y. Srinivasan <kys@microsoft.com>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
---
drivers/net/hyperv/netvsc_drv.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
FYI: these are the related 3 paths which show the deadlock:
#1:
Workqueue: hv_vmbus_con vmbus_onmessage_work [hv_vmbus]
Call Trace:
schedule
schedule_preempt_disabled
__mutex_lock
__device_attach
bus_probe_device
device_add
vmbus_device_register
vmbus_onoffer
vmbus_onmessage_work
process_one_work
worker_thread
kthread
ret_from_fork
#2:
schedule
schedule_preempt_disabled
__mutex_lock
netvsc_probe
vmbus_probe
really_probe
__driver_attach
bus_for_each_dev
driver_attach_async
async_run_entry_fn
process_one_work
worker_thread
kthread
ret_from_fork
#3:
Workqueue: events netvsc_subchan_work [hv_netvsc]
Call Trace:
schedule
rndis_set_subchannel
netvsc_subchan_work
process_one_work
worker_thread
kthread
ret_from_fork
Before path #1 finishes, path #2 can start to run, because just before
the "bus_probe_device(dev);" in device_add() in path #1, there is a line
"object_uevent(&dev->kobj, KOBJ_ADD);", so systemd-udevd can
immediately try to load hv_netvsc and hence path #2 can start to run.
diff --git a/drivers/net/hyperv/netvsc_drv.c b/drivers/net/hyperv/netvsc_drv.c
index 1121a1ec..4fd14a0 100644
--- a/drivers/net/hyperv/netvsc_drv.c
+++ b/drivers/net/hyperv/netvsc_drv.c
@@ -2206,6 +2206,16 @@ static int netvsc_probe(struct hv_device *dev,
memcpy(net->dev_addr, device_info.mac_adr, ETH_ALEN);
+ /* We must get rtnl_lock before scheduling nvdev->subchan_work,
+ * otherwise netvsc_subchan_work() can get rtnl_lock first and wait
+ * all subchannels to show up, but that may not happen because
+ * netvsc_probe() can't get rtnl_lock and as a result vmbus_onoffer()
+ * -> ... -> device_add() -> ... -> __device_attach() can't get
+ * the device lock, so all the subchannels can't be processed --
+ * finally netvsc_subchan_work() hangs for ever.
+ */
+ rtnl_lock();
+
if (nvdev->num_chn > 1)
schedule_work(&nvdev->subchan_work);
@@ -2224,7 +2234,6 @@ static int netvsc_probe(struct hv_device *dev,
else
net->max_mtu = ETH_DATA_LEN;
- rtnl_lock();
ret = register_netdevice(net);
if (ret != 0) {
pr_err("Unable to register netdev.\n");
--
2.7.4
next reply other threads:[~2018-08-22 21:20 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-08-22 21:20 Dexuan Cui [this message]
2018-08-30 0:48 ` [PATCH] hv_netvsc: Fix a deadlock by getting rtnl_lock earlier in netvsc_probe() David Miller
2018-08-30 0:58 ` Dexuan Cui
2018-08-30 2:08 ` David Miller
2018-08-30 2:10 ` Dexuan Cui
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=PU1P153MB01692432D92B3A726FBDDC68BF300@PU1P153MB0169.APCP153.PROD.OUTLOOK.COM \
--to=decui@microsoft.com \
--cc=apw@canonical.com \
--cc=davem@davemloft.net \
--cc=devel@linuxdriverproject.org \
--cc=haiyangz@microsoft.com \
--cc=jasowang@redhat.com \
--cc=jopoulso@microsoft.com \
--cc=kys@microsoft.com \
--cc=linux-kernel@vger.kernel.org \
--cc=marcelo.cerri@canonical.com \
--cc=netdev@vger.kernel.org \
--cc=olaf@aepfle.de \
--cc=sthemmin@microsoft.com \
--cc=vkuznets@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).