[PATCH net 0/1] netvsc: race on sub channel open

[PATCH net 0/1] netvsc: race on sub channel open

[Stephen Hemminger]

Found this while testing mtu, queue and buffer size changes
in 4.13, but the problem goes back much further. The addition
of NAPI makes the race into a crash. Before that there was just
a risk of sending on an uninitialized channel.

Stephen Hemminger (1):
  netvsc: fix race on sub channel creation

 drivers/net/hyperv/hyperv_net.h   |  3 ++-
 drivers/net/hyperv/netvsc.c       |  1 +
 drivers/net/hyperv/rndis_filter.c | 14 ++++++++------
 3 files changed, 11 insertions(+), 7 deletions(-)

-- 
2.11.0

[PATCH net 1/1] netvsc: fix race on sub channel creation

[Stephen Hemminger]

The existing sub channel code did not wait for all the sub-channels
to completely initialize. This could lead to race causing crash
in napi_netif_del() from bad list. The existing code would send
an init message, then wait only for the initial response that
the init message was received. It thought it was waiting for
sub channels but really the init response did the wakeup.

The new code keeps track of the number of open channels and
waits until that many are open.

Other issues here were:
  * host might return less sub-channels than was requested.
  * the new init status is not valid until after init was completed.

Fixes: b3e6b82a0099 ("hv_netvsc: Wait for sub-channels to be processed during probe")
Signed-off-by: Stephen Hemminger <sthemmin@microsoft.com>
---
 drivers/net/hyperv/hyperv_net.h   |  3 ++-
 drivers/net/hyperv/netvsc.c       |  1 +
 drivers/net/hyperv/rndis_filter.c | 14 ++++++++------
 3 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/drivers/net/hyperv/hyperv_net.h b/drivers/net/hyperv/hyperv_net.h
index d6c25580f8dd..12cc64bfcff8 100644
--- a/drivers/net/hyperv/hyperv_net.h
+++ b/drivers/net/hyperv/hyperv_net.h
@@ -765,7 +765,8 @@ struct netvsc_device {
 	u32 max_chn;
 	u32 num_chn;
 
-	refcount_t sc_offered;
+	atomic_t open_chn;
+	wait_queue_head_t subchan_open;
 
 	struct rndis_device *extension;
 
diff --git a/drivers/net/hyperv/netvsc.c b/drivers/net/hyperv/netvsc.c
index 96f90c75d1b7..d18c3326a1f7 100644
--- a/drivers/net/hyperv/netvsc.c
+++ b/drivers/net/hyperv/netvsc.c
@@ -78,6 +78,7 @@ static struct netvsc_device *alloc_net_device(void)
 	net_device->max_pkt = RNDIS_MAX_PKT_DEFAULT;
 	net_device->pkt_align = RNDIS_PKT_ALIGN_DEFAULT;
 	init_completion(&net_device->channel_init_wait);
+	init_waitqueue_head(&net_device->subchan_open);
 
 	return net_device;
 }
diff --git a/drivers/net/hyperv/rndis_filter.c b/drivers/net/hyperv/rndis_filter.c
index 85c00e1c52b6..d6308ffda53e 100644
--- a/drivers/net/hyperv/rndis_filter.c
+++ b/drivers/net/hyperv/rndis_filter.c
@@ -1048,8 +1048,8 @@ static void netvsc_sc_open(struct vmbus_channel *new_sc)
 	else
 		netif_napi_del(&nvchan->napi);
 
-	if (refcount_dec_and_test(&nvscdev->sc_offered))
-		complete(&nvscdev->channel_init_wait);
+	atomic_inc(&nvscdev->open_chn);
+	wake_up(&nvscdev->subchan_open);
 }
 
 int rndis_filter_device_add(struct hv_device *dev,
@@ -1090,8 +1090,6 @@ int rndis_filter_device_add(struct hv_device *dev,
 	net_device->max_chn = 1;
 	net_device->num_chn = 1;
 
-	refcount_set(&net_device->sc_offered, 0);
-
 	net_device->extension = rndis_device;
 	rndis_device->ndev = net;
 
@@ -1221,11 +1219,11 @@ int rndis_filter_device_add(struct hv_device *dev,
 		rndis_device->ind_table[i] = ethtool_rxfh_indir_default(i,
 							net_device->num_chn);
 
+	atomic_set(&net_device->open_chn, 1);
 	num_rss_qs = net_device->num_chn - 1;
 	if (num_rss_qs == 0)
 		return 0;
 
-	refcount_set(&net_device->sc_offered, num_rss_qs);
 	vmbus_set_sc_create_callback(dev->channel, netvsc_sc_open);
 
 	init_packet = &net_device->channel_init_pkt;
@@ -1242,15 +1240,19 @@ int rndis_filter_device_add(struct hv_device *dev,
 	if (ret)
 		goto out;
 
+	wait_for_completion(&net_device->channel_init_wait);
 	if (init_packet->msg.v5_msg.subchn_comp.status != NVSP_STAT_SUCCESS) {
 		ret = -ENODEV;
 		goto out;
 	}
-	wait_for_completion(&net_device->channel_init_wait);
 
 	net_device->num_chn = 1 +
 		init_packet->msg.v5_msg.subchn_comp.num_subchannels;
 
+	/* wait for all sub channels to open */
+	wait_event(net_device->subchan_open,
+		   atomic_read(&net_device->open_chn) == net_device->num_chn);
+
 	/* ignore failues from setting rss parameters, still have channels */
 	rndis_filter_set_rss_param(rndis_device, netvsc_hash_key,
 				   net_device->num_chn);
-- 
2.11.0

Re: [PATCH net 1/1] netvsc: fix race on sub channel creation

[David Miller]

From: Stephen Hemminger <stephen@networkplumber.org>
Date: Thu,  3 Aug 2017 17:13:54 -0700

> The existing sub channel code did not wait for all the sub-channels
> to completely initialize. This could lead to race causing crash
> in napi_netif_del() from bad list. The existing code would send
> an init message, then wait only for the initial response that
> the init message was received. It thought it was waiting for
> sub channels but really the init response did the wakeup.
> 
> The new code keeps track of the number of open channels and
> waits until that many are open.
> 
> Other issues here were:
>   * host might return less sub-channels than was requested.
>   * the new init status is not valid until after init was completed.
> 
> Fixes: b3e6b82a0099 ("hv_netvsc: Wait for sub-channels to be processed during probe")
> Signed-off-by: Stephen Hemminger <sthemmin@microsoft.com>

Applied, thanks.