[PATCH net] hv_netvsc: Fix race between VF offering and VF association message from host

[PATCH net] hv_netvsc: Fix race between VF offering and VF association message from host

[Gaurav Kohli]

During vm boot, there might be possibility that vf registration
call comes before the vf association from host to vm.

And this might break netvsc vf path, To prevent the same block
vf registration until vf bind message comes from host.

Cc: stable@vger.kernel.org
Fixes: 00d7ddba11436 ("hv_netvsc: pair VF based on serial number")
Signed-off-by: Gaurav Kohli <gauravkohli@linux.microsoft.com>
---
 drivers/net/hyperv/hyperv_net.h |  3 ++-
 drivers/net/hyperv/netvsc.c     |  5 +++++
 drivers/net/hyperv/netvsc_drv.c | 18 ++++++++++++++++++
 3 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/drivers/net/hyperv/hyperv_net.h b/drivers/net/hyperv/hyperv_net.h
index 25b38a374e3c..dd5919ec408b 100644
--- a/drivers/net/hyperv/hyperv_net.h
+++ b/drivers/net/hyperv/hyperv_net.h
@@ -1051,7 +1051,8 @@ struct net_device_context {
 	u32 vf_alloc;
 	/* Serial number of the VF to team with */
 	u32 vf_serial;
-
+	/* completion variable to confirm vf association */
+	struct completion vf_add;
 	/* Is the current data path through the VF NIC? */
 	bool  data_path_is_vf;
 
diff --git a/drivers/net/hyperv/netvsc.c b/drivers/net/hyperv/netvsc.c
index 8bcba6f21aa9..6531d8d9f17f 100644
--- a/drivers/net/hyperv/netvsc.c
+++ b/drivers/net/hyperv/netvsc.c
@@ -1582,6 +1582,11 @@ static void netvsc_send_vf(struct net_device *ndev,
 
 	net_device_ctx->vf_alloc = nvmsg->msg.v4_msg.vf_assoc.allocated;
 	net_device_ctx->vf_serial = nvmsg->msg.v4_msg.vf_assoc.serial;
+
+	if (net_device_ctx->vf_alloc)
+		complete(&net_device_ctx->vf_add);
+	else
+		reinit_completion(&net_device_ctx->vf_add);
 	netdev_info(ndev, "VF slot %u %s\n",
 		    net_device_ctx->vf_serial,
 		    net_device_ctx->vf_alloc ? "added" : "removed");
diff --git a/drivers/net/hyperv/netvsc_drv.c b/drivers/net/hyperv/netvsc_drv.c
index c27cb1267ca5..393c69430147 100644
--- a/drivers/net/hyperv/netvsc_drv.c
+++ b/drivers/net/hyperv/netvsc_drv.c
@@ -2309,6 +2309,18 @@ static struct net_device *get_netvsc_byslot(const struct net_device *vf_netdev)
 
 	}
 
+	/* Fallback path to check synthetic vf with
+	 * help of mac addr
+	 */
+	list_for_each_entry(ndev_ctx, &netvsc_dev_list, list) {
+		ndev = hv_get_drvdata(ndev_ctx->device_ctx);
+		if (ether_addr_equal(vf_netdev->perm_addr, ndev->perm_addr)) {
+			netdev_notice(vf_netdev,
+				      "falling back to mac addr based matching\n");
+			return ndev;
+		}
+	}
+
 	netdev_notice(vf_netdev,
 		      "no netdev found for vf serial:%u\n", serial);
 	return NULL;
@@ -2405,6 +2417,11 @@ static int netvsc_vf_changed(struct net_device *vf_netdev, unsigned long event)
 	if (net_device_ctx->data_path_is_vf == vf_is_up)
 		return NOTIFY_OK;
 
+	if (vf_is_up && !net_device_ctx->vf_alloc) {
+		netdev_info(ndev, "Waiting for the VF association from host\n");
+		wait_for_completion(&net_device_ctx->vf_add);
+	}
+
 	ret = netvsc_switch_datapath(ndev, vf_is_up);
 
 	if (ret) {
@@ -2475,6 +2492,7 @@ static int netvsc_probe(struct hv_device *dev,
 
 	INIT_DELAYED_WORK(&net_device_ctx->dwork, netvsc_link_change);
 
+	init_completion(&net_device_ctx->vf_add);
 	spin_lock_init(&net_device_ctx->lock);
 	INIT_LIST_HEAD(&net_device_ctx->reconfig_events);
 	INIT_DELAYED_WORK(&net_device_ctx->vf_takeover, netvsc_vf_setup);
-- 
2.17.1

RE: [PATCH net] hv_netvsc: Fix race between VF offering and VF association message from host

[Haiyang Zhang]

> -----Original Message-----
> From: Gaurav Kohli <gauravkohli@linux.microsoft.com>
> Sent: Wednesday, September 28, 2022 9:49 AM
> To: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
> <haiyangz@microsoft.com>; Stephen Hemminger
> <sthemmin@microsoft.com>; wei.liu@kernel.org; Dexuan Cui
> <decui@microsoft.com>; linux-hyperv@vger.kernel.org;
> netdev@vger.kernel.org
> Subject: [PATCH net] hv_netvsc: Fix race between VF offering and VF
> association message from host
> 
> During vm boot, there might be possibility that vf registration
> call comes before the vf association from host to vm.
> 
> And this might break netvsc vf path, To prevent the same block
> vf registration until vf bind message comes from host.
> 
> Cc: stable@vger.kernel.org
> Fixes: 00d7ddba11436 ("hv_netvsc: pair VF based on serial number")
> Signed-off-by: Gaurav Kohli <gauravkohli@linux.microsoft.com>

Reviewed-by: Haiyang Zhang <haiyangz@microsoft.com>

By the way, did you use "git send-email"? I didn't see the stable@vger.kernel.org cc-ed in your original email.

RE: [PATCH net] hv_netvsc: Fix race between VF offering and VF association message from host

[Stephen Hemminger]

The policy of network development is to not copy the stable list. Instead the netdev maintainers want to curate the patches going to stable.

-----Original Message-----
From: Haiyang Zhang <haiyangz@microsoft.com> 
Sent: Thursday, September 29, 2022 1:40 PM
To: Gaurav Kohli <gauravkohli@linux.microsoft.com>; KY Srinivasan <kys@microsoft.com>; Stephen Hemminger <sthemmin@microsoft.com>; wei.liu@kernel.org; Dexuan Cui <decui@microsoft.com>; linux-hyperv@vger.kernel.org; netdev@vger.kernel.org
Cc: stable@vger.kernel.org
Subject: RE: [PATCH net] hv_netvsc: Fix race between VF offering and VF association message from host



> -----Original Message-----
> From: Gaurav Kohli <gauravkohli@linux.microsoft.com>
> Sent: Wednesday, September 28, 2022 9:49 AM
> To: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
> <haiyangz@microsoft.com>; Stephen Hemminger
> <sthemmin@microsoft.com>; wei.liu@kernel.org; Dexuan Cui
> <decui@microsoft.com>; linux-hyperv@vger.kernel.org;
> netdev@vger.kernel.org
> Subject: [PATCH net] hv_netvsc: Fix race between VF offering and VF
> association message from host
> 
> During vm boot, there might be possibility that vf registration
> call comes before the vf association from host to vm.
> 
> And this might break netvsc vf path, To prevent the same block
> vf registration until vf bind message comes from host.
> 
> Cc: stable@vger.kernel.org
> Fixes: 00d7ddba11436 ("hv_netvsc: pair VF based on serial number")
> Signed-off-by: Gaurav Kohli <gauravkohli@linux.microsoft.com>

Reviewed-by: Haiyang Zhang <haiyangz@microsoft.com>

By the way, did you use "git send-email"? I didn't see the stable@vger.kernel.org cc-ed in your original email.

RE: [PATCH net] hv_netvsc: Fix race between VF offering and VF association message from host

[Haiyang Zhang]

> -----Original Message-----
> From: Stephen Hemminger <sthemmin@microsoft.com>
> Sent: Thursday, September 29, 2022 5:10 PM
> To: Haiyang Zhang <haiyangz@microsoft.com>; Gaurav Kohli
> <gauravkohli@linux.microsoft.com>; KY Srinivasan <kys@microsoft.com>;
> wei.liu@kernel.org; Dexuan Cui <decui@microsoft.com>; linux-
> hyperv@vger.kernel.org
> Subject: RE: [PATCH net] hv_netvsc: Fix race between VF offering and VF
> association message from host
> 
> The policy of network development is to not copy the stable list. Instead the
> netdev maintainers want to curate the patches going to stable.

Thanks for the info.
How do we indicate if a patch to netdev list should be applied to stable trees too?

Thanks,
- Haiyang

RE: [PATCH net] hv_netvsc: Fix race between VF offering and VF association message from host

[Stephen Hemminger]

I would add a comment to the commit message after the --- break

-----Original Message-----
From: Haiyang Zhang <haiyangz@microsoft.com> 
Sent: Thursday, September 29, 2022 2:33 PM
To: Stephen Hemminger <sthemmin@microsoft.com>; Gaurav Kohli <gauravkohli@linux.microsoft.com>; KY Srinivasan <kys@microsoft.com>; wei.liu@kernel.org; Dexuan Cui <decui@microsoft.com>; linux-hyperv@vger.kernel.org
Subject: RE: [PATCH net] hv_netvsc: Fix race between VF offering and VF association message from host



> -----Original Message-----
> From: Stephen Hemminger <sthemmin@microsoft.com>
> Sent: Thursday, September 29, 2022 5:10 PM
> To: Haiyang Zhang <haiyangz@microsoft.com>; Gaurav Kohli
> <gauravkohli@linux.microsoft.com>; KY Srinivasan <kys@microsoft.com>;
> wei.liu@kernel.org; Dexuan Cui <decui@microsoft.com>; linux-
> hyperv@vger.kernel.org
> Subject: RE: [PATCH net] hv_netvsc: Fix race between VF offering and VF
> association message from host
> 
> The policy of network development is to not copy the stable list. Instead the
> netdev maintainers want to curate the patches going to stable.

Thanks for the info.
How do we indicate if a patch to netdev list should be applied to stable trees too?

Thanks,
- Haiyang

Re: [PATCH net] hv_netvsc: Fix race between VF offering and VF association message from host

[Jakub Kicinski]

On Wed, 28 Sep 2022 06:48:33 -0700 Gaurav Kohli wrote:
> During vm boot, there might be possibility that vf registration
> call comes before the vf association from host to vm.
> 
> And this might break netvsc vf path, To prevent the same block
> vf registration until vf bind message comes from host.
> 
> Cc: stable@vger.kernel.org
> Fixes: 00d7ddba11436 ("hv_netvsc: pair VF based on serial number")
> Signed-off-by: Gaurav Kohli <gauravkohli@linux.microsoft.com>

Is it possible to add a timeout or such? Waiting for an external 
event while holding rtnl lock seems a little scary.

The other question is - what protects the completion and ->vf_alloc
from races? Is there some locking? ->vf_alloc only goes from 0 to 1
and never back?

RE: [PATCH net] hv_netvsc: Fix race between VF offering and VF association message from host

[Haiyang Zhang]

> -----Original Message-----
> From: Jakub Kicinski <kuba@kernel.org>
> Sent: Thursday, September 29, 2022 10:26 PM
> To: Gaurav Kohli <gauravkohli@linux.microsoft.com>
> Cc: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
> <haiyangz@microsoft.com>; Stephen Hemminger
> <sthemmin@microsoft.com>; wei.liu@kernel.org; Dexuan Cui
> <decui@microsoft.com>; linux-hyperv@vger.kernel.org;
> netdev@vger.kernel.org
> Subject: Re: [PATCH net] hv_netvsc: Fix race between VF offering and VF
> association message from host
> 
> On Wed, 28 Sep 2022 06:48:33 -0700 Gaurav Kohli wrote:
> > During vm boot, there might be possibility that vf registration
> > call comes before the vf association from host to vm.
> >
> > And this might break netvsc vf path, To prevent the same block
> > vf registration until vf bind message comes from host.
> >
> > Cc: stable@vger.kernel.org
> > Fixes: 00d7ddba11436 ("hv_netvsc: pair VF based on serial number")
> > Signed-off-by: Gaurav Kohli <gauravkohli@linux.microsoft.com>
> 
> Is it possible to add a timeout or such? Waiting for an external
> event while holding rtnl lock seems a little scary.

We used to have time-out in many places of this driver. But there is
no protocol guarantees of the host response time, so the time out value
cannot be set. These time-outs were removed several years ago.


> The other question is - what protects the completion and ->vf_alloc
> from races? Is there some locking? ->vf_alloc only goes from 0 to 1
> and never back?

When Vf is removed, the vf_assoc msg will set it to 0 here:
        net_device_ctx->vf_alloc = nvmsg->msg.v4_msg.vf_assoc.allocated;
        net_device_ctx->vf_serial = nvmsg->msg.v4_msg.vf_assoc.serial;

Also, I think this condition can be changed from:
+	if (vf_is_up && !net_device_ctx->vf_alloc) {
to: 
+	if (vf_is_up) {
So when VF comes up, it always wait for the completion without depending
on the vf_alloc.

Thanks,
- Haiyang

Re: [PATCH net] hv_netvsc: Fix race between VF offering and VF association message from host

[Gaurav Kohli]

On 9/30/2022 6:33 PM, Haiyang Zhang wrote:
>
>> -----Original Message-----
>> From: Jakub Kicinski <kuba@kernel.org>
>> Sent: Thursday, September 29, 2022 10:26 PM
>> To: Gaurav Kohli <gauravkohli@linux.microsoft.com>
>> Cc: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
>> <haiyangz@microsoft.com>; Stephen Hemminger
>> <sthemmin@microsoft.com>; wei.liu@kernel.org; Dexuan Cui
>> <decui@microsoft.com>; linux-hyperv@vger.kernel.org;
>> netdev@vger.kernel.org
>> Subject: Re: [PATCH net] hv_netvsc: Fix race between VF offering and VF
>> association message from host
>>
>> On Wed, 28 Sep 2022 06:48:33 -0700 Gaurav Kohli wrote:
>>> During vm boot, there might be possibility that vf registration
>>> call comes before the vf association from host to vm.
>>>
>>> And this might break netvsc vf path, To prevent the same block
>>> vf registration until vf bind message comes from host.
>>>
>>> Cc: stable@vger.kernel.org
>>> Fixes: 00d7ddba11436 ("hv_netvsc: pair VF based on serial number")
>>> Signed-off-by: Gaurav Kohli <gauravkohli@linux.microsoft.com>
>> Is it possible to add a timeout or such? Waiting for an external
>> event while holding rtnl lock seems a little scary.
> We used to have time-out in many places of this driver. But there is
> no protocol guarantees of the host response time, so the time out value
> cannot be set. These time-outs were removed several years ago.
>
>
>> The other question is - what protects the completion and ->vf_alloc
>> from races? Is there some locking? ->vf_alloc only goes from 0 to 1
>> and never back?

Thanks for the comment, i understand your concern for vf_alloc and 
reinit completion part, I think

we can move reinit completion to unregistration part of vf code.

Let me send v2 patch.

> When Vf is removed, the vf_assoc msg will set it to 0 here:
>          net_device_ctx->vf_alloc = nvmsg->msg.v4_msg.vf_assoc.allocated;
>          net_device_ctx->vf_serial = nvmsg->msg.v4_msg.vf_assoc.serial;
>
> Also, I think this condition can be changed from:
> +	if (vf_is_up && !net_device_ctx->vf_alloc) {

Thanks for the comment.

This is needed to maintain state machine, as netvsc change event can 
comes multiple time. That's why i have put

extra check to avoid any deadlock.

> to:
> +	if (vf_is_up) {
> So when VF comes up, it always wait for the completion without depending
> on the vf_alloc.
>
> Thanks,
> - Haiyang