All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] vmbus: remove hv_event_tasklet_disable/enable
@ 2017-03-02 12:32 ` Dexuan Cui
  0 siblings, 0 replies; 8+ messages in thread
From: Dexuan Cui @ 2017-03-02 12:32 UTC (permalink / raw)
  To: gregkh, Stephen Hemminger, KY Srinivasan, Haiyang Zhang, driverdev-devel
  Cc: linux-kernel

With the recent introduction of per-channel tasklet, we need to update
the way we handle the 3 concurrency issues:

1. hv_process_channel_removal -> percpu_channel_deq vs.
   vmbus_chan_sched -> list_for_each_entry(..., percpu_list);

2. vmbus_process_offer -> percpu_channel_enq/deq vs. vmbus_chan_sched.

3. vmbus_close_internal vs. the per-channel tasklet vmbus_on_event;

The first 2 issues can be handled by Stephen's recent patch
"vmbus: use rcu for per-cpu channel list", and the third issue
can be handled by calling tasklet_disable in vmbus_close_internal here.

We don't need the original hv_event_tasklet_disable/enable since we
now use per-channel tasklet instead of the previous per-CPU tasklet,
and actually we must remove them due to the side effect now:
vmbus_process_offer -> hv_event_tasklet_enable -> tasklet_schedule will
start the per-channel callback prematurely, cauing NULL dereferencing
(the channel may haven't been properly configured to run the callback yet).

Fixes: 631e63a9f346 ("vmbus: change to per channel tasklet")

Signed-off-by: Dexuan Cui <decui@microsoft.com>
Cc: "K. Y. Srinivasan" <kys@microsoft.com>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Stephen Hemminger <sthemmin@microsoft.com>
---

Stephen Hemminger's patch ("vmbus: use rcu for per-cpu channel list")
was posted to the community last month, but it hasn't been in char-misc
yet, as far as I know.

 drivers/hv/channel.c      | 12 ++++--------
 drivers/hv/channel_mgmt.c | 19 -------------------
 include/linux/hyperv.h    |  3 ---
 3 files changed, 4 insertions(+), 30 deletions(-)

diff --git a/drivers/hv/channel.c b/drivers/hv/channel.c
index bd0d198..57b2958 100644
--- a/drivers/hv/channel.c
+++ b/drivers/hv/channel.c
@@ -530,15 +530,13 @@ static int vmbus_close_internal(struct vmbus_channel *channel)
 	int ret;
 
 	/*
-	 * vmbus_on_event(), running in the tasklet, can race
+	 * vmbus_on_event(), running in the per-channel tasklet, can race
 	 * with vmbus_close_internal() in the case of SMP guest, e.g., when
 	 * the former is accessing channel->inbound.ring_buffer, the latter
-	 * could be freeing the ring_buffer pages.
-	 *
-	 * To resolve the race, we can serialize them by disabling the
-	 * tasklet when the latter is running here.
+	 * could be freeing the ring_buffer pages, so here we must stop it
+	 * first.
 	 */
-	hv_event_tasklet_disable(channel);
+	tasklet_disable(&channel->callback_event);
 
 	/*
 	 * In case a device driver's probe() fails (e.g.,
@@ -605,8 +603,6 @@ static int vmbus_close_internal(struct vmbus_channel *channel)
 		get_order(channel->ringbuffer_pagecount * PAGE_SIZE));
 
 out:
-	hv_event_tasklet_enable(channel);
-
 	return ret;
 }
 
diff --git a/drivers/hv/channel_mgmt.c b/drivers/hv/channel_mgmt.c
index d2cfa3e..bf846d0 100644
--- a/drivers/hv/channel_mgmt.c
+++ b/drivers/hv/channel_mgmt.c
@@ -382,19 +382,6 @@ static void vmbus_release_relid(u32 relid)
 		       true);
 }
 
-void hv_event_tasklet_disable(struct vmbus_channel *channel)
-{
-	tasklet_disable(&channel->callback_event);
-}
-
-void hv_event_tasklet_enable(struct vmbus_channel *channel)
-{
-	tasklet_enable(&channel->callback_event);
-
-	/* In case there is any pending event */
-	tasklet_schedule(&channel->callback_event);
-}
-
 void hv_process_channel_removal(struct vmbus_channel *channel, u32 relid)
 {
 	unsigned long flags;
@@ -403,7 +390,6 @@ void hv_process_channel_removal(struct vmbus_channel *channel, u32 relid)
 	BUG_ON(!channel->rescind);
 	BUG_ON(!mutex_is_locked(&vmbus_connection.channel_mutex));
 
-	hv_event_tasklet_disable(channel);
 	if (channel->target_cpu != get_cpu()) {
 		put_cpu();
 		smp_call_function_single(channel->target_cpu,
@@ -412,7 +398,6 @@ void hv_process_channel_removal(struct vmbus_channel *channel, u32 relid)
 		percpu_channel_deq(channel);
 		put_cpu();
 	}
-	hv_event_tasklet_enable(channel);
 
 	if (channel->primary_channel == NULL) {
 		list_del(&channel->listentry);
@@ -506,7 +491,6 @@ static void vmbus_process_offer(struct vmbus_channel *newchannel)
 
 	init_vp_index(newchannel, dev_type);
 
-	hv_event_tasklet_disable(newchannel);
 	if (newchannel->target_cpu != get_cpu()) {
 		put_cpu();
 		smp_call_function_single(newchannel->target_cpu,
@@ -516,7 +500,6 @@ static void vmbus_process_offer(struct vmbus_channel *newchannel)
 		percpu_channel_enq(newchannel);
 		put_cpu();
 	}
-	hv_event_tasklet_enable(newchannel);
 
 	/*
 	 * This state is used to indicate a successful open
@@ -566,7 +549,6 @@ static void vmbus_process_offer(struct vmbus_channel *newchannel)
 	list_del(&newchannel->listentry);
 	mutex_unlock(&vmbus_connection.channel_mutex);
 
-	hv_event_tasklet_disable(newchannel);
 	if (newchannel->target_cpu != get_cpu()) {
 		put_cpu();
 		smp_call_function_single(newchannel->target_cpu,
@@ -575,7 +557,6 @@ static void vmbus_process_offer(struct vmbus_channel *newchannel)
 		percpu_channel_deq(newchannel);
 		put_cpu();
 	}
-	hv_event_tasklet_enable(newchannel);
 
 	vmbus_release_relid(newchannel->offermsg.child_relid);
 
diff --git a/include/linux/hyperv.h b/include/linux/hyperv.h
index c4c7ae9..970771a 100644
--- a/include/linux/hyperv.h
+++ b/include/linux/hyperv.h
@@ -1437,9 +1437,6 @@ extern bool vmbus_prep_negotiate_resp(struct icmsg_hdr *icmsghdrp, u8 *buf,
 				const int *srv_version, int srv_vercnt,
 				int *nego_fw_version, int *nego_srv_version);
 
-void hv_event_tasklet_disable(struct vmbus_channel *channel);
-void hv_event_tasklet_enable(struct vmbus_channel *channel);
-
 void hv_process_channel_removal(struct vmbus_channel *channel, u32 relid);
 
 void vmbus_setevent(struct vmbus_channel *channel);
-- 
2.7.4

^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH] vmbus: remove hv_event_tasklet_disable/enable
@ 2017-03-02 12:32 ` Dexuan Cui
  0 siblings, 0 replies; 8+ messages in thread
From: Dexuan Cui @ 2017-03-02 12:32 UTC (permalink / raw)
  To: gregkh, Stephen Hemminger, KY Srinivasan, Haiyang Zhang, driverdev-devel
  Cc: linux-kernel

With the recent introduction of per-channel tasklet, we need to update
the way we handle the 3 concurrency issues:

1. hv_process_channel_removal -> percpu_channel_deq vs.
   vmbus_chan_sched -> list_for_each_entry(..., percpu_list);

2. vmbus_process_offer -> percpu_channel_enq/deq vs. vmbus_chan_sched.

3. vmbus_close_internal vs. the per-channel tasklet vmbus_on_event;

The first 2 issues can be handled by Stephen's recent patch
"vmbus: use rcu for per-cpu channel list", and the third issue
can be handled by calling tasklet_disable in vmbus_close_internal here.

We don't need the original hv_event_tasklet_disable/enable since we
now use per-channel tasklet instead of the previous per-CPU tasklet,
and actually we must remove them due to the side effect now:
vmbus_process_offer -> hv_event_tasklet_enable -> tasklet_schedule will
start the per-channel callback prematurely, cauing NULL dereferencing
(the channel may haven't been properly configured to run the callback yet).

Fixes: 631e63a9f346 ("vmbus: change to per channel tasklet")

Signed-off-by: Dexuan Cui <decui@microsoft.com>
Cc: "K. Y. Srinivasan" <kys@microsoft.com>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Stephen Hemminger <sthemmin@microsoft.com>
---

Stephen Hemminger's patch ("vmbus: use rcu for per-cpu channel list")
was posted to the community last month, but it hasn't been in char-misc
yet, as far as I know.

 drivers/hv/channel.c      | 12 ++++--------
 drivers/hv/channel_mgmt.c | 19 -------------------
 include/linux/hyperv.h    |  3 ---
 3 files changed, 4 insertions(+), 30 deletions(-)

diff --git a/drivers/hv/channel.c b/drivers/hv/channel.c
index bd0d198..57b2958 100644
--- a/drivers/hv/channel.c
+++ b/drivers/hv/channel.c
@@ -530,15 +530,13 @@ static int vmbus_close_internal(struct vmbus_channel *channel)
 	int ret;
 
 	/*
-	 * vmbus_on_event(), running in the tasklet, can race
+	 * vmbus_on_event(), running in the per-channel tasklet, can race
 	 * with vmbus_close_internal() in the case of SMP guest, e.g., when
 	 * the former is accessing channel->inbound.ring_buffer, the latter
-	 * could be freeing the ring_buffer pages.
-	 *
-	 * To resolve the race, we can serialize them by disabling the
-	 * tasklet when the latter is running here.
+	 * could be freeing the ring_buffer pages, so here we must stop it
+	 * first.
 	 */
-	hv_event_tasklet_disable(channel);
+	tasklet_disable(&channel->callback_event);
 
 	/*
 	 * In case a device driver's probe() fails (e.g.,
@@ -605,8 +603,6 @@ static int vmbus_close_internal(struct vmbus_channel *channel)
 		get_order(channel->ringbuffer_pagecount * PAGE_SIZE));
 
 out:
-	hv_event_tasklet_enable(channel);
-
 	return ret;
 }
 
diff --git a/drivers/hv/channel_mgmt.c b/drivers/hv/channel_mgmt.c
index d2cfa3e..bf846d0 100644
--- a/drivers/hv/channel_mgmt.c
+++ b/drivers/hv/channel_mgmt.c
@@ -382,19 +382,6 @@ static void vmbus_release_relid(u32 relid)
 		       true);
 }
 
-void hv_event_tasklet_disable(struct vmbus_channel *channel)
-{
-	tasklet_disable(&channel->callback_event);
-}
-
-void hv_event_tasklet_enable(struct vmbus_channel *channel)
-{
-	tasklet_enable(&channel->callback_event);
-
-	/* In case there is any pending event */
-	tasklet_schedule(&channel->callback_event);
-}
-
 void hv_process_channel_removal(struct vmbus_channel *channel, u32 relid)
 {
 	unsigned long flags;
@@ -403,7 +390,6 @@ void hv_process_channel_removal(struct vmbus_channel *channel, u32 relid)
 	BUG_ON(!channel->rescind);
 	BUG_ON(!mutex_is_locked(&vmbus_connection.channel_mutex));
 
-	hv_event_tasklet_disable(channel);
 	if (channel->target_cpu != get_cpu()) {
 		put_cpu();
 		smp_call_function_single(channel->target_cpu,
@@ -412,7 +398,6 @@ void hv_process_channel_removal(struct vmbus_channel *channel, u32 relid)
 		percpu_channel_deq(channel);
 		put_cpu();
 	}
-	hv_event_tasklet_enable(channel);
 
 	if (channel->primary_channel == NULL) {
 		list_del(&channel->listentry);
@@ -506,7 +491,6 @@ static void vmbus_process_offer(struct vmbus_channel *newchannel)
 
 	init_vp_index(newchannel, dev_type);
 
-	hv_event_tasklet_disable(newchannel);
 	if (newchannel->target_cpu != get_cpu()) {
 		put_cpu();
 		smp_call_function_single(newchannel->target_cpu,
@@ -516,7 +500,6 @@ static void vmbus_process_offer(struct vmbus_channel *newchannel)
 		percpu_channel_enq(newchannel);
 		put_cpu();
 	}
-	hv_event_tasklet_enable(newchannel);
 
 	/*
 	 * This state is used to indicate a successful open
@@ -566,7 +549,6 @@ static void vmbus_process_offer(struct vmbus_channel *newchannel)
 	list_del(&newchannel->listentry);
 	mutex_unlock(&vmbus_connection.channel_mutex);
 
-	hv_event_tasklet_disable(newchannel);
 	if (newchannel->target_cpu != get_cpu()) {
 		put_cpu();
 		smp_call_function_single(newchannel->target_cpu,
@@ -575,7 +557,6 @@ static void vmbus_process_offer(struct vmbus_channel *newchannel)
 		percpu_channel_deq(newchannel);
 		put_cpu();
 	}
-	hv_event_tasklet_enable(newchannel);
 
 	vmbus_release_relid(newchannel->offermsg.child_relid);
 
diff --git a/include/linux/hyperv.h b/include/linux/hyperv.h
index c4c7ae9..970771a 100644
--- a/include/linux/hyperv.h
+++ b/include/linux/hyperv.h
@@ -1437,9 +1437,6 @@ extern bool vmbus_prep_negotiate_resp(struct icmsg_hdr *icmsghdrp, u8 *buf,
 				const int *srv_version, int srv_vercnt,
 				int *nego_fw_version, int *nego_srv_version);
 
-void hv_event_tasklet_disable(struct vmbus_channel *channel);
-void hv_event_tasklet_enable(struct vmbus_channel *channel);
-
 void hv_process_channel_removal(struct vmbus_channel *channel, u32 relid);
 
 void vmbus_setevent(struct vmbus_channel *channel);
-- 
2.7.4

_______________________________________________
devel mailing list
devel@linuxdriverproject.org
http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel

^ permalink raw reply related	[flat|nested] 8+ messages in thread

* Re: [PATCH] vmbus: remove hv_event_tasklet_disable/enable
  2017-03-02 12:32 ` Dexuan Cui
@ 2017-03-03 16:40   ` Vitaly Kuznetsov
  -1 siblings, 0 replies; 8+ messages in thread
From: Vitaly Kuznetsov @ 2017-03-03 16:40 UTC (permalink / raw)
  To: Dexuan Cui
  Cc: gregkh, Stephen Hemminger, KY Srinivasan, Haiyang Zhang,
	driverdev-devel, linux-kernel

Dexuan Cui <decui@microsoft.com> writes:

> With the recent introduction of per-channel tasklet, we need to update
> the way we handle the 3 concurrency issues:
>
> 1. hv_process_channel_removal -> percpu_channel_deq vs.
>    vmbus_chan_sched -> list_for_each_entry(..., percpu_list);
>
> 2. vmbus_process_offer -> percpu_channel_enq/deq vs. vmbus_chan_sched.
>
> 3. vmbus_close_internal vs. the per-channel tasklet vmbus_on_event;
>
> The first 2 issues can be handled by Stephen's recent patch
> "vmbus: use rcu for per-cpu channel list", and the third issue
> can be handled by calling tasklet_disable in vmbus_close_internal here.
>
> We don't need the original hv_event_tasklet_disable/enable since we
> now use per-channel tasklet instead of the previous per-CPU tasklet,
> and actually we must remove them due to the side effect now:
> vmbus_process_offer -> hv_event_tasklet_enable -> tasklet_schedule will
> start the per-channel callback prematurely, cauing NULL dereferencing
> (the channel may haven't been properly configured to run the callback yet).
>
> Fixes: 631e63a9f346 ("vmbus: change to per channel tasklet")
>
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> Cc: "K. Y. Srinivasan" <kys@microsoft.com>
> Cc: Haiyang Zhang <haiyangz@microsoft.com>
> Cc: Stephen Hemminger <sthemmin@microsoft.com>

Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>

This patch fixes the following crash on boot:

[    1.451648] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
[    1.452255] IP: netvsc_channel_cb+0x90/0x7b0 [hv_netvsc]
[    1.452255] PGD 0 
[    1.452255] 
[    1.452255] Oops: 0000 [#1] SMP
[    1.452255] Modules linked in: hv_storvsc hv_netvsc(+) scsi_transport_fc hyperv_fb hyperv_keyboard hid_hyperv hv_vmbus
[    1.452255] CPU: 1 PID: 19 Comm: ksoftirqd/1 Not tainted 4.10.0_test+ #911
[    1.452255] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v1.0 11/26/2012
[    1.452255] task: ffff880007fd2b00 task.stack: ffffc90000e34000
[    1.452255] RIP: 0010:netvsc_channel_cb+0x90/0x7b0 [hv_netvsc]
...
[    1.452255] Call Trace:
[    1.452255]  vmbus_on_event+0x22/0x90 [hv_vmbus]
[    1.452255]  tasklet_action+0x5e/0x110
[    1.452255]  __do_softirq+0x104/0x2af
[    1.452255]  run_ksoftirqd+0x29/0x40
...
[    1.548068] RIP: netvsc_channel_cb+0x90/0x7b0 [hv_netvsc] RSP: ffffc90000e37d88
[    1.548068] CR2: 0000000000000004
[    1.548068] ---[ end trace 601fd9d6588b21e5 ]---
[    1.548068] Kernel panic - not syncing: Fatal exception in interrupt
[    1.548068] Kernel Offset: disabled
[    1.548068] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
[    1.572155] ------------[ cut here ]------------

The crash is not imminent but it happens pretty often on boot, I think
we need to push it to 4.11.

-- 
  Vitaly

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] vmbus: remove hv_event_tasklet_disable/enable
@ 2017-03-03 16:40   ` Vitaly Kuznetsov
  0 siblings, 0 replies; 8+ messages in thread
From: Vitaly Kuznetsov @ 2017-03-03 16:40 UTC (permalink / raw)
  To: Dexuan Cui
  Cc: Stephen Hemminger, gregkh, Haiyang Zhang, driverdev-devel, linux-kernel

Dexuan Cui <decui@microsoft.com> writes:

> With the recent introduction of per-channel tasklet, we need to update
> the way we handle the 3 concurrency issues:
>
> 1. hv_process_channel_removal -> percpu_channel_deq vs.
>    vmbus_chan_sched -> list_for_each_entry(..., percpu_list);
>
> 2. vmbus_process_offer -> percpu_channel_enq/deq vs. vmbus_chan_sched.
>
> 3. vmbus_close_internal vs. the per-channel tasklet vmbus_on_event;
>
> The first 2 issues can be handled by Stephen's recent patch
> "vmbus: use rcu for per-cpu channel list", and the third issue
> can be handled by calling tasklet_disable in vmbus_close_internal here.
>
> We don't need the original hv_event_tasklet_disable/enable since we
> now use per-channel tasklet instead of the previous per-CPU tasklet,
> and actually we must remove them due to the side effect now:
> vmbus_process_offer -> hv_event_tasklet_enable -> tasklet_schedule will
> start the per-channel callback prematurely, cauing NULL dereferencing
> (the channel may haven't been properly configured to run the callback yet).
>
> Fixes: 631e63a9f346 ("vmbus: change to per channel tasklet")
>
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> Cc: "K. Y. Srinivasan" <kys@microsoft.com>
> Cc: Haiyang Zhang <haiyangz@microsoft.com>
> Cc: Stephen Hemminger <sthemmin@microsoft.com>

Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>

This patch fixes the following crash on boot:

[    1.451648] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
[    1.452255] IP: netvsc_channel_cb+0x90/0x7b0 [hv_netvsc]
[    1.452255] PGD 0 
[    1.452255] 
[    1.452255] Oops: 0000 [#1] SMP
[    1.452255] Modules linked in: hv_storvsc hv_netvsc(+) scsi_transport_fc hyperv_fb hyperv_keyboard hid_hyperv hv_vmbus
[    1.452255] CPU: 1 PID: 19 Comm: ksoftirqd/1 Not tainted 4.10.0_test+ #911
[    1.452255] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v1.0 11/26/2012
[    1.452255] task: ffff880007fd2b00 task.stack: ffffc90000e34000
[    1.452255] RIP: 0010:netvsc_channel_cb+0x90/0x7b0 [hv_netvsc]
...
[    1.452255] Call Trace:
[    1.452255]  vmbus_on_event+0x22/0x90 [hv_vmbus]
[    1.452255]  tasklet_action+0x5e/0x110
[    1.452255]  __do_softirq+0x104/0x2af
[    1.452255]  run_ksoftirqd+0x29/0x40
...
[    1.548068] RIP: netvsc_channel_cb+0x90/0x7b0 [hv_netvsc] RSP: ffffc90000e37d88
[    1.548068] CR2: 0000000000000004
[    1.548068] ---[ end trace 601fd9d6588b21e5 ]---
[    1.548068] Kernel panic - not syncing: Fatal exception in interrupt
[    1.548068] Kernel Offset: disabled
[    1.548068] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
[    1.572155] ------------[ cut here ]------------

The crash is not imminent but it happens pretty often on boot, I think
we need to push it to 4.11.

-- 
  Vitaly
_______________________________________________
devel mailing list
devel@linuxdriverproject.org
http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] vmbus: remove hv_event_tasklet_disable/enable
  2017-03-03 16:40   ` Vitaly Kuznetsov
@ 2017-03-03 17:12     ` Stephen Hemminger
  -1 siblings, 0 replies; 8+ messages in thread
From: Stephen Hemminger @ 2017-03-03 17:12 UTC (permalink / raw)
  To: Vitaly Kuznetsov
  Cc: Dexuan Cui, Stephen Hemminger, gregkh, Haiyang Zhang,
	driverdev-devel, linux-kernel

On Fri, 03 Mar 2017 17:40:47 +0100
Vitaly Kuznetsov <vkuznets@redhat.com> wrote:

> Dexuan Cui <decui@microsoft.com> writes:
> 
> > With the recent introduction of per-channel tasklet, we need to update
> > the way we handle the 3 concurrency issues:
> >
> > 1. hv_process_channel_removal -> percpu_channel_deq vs.
> >    vmbus_chan_sched -> list_for_each_entry(..., percpu_list);
> >
> > 2. vmbus_process_offer -> percpu_channel_enq/deq vs. vmbus_chan_sched.
> >
> > 3. vmbus_close_internal vs. the per-channel tasklet vmbus_on_event;
> >
> > The first 2 issues can be handled by Stephen's recent patch
> > "vmbus: use rcu for per-cpu channel list", and the third issue
> > can be handled by calling tasklet_disable in vmbus_close_internal here.
> >
> > We don't need the original hv_event_tasklet_disable/enable since we
> > now use per-channel tasklet instead of the previous per-CPU tasklet,
> > and actually we must remove them due to the side effect now:
> > vmbus_process_offer -> hv_event_tasklet_enable -> tasklet_schedule will
> > start the per-channel callback prematurely, cauing NULL dereferencing
> > (the channel may haven't been properly configured to run the callback yet).
> >
> > Fixes: 631e63a9f346 ("vmbus: change to per channel tasklet")
> >
> > Signed-off-by: Dexuan Cui <decui@microsoft.com>
> > Cc: "K. Y. Srinivasan" <kys@microsoft.com>
> > Cc: Haiyang Zhang <haiyangz@microsoft.com>
> > Cc: Stephen Hemminger <sthemmin@microsoft.com>  
> 
> Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> 
> This patch fixes the following crash on boot:
> 
> [    1.451648] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
> [    1.452255] IP: netvsc_channel_cb+0x90/0x7b0 [hv_netvsc]
> [    1.452255] PGD 0 
> [    1.452255] 
> [    1.452255] Oops: 0000 [#1] SMP
> [    1.452255] Modules linked in: hv_storvsc hv_netvsc(+) scsi_transport_fc hyperv_fb hyperv_keyboard hid_hyperv hv_vmbus
> [    1.452255] CPU: 1 PID: 19 Comm: ksoftirqd/1 Not tainted 4.10.0_test+ #911
> [    1.452255] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v1.0 11/26/2012
> [    1.452255] task: ffff880007fd2b00 task.stack: ffffc90000e34000
> [    1.452255] RIP: 0010:netvsc_channel_cb+0x90/0x7b0 [hv_netvsc]
> ...
> [    1.452255] Call Trace:
> [    1.452255]  vmbus_on_event+0x22/0x90 [hv_vmbus]
> [    1.452255]  tasklet_action+0x5e/0x110
> [    1.452255]  __do_softirq+0x104/0x2af
> [    1.452255]  run_ksoftirqd+0x29/0x40
> ...
> [    1.548068] RIP: netvsc_channel_cb+0x90/0x7b0 [hv_netvsc] RSP: ffffc90000e37d88
> [    1.548068] CR2: 0000000000000004
> [    1.548068] ---[ end trace 601fd9d6588b21e5 ]---
> [    1.548068] Kernel panic - not syncing: Fatal exception in interrupt
> [    1.548068] Kernel Offset: disabled
> [    1.548068] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
> [    1.572155] ------------[ cut here ]------------
> 
> The crash is not imminent but it happens pretty often on boot, I think
> we need to push it to 4.11.
> 

Agree that this needs to be in 4.11, but when NAPI is merged it also will not happen.

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] vmbus: remove hv_event_tasklet_disable/enable
@ 2017-03-03 17:12     ` Stephen Hemminger
  0 siblings, 0 replies; 8+ messages in thread
From: Stephen Hemminger @ 2017-03-03 17:12 UTC (permalink / raw)
  To: Vitaly Kuznetsov
  Cc: Stephen Hemminger, gregkh, Haiyang Zhang, driverdev-devel, linux-kernel

On Fri, 03 Mar 2017 17:40:47 +0100
Vitaly Kuznetsov <vkuznets@redhat.com> wrote:

> Dexuan Cui <decui@microsoft.com> writes:
> 
> > With the recent introduction of per-channel tasklet, we need to update
> > the way we handle the 3 concurrency issues:
> >
> > 1. hv_process_channel_removal -> percpu_channel_deq vs.
> >    vmbus_chan_sched -> list_for_each_entry(..., percpu_list);
> >
> > 2. vmbus_process_offer -> percpu_channel_enq/deq vs. vmbus_chan_sched.
> >
> > 3. vmbus_close_internal vs. the per-channel tasklet vmbus_on_event;
> >
> > The first 2 issues can be handled by Stephen's recent patch
> > "vmbus: use rcu for per-cpu channel list", and the third issue
> > can be handled by calling tasklet_disable in vmbus_close_internal here.
> >
> > We don't need the original hv_event_tasklet_disable/enable since we
> > now use per-channel tasklet instead of the previous per-CPU tasklet,
> > and actually we must remove them due to the side effect now:
> > vmbus_process_offer -> hv_event_tasklet_enable -> tasklet_schedule will
> > start the per-channel callback prematurely, cauing NULL dereferencing
> > (the channel may haven't been properly configured to run the callback yet).
> >
> > Fixes: 631e63a9f346 ("vmbus: change to per channel tasklet")
> >
> > Signed-off-by: Dexuan Cui <decui@microsoft.com>
> > Cc: "K. Y. Srinivasan" <kys@microsoft.com>
> > Cc: Haiyang Zhang <haiyangz@microsoft.com>
> > Cc: Stephen Hemminger <sthemmin@microsoft.com>  
> 
> Tested-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> 
> This patch fixes the following crash on boot:
> 
> [    1.451648] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
> [    1.452255] IP: netvsc_channel_cb+0x90/0x7b0 [hv_netvsc]
> [    1.452255] PGD 0 
> [    1.452255] 
> [    1.452255] Oops: 0000 [#1] SMP
> [    1.452255] Modules linked in: hv_storvsc hv_netvsc(+) scsi_transport_fc hyperv_fb hyperv_keyboard hid_hyperv hv_vmbus
> [    1.452255] CPU: 1 PID: 19 Comm: ksoftirqd/1 Not tainted 4.10.0_test+ #911
> [    1.452255] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v1.0 11/26/2012
> [    1.452255] task: ffff880007fd2b00 task.stack: ffffc90000e34000
> [    1.452255] RIP: 0010:netvsc_channel_cb+0x90/0x7b0 [hv_netvsc]
> ...
> [    1.452255] Call Trace:
> [    1.452255]  vmbus_on_event+0x22/0x90 [hv_vmbus]
> [    1.452255]  tasklet_action+0x5e/0x110
> [    1.452255]  __do_softirq+0x104/0x2af
> [    1.452255]  run_ksoftirqd+0x29/0x40
> ...
> [    1.548068] RIP: netvsc_channel_cb+0x90/0x7b0 [hv_netvsc] RSP: ffffc90000e37d88
> [    1.548068] CR2: 0000000000000004
> [    1.548068] ---[ end trace 601fd9d6588b21e5 ]---
> [    1.548068] Kernel panic - not syncing: Fatal exception in interrupt
> [    1.548068] Kernel Offset: disabled
> [    1.548068] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
> [    1.572155] ------------[ cut here ]------------
> 
> The crash is not imminent but it happens pretty often on boot, I think
> we need to push it to 4.11.
> 

Agree that this needs to be in 4.11, but when NAPI is merged it also will not happen.
_______________________________________________
devel mailing list
devel@linuxdriverproject.org
http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] vmbus: remove hv_event_tasklet_disable/enable
  2017-03-02 12:32 ` Dexuan Cui
@ 2017-03-17 23:15   ` Stephen Hemminger
  -1 siblings, 0 replies; 8+ messages in thread
From: Stephen Hemminger @ 2017-03-17 23:15 UTC (permalink / raw)
  To: Dexuan Cui
  Cc: gregkh, Stephen Hemminger, KY Srinivasan, Haiyang Zhang,
	driverdev-devel, linux-kernel

On Thu, 2 Mar 2017 12:32:55 +0000
Dexuan Cui <decui@microsoft.com> wrote:

> With the recent introduction of per-channel tasklet, we need to update
> the way we handle the 3 concurrency issues:
> 
> 1. hv_process_channel_removal -> percpu_channel_deq vs.
>    vmbus_chan_sched -> list_for_each_entry(..., percpu_list);
> 
> 2. vmbus_process_offer -> percpu_channel_enq/deq vs. vmbus_chan_sched.
> 
> 3. vmbus_close_internal vs. the per-channel tasklet vmbus_on_event;
> 
> The first 2 issues can be handled by Stephen's recent patch
> "vmbus: use rcu for per-cpu channel list", and the third issue
> can be handled by calling tasklet_disable in vmbus_close_internal here.
> 
> We don't need the original hv_event_tasklet_disable/enable since we
> now use per-channel tasklet instead of the previous per-CPU tasklet,
> and actually we must remove them due to the side effect now:
> vmbus_process_offer -> hv_event_tasklet_enable -> tasklet_schedule will
> start the per-channel callback prematurely, cauing NULL dereferencing
> (the channel may haven't been properly configured to run the callback yet).
> 
> Fixes: 631e63a9f346 ("vmbus: change to per channel tasklet")
> 
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> Cc: "K. Y. Srinivasan" <kys@microsoft.com>
> Cc: Haiyang Zhang <haiyangz@microsoft.com>
> Cc: Stephen Hemminger <sthemmin@microsoft.com>

It also fixes a number of missed interrupts causing stuck tasks on boot.
Please put it in 4.11

Reviewed-by: Stephen Hemminger <sthemmin@microsoft.com>

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] vmbus: remove hv_event_tasklet_disable/enable
@ 2017-03-17 23:15   ` Stephen Hemminger
  0 siblings, 0 replies; 8+ messages in thread
From: Stephen Hemminger @ 2017-03-17 23:15 UTC (permalink / raw)
  To: Dexuan Cui
  Cc: Stephen Hemminger, gregkh, Haiyang Zhang, driverdev-devel, linux-kernel

On Thu, 2 Mar 2017 12:32:55 +0000
Dexuan Cui <decui@microsoft.com> wrote:

> With the recent introduction of per-channel tasklet, we need to update
> the way we handle the 3 concurrency issues:
> 
> 1. hv_process_channel_removal -> percpu_channel_deq vs.
>    vmbus_chan_sched -> list_for_each_entry(..., percpu_list);
> 
> 2. vmbus_process_offer -> percpu_channel_enq/deq vs. vmbus_chan_sched.
> 
> 3. vmbus_close_internal vs. the per-channel tasklet vmbus_on_event;
> 
> The first 2 issues can be handled by Stephen's recent patch
> "vmbus: use rcu for per-cpu channel list", and the third issue
> can be handled by calling tasklet_disable in vmbus_close_internal here.
> 
> We don't need the original hv_event_tasklet_disable/enable since we
> now use per-channel tasklet instead of the previous per-CPU tasklet,
> and actually we must remove them due to the side effect now:
> vmbus_process_offer -> hv_event_tasklet_enable -> tasklet_schedule will
> start the per-channel callback prematurely, cauing NULL dereferencing
> (the channel may haven't been properly configured to run the callback yet).
> 
> Fixes: 631e63a9f346 ("vmbus: change to per channel tasklet")
> 
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> Cc: "K. Y. Srinivasan" <kys@microsoft.com>
> Cc: Haiyang Zhang <haiyangz@microsoft.com>
> Cc: Stephen Hemminger <sthemmin@microsoft.com>

It also fixes a number of missed interrupts causing stuck tasks on boot.
Please put it in 4.11

Reviewed-by: Stephen Hemminger <sthemmin@microsoft.com>

_______________________________________________
devel mailing list
devel@linuxdriverproject.org
http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2017-03-17 23:43 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-03-02 12:32 [PATCH] vmbus: remove hv_event_tasklet_disable/enable Dexuan Cui
2017-03-02 12:32 ` Dexuan Cui
2017-03-03 16:40 ` Vitaly Kuznetsov
2017-03-03 16:40   ` Vitaly Kuznetsov
2017-03-03 17:12   ` Stephen Hemminger
2017-03-03 17:12     ` Stephen Hemminger
2017-03-17 23:15 ` Stephen Hemminger
2017-03-17 23:15   ` Stephen Hemminger

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.