linux-hyperv.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver
@ 2021-08-31 14:39 Vitaly Kuznetsov
  2021-09-01 11:25 ` Wei Liu
                   ` (3 more replies)
  0 siblings, 4 replies; 6+ messages in thread
From: Vitaly Kuznetsov @ 2021-08-31 14:39 UTC (permalink / raw)
  To: linux-hyperv
  Cc: Andres Beltran, Michael Kelley, Andrea Parri (Microsoft),
	Dexuan Cui, Wei Liu, Stephen Hemminger, Haiyang Zhang,
	K. Y. Srinivasan, linux-kernel

The following crash happens when a never-used device is unbound from
uio_hv_generic driver:

 kernel BUG at mm/slub.c:321!
 invalid opcode: 0000 [#1] SMP PTI
 CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G               X --------- ---  5.14.0-0.rc2.23.el9.x86_64 #1
 Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
 RIP: 0010:__slab_free+0x1d5/0x3d0
...
 Call Trace:
  ? pick_next_task_fair+0x18e/0x3b0
  ? __cond_resched+0x16/0x40
  ? vunmap_pmd_range.isra.0+0x154/0x1c0
  ? __vunmap+0x22d/0x290
  ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
  kfree+0x331/0x380
  ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
  hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
  vmbus_free_ring+0x21/0x60 [hv_vmbus]
  hv_uio_remove+0x4f/0x60 [uio_hv_generic]
  vmbus_remove+0x23/0x30 [hv_vmbus]
  __device_release_driver+0x17a/0x230
  device_driver_detach+0x3c/0xa0
  unbind_store+0x113/0x130
...

The problem appears to be that we free 'ring_info->pkt_buffer' twice:
first, when the device is unbound from in-kernel driver (netvsc in this
case) and second from hv_uio_remove(). Normally, ring buffer is supposed
to be re-initialized from hv_uio_open() but this happens when UIO device
is being opened and this is not guaranteed to happen.

Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same
channel (which is being handed over between in-kernel drivers and UIO) even
if we didn't call hv_ringbuffer_init() in between. We, however, need to
avoid kfree() call for an already freed pointer.

Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 drivers/hv/ring_buffer.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
index 2aee356840a2..314015d9e912 100644
--- a/drivers/hv/ring_buffer.c
+++ b/drivers/hv/ring_buffer.c
@@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
 	mutex_unlock(&ring_info->ring_buffer_mutex);
 
 	kfree(ring_info->pkt_buffer);
+	ring_info->pkt_buffer = NULL;
 	ring_info->pkt_buffer_size = 0;
 }
 
-- 
2.31.1


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver
  2021-08-31 14:39 [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver Vitaly Kuznetsov
@ 2021-09-01 11:25 ` Wei Liu
  2021-09-01 11:40   ` Vitaly Kuznetsov
  2021-09-01 15:25 ` Andrea Parri
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 6+ messages in thread
From: Wei Liu @ 2021-09-01 11:25 UTC (permalink / raw)
  To: Vitaly Kuznetsov
  Cc: linux-hyperv, Andres Beltran, Michael Kelley,
	Andrea Parri (Microsoft),
	Dexuan Cui, Wei Liu, Stephen Hemminger, Haiyang Zhang,
	K. Y. Srinivasan, linux-kernel

On Tue, Aug 31, 2021 at 04:39:16PM +0200, Vitaly Kuznetsov wrote:
> The following crash happens when a never-used device is unbound from
> uio_hv_generic driver:
> 
>  kernel BUG at mm/slub.c:321!
>  invalid opcode: 0000 [#1] SMP PTI
>  CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G               X --------- ---  5.14.0-0.rc2.23.el9.x86_64 #1
>  Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
>  RIP: 0010:__slab_free+0x1d5/0x3d0
> ...
>  Call Trace:
>   ? pick_next_task_fair+0x18e/0x3b0
>   ? __cond_resched+0x16/0x40
>   ? vunmap_pmd_range.isra.0+0x154/0x1c0
>   ? __vunmap+0x22d/0x290
>   ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
>   kfree+0x331/0x380
>   ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
>   hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
>   vmbus_free_ring+0x21/0x60 [hv_vmbus]
>   hv_uio_remove+0x4f/0x60 [uio_hv_generic]
>   vmbus_remove+0x23/0x30 [hv_vmbus]
>   __device_release_driver+0x17a/0x230
>   device_driver_detach+0x3c/0xa0
>   unbind_store+0x113/0x130
> ...
> 
> The problem appears to be that we free 'ring_info->pkt_buffer' twice:
> first, when the device is unbound from in-kernel driver (netvsc in this
> case) and second from hv_uio_remove(). Normally, ring buffer is supposed
> to be re-initialized from hv_uio_open() but this happens when UIO device
> is being opened and this is not guaranteed to happen.
> 
> Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same
> channel (which is being handed over between in-kernel drivers and UIO) even
> if we didn't call hv_ringbuffer_init() in between. We, however, need to
> avoid kfree() call for an already freed pointer.
> 
> Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  drivers/hv/ring_buffer.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
> index 2aee356840a2..314015d9e912 100644
> --- a/drivers/hv/ring_buffer.c
> +++ b/drivers/hv/ring_buffer.c
> @@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
>  	mutex_unlock(&ring_info->ring_buffer_mutex);
>  
>  	kfree(ring_info->pkt_buffer);
> +	ring_info->pkt_buffer = NULL;

This certainly won't hurt.

I would however like to wait till Andrea and Michael go over the
reasoning of this patch before doing anything.

Wei.

>  	ring_info->pkt_buffer_size = 0;
>  }
>  
> -- 
> 2.31.1
> 

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver
  2021-09-01 11:25 ` Wei Liu
@ 2021-09-01 11:40   ` Vitaly Kuznetsov
  0 siblings, 0 replies; 6+ messages in thread
From: Vitaly Kuznetsov @ 2021-09-01 11:40 UTC (permalink / raw)
  To: Wei Liu
  Cc: linux-hyperv, Andres Beltran, Michael Kelley,
	Andrea Parri (Microsoft),
	Dexuan Cui, Wei Liu, Stephen Hemminger, Haiyang Zhang,
	K. Y. Srinivasan, linux-kernel

Wei Liu <wei.liu@kernel.org> writes:

> On Tue, Aug 31, 2021 at 04:39:16PM +0200, Vitaly Kuznetsov wrote:
>> The following crash happens when a never-used device is unbound from
>> uio_hv_generic driver:
>> 
>>  kernel BUG at mm/slub.c:321!
>>  invalid opcode: 0000 [#1] SMP PTI
>>  CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G               X --------- ---  5.14.0-0.rc2.23.el9.x86_64 #1
>>  Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
>>  RIP: 0010:__slab_free+0x1d5/0x3d0
>> ...
>>  Call Trace:
>>   ? pick_next_task_fair+0x18e/0x3b0
>>   ? __cond_resched+0x16/0x40
>>   ? vunmap_pmd_range.isra.0+0x154/0x1c0
>>   ? __vunmap+0x22d/0x290
>>   ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
>>   kfree+0x331/0x380
>>   ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
>>   hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
>>   vmbus_free_ring+0x21/0x60 [hv_vmbus]
>>   hv_uio_remove+0x4f/0x60 [uio_hv_generic]
>>   vmbus_remove+0x23/0x30 [hv_vmbus]
>>   __device_release_driver+0x17a/0x230
>>   device_driver_detach+0x3c/0xa0
>>   unbind_store+0x113/0x130
>> ...
>> 
>> The problem appears to be that we free 'ring_info->pkt_buffer' twice:
>> first, when the device is unbound from in-kernel driver (netvsc in this
>> case) and second from hv_uio_remove(). Normally, ring buffer is supposed
>> to be re-initialized from hv_uio_open() but this happens when UIO device
>> is being opened and this is not guaranteed to happen.
>> 
>> Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same
>> channel (which is being handed over between in-kernel drivers and UIO) even
>> if we didn't call hv_ringbuffer_init() in between. We, however, need to
>> avoid kfree() call for an already freed pointer.
>> 
>> Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> ---
>>  drivers/hv/ring_buffer.c | 1 +
>>  1 file changed, 1 insertion(+)
>> 
>> diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
>> index 2aee356840a2..314015d9e912 100644
>> --- a/drivers/hv/ring_buffer.c
>> +++ b/drivers/hv/ring_buffer.c
>> @@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
>>  	mutex_unlock(&ring_info->ring_buffer_mutex);
>>  
>>  	kfree(ring_info->pkt_buffer);
>> +	ring_info->pkt_buffer = NULL;
>
> This certainly won't hurt.
>
> I would however like to wait till Andrea and Michael go over the
> reasoning of this patch before doing anything.
>

Thanks,

the counter-intuitive thing here is that the channel sometimes continues
to live after hv_ringbuffer_cleanup(): when we unbind a device from
in-kernel driver and give it to uio_hv_generic the channel is handed
over from one driver to another.

> Wei.
>
>>  	ring_info->pkt_buffer_size = 0;
>>  }
>>  
>> -- 
>> 2.31.1
>> 
>

-- 
Vitaly


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver
  2021-08-31 14:39 [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver Vitaly Kuznetsov
  2021-09-01 11:25 ` Wei Liu
@ 2021-09-01 15:25 ` Andrea Parri
  2021-09-02 17:09 ` Michael Kelley
  2021-09-03 11:00 ` Wei Liu
  3 siblings, 0 replies; 6+ messages in thread
From: Andrea Parri @ 2021-09-01 15:25 UTC (permalink / raw)
  To: Vitaly Kuznetsov
  Cc: linux-hyperv, Andres Beltran, Michael Kelley, Dexuan Cui,
	Wei Liu, Stephen Hemminger, Haiyang Zhang, K. Y. Srinivasan,
	linux-kernel

On Tue, Aug 31, 2021 at 04:39:16PM +0200, Vitaly Kuznetsov wrote:
> The following crash happens when a never-used device is unbound from
> uio_hv_generic driver:
> 
>  kernel BUG at mm/slub.c:321!
>  invalid opcode: 0000 [#1] SMP PTI
>  CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G               X --------- ---  5.14.0-0.rc2.23.el9.x86_64 #1
>  Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
>  RIP: 0010:__slab_free+0x1d5/0x3d0
> ...
>  Call Trace:
>   ? pick_next_task_fair+0x18e/0x3b0
>   ? __cond_resched+0x16/0x40
>   ? vunmap_pmd_range.isra.0+0x154/0x1c0
>   ? __vunmap+0x22d/0x290
>   ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
>   kfree+0x331/0x380
>   ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
>   hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
>   vmbus_free_ring+0x21/0x60 [hv_vmbus]
>   hv_uio_remove+0x4f/0x60 [uio_hv_generic]
>   vmbus_remove+0x23/0x30 [hv_vmbus]
>   __device_release_driver+0x17a/0x230
>   device_driver_detach+0x3c/0xa0
>   unbind_store+0x113/0x130
> ...
> 
> The problem appears to be that we free 'ring_info->pkt_buffer' twice:
> first, when the device is unbound from in-kernel driver (netvsc in this
> case) and second from hv_uio_remove(). Normally, ring buffer is supposed
> to be re-initialized from hv_uio_open() but this happens when UIO device
> is being opened and this is not guaranteed to happen.
> 
> Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same
> channel (which is being handed over between in-kernel drivers and UIO) even
> if we didn't call hv_ringbuffer_init() in between. We, however, need to
> avoid kfree() call for an already freed pointer.
> 
> Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>

LGTM.

Reviewed-by: Andrea Parri <parri.andrea@gmail.com>

ae6935ed7d424f appears to have anticipated this problem on ->ring_buffer
and adopted the solution proposed here by Vitaly.

  Andrea


> ---
>  drivers/hv/ring_buffer.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
> index 2aee356840a2..314015d9e912 100644
> --- a/drivers/hv/ring_buffer.c
> +++ b/drivers/hv/ring_buffer.c
> @@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
>  	mutex_unlock(&ring_info->ring_buffer_mutex);
>  
>  	kfree(ring_info->pkt_buffer);
> +	ring_info->pkt_buffer = NULL;
>  	ring_info->pkt_buffer_size = 0;
>  }
>  
> -- 
> 2.31.1
> 

^ permalink raw reply	[flat|nested] 6+ messages in thread

* RE: [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver
  2021-08-31 14:39 [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver Vitaly Kuznetsov
  2021-09-01 11:25 ` Wei Liu
  2021-09-01 15:25 ` Andrea Parri
@ 2021-09-02 17:09 ` Michael Kelley
  2021-09-03 11:00 ` Wei Liu
  3 siblings, 0 replies; 6+ messages in thread
From: Michael Kelley @ 2021-09-02 17:09 UTC (permalink / raw)
  To: vkuznets, linux-hyperv
  Cc: Andres Beltran, Andrea Parri (Microsoft),
	Dexuan Cui, Wei Liu, Stephen Hemminger, Haiyang Zhang,
	KY Srinivasan, linux-kernel

From: Vitaly Kuznetsov <vkuznets@redhat.com> Sent: Tuesday, August 31, 2021 7:39 AM
> 
> The following crash happens when a never-used device is unbound from
> uio_hv_generic driver:
> 
>  kernel BUG at mm/slub.c:321!
>  invalid opcode: 0000 [#1] SMP PTI
>  CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G               X --------- ---  5.14.0-0.rc2.23.el9.x86_64 #1
>  Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
>  RIP: 0010:__slab_free+0x1d5/0x3d0
> ...
>  Call Trace:
>   ? pick_next_task_fair+0x18e/0x3b0
>   ? __cond_resched+0x16/0x40
>   ? vunmap_pmd_range.isra.0+0x154/0x1c0
>   ? __vunmap+0x22d/0x290
>   ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
>   kfree+0x331/0x380
>   ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
>   hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
>   vmbus_free_ring+0x21/0x60 [hv_vmbus]
>   hv_uio_remove+0x4f/0x60 [uio_hv_generic]
>   vmbus_remove+0x23/0x30 [hv_vmbus]
>   __device_release_driver+0x17a/0x230
>   device_driver_detach+0x3c/0xa0
>   unbind_store+0x113/0x130
> ...
> 
> The problem appears to be that we free 'ring_info->pkt_buffer' twice:
> first, when the device is unbound from in-kernel driver (netvsc in this
> case) and second from hv_uio_remove(). Normally, ring buffer is supposed
> to be re-initialized from hv_uio_open() but this happens when UIO device
> is being opened and this is not guaranteed to happen.
> 
> Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same
> channel (which is being handed over between in-kernel drivers and UIO) even
> if we didn't call hv_ringbuffer_init() in between. We, however, need to
> avoid kfree() call for an already freed pointer.
> 
> Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  drivers/hv/ring_buffer.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
> index 2aee356840a2..314015d9e912 100644
> --- a/drivers/hv/ring_buffer.c
> +++ b/drivers/hv/ring_buffer.c
> @@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
>  	mutex_unlock(&ring_info->ring_buffer_mutex);
> 
>  	kfree(ring_info->pkt_buffer);
> +	ring_info->pkt_buffer = NULL;
>  	ring_info->pkt_buffer_size = 0;
>  }
> 
> --
> 2.31.1

Reviewed-by: Michael Kelley <mikelley@microsoft.com>


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver
  2021-08-31 14:39 [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver Vitaly Kuznetsov
                   ` (2 preceding siblings ...)
  2021-09-02 17:09 ` Michael Kelley
@ 2021-09-03 11:00 ` Wei Liu
  3 siblings, 0 replies; 6+ messages in thread
From: Wei Liu @ 2021-09-03 11:00 UTC (permalink / raw)
  To: Vitaly Kuznetsov
  Cc: linux-hyperv, Andres Beltran, Michael Kelley,
	Andrea Parri (Microsoft),
	Dexuan Cui, Wei Liu, Stephen Hemminger, Haiyang Zhang,
	K. Y. Srinivasan, linux-kernel

On Tue, Aug 31, 2021 at 04:39:16PM +0200, Vitaly Kuznetsov wrote:
> The following crash happens when a never-used device is unbound from
> uio_hv_generic driver:
> 
[...]
> Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>

Applied to hyperv-fixes. Thanks.

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2021-09-03 11:00 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-31 14:39 [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver Vitaly Kuznetsov
2021-09-01 11:25 ` Wei Liu
2021-09-01 11:40   ` Vitaly Kuznetsov
2021-09-01 15:25 ` Andrea Parri
2021-09-02 17:09 ` Michael Kelley
2021-09-03 11:00 ` Wei Liu

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).