* [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver
@ 2021-08-31 14:39 Vitaly Kuznetsov
2021-09-01 11:25 ` Wei Liu
` (3 more replies)
0 siblings, 4 replies; 6+ messages in thread
From: Vitaly Kuznetsov @ 2021-08-31 14:39 UTC (permalink / raw)
To: linux-hyperv
Cc: Andres Beltran, Michael Kelley, Andrea Parri (Microsoft),
Dexuan Cui, Wei Liu, Stephen Hemminger, Haiyang Zhang,
K. Y. Srinivasan, linux-kernel
The following crash happens when a never-used device is unbound from
uio_hv_generic driver:
kernel BUG at mm/slub.c:321!
invalid opcode: 0000 [#1] SMP PTI
CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G X --------- --- 5.14.0-0.rc2.23.el9.x86_64 #1
Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
RIP: 0010:__slab_free+0x1d5/0x3d0
...
Call Trace:
? pick_next_task_fair+0x18e/0x3b0
? __cond_resched+0x16/0x40
? vunmap_pmd_range.isra.0+0x154/0x1c0
? __vunmap+0x22d/0x290
? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
kfree+0x331/0x380
? hv_uio_remove+0x43/0x60 [uio_hv_generic]
hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
vmbus_free_ring+0x21/0x60 [hv_vmbus]
hv_uio_remove+0x4f/0x60 [uio_hv_generic]
vmbus_remove+0x23/0x30 [hv_vmbus]
__device_release_driver+0x17a/0x230
device_driver_detach+0x3c/0xa0
unbind_store+0x113/0x130
...
The problem appears to be that we free 'ring_info->pkt_buffer' twice:
first, when the device is unbound from in-kernel driver (netvsc in this
case) and second from hv_uio_remove(). Normally, ring buffer is supposed
to be re-initialized from hv_uio_open() but this happens when UIO device
is being opened and this is not guaranteed to happen.
Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same
channel (which is being handed over between in-kernel drivers and UIO) even
if we didn't call hv_ringbuffer_init() in between. We, however, need to
avoid kfree() call for an already freed pointer.
Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
drivers/hv/ring_buffer.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
index 2aee356840a2..314015d9e912 100644
--- a/drivers/hv/ring_buffer.c
+++ b/drivers/hv/ring_buffer.c
@@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
mutex_unlock(&ring_info->ring_buffer_mutex);
kfree(ring_info->pkt_buffer);
+ ring_info->pkt_buffer = NULL;
ring_info->pkt_buffer_size = 0;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver
2021-08-31 14:39 [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver Vitaly Kuznetsov
@ 2021-09-01 11:25 ` Wei Liu
2021-09-01 11:40 ` Vitaly Kuznetsov
2021-09-01 15:25 ` Andrea Parri
` (2 subsequent siblings)
3 siblings, 1 reply; 6+ messages in thread
From: Wei Liu @ 2021-09-01 11:25 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: linux-hyperv, Andres Beltran, Michael Kelley,
Andrea Parri (Microsoft),
Dexuan Cui, Wei Liu, Stephen Hemminger, Haiyang Zhang,
K. Y. Srinivasan, linux-kernel
On Tue, Aug 31, 2021 at 04:39:16PM +0200, Vitaly Kuznetsov wrote:
> The following crash happens when a never-used device is unbound from
> uio_hv_generic driver:
>
> kernel BUG at mm/slub.c:321!
> invalid opcode: 0000 [#1] SMP PTI
> CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G X --------- --- 5.14.0-0.rc2.23.el9.x86_64 #1
> Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
> RIP: 0010:__slab_free+0x1d5/0x3d0
> ...
> Call Trace:
> ? pick_next_task_fair+0x18e/0x3b0
> ? __cond_resched+0x16/0x40
> ? vunmap_pmd_range.isra.0+0x154/0x1c0
> ? __vunmap+0x22d/0x290
> ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
> kfree+0x331/0x380
> ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
> hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
> vmbus_free_ring+0x21/0x60 [hv_vmbus]
> hv_uio_remove+0x4f/0x60 [uio_hv_generic]
> vmbus_remove+0x23/0x30 [hv_vmbus]
> __device_release_driver+0x17a/0x230
> device_driver_detach+0x3c/0xa0
> unbind_store+0x113/0x130
> ...
>
> The problem appears to be that we free 'ring_info->pkt_buffer' twice:
> first, when the device is unbound from in-kernel driver (netvsc in this
> case) and second from hv_uio_remove(). Normally, ring buffer is supposed
> to be re-initialized from hv_uio_open() but this happens when UIO device
> is being opened and this is not guaranteed to happen.
>
> Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same
> channel (which is being handed over between in-kernel drivers and UIO) even
> if we didn't call hv_ringbuffer_init() in between. We, however, need to
> avoid kfree() call for an already freed pointer.
>
> Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
> drivers/hv/ring_buffer.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
> index 2aee356840a2..314015d9e912 100644
> --- a/drivers/hv/ring_buffer.c
> +++ b/drivers/hv/ring_buffer.c
> @@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
> mutex_unlock(&ring_info->ring_buffer_mutex);
>
> kfree(ring_info->pkt_buffer);
> + ring_info->pkt_buffer = NULL;
This certainly won't hurt.
I would however like to wait till Andrea and Michael go over the
reasoning of this patch before doing anything.
Wei.
> ring_info->pkt_buffer_size = 0;
> }
>
> --
> 2.31.1
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver
2021-09-01 11:25 ` Wei Liu
@ 2021-09-01 11:40 ` Vitaly Kuznetsov
0 siblings, 0 replies; 6+ messages in thread
From: Vitaly Kuznetsov @ 2021-09-01 11:40 UTC (permalink / raw)
To: Wei Liu
Cc: linux-hyperv, Andres Beltran, Michael Kelley,
Andrea Parri (Microsoft),
Dexuan Cui, Wei Liu, Stephen Hemminger, Haiyang Zhang,
K. Y. Srinivasan, linux-kernel
Wei Liu <wei.liu@kernel.org> writes:
> On Tue, Aug 31, 2021 at 04:39:16PM +0200, Vitaly Kuznetsov wrote:
>> The following crash happens when a never-used device is unbound from
>> uio_hv_generic driver:
>>
>> kernel BUG at mm/slub.c:321!
>> invalid opcode: 0000 [#1] SMP PTI
>> CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G X --------- --- 5.14.0-0.rc2.23.el9.x86_64 #1
>> Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
>> RIP: 0010:__slab_free+0x1d5/0x3d0
>> ...
>> Call Trace:
>> ? pick_next_task_fair+0x18e/0x3b0
>> ? __cond_resched+0x16/0x40
>> ? vunmap_pmd_range.isra.0+0x154/0x1c0
>> ? __vunmap+0x22d/0x290
>> ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
>> kfree+0x331/0x380
>> ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
>> hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
>> vmbus_free_ring+0x21/0x60 [hv_vmbus]
>> hv_uio_remove+0x4f/0x60 [uio_hv_generic]
>> vmbus_remove+0x23/0x30 [hv_vmbus]
>> __device_release_driver+0x17a/0x230
>> device_driver_detach+0x3c/0xa0
>> unbind_store+0x113/0x130
>> ...
>>
>> The problem appears to be that we free 'ring_info->pkt_buffer' twice:
>> first, when the device is unbound from in-kernel driver (netvsc in this
>> case) and second from hv_uio_remove(). Normally, ring buffer is supposed
>> to be re-initialized from hv_uio_open() but this happens when UIO device
>> is being opened and this is not guaranteed to happen.
>>
>> Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same
>> channel (which is being handed over between in-kernel drivers and UIO) even
>> if we didn't call hv_ringbuffer_init() in between. We, however, need to
>> avoid kfree() call for an already freed pointer.
>>
>> Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> ---
>> drivers/hv/ring_buffer.c | 1 +
>> 1 file changed, 1 insertion(+)
>>
>> diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
>> index 2aee356840a2..314015d9e912 100644
>> --- a/drivers/hv/ring_buffer.c
>> +++ b/drivers/hv/ring_buffer.c
>> @@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
>> mutex_unlock(&ring_info->ring_buffer_mutex);
>>
>> kfree(ring_info->pkt_buffer);
>> + ring_info->pkt_buffer = NULL;
>
> This certainly won't hurt.
>
> I would however like to wait till Andrea and Michael go over the
> reasoning of this patch before doing anything.
>
Thanks,
the counter-intuitive thing here is that the channel sometimes continues
to live after hv_ringbuffer_cleanup(): when we unbind a device from
in-kernel driver and give it to uio_hv_generic the channel is handed
over from one driver to another.
> Wei.
>
>> ring_info->pkt_buffer_size = 0;
>> }
>>
>> --
>> 2.31.1
>>
>
--
Vitaly
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver
2021-08-31 14:39 [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver Vitaly Kuznetsov
2021-09-01 11:25 ` Wei Liu
@ 2021-09-01 15:25 ` Andrea Parri
2021-09-02 17:09 ` Michael Kelley
2021-09-03 11:00 ` Wei Liu
3 siblings, 0 replies; 6+ messages in thread
From: Andrea Parri @ 2021-09-01 15:25 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: linux-hyperv, Andres Beltran, Michael Kelley, Dexuan Cui,
Wei Liu, Stephen Hemminger, Haiyang Zhang, K. Y. Srinivasan,
linux-kernel
On Tue, Aug 31, 2021 at 04:39:16PM +0200, Vitaly Kuznetsov wrote:
> The following crash happens when a never-used device is unbound from
> uio_hv_generic driver:
>
> kernel BUG at mm/slub.c:321!
> invalid opcode: 0000 [#1] SMP PTI
> CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G X --------- --- 5.14.0-0.rc2.23.el9.x86_64 #1
> Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
> RIP: 0010:__slab_free+0x1d5/0x3d0
> ...
> Call Trace:
> ? pick_next_task_fair+0x18e/0x3b0
> ? __cond_resched+0x16/0x40
> ? vunmap_pmd_range.isra.0+0x154/0x1c0
> ? __vunmap+0x22d/0x290
> ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
> kfree+0x331/0x380
> ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
> hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
> vmbus_free_ring+0x21/0x60 [hv_vmbus]
> hv_uio_remove+0x4f/0x60 [uio_hv_generic]
> vmbus_remove+0x23/0x30 [hv_vmbus]
> __device_release_driver+0x17a/0x230
> device_driver_detach+0x3c/0xa0
> unbind_store+0x113/0x130
> ...
>
> The problem appears to be that we free 'ring_info->pkt_buffer' twice:
> first, when the device is unbound from in-kernel driver (netvsc in this
> case) and second from hv_uio_remove(). Normally, ring buffer is supposed
> to be re-initialized from hv_uio_open() but this happens when UIO device
> is being opened and this is not guaranteed to happen.
>
> Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same
> channel (which is being handed over between in-kernel drivers and UIO) even
> if we didn't call hv_ringbuffer_init() in between. We, however, need to
> avoid kfree() call for an already freed pointer.
>
> Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
LGTM.
Reviewed-by: Andrea Parri <parri.andrea@gmail.com>
ae6935ed7d424f appears to have anticipated this problem on ->ring_buffer
and adopted the solution proposed here by Vitaly.
Andrea
> ---
> drivers/hv/ring_buffer.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
> index 2aee356840a2..314015d9e912 100644
> --- a/drivers/hv/ring_buffer.c
> +++ b/drivers/hv/ring_buffer.c
> @@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
> mutex_unlock(&ring_info->ring_buffer_mutex);
>
> kfree(ring_info->pkt_buffer);
> + ring_info->pkt_buffer = NULL;
> ring_info->pkt_buffer_size = 0;
> }
>
> --
> 2.31.1
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver
2021-08-31 14:39 [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver Vitaly Kuznetsov
2021-09-01 11:25 ` Wei Liu
2021-09-01 15:25 ` Andrea Parri
@ 2021-09-02 17:09 ` Michael Kelley
2021-09-03 11:00 ` Wei Liu
3 siblings, 0 replies; 6+ messages in thread
From: Michael Kelley @ 2021-09-02 17:09 UTC (permalink / raw)
To: vkuznets, linux-hyperv
Cc: Andres Beltran, Andrea Parri (Microsoft),
Dexuan Cui, Wei Liu, Stephen Hemminger, Haiyang Zhang,
KY Srinivasan, linux-kernel
From: Vitaly Kuznetsov <vkuznets@redhat.com> Sent: Tuesday, August 31, 2021 7:39 AM
>
> The following crash happens when a never-used device is unbound from
> uio_hv_generic driver:
>
> kernel BUG at mm/slub.c:321!
> invalid opcode: 0000 [#1] SMP PTI
> CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G X --------- --- 5.14.0-0.rc2.23.el9.x86_64 #1
> Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
> RIP: 0010:__slab_free+0x1d5/0x3d0
> ...
> Call Trace:
> ? pick_next_task_fair+0x18e/0x3b0
> ? __cond_resched+0x16/0x40
> ? vunmap_pmd_range.isra.0+0x154/0x1c0
> ? __vunmap+0x22d/0x290
> ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
> kfree+0x331/0x380
> ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
> hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
> vmbus_free_ring+0x21/0x60 [hv_vmbus]
> hv_uio_remove+0x4f/0x60 [uio_hv_generic]
> vmbus_remove+0x23/0x30 [hv_vmbus]
> __device_release_driver+0x17a/0x230
> device_driver_detach+0x3c/0xa0
> unbind_store+0x113/0x130
> ...
>
> The problem appears to be that we free 'ring_info->pkt_buffer' twice:
> first, when the device is unbound from in-kernel driver (netvsc in this
> case) and second from hv_uio_remove(). Normally, ring buffer is supposed
> to be re-initialized from hv_uio_open() but this happens when UIO device
> is being opened and this is not guaranteed to happen.
>
> Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same
> channel (which is being handed over between in-kernel drivers and UIO) even
> if we didn't call hv_ringbuffer_init() in between. We, however, need to
> avoid kfree() call for an already freed pointer.
>
> Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
> drivers/hv/ring_buffer.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
> index 2aee356840a2..314015d9e912 100644
> --- a/drivers/hv/ring_buffer.c
> +++ b/drivers/hv/ring_buffer.c
> @@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
> mutex_unlock(&ring_info->ring_buffer_mutex);
>
> kfree(ring_info->pkt_buffer);
> + ring_info->pkt_buffer = NULL;
> ring_info->pkt_buffer_size = 0;
> }
>
> --
> 2.31.1
Reviewed-by: Michael Kelley <mikelley@microsoft.com>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver
2021-08-31 14:39 [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver Vitaly Kuznetsov
` (2 preceding siblings ...)
2021-09-02 17:09 ` Michael Kelley
@ 2021-09-03 11:00 ` Wei Liu
3 siblings, 0 replies; 6+ messages in thread
From: Wei Liu @ 2021-09-03 11:00 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: linux-hyperv, Andres Beltran, Michael Kelley,
Andrea Parri (Microsoft),
Dexuan Cui, Wei Liu, Stephen Hemminger, Haiyang Zhang,
K. Y. Srinivasan, linux-kernel
On Tue, Aug 31, 2021 at 04:39:16PM +0200, Vitaly Kuznetsov wrote:
> The following crash happens when a never-used device is unbound from
> uio_hv_generic driver:
>
[...]
> Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Applied to hyperv-fixes. Thanks.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-09-03 11:00 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-31 14:39 [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver Vitaly Kuznetsov
2021-09-01 11:25 ` Wei Liu
2021-09-01 11:40 ` Vitaly Kuznetsov
2021-09-01 15:25 ` Andrea Parri
2021-09-02 17:09 ` Michael Kelley
2021-09-03 11:00 ` Wei Liu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).