Re: [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver

From: Vitaly Kuznetsov <vkuznets@redhat.com>
To: Wei Liu <wei.liu@kernel.org>
Cc: linux-hyperv@vger.kernel.org,
	Andres Beltran <lkmlabelt@gmail.com>,
	Michael Kelley <mikelley@microsoft.com>,
	"Andrea Parri (Microsoft)" <parri.andrea@gmail.com>,
	Dexuan Cui <decui@microsoft.com>, Wei Liu <wei.liu@kernel.org>,
	Stephen Hemminger <sthemmin@microsoft.com>,
	Haiyang Zhang <haiyangz@microsoft.com>,
	"K. Y. Srinivasan" <kys@microsoft.com>,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH] Drivers: hv: vmbus: Fix kernel crash upon unbinding a device from uio_hv_generic driver
Date: Wed, 01 Sep 2021 13:40:09 +0200	[thread overview]
Message-ID: <878s0go76u.fsf@vitty.brq.redhat.com> (raw)
In-Reply-To: <20210901112500.7q4oxjtesiuniop3@liuwe-devbox-debian-v2>

Wei Liu <wei.liu@kernel.org> writes:

> On Tue, Aug 31, 2021 at 04:39:16PM +0200, Vitaly Kuznetsov wrote:
>> The following crash happens when a never-used device is unbound from
>> uio_hv_generic driver:
>> 
>>  kernel BUG at mm/slub.c:321!
>>  invalid opcode: 0000 [#1] SMP PTI
>>  CPU: 0 PID: 4001 Comm: bash Kdump: loaded Tainted: G               X --------- ---  5.14.0-0.rc2.23.el9.x86_64 #1
>>  Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
>>  RIP: 0010:__slab_free+0x1d5/0x3d0
>> ...
>>  Call Trace:
>>   ? pick_next_task_fair+0x18e/0x3b0
>>   ? __cond_resched+0x16/0x40
>>   ? vunmap_pmd_range.isra.0+0x154/0x1c0
>>   ? __vunmap+0x22d/0x290
>>   ? hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
>>   kfree+0x331/0x380
>>   ? hv_uio_remove+0x43/0x60 [uio_hv_generic]
>>   hv_ringbuffer_cleanup+0x36/0x40 [hv_vmbus]
>>   vmbus_free_ring+0x21/0x60 [hv_vmbus]
>>   hv_uio_remove+0x4f/0x60 [uio_hv_generic]
>>   vmbus_remove+0x23/0x30 [hv_vmbus]
>>   __device_release_driver+0x17a/0x230
>>   device_driver_detach+0x3c/0xa0
>>   unbind_store+0x113/0x130
>> ...
>> 
>> The problem appears to be that we free 'ring_info->pkt_buffer' twice:
>> first, when the device is unbound from in-kernel driver (netvsc in this
>> case) and second from hv_uio_remove(). Normally, ring buffer is supposed
>> to be re-initialized from hv_uio_open() but this happens when UIO device
>> is being opened and this is not guaranteed to happen.
>> 
>> Generally, it is OK to call hv_ringbuffer_cleanup() twice for the same
>> channel (which is being handed over between in-kernel drivers and UIO) even
>> if we didn't call hv_ringbuffer_init() in between. We, however, need to
>> avoid kfree() call for an already freed pointer.
>> 
>> Fixes: adae1e931acd ("Drivers: hv: vmbus: Copy packets sent by Hyper-V out of the ring buffer")
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> ---
>>  drivers/hv/ring_buffer.c | 1 +
>>  1 file changed, 1 insertion(+)
>> 
>> diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
>> index 2aee356840a2..314015d9e912 100644
>> --- a/drivers/hv/ring_buffer.c
>> +++ b/drivers/hv/ring_buffer.c
>> @@ -245,6 +245,7 @@ void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info)
>>  	mutex_unlock(&ring_info->ring_buffer_mutex);
>>  
>>  	kfree(ring_info->pkt_buffer);
>> +	ring_info->pkt_buffer = NULL;
>
> This certainly won't hurt.
>
> I would however like to wait till Andrea and Michael go over the
> reasoning of this patch before doing anything.
>

Thanks,

the counter-intuitive thing here is that the channel sometimes continues
to live after hv_ringbuffer_cleanup(): when we unbind a device from
in-kernel driver and give it to uio_hv_generic the channel is handed
over from one driver to another.

> Wei.
>
>>  	ring_info->pkt_buffer_size = 0;
>>  }
>>  
>> -- 
>> 2.31.1
>> 
>

-- 
Vitaly