Re: [PATCH 2/3] virito_pci: add timeout to reset device operation

From: Max Gurtovoy <mgurtovoy@nvidia.com>
To: "Michael S. Tsirkin" <mst@redhat.com>
Cc: <kvm@vger.kernel.org>,
	<virtualization@lists.linux-foundation.org>,
	<jasowang@redhat.com>, <oren@nvidia.com>, <nitzanc@nvidia.com>
Subject: Re: [PATCH 2/3] virito_pci: add timeout to reset device operation
Date: Wed, 7 Apr 2021 17:06:22 +0300	[thread overview]
Message-ID: <02cc0471-af5f-8cae-51a5-855bbe90021b@nvidia.com> (raw)
In-Reply-To: <20210407094228-mutt-send-email-mst@kernel.org>

On 4/7/2021 4:45 PM, Michael S. Tsirkin wrote:
> On Wed, Apr 07, 2021 at 12:09:23PM +0000, Max Gurtovoy wrote:
>> According to the spec after writing 0 to device_status, the driver MUST
>> wait for a read of device_status to return 0 before reinitializing the
>> device. In case we have a device that won't return 0, the reset
>> operation will loop forever and cause the host/vm to stuck. Set timeout
>> for 3 minutes before giving up on the device.
>>
>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>> ---
>>   drivers/virtio/virtio_pci_modern.c | 10 +++++++++-
>>   1 file changed, 9 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c
>> index cc3412a96a17..dcee616e8d21 100644
>> --- a/drivers/virtio/virtio_pci_modern.c
>> +++ b/drivers/virtio/virtio_pci_modern.c
>> @@ -162,6 +162,7 @@ static int vp_reset(struct virtio_device *vdev)
>>   {
>>   	struct virtio_pci_device *vp_dev = to_vp_device(vdev);
>>   	struct virtio_pci_modern_device *mdev = &vp_dev->mdev;
>> +	unsigned long timeout = jiffies + msecs_to_jiffies(180000);
>>   
>>   	/* 0 status means a reset. */
>>   	vp_modern_set_status(mdev, 0);
>> @@ -169,9 +170,16 @@ static int vp_reset(struct virtio_device *vdev)
>>   	 * device_status to return 0 before reinitializing the device.
>>   	 * This will flush out the status write, and flush in device writes,
>>   	 * including MSI-X interrupts, if any.
>> +	 * Set a timeout before giving up on the device.
>>   	 */
>> -	while (vp_modern_get_status(mdev))
>> +	while (vp_modern_get_status(mdev)) {
>> +		if (time_after(jiffies, timeout)) {
>> +			dev_err(&vdev->dev, "virtio: device not ready. "
>> +				"Aborting. Try again later\n");
>> +			return -EAGAIN;
>> +		}
>>   		msleep(1);
>> +	}
>>   	/* Flush pending VQ/configuration callbacks. */
>>   	vp_synchronize_vectors(vdev);
>>   	return 0;
> Problem is everyone just ignores the return code from reset.
> Timing out like that has a chance to cause a lot of trouble
> if the device remains active - we need to make reset robust.

But in commit 1/3 I added a code that doesn't ignore the reset return code.

>
> What exactly is going on with the device that
> get status never returns 0? E.g. maybe it's in a state
> where it's returning all 1's because it's wedged permanently -
> using that would be better...

In HW devices you might have situations that the controller is in bad 
state (maybe bad FW) but still can be seen under the PCI bus.

As long as the device is not returning 0, this is legal. But in today's 
code, it will cause the kernel to be in endless while loop because of 
one bad device (that might recover later).

If we have 10 devices, and the first will stuck, all the others will 
wait forever to be probed.

By Virtio spec, setting FAILED is allowed in case "..driver didn’t like 
the device for some reason, or even a fatal error during device operation."

For example, in the NVMe spec there is TO (timeout) register that "is 
the worst case time that host software shall wait for CSTS.RDY to 
transition from: ..." and the driver wait for this time until it 
understands that the device is not ready to operate.

I tried to add similar logic to virtio.

>
>
>
>> -- 
>> 2.25.4