All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Michael S. Tsirkin" <mst@redhat.com>
To: Max Gurtovoy <mgurtovoy@nvidia.com>
Cc: kvm@vger.kernel.org, virtualization@lists.linux-foundation.org,
	jasowang@redhat.com, oren@nvidia.com, nitzanc@nvidia.com
Subject: Re: [PATCH 2/3] virito_pci: add timeout to reset device operation
Date: Wed, 7 Apr 2021 09:45:16 -0400	[thread overview]
Message-ID: <20210407094228-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <20210407120924.133294-2-mgurtovoy@nvidia.com>

On Wed, Apr 07, 2021 at 12:09:23PM +0000, Max Gurtovoy wrote:
> According to the spec after writing 0 to device_status, the driver MUST
> wait for a read of device_status to return 0 before reinitializing the
> device. In case we have a device that won't return 0, the reset
> operation will loop forever and cause the host/vm to stuck. Set timeout
> for 3 minutes before giving up on the device.
> 
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> ---
>  drivers/virtio/virtio_pci_modern.c | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c
> index cc3412a96a17..dcee616e8d21 100644
> --- a/drivers/virtio/virtio_pci_modern.c
> +++ b/drivers/virtio/virtio_pci_modern.c
> @@ -162,6 +162,7 @@ static int vp_reset(struct virtio_device *vdev)
>  {
>  	struct virtio_pci_device *vp_dev = to_vp_device(vdev);
>  	struct virtio_pci_modern_device *mdev = &vp_dev->mdev;
> +	unsigned long timeout = jiffies + msecs_to_jiffies(180000);
>  
>  	/* 0 status means a reset. */
>  	vp_modern_set_status(mdev, 0);
> @@ -169,9 +170,16 @@ static int vp_reset(struct virtio_device *vdev)
>  	 * device_status to return 0 before reinitializing the device.
>  	 * This will flush out the status write, and flush in device writes,
>  	 * including MSI-X interrupts, if any.
> +	 * Set a timeout before giving up on the device.
>  	 */
> -	while (vp_modern_get_status(mdev))
> +	while (vp_modern_get_status(mdev)) {
> +		if (time_after(jiffies, timeout)) {
> +			dev_err(&vdev->dev, "virtio: device not ready. "
> +				"Aborting. Try again later\n");
> +			return -EAGAIN;
> +		}
>  		msleep(1);
> +	}
>  	/* Flush pending VQ/configuration callbacks. */
>  	vp_synchronize_vectors(vdev);
>  	return 0;

Problem is everyone just ignores the return code from reset.
Timing out like that has a chance to cause a lot of trouble
if the device remains active - we need to make reset robust.

What exactly is going on with the device that
get status never returns 0? E.g. maybe it's in a state
where it's returning all 1's because it's wedged permanently -
using that would be better...



> -- 
> 2.25.4


WARNING: multiple messages have this Message-ID (diff)
From: "Michael S. Tsirkin" <mst@redhat.com>
To: Max Gurtovoy <mgurtovoy@nvidia.com>
Cc: nitzanc@nvidia.com, oren@nvidia.com, kvm@vger.kernel.org,
	virtualization@lists.linux-foundation.org
Subject: Re: [PATCH 2/3] virito_pci: add timeout to reset device operation
Date: Wed, 7 Apr 2021 09:45:16 -0400	[thread overview]
Message-ID: <20210407094228-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <20210407120924.133294-2-mgurtovoy@nvidia.com>

On Wed, Apr 07, 2021 at 12:09:23PM +0000, Max Gurtovoy wrote:
> According to the spec after writing 0 to device_status, the driver MUST
> wait for a read of device_status to return 0 before reinitializing the
> device. In case we have a device that won't return 0, the reset
> operation will loop forever and cause the host/vm to stuck. Set timeout
> for 3 minutes before giving up on the device.
> 
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> ---
>  drivers/virtio/virtio_pci_modern.c | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c
> index cc3412a96a17..dcee616e8d21 100644
> --- a/drivers/virtio/virtio_pci_modern.c
> +++ b/drivers/virtio/virtio_pci_modern.c
> @@ -162,6 +162,7 @@ static int vp_reset(struct virtio_device *vdev)
>  {
>  	struct virtio_pci_device *vp_dev = to_vp_device(vdev);
>  	struct virtio_pci_modern_device *mdev = &vp_dev->mdev;
> +	unsigned long timeout = jiffies + msecs_to_jiffies(180000);
>  
>  	/* 0 status means a reset. */
>  	vp_modern_set_status(mdev, 0);
> @@ -169,9 +170,16 @@ static int vp_reset(struct virtio_device *vdev)
>  	 * device_status to return 0 before reinitializing the device.
>  	 * This will flush out the status write, and flush in device writes,
>  	 * including MSI-X interrupts, if any.
> +	 * Set a timeout before giving up on the device.
>  	 */
> -	while (vp_modern_get_status(mdev))
> +	while (vp_modern_get_status(mdev)) {
> +		if (time_after(jiffies, timeout)) {
> +			dev_err(&vdev->dev, "virtio: device not ready. "
> +				"Aborting. Try again later\n");
> +			return -EAGAIN;
> +		}
>  		msleep(1);
> +	}
>  	/* Flush pending VQ/configuration callbacks. */
>  	vp_synchronize_vectors(vdev);
>  	return 0;

Problem is everyone just ignores the return code from reset.
Timing out like that has a chance to cause a lot of trouble
if the device remains active - we need to make reset robust.

What exactly is going on with the device that
get status never returns 0? E.g. maybe it's in a state
where it's returning all 1's because it's wedged permanently -
using that would be better...



> -- 
> 2.25.4

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

  reply	other threads:[~2021-04-07 13:45 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-07 12:09 [PATCH 1/3] virtio: update reset callback to return status Max Gurtovoy
2021-04-07 12:09 ` [PATCH 2/3] virito_pci: add timeout to reset device operation Max Gurtovoy
2021-04-07 13:45   ` Michael S. Tsirkin [this message]
2021-04-07 13:45     ` Michael S. Tsirkin
2021-04-07 14:06     ` Max Gurtovoy
2021-04-07 12:09 ` [PATCH 3/3] virtio_pci: add module param for reset_timeout Max Gurtovoy
2021-04-07 13:44 ` [PATCH 1/3] virtio: update reset callback to return status Cornelia Huck
2021-04-07 13:44   ` Cornelia Huck
2021-04-07 14:35   ` Max Gurtovoy
2021-04-07 17:51 ` kernel test robot
2021-04-07 17:51   ` kernel test robot
2021-04-07 17:51   ` kernel test robot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210407094228-mutt-send-email-mst@kernel.org \
    --to=mst@redhat.com \
    --cc=jasowang@redhat.com \
    --cc=kvm@vger.kernel.org \
    --cc=mgurtovoy@nvidia.com \
    --cc=nitzanc@nvidia.com \
    --cc=oren@nvidia.com \
    --cc=virtualization@lists.linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.