Re: [PATCH v4 0/4] Implement vdpasim stop operation

From: Jason Wang <jasowang@redhat.com>
To: Parav Pandit <parav@nvidia.com>
Cc: "Eugenio Pérez" <eperezma@redhat.com>,
	"Michael S. Tsirkin" <mst@redhat.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"virtualization@lists.linux-foundation.org"
	<virtualization@lists.linux-foundation.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"martinh@xilinx.com" <martinh@xilinx.com>,
	"Stefano Garzarella" <sgarzare@redhat.com>,
	"martinpo@xilinx.com" <martinpo@xilinx.com>,
	"lvivier@redhat.com" <lvivier@redhat.com>,
	"pabloc@xilinx.com" <pabloc@xilinx.com>,
	"Eli Cohen" <elic@nvidia.com>,
	"Dan Carpenter" <dan.carpenter@oracle.com>,
	"Xie Yongji" <xieyongji@bytedance.com>,
	"Christophe JAILLET" <christophe.jaillet@wanadoo.fr>,
	"Zhang Min" <zhang.min9@zte.com.cn>,
	"Wu Zongyong" <wuzongyong@linux.alibaba.com>,
	"lulu@redhat.com" <lulu@redhat.com>,
	"Zhu Lingshan" <lingshan.zhu@intel.com>,
	"Piotr.Uminski@intel.com" <Piotr.Uminski@intel.com>,
	"Si-Wei Liu" <si-wei.liu@oracle.com>,
	"ecree.xilinx@gmail.com" <ecree.xilinx@gmail.com>,
	"gautam.dawar@amd.com" <gautam.dawar@amd.com>,
	"habetsm.xilinx@gmail.com" <habetsm.xilinx@gmail.com>,
	"tanuj.kamde@amd.com" <tanuj.kamde@amd.com>,
	"hanand@xilinx.com" <hanand@xilinx.com>,
	"dinang@xilinx.com" <dinang@xilinx.com>,
	Longpeng <longpeng2@huawei.com>
Subject: Re: [PATCH v4 0/4] Implement vdpasim stop operation
Date: Fri, 27 May 2022 10:26:35 +0800	[thread overview]
Message-ID: <CACGkMEu1YenjBHAssP=FvKX6WxDQ5Aa50r-BsnkfR4zqNTk6hg@mail.gmail.com> (raw)
In-Reply-To: <PH0PR12MB54819C6C6DAF6572AEADC1AEDCD99@PH0PR12MB5481.namprd12.prod.outlook.com>

On Thu, May 26, 2022 at 8:54 PM Parav Pandit <parav@nvidia.com> wrote:
>
>
>
> > From: Eugenio Pérez <eperezma@redhat.com>
> > Sent: Thursday, May 26, 2022 8:44 AM
>
> > Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer
> >
> > that backend feature and userspace can effectively stop the device.
> >
> >
> >
> > This is a must before get virtqueue indexes (base) for live migration,
> >
> > since the device could modify them after userland gets them. There are
> >
> > individual ways to perform that action for some devices
> >
> > (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there
> > was no
> >
> > way to perform it for any vhost device (and, in particular, vhost-vdpa).
> >
> >
> >
> > After the return of ioctl with stop != 0, the device MUST finish any
> >
> > pending operations like in flight requests. It must also preserve all
> >
> > the necessary state (the virtqueue vring base plus the possible device
> >
> > specific states) that is required for restoring in the future. The
> >
> > device must not change its configuration after that point.
> >
> >
> >
> > After the return of ioctl with stop == 0, the device can continue
> >
> > processing buffers as long as typical conditions are met (vq is enabled,
> >
> > DRIVER_OK status bit is enabled, etc).
>
> Just to be clear, we are adding vdpa level new ioctl() that doesn’t map to any mechanism in the virtio spec.

We try to provide forward compatibility to VIRTIO_CONFIG_S_STOP. That
means it is expected to implement at least a subset of
VIRTIO_CONFIG_S_STOP.

>
> Why can't we use this ioctl() to indicate driver to start/stop the device instead of driving it through the driver_ok?

So the idea is to add capability that does not exist in the spec. Then
came the stop/resume which can't be done via DRIVER_OK. I think we
should only allow the stop/resume to succeed after DRIVER_OK is set.

> This is in the context of other discussion we had in the LM series.

Do you see any issue that blocks the live migration?

Thanks