[PATCH v6 0/4] vdpa: Add resume operation

[PATCH v6 0/4] vdpa: Add resume operation

[sebastien.boeuf]

From: Sebastien Boeuf <sebastien.boeuf@intel.com>

This series introduces a new operation for vdpa devices. It allows them
to be resumed after they have been suspended. A new feature bit is
introduced for devices to advertise their ability to be resumed after
they have been suspended. This feature bit is different from the one
advertising the ability to be suspended, meaning a device that can be
suspended might not have the ability to be resumed.

Even if it is already possible to restore a device that has been
suspended, which is very convenient for live migrating virtual machines,
there is a major drawback as the device must be fully reset. There is no
way to resume a device that has been suspended without having to
configure the device again and without having to recreate the IOMMU
mappings. This new operation aims at filling this gap by allowing the
device to resume processing the virtqueue descriptors without having to
reset it. This is particularly useful for performing virtual machine
offline migration, also called snapshot/restore, as it allows a virtual
machine to resume to a running state after it was paused and a snapshot
of the entire system was taken.

Sebastien Boeuf (4):
  vdpa: Add resume operation
  vhost-vdpa: Introduce RESUME backend feature bit
  vhost-vdpa: uAPI to resume the device
  vdpa_sim: Implement resume vdpa op

 drivers/vdpa/vdpa_sim/vdpa_sim.c | 29 +++++++++++++++++++++++++++
 drivers/vdpa/vdpa_sim/vdpa_sim.h |  1 +
 drivers/vhost/vdpa.c             | 34 +++++++++++++++++++++++++++++++-
 include/linux/vdpa.h             |  6 +++++-
 include/uapi/linux/vhost.h       |  8 ++++++++
 include/uapi/linux/vhost_types.h |  2 ++
 6 files changed, 78 insertions(+), 2 deletions(-)

-- 
2.37.2

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 5 208 026.16 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH v6 1/4] vdpa: Add resume operation

[sebastien.boeuf]

From: Sebastien Boeuf <sebastien.boeuf@intel.com>

Add a new operation to allow a vDPA device to be resumed after it has
been suspended. Trying to resume a device that wasn't suspended will
result in a no-op.

This operation is optional. If it's not implemented, the associated
backend feature bit will not be exposed. And if the feature bit is not
exposed, invoking this operation will return an error.

Acked-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
---
 include/linux/vdpa.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/include/linux/vdpa.h b/include/linux/vdpa.h
index 6d0f5e4e82c2..96d308cbf97b 100644
--- a/include/linux/vdpa.h
+++ b/include/linux/vdpa.h
@@ -219,7 +219,10 @@ struct vdpa_map_file {
  * @reset:			Reset device
  *				@vdev: vdpa device
  *				Returns integer: success (0) or error (< 0)
- * @suspend:			Suspend or resume the device (optional)
+ * @suspend:			Suspend the device (optional)
+ *				@vdev: vdpa device
+ *				Returns integer: success (0) or error (< 0)
+ * @resume:			Resume the device (optional)
  *				@vdev: vdpa device
  *				Returns integer: success (0) or error (< 0)
  * @get_config_size:		Get the size of the configuration space includes
@@ -324,6 +327,7 @@ struct vdpa_config_ops {
 	void (*set_status)(struct vdpa_device *vdev, u8 status);
 	int (*reset)(struct vdpa_device *vdev);
 	int (*suspend)(struct vdpa_device *vdev);
+	int (*resume)(struct vdpa_device *vdev);
 	size_t (*get_config_size)(struct vdpa_device *vdev);
 	void (*get_config)(struct vdpa_device *vdev, unsigned int offset,
 			   void *buf, unsigned int len);
-- 
2.37.2

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 5 208 026.16 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH v6 2/4] vhost-vdpa: Introduce RESUME backend feature bit

[sebastien.boeuf]

From: Sebastien Boeuf <sebastien.boeuf@intel.com>

Userspace knows if the device can be resumed or not by checking this
feature bit.

It's only exposed if the vdpa driver backend implements the resume()
operation callback. Userspace trying to negotiate this feature when it
hasn't been exposed will result in an error.

Acked-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
---
 drivers/vhost/vdpa.c             | 16 +++++++++++++++-
 include/uapi/linux/vhost_types.h |  2 ++
 2 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
index 166044642fd5..833617d00ef6 100644
--- a/drivers/vhost/vdpa.c
+++ b/drivers/vhost/vdpa.c
@@ -355,6 +355,14 @@ static bool vhost_vdpa_can_suspend(const struct vhost_vdpa *v)
 	return ops->suspend;
 }
 
+static bool vhost_vdpa_can_resume(const struct vhost_vdpa *v)
+{
+	struct vdpa_device *vdpa = v->vdpa;
+	const struct vdpa_config_ops *ops = vdpa->config;
+
+	return ops->resume;
+}
+
 static long vhost_vdpa_get_features(struct vhost_vdpa *v, u64 __user *featurep)
 {
 	struct vdpa_device *vdpa = v->vdpa;
@@ -602,11 +610,15 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
 		if (copy_from_user(&features, featurep, sizeof(features)))
 			return -EFAULT;
 		if (features & ~(VHOST_VDPA_BACKEND_FEATURES |
-				 BIT_ULL(VHOST_BACKEND_F_SUSPEND)))
+				 BIT_ULL(VHOST_BACKEND_F_SUSPEND) |
+				 BIT_ULL(VHOST_BACKEND_F_RESUME)))
 			return -EOPNOTSUPP;
 		if ((features & BIT_ULL(VHOST_BACKEND_F_SUSPEND)) &&
 		     !vhost_vdpa_can_suspend(v))
 			return -EOPNOTSUPP;
+		if ((features & BIT_ULL(VHOST_BACKEND_F_RESUME)) &&
+		     !vhost_vdpa_can_resume(v))
+			return -EOPNOTSUPP;
 		vhost_set_backend_features(&v->vdev, features);
 		return 0;
 	}
@@ -658,6 +670,8 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
 		features = VHOST_VDPA_BACKEND_FEATURES;
 		if (vhost_vdpa_can_suspend(v))
 			features |= BIT_ULL(VHOST_BACKEND_F_SUSPEND);
+		if (vhost_vdpa_can_resume(v))
+			features |= BIT_ULL(VHOST_BACKEND_F_RESUME);
 		if (copy_to_user(featurep, &features, sizeof(features)))
 			r = -EFAULT;
 		break;
diff --git a/include/uapi/linux/vhost_types.h b/include/uapi/linux/vhost_types.h
index 53601ce2c20a..c5690a8992d8 100644
--- a/include/uapi/linux/vhost_types.h
+++ b/include/uapi/linux/vhost_types.h
@@ -163,5 +163,7 @@ struct vhost_vdpa_iova_range {
 #define VHOST_BACKEND_F_IOTLB_ASID  0x3
 /* Device can be suspended */
 #define VHOST_BACKEND_F_SUSPEND  0x4
+/* Device can be resumed */
+#define VHOST_BACKEND_F_RESUME  0x5
 
 #endif
-- 
2.37.2

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 5 208 026.16 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH v6 3/4] vhost-vdpa: uAPI to resume the device

[sebastien.boeuf]

From: Sebastien Boeuf <sebastien.boeuf@intel.com>

This new ioctl adds support for resuming the device from userspace.

This is required when trying to restore the device in a functioning
state after it's been suspended. It is already possible to reset a
suspended device, but that means the device must be reconfigured and
all the IOMMU/IOTLB mappings must be recreated. This new operation
allows the device to be resumed without going through a full reset.

This is particularly useful when trying to perform offline migration of
a virtual machine (also known as snapshot/restore) as it allows the VMM
to resume the virtual machine back to a running state after the snapshot
is performed.

Acked-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
---
 drivers/vhost/vdpa.c       | 18 ++++++++++++++++++
 include/uapi/linux/vhost.h |  8 ++++++++
 2 files changed, 26 insertions(+)

diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
index 833617d00ef6..1db7bd39fb63 100644
--- a/drivers/vhost/vdpa.c
+++ b/drivers/vhost/vdpa.c
@@ -502,6 +502,21 @@ static long vhost_vdpa_suspend(struct vhost_vdpa *v)
 	return ops->suspend(vdpa);
 }
 
+/* After a successful return of this ioctl the device resumes processing
+ * virtqueue descriptors. The device becomes fully operational the same way it
+ * was before it was suspended.
+ */
+static long vhost_vdpa_resume(struct vhost_vdpa *v)
+{
+	struct vdpa_device *vdpa = v->vdpa;
+	const struct vdpa_config_ops *ops = vdpa->config;
+
+	if (!ops->resume)
+		return -EOPNOTSUPP;
+
+	return ops->resume(vdpa);
+}
+
 static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
 				   void __user *argp)
 {
@@ -687,6 +702,9 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
 	case VHOST_VDPA_SUSPEND:
 		r = vhost_vdpa_suspend(v);
 		break;
+	case VHOST_VDPA_RESUME:
+		r = vhost_vdpa_resume(v);
+		break;
 	default:
 		r = vhost_dev_ioctl(&v->vdev, cmd, argp);
 		if (r == -ENOIOCTLCMD)
diff --git a/include/uapi/linux/vhost.h b/include/uapi/linux/vhost.h
index f9f115a7c75b..92e1b700b51c 100644
--- a/include/uapi/linux/vhost.h
+++ b/include/uapi/linux/vhost.h
@@ -180,4 +180,12 @@
  */
 #define VHOST_VDPA_SUSPEND		_IO(VHOST_VIRTIO, 0x7D)
 
+/* Resume a device so it can resume processing virtqueue requests
+ *
+ * After the return of this ioctl the device will have restored all the
+ * necessary states and it is fully operational to continue processing the
+ * virtqueue descriptors.
+ */
+#define VHOST_VDPA_RESUME		_IO(VHOST_VIRTIO, 0x7E)
+
 #endif
-- 
2.37.2

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 5 208 026.16 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH v6 4/4] vdpa_sim: Implement resume vdpa op

[sebastien.boeuf]

From: Sebastien Boeuf <sebastien.boeuf@intel.com>

Implement resume operation for vdpa_sim devices, so vhost-vdpa will
offer that backend feature and userspace can effectively resume the
device.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
---
 drivers/vdpa/vdpa_sim/vdpa_sim.c | 29 +++++++++++++++++++++++++++++
 drivers/vdpa/vdpa_sim/vdpa_sim.h |  1 +
 2 files changed, 30 insertions(+)

diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
index b071f0d842fb..756a5db0109c 100644
--- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
@@ -357,6 +357,12 @@ static void vdpasim_kick_vq(struct vdpa_device *vdpa, u16 idx)
 	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
 	struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx];
 
+	if (!vdpasim->running &&
+	    (vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK)) {
+		vdpasim->pending_kick = true;
+		return;
+	}
+
 	if (vq->ready)
 		schedule_work(&vdpasim->work);
 }
@@ -527,6 +533,27 @@ static int vdpasim_suspend(struct vdpa_device *vdpa)
 	return 0;
 }
 
+static int vdpasim_resume(struct vdpa_device *vdpa)
+{
+	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
+	int i;
+
+	spin_lock(&vdpasim->lock);
+	vdpasim->running = true;
+
+	if (vdpasim->pending_kick) {
+		/* Process pending descriptors */
+		for (i = 0; i < vdpasim->dev_attr.nvqs; ++i)
+			vdpasim_kick_vq(vdpa, i);
+
+		vdpasim->pending_kick = false;
+	}
+
+	spin_unlock(&vdpasim->lock);
+
+	return 0;
+}
+
 static size_t vdpasim_get_config_size(struct vdpa_device *vdpa)
 {
 	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
@@ -717,6 +744,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
 	.set_status             = vdpasim_set_status,
 	.reset			= vdpasim_reset,
 	.suspend		= vdpasim_suspend,
+	.resume			= vdpasim_resume,
 	.get_config_size        = vdpasim_get_config_size,
 	.get_config             = vdpasim_get_config,
 	.set_config             = vdpasim_set_config,
@@ -750,6 +778,7 @@ static const struct vdpa_config_ops vdpasim_batch_config_ops = {
 	.set_status             = vdpasim_set_status,
 	.reset			= vdpasim_reset,
 	.suspend		= vdpasim_suspend,
+	.resume			= vdpasim_resume,
 	.get_config_size        = vdpasim_get_config_size,
 	.get_config             = vdpasim_get_config,
 	.set_config             = vdpasim_set_config,
diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.h b/drivers/vdpa/vdpa_sim/vdpa_sim.h
index 0e78737dcc16..a745605589e2 100644
--- a/drivers/vdpa/vdpa_sim/vdpa_sim.h
+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.h
@@ -67,6 +67,7 @@ struct vdpasim {
 	u64 features;
 	u32 groups;
 	bool running;
+	bool pending_kick;
 	/* spinlock to synchronize iommu table */
 	spinlock_t iommu_lock;
 };
-- 
2.37.2

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 5 208 026.16 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: [PATCH v6 4/4] vdpa_sim: Implement resume vdpa op

[Jason Wang]

On Tue, Jan 3, 2023 at 6:51 PM <sebastien.boeuf@intel.com> wrote:
>
> From: Sebastien Boeuf <sebastien.boeuf@intel.com>
>
> Implement resume operation for vdpa_sim devices, so vhost-vdpa will
> offer that backend feature and userspace can effectively resume the
> device.
>
> Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>

Acked-by: Jason Wang <jasowang@redhat.com>

Thanks

> ---
>  drivers/vdpa/vdpa_sim/vdpa_sim.c | 29 +++++++++++++++++++++++++++++
>  drivers/vdpa/vdpa_sim/vdpa_sim.h |  1 +
>  2 files changed, 30 insertions(+)
>
> diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
> index b071f0d842fb..756a5db0109c 100644
> --- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
> +++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
> @@ -357,6 +357,12 @@ static void vdpasim_kick_vq(struct vdpa_device *vdpa, u16 idx)
>         struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
>         struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx];
>
> +       if (!vdpasim->running &&
> +           (vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK)) {
> +               vdpasim->pending_kick = true;
> +               return;
> +       }
> +
>         if (vq->ready)
>                 schedule_work(&vdpasim->work);
>  }
> @@ -527,6 +533,27 @@ static int vdpasim_suspend(struct vdpa_device *vdpa)
>         return 0;
>  }
>
> +static int vdpasim_resume(struct vdpa_device *vdpa)
> +{
> +       struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
> +       int i;
> +
> +       spin_lock(&vdpasim->lock);
> +       vdpasim->running = true;
> +
> +       if (vdpasim->pending_kick) {
> +               /* Process pending descriptors */
> +               for (i = 0; i < vdpasim->dev_attr.nvqs; ++i)
> +                       vdpasim_kick_vq(vdpa, i);
> +
> +               vdpasim->pending_kick = false;
> +       }
> +
> +       spin_unlock(&vdpasim->lock);
> +
> +       return 0;
> +}
> +
>  static size_t vdpasim_get_config_size(struct vdpa_device *vdpa)
>  {
>         struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
> @@ -717,6 +744,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
>         .set_status             = vdpasim_set_status,
>         .reset                  = vdpasim_reset,
>         .suspend                = vdpasim_suspend,
> +       .resume                 = vdpasim_resume,
>         .get_config_size        = vdpasim_get_config_size,
>         .get_config             = vdpasim_get_config,
>         .set_config             = vdpasim_set_config,
> @@ -750,6 +778,7 @@ static const struct vdpa_config_ops vdpasim_batch_config_ops = {
>         .set_status             = vdpasim_set_status,
>         .reset                  = vdpasim_reset,
>         .suspend                = vdpasim_suspend,
> +       .resume                 = vdpasim_resume,
>         .get_config_size        = vdpasim_get_config_size,
>         .get_config             = vdpasim_get_config,
>         .set_config             = vdpasim_set_config,
> diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.h b/drivers/vdpa/vdpa_sim/vdpa_sim.h
> index 0e78737dcc16..a745605589e2 100644
> --- a/drivers/vdpa/vdpa_sim/vdpa_sim.h
> +++ b/drivers/vdpa/vdpa_sim/vdpa_sim.h
> @@ -67,6 +67,7 @@ struct vdpasim {
>         u64 features;
>         u32 groups;
>         bool running;
> +       bool pending_kick;
>         /* spinlock to synchronize iommu table */
>         spinlock_t iommu_lock;
>  };
> --
> 2.37.2
>
> ---------------------------------------------------------------------
> Intel Corporation SAS (French simplified joint stock company)
> Registered headquarters: "Les Montalets"- 2, rue de Paris,
> 92196 Meudon Cedex, France
> Registration Number:  302 456 199 R.C.S. NANTERRE
> Capital: 5 208 026.16 Euros
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH v6 4/4] vdpa_sim: Implement resume vdpa op

[Jason Wang]

On Tue, Jan 3, 2023 at 6:51 PM <sebastien.boeuf@intel.com> wrote:
>
> From: Sebastien Boeuf <sebastien.boeuf@intel.com>
>
> Implement resume operation for vdpa_sim devices, so vhost-vdpa will
> offer that backend feature and userspace can effectively resume the
> device.
>
> Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>

Acked-by: Jason Wang <jasowang@redhat.com>

Thanks

> ---
>  drivers/vdpa/vdpa_sim/vdpa_sim.c | 29 +++++++++++++++++++++++++++++
>  drivers/vdpa/vdpa_sim/vdpa_sim.h |  1 +
>  2 files changed, 30 insertions(+)
>
> diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
> index b071f0d842fb..756a5db0109c 100644
> --- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
> +++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
> @@ -357,6 +357,12 @@ static void vdpasim_kick_vq(struct vdpa_device *vdpa, u16 idx)
>         struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
>         struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx];
>
> +       if (!vdpasim->running &&
> +           (vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK)) {
> +               vdpasim->pending_kick = true;
> +               return;
> +       }
> +
>         if (vq->ready)
>                 schedule_work(&vdpasim->work);
>  }
> @@ -527,6 +533,27 @@ static int vdpasim_suspend(struct vdpa_device *vdpa)
>         return 0;
>  }
>
> +static int vdpasim_resume(struct vdpa_device *vdpa)
> +{
> +       struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
> +       int i;
> +
> +       spin_lock(&vdpasim->lock);
> +       vdpasim->running = true;
> +
> +       if (vdpasim->pending_kick) {
> +               /* Process pending descriptors */
> +               for (i = 0; i < vdpasim->dev_attr.nvqs; ++i)
> +                       vdpasim_kick_vq(vdpa, i);
> +
> +               vdpasim->pending_kick = false;
> +       }
> +
> +       spin_unlock(&vdpasim->lock);
> +
> +       return 0;
> +}
> +
>  static size_t vdpasim_get_config_size(struct vdpa_device *vdpa)
>  {
>         struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
> @@ -717,6 +744,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
>         .set_status             = vdpasim_set_status,
>         .reset                  = vdpasim_reset,
>         .suspend                = vdpasim_suspend,
> +       .resume                 = vdpasim_resume,
>         .get_config_size        = vdpasim_get_config_size,
>         .get_config             = vdpasim_get_config,
>         .set_config             = vdpasim_set_config,
> @@ -750,6 +778,7 @@ static const struct vdpa_config_ops vdpasim_batch_config_ops = {
>         .set_status             = vdpasim_set_status,
>         .reset                  = vdpasim_reset,
>         .suspend                = vdpasim_suspend,
> +       .resume                 = vdpasim_resume,
>         .get_config_size        = vdpasim_get_config_size,
>         .get_config             = vdpasim_get_config,
>         .set_config             = vdpasim_set_config,
> diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.h b/drivers/vdpa/vdpa_sim/vdpa_sim.h
> index 0e78737dcc16..a745605589e2 100644
> --- a/drivers/vdpa/vdpa_sim/vdpa_sim.h
> +++ b/drivers/vdpa/vdpa_sim/vdpa_sim.h
> @@ -67,6 +67,7 @@ struct vdpasim {
>         u64 features;
>         u32 groups;
>         bool running;
> +       bool pending_kick;
>         /* spinlock to synchronize iommu table */
>         spinlock_t iommu_lock;
>  };
> --
> 2.37.2
>
> ---------------------------------------------------------------------
> Intel Corporation SAS (French simplified joint stock company)
> Registered headquarters: "Les Montalets"- 2, rue de Paris,
> 92196 Meudon Cedex, France
> Registration Number:  302 456 199 R.C.S. NANTERRE
> Capital: 5 208 026.16 Euros
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>

Re: [PATCH v6 1/4] vdpa: Add resume operation

[Stefano Garzarella]

On Tue, Jan 03, 2023 at 11:51:05AM +0100, sebastien.boeuf@intel.com wrote:
>From: Sebastien Boeuf <sebastien.boeuf@intel.com>
>
>Add a new operation to allow a vDPA device to be resumed after it has
>been suspended. Trying to resume a device that wasn't suspended will
>result in a no-op.
>
>This operation is optional. If it's not implemented, the associated
>backend feature bit will not be exposed. And if the feature bit is not
>exposed, invoking this operation will return an error.
>
>Acked-by: Jason Wang <jasowang@redhat.com>
>Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
>---
> include/linux/vdpa.h | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)

Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>

>
>diff --git a/include/linux/vdpa.h b/include/linux/vdpa.h
>index 6d0f5e4e82c2..96d308cbf97b 100644
>--- a/include/linux/vdpa.h
>+++ b/include/linux/vdpa.h
>@@ -219,7 +219,10 @@ struct vdpa_map_file {
>  * @reset:			Reset device
>  *				@vdev: vdpa device
>  *				Returns integer: success (0) or error (< 0)
>- * @suspend:			Suspend or resume the device (optional)
>+ * @suspend:			Suspend the device (optional)
>+ *				@vdev: vdpa device
>+ *				Returns integer: success (0) or error (< 0)
>+ * @resume:			Resume the device (optional)
>  *				@vdev: vdpa device
>  *				Returns integer: success (0) or error (< 0)
>  * @get_config_size:		Get the size of the configuration space includes
>@@ -324,6 +327,7 @@ struct vdpa_config_ops {
> 	void (*set_status)(struct vdpa_device *vdev, u8 status);
> 	int (*reset)(struct vdpa_device *vdev);
> 	int (*suspend)(struct vdpa_device *vdev);
>+	int (*resume)(struct vdpa_device *vdev);
> 	size_t (*get_config_size)(struct vdpa_device *vdev);
> 	void (*get_config)(struct vdpa_device *vdev, unsigned int offset,
> 			   void *buf, unsigned int len);
>-- 
>2.37.2
>
>---------------------------------------------------------------------
>Intel Corporation SAS (French simplified joint stock company)
>Registered headquarters: "Les Montalets"- 2, rue de Paris,
>92196 Meudon Cedex, France
>Registration Number:  302 456 199 R.C.S. NANTERRE
>Capital: 5 208 026.16 Euros
>
>This e-mail and any attachments may contain confidential material for
>the sole use of the intended recipient(s). Any review or distribution
>by others is strictly prohibited. If you are not the intended
>recipient, please contact the sender and delete all copies.
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH v6 1/4] vdpa: Add resume operation

[Stefano Garzarella]

On Tue, Jan 03, 2023 at 11:51:05AM +0100, sebastien.boeuf@intel.com wrote:
>From: Sebastien Boeuf <sebastien.boeuf@intel.com>
>
>Add a new operation to allow a vDPA device to be resumed after it has
>been suspended. Trying to resume a device that wasn't suspended will
>result in a no-op.
>
>This operation is optional. If it's not implemented, the associated
>backend feature bit will not be exposed. And if the feature bit is not
>exposed, invoking this operation will return an error.
>
>Acked-by: Jason Wang <jasowang@redhat.com>
>Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
>---
> include/linux/vdpa.h | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)

Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>

>
>diff --git a/include/linux/vdpa.h b/include/linux/vdpa.h
>index 6d0f5e4e82c2..96d308cbf97b 100644
>--- a/include/linux/vdpa.h
>+++ b/include/linux/vdpa.h
>@@ -219,7 +219,10 @@ struct vdpa_map_file {
>  * @reset:			Reset device
>  *				@vdev: vdpa device
>  *				Returns integer: success (0) or error (< 0)
>- * @suspend:			Suspend or resume the device (optional)
>+ * @suspend:			Suspend the device (optional)
>+ *				@vdev: vdpa device
>+ *				Returns integer: success (0) or error (< 0)
>+ * @resume:			Resume the device (optional)
>  *				@vdev: vdpa device
>  *				Returns integer: success (0) or error (< 0)
>  * @get_config_size:		Get the size of the configuration space includes
>@@ -324,6 +327,7 @@ struct vdpa_config_ops {
> 	void (*set_status)(struct vdpa_device *vdev, u8 status);
> 	int (*reset)(struct vdpa_device *vdev);
> 	int (*suspend)(struct vdpa_device *vdev);
>+	int (*resume)(struct vdpa_device *vdev);
> 	size_t (*get_config_size)(struct vdpa_device *vdev);
> 	void (*get_config)(struct vdpa_device *vdev, unsigned int offset,
> 			   void *buf, unsigned int len);
>-- 
>2.37.2
>
>---------------------------------------------------------------------
>Intel Corporation SAS (French simplified joint stock company)
>Registered headquarters: "Les Montalets"- 2, rue de Paris,
>92196 Meudon Cedex, France
>Registration Number:  302 456 199 R.C.S. NANTERRE
>Capital: 5 208 026.16 Euros
>
>This e-mail and any attachments may contain confidential material for
>the sole use of the intended recipient(s). Any review or distribution
>by others is strictly prohibited. If you are not the intended
>recipient, please contact the sender and delete all copies.
>

Re: [PATCH v6 2/4] vhost-vdpa: Introduce RESUME backend feature bit

[Stefano Garzarella]

On Tue, Jan 03, 2023 at 11:51:06AM +0100, sebastien.boeuf@intel.com wrote:
>From: Sebastien Boeuf <sebastien.boeuf@intel.com>
>
>Userspace knows if the device can be resumed or not by checking this
>feature bit.
>
>It's only exposed if the vdpa driver backend implements the resume()
>operation callback. Userspace trying to negotiate this feature when it
>hasn't been exposed will result in an error.
>
>Acked-by: Jason Wang <jasowang@redhat.com>
>Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
>---
> drivers/vhost/vdpa.c             | 16 +++++++++++++++-
> include/uapi/linux/vhost_types.h |  2 ++
> 2 files changed, 17 insertions(+), 1 deletion(-)
>
>diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
>index 166044642fd5..833617d00ef6 100644
>--- a/drivers/vhost/vdpa.c
>+++ b/drivers/vhost/vdpa.c
>@@ -355,6 +355,14 @@ static bool vhost_vdpa_can_suspend(const struct vhost_vdpa *v)
> 	return ops->suspend;
> }
>
>+static bool vhost_vdpa_can_resume(const struct vhost_vdpa *v)
>+{
>+	struct vdpa_device *vdpa = v->vdpa;
>+	const struct vdpa_config_ops *ops = vdpa->config;
>+
>+	return ops->resume;
>+}
>+
> static long vhost_vdpa_get_features(struct vhost_vdpa *v, u64 __user *featurep)
> {
> 	struct vdpa_device *vdpa = v->vdpa;
>@@ -602,11 +610,15 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
> 		if (copy_from_user(&features, featurep, sizeof(features)))
> 			return -EFAULT;
> 		if (features & ~(VHOST_VDPA_BACKEND_FEATURES |
>-				 BIT_ULL(VHOST_BACKEND_F_SUSPEND)))
>+				 BIT_ULL(VHOST_BACKEND_F_SUSPEND) |
>+				 BIT_ULL(VHOST_BACKEND_F_RESUME)))
> 			return -EOPNOTSUPP;
> 		if ((features & BIT_ULL(VHOST_BACKEND_F_SUSPEND)) &&
> 		     !vhost_vdpa_can_suspend(v))
> 			return -EOPNOTSUPP;
>+		if ((features & BIT_ULL(VHOST_BACKEND_F_RESUME)) &&
>+		     !vhost_vdpa_can_resume(v))
>+			return -EOPNOTSUPP;

Not for this patch, but I'd like to refactor this code a bit to fill a 
`backend_features` field in vhost_vdpa during the vhost_vdpa_probe(), so 
we don't need to change this code or the VHOST_GET_BACKEND_FEATURES for 
every new backend feature.

I'll send a patch.

This LGTM:

Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>


> 		vhost_set_backend_features(&v->vdev, features);
> 		return 0;
> 	}
>@@ -658,6 +670,8 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
> 		features = VHOST_VDPA_BACKEND_FEATURES;
> 		if (vhost_vdpa_can_suspend(v))
> 			features |= BIT_ULL(VHOST_BACKEND_F_SUSPEND);
>+		if (vhost_vdpa_can_resume(v))
>+			features |= BIT_ULL(VHOST_BACKEND_F_RESUME);
> 		if (copy_to_user(featurep, &features, sizeof(features)))
> 			r = -EFAULT;
> 		break;
>diff --git a/include/uapi/linux/vhost_types.h b/include/uapi/linux/vhost_types.h
>index 53601ce2c20a..c5690a8992d8 100644
>--- a/include/uapi/linux/vhost_types.h
>+++ b/include/uapi/linux/vhost_types.h
>@@ -163,5 +163,7 @@ struct vhost_vdpa_iova_range {
> #define VHOST_BACKEND_F_IOTLB_ASID  0x3
> /* Device can be suspended */
> #define VHOST_BACKEND_F_SUSPEND  0x4
>+/* Device can be resumed */
>+#define VHOST_BACKEND_F_RESUME  0x5
>
> #endif
>-- 
>2.37.2
>
>---------------------------------------------------------------------
>Intel Corporation SAS (French simplified joint stock company)
>Registered headquarters: "Les Montalets"- 2, rue de Paris,
>92196 Meudon Cedex, France
>Registration Number:  302 456 199 R.C.S. NANTERRE
>Capital: 5 208 026.16 Euros
>
>This e-mail and any attachments may contain confidential material for
>the sole use of the intended recipient(s). Any review or distribution
>by others is strictly prohibited. If you are not the intended
>recipient, please contact the sender and delete all copies.
>

Re: [PATCH v6 2/4] vhost-vdpa: Introduce RESUME backend feature bit

[Stefano Garzarella]

On Tue, Jan 03, 2023 at 11:51:06AM +0100, sebastien.boeuf@intel.com wrote:
>From: Sebastien Boeuf <sebastien.boeuf@intel.com>
>
>Userspace knows if the device can be resumed or not by checking this
>feature bit.
>
>It's only exposed if the vdpa driver backend implements the resume()
>operation callback. Userspace trying to negotiate this feature when it
>hasn't been exposed will result in an error.
>
>Acked-by: Jason Wang <jasowang@redhat.com>
>Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
>---
> drivers/vhost/vdpa.c             | 16 +++++++++++++++-
> include/uapi/linux/vhost_types.h |  2 ++
> 2 files changed, 17 insertions(+), 1 deletion(-)
>
>diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
>index 166044642fd5..833617d00ef6 100644
>--- a/drivers/vhost/vdpa.c
>+++ b/drivers/vhost/vdpa.c
>@@ -355,6 +355,14 @@ static bool vhost_vdpa_can_suspend(const struct vhost_vdpa *v)
> 	return ops->suspend;
> }
>
>+static bool vhost_vdpa_can_resume(const struct vhost_vdpa *v)
>+{
>+	struct vdpa_device *vdpa = v->vdpa;
>+	const struct vdpa_config_ops *ops = vdpa->config;
>+
>+	return ops->resume;
>+}
>+
> static long vhost_vdpa_get_features(struct vhost_vdpa *v, u64 __user *featurep)
> {
> 	struct vdpa_device *vdpa = v->vdpa;
>@@ -602,11 +610,15 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
> 		if (copy_from_user(&features, featurep, sizeof(features)))
> 			return -EFAULT;
> 		if (features & ~(VHOST_VDPA_BACKEND_FEATURES |
>-				 BIT_ULL(VHOST_BACKEND_F_SUSPEND)))
>+				 BIT_ULL(VHOST_BACKEND_F_SUSPEND) |
>+				 BIT_ULL(VHOST_BACKEND_F_RESUME)))
> 			return -EOPNOTSUPP;
> 		if ((features & BIT_ULL(VHOST_BACKEND_F_SUSPEND)) &&
> 		     !vhost_vdpa_can_suspend(v))
> 			return -EOPNOTSUPP;
>+		if ((features & BIT_ULL(VHOST_BACKEND_F_RESUME)) &&
>+		     !vhost_vdpa_can_resume(v))
>+			return -EOPNOTSUPP;

Not for this patch, but I'd like to refactor this code a bit to fill a 
`backend_features` field in vhost_vdpa during the vhost_vdpa_probe(), so 
we don't need to change this code or the VHOST_GET_BACKEND_FEATURES for 
every new backend feature.

I'll send a patch.

This LGTM:

Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>


> 		vhost_set_backend_features(&v->vdev, features);
> 		return 0;
> 	}
>@@ -658,6 +670,8 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
> 		features = VHOST_VDPA_BACKEND_FEATURES;
> 		if (vhost_vdpa_can_suspend(v))
> 			features |= BIT_ULL(VHOST_BACKEND_F_SUSPEND);
>+		if (vhost_vdpa_can_resume(v))
>+			features |= BIT_ULL(VHOST_BACKEND_F_RESUME);
> 		if (copy_to_user(featurep, &features, sizeof(features)))
> 			r = -EFAULT;
> 		break;
>diff --git a/include/uapi/linux/vhost_types.h b/include/uapi/linux/vhost_types.h
>index 53601ce2c20a..c5690a8992d8 100644
>--- a/include/uapi/linux/vhost_types.h
>+++ b/include/uapi/linux/vhost_types.h
>@@ -163,5 +163,7 @@ struct vhost_vdpa_iova_range {
> #define VHOST_BACKEND_F_IOTLB_ASID  0x3
> /* Device can be suspended */
> #define VHOST_BACKEND_F_SUSPEND  0x4
>+/* Device can be resumed */
>+#define VHOST_BACKEND_F_RESUME  0x5
>
> #endif
>-- 
>2.37.2
>
>---------------------------------------------------------------------
>Intel Corporation SAS (French simplified joint stock company)
>Registered headquarters: "Les Montalets"- 2, rue de Paris,
>92196 Meudon Cedex, France
>Registration Number:  302 456 199 R.C.S. NANTERRE
>Capital: 5 208 026.16 Euros
>
>This e-mail and any attachments may contain confidential material for
>the sole use of the intended recipient(s). Any review or distribution
>by others is strictly prohibited. If you are not the intended
>recipient, please contact the sender and delete all copies.
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH v6 3/4] vhost-vdpa: uAPI to resume the device

[Stefano Garzarella]

On Tue, Jan 03, 2023 at 11:51:07AM +0100, sebastien.boeuf@intel.com wrote:
>From: Sebastien Boeuf <sebastien.boeuf@intel.com>
>
>This new ioctl adds support for resuming the device from userspace.
>
>This is required when trying to restore the device in a functioning
>state after it's been suspended. It is already possible to reset a
>suspended device, but that means the device must be reconfigured and
>all the IOMMU/IOTLB mappings must be recreated. This new operation
>allows the device to be resumed without going through a full reset.
>
>This is particularly useful when trying to perform offline migration of
>a virtual machine (also known as snapshot/restore) as it allows the VMM
>to resume the virtual machine back to a running state after the snapshot
>is performed.
>
>Acked-by: Jason Wang <jasowang@redhat.com>
>Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
>---
> drivers/vhost/vdpa.c       | 18 ++++++++++++++++++
> include/uapi/linux/vhost.h |  8 ++++++++
> 2 files changed, 26 insertions(+)
>
>diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
>index 833617d00ef6..1db7bd39fb63 100644
>--- a/drivers/vhost/vdpa.c
>+++ b/drivers/vhost/vdpa.c
>@@ -502,6 +502,21 @@ static long vhost_vdpa_suspend(struct vhost_vdpa *v)
> 	return ops->suspend(vdpa);
> }
>
>+/* After a successful return of this ioctl the device resumes processing
>+ * virtqueue descriptors. The device becomes fully operational the same way it
>+ * was before it was suspended.
>+ */
>+static long vhost_vdpa_resume(struct vhost_vdpa *v)
>+{
>+	struct vdpa_device *vdpa = v->vdpa;
>+	const struct vdpa_config_ops *ops = vdpa->config;
>+
>+	if (!ops->resume)
>+		return -EOPNOTSUPP;
>+
>+	return ops->resume(vdpa);
>+}
>+
> static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
> 				   void __user *argp)
> {
>@@ -687,6 +702,9 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
> 	case VHOST_VDPA_SUSPEND:
> 		r = vhost_vdpa_suspend(v);
> 		break;
>+	case VHOST_VDPA_RESUME:
>+		r = vhost_vdpa_resume(v);
>+		break;
> 	default:
> 		r = vhost_dev_ioctl(&v->vdev, cmd, argp);
> 		if (r == -ENOIOCTLCMD)
>diff --git a/include/uapi/linux/vhost.h b/include/uapi/linux/vhost.h
>index f9f115a7c75b..92e1b700b51c 100644
>--- a/include/uapi/linux/vhost.h
>+++ b/include/uapi/linux/vhost.h
>@@ -180,4 +180,12 @@
>  */
> #define VHOST_VDPA_SUSPEND		_IO(VHOST_VIRTIO, 0x7D)
>
>+/* Resume a device so it can resume processing virtqueue requests
>+ *
>+ * After the return of this ioctl the device will have restored all the
>+ * necessary states and it is fully operational to continue processing the
>+ * virtqueue descriptors.

IIUC this is a no-op if the device wasn't suspended.
If you have to resend, maybe add this info also here in the user 
documentation.

Anyway, the patch LGTM:

Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>

>+ */
>+#define VHOST_VDPA_RESUME		_IO(VHOST_VIRTIO, 0x7E)
>+
> #endif
>-- 
>2.37.2
>
>---------------------------------------------------------------------
>Intel Corporation SAS (French simplified joint stock company)
>Registered headquarters: "Les Montalets"- 2, rue de Paris,
>92196 Meudon Cedex, France
>Registration Number:  302 456 199 R.C.S. NANTERRE
>Capital: 5 208 026.16 Euros
>
>This e-mail and any attachments may contain confidential material for
>the sole use of the intended recipient(s). Any review or distribution
>by others is strictly prohibited. If you are not the intended
>recipient, please contact the sender and delete all copies.
>

Re: [PATCH v6 3/4] vhost-vdpa: uAPI to resume the device

[Stefano Garzarella]

On Tue, Jan 03, 2023 at 11:51:07AM +0100, sebastien.boeuf@intel.com wrote:
>From: Sebastien Boeuf <sebastien.boeuf@intel.com>
>
>This new ioctl adds support for resuming the device from userspace.
>
>This is required when trying to restore the device in a functioning
>state after it's been suspended. It is already possible to reset a
>suspended device, but that means the device must be reconfigured and
>all the IOMMU/IOTLB mappings must be recreated. This new operation
>allows the device to be resumed without going through a full reset.
>
>This is particularly useful when trying to perform offline migration of
>a virtual machine (also known as snapshot/restore) as it allows the VMM
>to resume the virtual machine back to a running state after the snapshot
>is performed.
>
>Acked-by: Jason Wang <jasowang@redhat.com>
>Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
>---
> drivers/vhost/vdpa.c       | 18 ++++++++++++++++++
> include/uapi/linux/vhost.h |  8 ++++++++
> 2 files changed, 26 insertions(+)
>
>diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
>index 833617d00ef6..1db7bd39fb63 100644
>--- a/drivers/vhost/vdpa.c
>+++ b/drivers/vhost/vdpa.c
>@@ -502,6 +502,21 @@ static long vhost_vdpa_suspend(struct vhost_vdpa *v)
> 	return ops->suspend(vdpa);
> }
>
>+/* After a successful return of this ioctl the device resumes processing
>+ * virtqueue descriptors. The device becomes fully operational the same way it
>+ * was before it was suspended.
>+ */
>+static long vhost_vdpa_resume(struct vhost_vdpa *v)
>+{
>+	struct vdpa_device *vdpa = v->vdpa;
>+	const struct vdpa_config_ops *ops = vdpa->config;
>+
>+	if (!ops->resume)
>+		return -EOPNOTSUPP;
>+
>+	return ops->resume(vdpa);
>+}
>+
> static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
> 				   void __user *argp)
> {
>@@ -687,6 +702,9 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
> 	case VHOST_VDPA_SUSPEND:
> 		r = vhost_vdpa_suspend(v);
> 		break;
>+	case VHOST_VDPA_RESUME:
>+		r = vhost_vdpa_resume(v);
>+		break;
> 	default:
> 		r = vhost_dev_ioctl(&v->vdev, cmd, argp);
> 		if (r == -ENOIOCTLCMD)
>diff --git a/include/uapi/linux/vhost.h b/include/uapi/linux/vhost.h
>index f9f115a7c75b..92e1b700b51c 100644
>--- a/include/uapi/linux/vhost.h
>+++ b/include/uapi/linux/vhost.h
>@@ -180,4 +180,12 @@
>  */
> #define VHOST_VDPA_SUSPEND		_IO(VHOST_VIRTIO, 0x7D)
>
>+/* Resume a device so it can resume processing virtqueue requests
>+ *
>+ * After the return of this ioctl the device will have restored all the
>+ * necessary states and it is fully operational to continue processing the
>+ * virtqueue descriptors.

IIUC this is a no-op if the device wasn't suspended.
If you have to resend, maybe add this info also here in the user 
documentation.

Anyway, the patch LGTM:

Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>

>+ */
>+#define VHOST_VDPA_RESUME		_IO(VHOST_VIRTIO, 0x7E)
>+
> #endif
>-- 
>2.37.2
>
>---------------------------------------------------------------------
>Intel Corporation SAS (French simplified joint stock company)
>Registered headquarters: "Les Montalets"- 2, rue de Paris,
>92196 Meudon Cedex, France
>Registration Number:  302 456 199 R.C.S. NANTERRE
>Capital: 5 208 026.16 Euros
>
>This e-mail and any attachments may contain confidential material for
>the sole use of the intended recipient(s). Any review or distribution
>by others is strictly prohibited. If you are not the intended
>recipient, please contact the sender and delete all copies.
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH v6 4/4] vdpa_sim: Implement resume vdpa op

[Stefano Garzarella]

On Tue, Jan 03, 2023 at 11:51:08AM +0100, sebastien.boeuf@intel.com wrote:
>From: Sebastien Boeuf <sebastien.boeuf@intel.com>
>
>Implement resume operation for vdpa_sim devices, so vhost-vdpa will
>offer that backend feature and userspace can effectively resume the
>device.
>
>Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
>---
> drivers/vdpa/vdpa_sim/vdpa_sim.c | 29 +++++++++++++++++++++++++++++
> drivers/vdpa/vdpa_sim/vdpa_sim.h |  1 +
> 2 files changed, 30 insertions(+)
>
>diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
>index b071f0d842fb..756a5db0109c 100644
>--- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
>+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
>@@ -357,6 +357,12 @@ static void vdpasim_kick_vq(struct vdpa_device *vdpa, u16 idx)
> 	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
> 	struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx];
>
>+	if (!vdpasim->running &&
>+	    (vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK)) {
>+		vdpasim->pending_kick = true;
>+		return;
>+	}
>+
> 	if (vq->ready)
> 		schedule_work(&vdpasim->work);
> }
>@@ -527,6 +533,27 @@ static int vdpasim_suspend(struct vdpa_device *vdpa)
> 	return 0;
> }
>
>+static int vdpasim_resume(struct vdpa_device *vdpa)
>+{
>+	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
>+	int i;
>+
>+	spin_lock(&vdpasim->lock);
>+	vdpasim->running = true;
>+
>+	if (vdpasim->pending_kick) {

IIUC if one of the vq receive a kick while the device is suspended, we 
will kick all the vq.

At this point perhaps we should either send the kick only to the vqs we 
should notify, or send it to all of them indiscriminately (I don't know 
if it is correct to send a spurious kick).

Thanks,
Stefano

>+		/* Process pending descriptors */
>+		for (i = 0; i < vdpasim->dev_attr.nvqs; ++i)
>+			vdpasim_kick_vq(vdpa, i);
>+
>+		vdpasim->pending_kick = false;
>+	}
>+
>+	spin_unlock(&vdpasim->lock);
>+
>+	return 0;
>+}
>+
> static size_t vdpasim_get_config_size(struct vdpa_device *vdpa)
> {
> 	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
>@@ -717,6 +744,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
> 	.set_status             = vdpasim_set_status,
> 	.reset			= vdpasim_reset,
> 	.suspend		= vdpasim_suspend,
>+	.resume			= vdpasim_resume,
> 	.get_config_size        = vdpasim_get_config_size,
> 	.get_config             = vdpasim_get_config,
> 	.set_config             = vdpasim_set_config,
>@@ -750,6 +778,7 @@ static const struct vdpa_config_ops vdpasim_batch_config_ops = {
> 	.set_status             = vdpasim_set_status,
> 	.reset			= vdpasim_reset,
> 	.suspend		= vdpasim_suspend,
>+	.resume			= vdpasim_resume,
> 	.get_config_size        = vdpasim_get_config_size,
> 	.get_config             = vdpasim_get_config,
> 	.set_config             = vdpasim_set_config,
>diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.h b/drivers/vdpa/vdpa_sim/vdpa_sim.h
>index 0e78737dcc16..a745605589e2 100644
>--- a/drivers/vdpa/vdpa_sim/vdpa_sim.h
>+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.h
>@@ -67,6 +67,7 @@ struct vdpasim {
> 	u64 features;
> 	u32 groups;
> 	bool running;
>+	bool pending_kick;
> 	/* spinlock to synchronize iommu table */
> 	spinlock_t iommu_lock;
> };
>-- 
>2.37.2
>
>---------------------------------------------------------------------
>Intel Corporation SAS (French simplified joint stock company)
>Registered headquarters: "Les Montalets"- 2, rue de Paris,
>92196 Meudon Cedex, France
>Registration Number:  302 456 199 R.C.S. NANTERRE
>Capital: 5 208 026.16 Euros
>
>This e-mail and any attachments may contain confidential material for
>the sole use of the intended recipient(s). Any review or distribution
>by others is strictly prohibited. If you are not the intended
>recipient, please contact the sender and delete all copies.
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH v6 4/4] vdpa_sim: Implement resume vdpa op

[Stefano Garzarella]

On Tue, Jan 03, 2023 at 11:51:08AM +0100, sebastien.boeuf@intel.com wrote:
>From: Sebastien Boeuf <sebastien.boeuf@intel.com>
>
>Implement resume operation for vdpa_sim devices, so vhost-vdpa will
>offer that backend feature and userspace can effectively resume the
>device.
>
>Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
>---
> drivers/vdpa/vdpa_sim/vdpa_sim.c | 29 +++++++++++++++++++++++++++++
> drivers/vdpa/vdpa_sim/vdpa_sim.h |  1 +
> 2 files changed, 30 insertions(+)
>
>diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
>index b071f0d842fb..756a5db0109c 100644
>--- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
>+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
>@@ -357,6 +357,12 @@ static void vdpasim_kick_vq(struct vdpa_device *vdpa, u16 idx)
> 	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
> 	struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx];
>
>+	if (!vdpasim->running &&
>+	    (vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK)) {
>+		vdpasim->pending_kick = true;
>+		return;
>+	}
>+
> 	if (vq->ready)
> 		schedule_work(&vdpasim->work);
> }
>@@ -527,6 +533,27 @@ static int vdpasim_suspend(struct vdpa_device *vdpa)
> 	return 0;
> }
>
>+static int vdpasim_resume(struct vdpa_device *vdpa)
>+{
>+	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
>+	int i;
>+
>+	spin_lock(&vdpasim->lock);
>+	vdpasim->running = true;
>+
>+	if (vdpasim->pending_kick) {

IIUC if one of the vq receive a kick while the device is suspended, we 
will kick all the vq.

At this point perhaps we should either send the kick only to the vqs we 
should notify, or send it to all of them indiscriminately (I don't know 
if it is correct to send a spurious kick).

Thanks,
Stefano

>+		/* Process pending descriptors */
>+		for (i = 0; i < vdpasim->dev_attr.nvqs; ++i)
>+			vdpasim_kick_vq(vdpa, i);
>+
>+		vdpasim->pending_kick = false;
>+	}
>+
>+	spin_unlock(&vdpasim->lock);
>+
>+	return 0;
>+}
>+
> static size_t vdpasim_get_config_size(struct vdpa_device *vdpa)
> {
> 	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
>@@ -717,6 +744,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
> 	.set_status             = vdpasim_set_status,
> 	.reset			= vdpasim_reset,
> 	.suspend		= vdpasim_suspend,
>+	.resume			= vdpasim_resume,
> 	.get_config_size        = vdpasim_get_config_size,
> 	.get_config             = vdpasim_get_config,
> 	.set_config             = vdpasim_set_config,
>@@ -750,6 +778,7 @@ static const struct vdpa_config_ops vdpasim_batch_config_ops = {
> 	.set_status             = vdpasim_set_status,
> 	.reset			= vdpasim_reset,
> 	.suspend		= vdpasim_suspend,
>+	.resume			= vdpasim_resume,
> 	.get_config_size        = vdpasim_get_config_size,
> 	.get_config             = vdpasim_get_config,
> 	.set_config             = vdpasim_set_config,
>diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.h b/drivers/vdpa/vdpa_sim/vdpa_sim.h
>index 0e78737dcc16..a745605589e2 100644
>--- a/drivers/vdpa/vdpa_sim/vdpa_sim.h
>+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.h
>@@ -67,6 +67,7 @@ struct vdpasim {
> 	u64 features;
> 	u32 groups;
> 	bool running;
>+	bool pending_kick;
> 	/* spinlock to synchronize iommu table */
> 	spinlock_t iommu_lock;
> };
>-- 
>2.37.2
>
>---------------------------------------------------------------------
>Intel Corporation SAS (French simplified joint stock company)
>Registered headquarters: "Les Montalets"- 2, rue de Paris,
>92196 Meudon Cedex, France
>Registration Number:  302 456 199 R.C.S. NANTERRE
>Capital: 5 208 026.16 Euros
>
>This e-mail and any attachments may contain confidential material for
>the sole use of the intended recipient(s). Any review or distribution
>by others is strictly prohibited. If you are not the intended
>recipient, please contact the sender and delete all copies.
>