[LINUX KERNEL PATCH v5 0/1] add freeze_mode for virtio_pci and add S3 support for virtgpu

[LINUX KERNEL PATCH v5 0/1] add freeze_mode for virtio_pci and add S3 support for virtgpu

[Jiqian Chen]

Hi all,
This is the v5 of kernel patches and v5 makes below changes:
* Since this series patches add a new mechanism that let virtgpu and Qemu can negotiate
  their reset behavior, and other guys hope me can improve this mechanism to virtio pci
  level, so that other virtio devices can also benefit from it. So instead of adding
  new feature flag VIRTIO_GPU_F_FREEZE_S3 only serves for virtgpu, v5 add a new parameter
  named freeze_mode to virtio_pci_common_cfg, when guest begin suspending, set freeze_mode
  to FREEZE_S3, and then all virtio devices can get this status, and change their reset
  behavior according freeze_mode. See the new commit "virtio_pci: Add freeze_mode for
  virtio_pci_common_cfg"
* The second commit "virtgpu: Add freeze and restore func to reinit vqs" is the same as
  before, it delete virtqueus in freeze() and re-initialize virtqueus in restore().

The link to trace this issue:
https://gitlab.com/qemu-project/qemu/-/issues/1860

Best regards,
Jiqian Chen

v4:
Link:
no v4 of kernel patch.
V4 of Qemu patch:
https://lore.kernel.org/qemu-devel/20230720120816.8751-1-Jiqian.Chen@amd.com/


v3:
makes below changes:
* Use enum for freeze mode, so this can be extended with more
  modes in the future.
* Rename functions and paratemers with "_S3" postfix.
And no functional changes.
Link:
https://lore.kernel.org/lkml/20230720115805.8206-1-Jiqian.Chen@amd.com/T/#t
V3 of Qemu patch:
https://lore.kernel.org/qemu-devel/20230719074726.1613088-1-Jiqian.Chen@amd.com/T/#t


v2:
makes below changes:
* Change VIRTIO_CPU_CMD_STATUS_FREEZING to 0x0400 (<0x1000)
* Add a new feature flag VIRTIO_GPU_F_FREEZING, so that guest and
  host can negotiate whenever freezing is supported or not.
Link:
https://lore.kernel.org/lkml/20230630073448.842767-1-Jiqian.Chen@amd.com/T/#t
V2 of Qemu patch:
https://lore.kernel.org/qemu-devel/20230630070016.841459-1-Jiqian.Chen@amd.com/T/#t


v1:
Hi all,
I am working to implement virtgpu S3 function on Xen.

Currently on Xen, if we start a guest who enables virtgpu, and then run "echo mem > 
/sys/power/state" to suspend guest. And run "sudo xl trigger <guest id> s3resume"
to resume guest. We can find that the guest kernel comes back, but the display doesn't.
It just shows a black screen.

In response to the above phenomenon, I have found two problems.

First, if we move mouse on the black screen, guest kernel still sends a cursor request
to Qemu, but Qemu doesn't response. Because when guest is suspending, it calls
device_suspend, and then call into Qemu to call virtio_reset->__virtio_queue_reset. In
__virtio_queue_reset, it clears all virtqueue information on Qemu end. So, after guest
resumes, Qemu can't get message from virtqueue.

Second, the reason why display can't come back is that when guest is suspending, it calls
into Qemu to call virtio_reset->virtio_gpu_gl_reset. In virtio_gpu_gl_reset, it destroys
all resources and resets renderer, which are used for display. So after guest resumes,
the display can't come back to the status when guest is suspended.

This patch initializes virtqueue when guest is resuming to solve first problem. And it
notifies Qemu that guest is suspending to prevent Qemu destroying resources, this is to
solve second problem. And then, I can bring the display back, and everything continues
their actions after guest resumes.
Link:
https://lore.kernel.org/lkml/20230608063857.1677973-1-Jiqian.Chen@amd.com/
V1 of Qemu patch:
https://lore.kernel.org/qemu-devel/20230608025655.1674357-2-Jiqian.Chen@amd.com/


Jiqian Chen (2):
  virtio_pci: Add freeze_mode for virtio_pci_common_cfg
  virtgpu: Add freeze and restore func to reinit vqs

 drivers/gpu/drm/virtio/virtgpu_drv.c   | 23 ++++++++++++++++++++
 drivers/gpu/drm/virtio/virtgpu_drv.h   |  1 +
 drivers/gpu/drm/virtio/virtgpu_kms.c   | 30 ++++++++++++++++++--------
 drivers/virtio/virtio.c                | 13 +++++++++++
 drivers/virtio/virtio_pci_modern.c     |  9 ++++++++
 drivers/virtio/virtio_pci_modern_dev.c | 16 ++++++++++++++
 include/linux/virtio_config.h          |  1 +
 include/linux/virtio_pci_modern.h      |  2 ++
 include/uapi/linux/virtio_pci.h        | 16 ++++++++++++--
 9 files changed, 100 insertions(+), 11 deletions(-)

-- 
2.34.1

[LINUX KERNEL PATCH v5 1/2] virtio_pci: Add freeze_mode for virtio_pci_common_cfg

[Jiqian Chen]

When guest vm does S3, Qemu will reset and clear some things of virtio
devices, but guest can't aware that, so that may cause some problems.
For excample, Qemu calls virtio_reset->virtio_gpu_gl_reset, that will
destroy render resources of virtio-gpu. As a result, after guest resume,
the display can't come back and we only saw a black screen. Due to guest
can't re-create all the resources, so we need to let Qemu not to destroy
them when S3.

For above purpose, this patch add a new parameter named freeze_mode to
struct virtio_pci_common_cfg, and when guest suspends, it can set
freeze_mode to be FREEZE_S3, so that virtio devices can change their
reset behavior on Qemu side according to that mode.

Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com>
---
 drivers/virtio/virtio.c                | 13 +++++++++++++
 drivers/virtio/virtio_pci_modern.c     |  9 +++++++++
 drivers/virtio/virtio_pci_modern_dev.c | 16 ++++++++++++++++
 include/linux/virtio_config.h          |  1 +
 include/linux/virtio_pci_modern.h      |  2 ++
 include/uapi/linux/virtio_pci.h        | 16 ++++++++++++++--
 6 files changed, 55 insertions(+), 2 deletions(-)

diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index 3893dc29eb26..b4eb8369d5a1 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -7,6 +7,7 @@
 #include <linux/idr.h>
 #include <linux/of.h>
 #include <uapi/linux/virtio_ids.h>
+#include <uapi/linux/virtio_pci.h>
 
 /* Unique numbering for virtio devices. */
 static DEFINE_IDA(virtio_index_ida);
@@ -486,10 +487,20 @@ void unregister_virtio_device(struct virtio_device *dev)
 EXPORT_SYMBOL_GPL(unregister_virtio_device);
 
 #ifdef CONFIG_PM_SLEEP
+static void virtio_set_freeze_mode(struct virtio_device *dev, u16 mode)
+{
+	if (!dev->config->set_freeze_mode)
+		return;
+	might_sleep();
+	dev->config->set_freeze_mode(dev, mode);
+}
+
 int virtio_device_freeze(struct virtio_device *dev)
 {
 	struct virtio_driver *drv = drv_to_virtio(dev->dev.driver);
 
+	virtio_set_freeze_mode(dev, VIRTIO_PCI_FREEZE_MODE_FREEZE_S3);
+
 	virtio_config_disable(dev);
 
 	dev->failed = dev->config->get_status(dev) & VIRTIO_CONFIG_S_FAILED;
@@ -544,6 +555,8 @@ int virtio_device_restore(struct virtio_device *dev)
 
 	virtio_config_enable(dev);
 
+	virtio_set_freeze_mode(dev, VIRTIO_PCI_FREEZE_MODE_UNFREEZE);
+
 	return 0;
 
 err:
diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c
index d6bb68ba84e5..846b70919cbd 100644
--- a/drivers/virtio/virtio_pci_modern.c
+++ b/drivers/virtio/virtio_pci_modern.c
@@ -491,6 +491,13 @@ static bool vp_get_shm_region(struct virtio_device *vdev,
 	return true;
 }
 
+static void vp_set_freeze_mode(struct virtio_device *vdev, u16 mode)
+{
+	struct virtio_pci_device *vp_dev = to_vp_device(vdev);
+
+	vp_modern_set_freeze_mode(&vp_dev->mdev, mode);
+}
+
 static const struct virtio_config_ops virtio_pci_config_nodev_ops = {
 	.get		= NULL,
 	.set		= NULL,
@@ -509,6 +516,7 @@ static const struct virtio_config_ops virtio_pci_config_nodev_ops = {
 	.get_shm_region  = vp_get_shm_region,
 	.disable_vq_and_reset = vp_modern_disable_vq_and_reset,
 	.enable_vq_after_reset = vp_modern_enable_vq_after_reset,
+	.set_freeze_mode = vp_set_freeze_mode,
 };
 
 static const struct virtio_config_ops virtio_pci_config_ops = {
@@ -529,6 +537,7 @@ static const struct virtio_config_ops virtio_pci_config_ops = {
 	.get_shm_region  = vp_get_shm_region,
 	.disable_vq_and_reset = vp_modern_disable_vq_and_reset,
 	.enable_vq_after_reset = vp_modern_enable_vq_after_reset,
+	.set_freeze_mode = vp_set_freeze_mode,
 };
 
 /* the PCI probing function */
diff --git a/drivers/virtio/virtio_pci_modern_dev.c b/drivers/virtio/virtio_pci_modern_dev.c
index aad7d9296e77..4a6f7d130b6e 100644
--- a/drivers/virtio/virtio_pci_modern_dev.c
+++ b/drivers/virtio/virtio_pci_modern_dev.c
@@ -203,6 +203,8 @@ static inline void check_offsets(void)
 		     offsetof(struct virtio_pci_common_cfg, queue_used_lo));
 	BUILD_BUG_ON(VIRTIO_PCI_COMMON_Q_USEDHI !=
 		     offsetof(struct virtio_pci_common_cfg, queue_used_hi));
+	BUILD_BUG_ON(VIRTIO_PCI_COMMON_F_MODE !=
+		     offsetof(struct virtio_pci_common_cfg, freeze_mode));
 }
 
 /*
@@ -714,6 +716,20 @@ void __iomem *vp_modern_map_vq_notify(struct virtio_pci_modern_device *mdev,
 }
 EXPORT_SYMBOL_GPL(vp_modern_map_vq_notify);
 
+/*
+ * vp_modern_set_freeze_mode - set freeze mode to device
+ * @mdev: the modern virtio-pci device
+ * @mode: the mode set to device
+ */
+void vp_modern_set_freeze_mode(struct virtio_pci_modern_device *mdev,
+				 u16 mode)
+{
+	struct virtio_pci_common_cfg __iomem *cfg = mdev->common;
+
+	vp_iowrite16(mode, &cfg->freeze_mode);
+}
+EXPORT_SYMBOL_GPL(vp_modern_set_freeze_mode);
+
 MODULE_VERSION("0.1");
 MODULE_DESCRIPTION("Modern Virtio PCI Device");
 MODULE_AUTHOR("Jason Wang <jasowang@redhat.com>");
diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
index 2b3438de2c4d..2a7443ff7f12 100644
--- a/include/linux/virtio_config.h
+++ b/include/linux/virtio_config.h
@@ -120,6 +120,7 @@ struct virtio_config_ops {
 			       struct virtio_shm_region *region, u8 id);
 	int (*disable_vq_and_reset)(struct virtqueue *vq);
 	int (*enable_vq_after_reset)(struct virtqueue *vq);
+	void (*set_freeze_mode)(struct virtio_device *vdev, u16 mode);
 };
 
 /* If driver didn't advertise the feature, it will never appear. */
diff --git a/include/linux/virtio_pci_modern.h b/include/linux/virtio_pci_modern.h
index 067ac1d789bc..ba6eed216ded 100644
--- a/include/linux/virtio_pci_modern.h
+++ b/include/linux/virtio_pci_modern.h
@@ -121,4 +121,6 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev);
 void vp_modern_remove(struct virtio_pci_modern_device *mdev);
 int vp_modern_get_queue_reset(struct virtio_pci_modern_device *mdev, u16 index);
 void vp_modern_set_queue_reset(struct virtio_pci_modern_device *mdev, u16 index);
+void vp_modern_set_freeze_mode(struct virtio_pci_modern_device *mdev,
+		   u16 mode);
 #endif
diff --git a/include/uapi/linux/virtio_pci.h b/include/uapi/linux/virtio_pci.h
index f703afc7ad31..725ace458a1b 100644
--- a/include/uapi/linux/virtio_pci.h
+++ b/include/uapi/linux/virtio_pci.h
@@ -140,6 +140,15 @@ struct virtio_pci_notify_cap {
 	__le32 notify_off_multiplier;	/* Multiplier for queue_notify_off. */
 };
 
+typedef enum {
+       VIRTIO_PCI_FREEZE_MODE_UNFREEZE = 0,
+       VIRTIO_PCI_FREEZE_MODE_FREEZE_S3 = 3,
+} virtio_pci_freeze_mode_t;
+
+#define VIRTIO_PCI_FREEZE_MODE_MASK \
+	((1 << VIRTIO_PCI_FREEZE_MODE_UNFREEZE) | \
+	(1 << VIRTIO_PCI_FREEZE_MODE_FREEZE_S3))
+
 /* Fields in VIRTIO_PCI_CAP_COMMON_CFG: */
 struct virtio_pci_common_cfg {
 	/* About the whole device. */
@@ -164,6 +173,8 @@ struct virtio_pci_common_cfg {
 	__le32 queue_avail_hi;		/* read-write */
 	__le32 queue_used_lo;		/* read-write */
 	__le32 queue_used_hi;		/* read-write */
+
+	__le16 freeze_mode;		/* read-write */
 };
 
 /* Fields in VIRTIO_PCI_CAP_PCI_CFG: */
@@ -202,8 +213,9 @@ struct virtio_pci_cfg_cap {
 #define VIRTIO_PCI_COMMON_Q_AVAILHI	44
 #define VIRTIO_PCI_COMMON_Q_USEDLO	48
 #define VIRTIO_PCI_COMMON_Q_USEDHI	52
-#define VIRTIO_PCI_COMMON_Q_NDATA	56
-#define VIRTIO_PCI_COMMON_Q_RESET	58
+#define VIRTIO_PCI_COMMON_F_MODE	56
+#define VIRTIO_PCI_COMMON_Q_NDATA	58
+#define VIRTIO_PCI_COMMON_Q_RESET	60
 
 #endif /* VIRTIO_PCI_NO_MODERN */
 
-- 
2.34.1

[LINUX KERNEL PATCH v5 2/2] virtgpu: Add freeze and restore func to reinit vqs

[Jiqian Chen]

When we suspended guest VM, it called into Qemu to call
virtio_reset->__virtio_queue_reset, this cleared all virtqueue information
of virtgpu on Qemu end, but guest kernel still keep the virtqueus. As a
result, after guest resumed, if guest sent ctrl/cursor requests to Qemu
through virtqueue, but now Qemu can't get requests from the virtqueue.
(Failed in function virtio_queue_notify, vq->vring.desc is NULL)

So, this patch add freeze and restore function for virtgpu driver. In
freeze function, it flushes all virtqueue works and deletes virtqueues. In
restore function, it re-initializes virtqueues. And then, Qemu and guest
can communicate normally.

Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com>
---
 drivers/gpu/drm/virtio/virtgpu_drv.c | 23 +++++++++++++++++++++
 drivers/gpu/drm/virtio/virtgpu_drv.h |  1 +
 drivers/gpu/drm/virtio/virtgpu_kms.c | 30 +++++++++++++++++++---------
 3 files changed, 45 insertions(+), 9 deletions(-)

diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.c b/drivers/gpu/drm/virtio/virtgpu_drv.c
index 644b8ee51009..8e751db129e4 100644
--- a/drivers/gpu/drm/virtio/virtgpu_drv.c
+++ b/drivers/gpu/drm/virtio/virtgpu_drv.c
@@ -130,6 +130,25 @@ static void virtio_gpu_config_changed(struct virtio_device *vdev)
 	schedule_work(&vgdev->config_changed_work);
 }
 
+#ifdef CONFIG_PM
+static int virtio_gpu_freeze(struct virtio_device *dev)
+{
+	struct drm_device *ddev = dev->priv;
+	struct virtio_gpu_device *vgdev = ddev->dev_private;
+
+	flush_work(&vgdev->ctrlq.dequeue_work);
+	flush_work(&vgdev->cursorq.dequeue_work);
+	vgdev->vdev->config->del_vqs(vgdev->vdev);
+
+	return 0;
+}
+
+static int virtio_gpu_restore(struct virtio_device *dev)
+{
+	return virtio_gpu_init_vqs(dev);
+}
+#endif
+
 static struct virtio_device_id id_table[] = {
 	{ VIRTIO_ID_GPU, VIRTIO_DEV_ANY_ID },
 	{ 0 },
@@ -156,6 +175,10 @@ static struct virtio_driver virtio_gpu_driver = {
 	.driver.owner = THIS_MODULE,
 	.id_table = id_table,
 	.probe = virtio_gpu_probe,
+#ifdef CONFIG_PM
+	.freeze = virtio_gpu_freeze,
+	.restore = virtio_gpu_restore,
+#endif
 	.remove = virtio_gpu_remove,
 	.config_changed = virtio_gpu_config_changed
 };
diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.h b/drivers/gpu/drm/virtio/virtgpu_drv.h
index 4126c384286b..d93dd53a947d 100644
--- a/drivers/gpu/drm/virtio/virtgpu_drv.h
+++ b/drivers/gpu/drm/virtio/virtgpu_drv.h
@@ -282,6 +282,7 @@ extern struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS];
 void virtio_gpu_create_context(struct drm_device *dev, struct drm_file *file);
 
 /* virtgpu_kms.c */
+int virtio_gpu_init_vqs(struct virtio_device *vdev);
 int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev);
 void virtio_gpu_deinit(struct drm_device *dev);
 void virtio_gpu_release(struct drm_device *dev);
diff --git a/drivers/gpu/drm/virtio/virtgpu_kms.c b/drivers/gpu/drm/virtio/virtgpu_kms.c
index 5a3b5aaed1f3..871b7ba98257 100644
--- a/drivers/gpu/drm/virtio/virtgpu_kms.c
+++ b/drivers/gpu/drm/virtio/virtgpu_kms.c
@@ -114,16 +114,33 @@ static void virtio_gpu_get_capsets(struct virtio_gpu_device *vgdev,
 	vgdev->num_capsets = num_capsets;
 }
 
-int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
+int virtio_gpu_init_vqs(struct virtio_device *vdev)
 {
 	static vq_callback_t *callbacks[] = {
 		virtio_gpu_ctrl_ack, virtio_gpu_cursor_ack
 	};
 	static const char * const names[] = { "control", "cursor" };
+	struct drm_device *dev = vdev->priv;
+	struct virtio_gpu_device *vgdev = dev->dev_private;
+	struct virtqueue *vqs[2];
+	int ret;
+
+	virtio_gpu_init_vq(&vgdev->ctrlq, virtio_gpu_dequeue_ctrl_func);
+	virtio_gpu_init_vq(&vgdev->cursorq, virtio_gpu_dequeue_cursor_func);
+
+	ret = virtio_find_vqs(vgdev->vdev, 2, vqs, callbacks, names, NULL);
+	if (ret) {
+		DRM_ERROR("failed to find virt queues\n");
+		return ret;
+	}
+	vgdev->ctrlq.vq = vqs[0];
+	vgdev->cursorq.vq = vqs[1];
+	return 0;
+}
 
+int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
+{
 	struct virtio_gpu_device *vgdev;
-	/* this will expand later */
-	struct virtqueue *vqs[2];
 	u32 num_scanouts, num_capsets;
 	int ret = 0;
 
@@ -144,8 +161,6 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
 	ida_init(&vgdev->ctx_id_ida);
 	ida_init(&vgdev->resource_ida);
 	init_waitqueue_head(&vgdev->resp_wq);
-	virtio_gpu_init_vq(&vgdev->ctrlq, virtio_gpu_dequeue_ctrl_func);
-	virtio_gpu_init_vq(&vgdev->cursorq, virtio_gpu_dequeue_cursor_func);
 
 	vgdev->fence_drv.context = dma_fence_context_alloc(1);
 	spin_lock_init(&vgdev->fence_drv.lock);
@@ -207,13 +222,10 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
 	DRM_INFO("features: %ccontext_init\n",
 		 vgdev->has_context_init ? '+' : '-');
 
-	ret = virtio_find_vqs(vgdev->vdev, 2, vqs, callbacks, names, NULL);
+	ret = virtio_gpu_init_vqs(vdev);
 	if (ret) {
-		DRM_ERROR("failed to find virt queues\n");
 		goto err_vqs;
 	}
-	vgdev->ctrlq.vq = vqs[0];
-	vgdev->cursorq.vq = vqs[1];
 	ret = virtio_gpu_alloc_vbufs(vgdev);
 	if (ret) {
 		DRM_ERROR("failed to alloc vbufs\n");
-- 
2.34.1

Re: [LINUX KERNEL PATCH v5 1/2] virtio_pci: Add freeze_mode for virtio_pci_common_cfg

[Michael S. Tsirkin]

On Tue, Sep 19, 2023 at 06:46:06PM +0800, Jiqian Chen wrote:
> When guest vm does S3, Qemu will reset and clear some things of virtio
> devices, but guest can't aware that, so that may cause some problems.
> For excample, Qemu calls virtio_reset->virtio_gpu_gl_reset, that will
> destroy render resources of virtio-gpu. As a result, after guest resume,
> the display can't come back and we only saw a black screen. Due to guest
> can't re-create all the resources, so we need to let Qemu not to destroy
> them when S3.
> 
> For above purpose, this patch add a new parameter named freeze_mode to
> struct virtio_pci_common_cfg, and when guest suspends, it can set
> freeze_mode to be FREEZE_S3, so that virtio devices can change their
> reset behavior on Qemu side according to that mode.
> 
> Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com>
> ---
>  drivers/virtio/virtio.c                | 13 +++++++++++++
>  drivers/virtio/virtio_pci_modern.c     |  9 +++++++++
>  drivers/virtio/virtio_pci_modern_dev.c | 16 ++++++++++++++++
>  include/linux/virtio_config.h          |  1 +
>  include/linux/virtio_pci_modern.h      |  2 ++
>  include/uapi/linux/virtio_pci.h        | 16 ++++++++++++++--
>  6 files changed, 55 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
> index 3893dc29eb26..b4eb8369d5a1 100644
> --- a/drivers/virtio/virtio.c
> +++ b/drivers/virtio/virtio.c
> @@ -7,6 +7,7 @@
>  #include <linux/idr.h>
>  #include <linux/of.h>
>  #include <uapi/linux/virtio_ids.h>
> +#include <uapi/linux/virtio_pci.h>
>  
>  /* Unique numbering for virtio devices. */
>  static DEFINE_IDA(virtio_index_ida);
> @@ -486,10 +487,20 @@ void unregister_virtio_device(struct virtio_device *dev)
>  EXPORT_SYMBOL_GPL(unregister_virtio_device);
>  
>  #ifdef CONFIG_PM_SLEEP
> +static void virtio_set_freeze_mode(struct virtio_device *dev, u16 mode)
> +{
> +	if (!dev->config->set_freeze_mode)
> +		return;
> +	might_sleep();
> +	dev->config->set_freeze_mode(dev, mode);
> +}
> +
>  int virtio_device_freeze(struct virtio_device *dev)
>  {
>  	struct virtio_driver *drv = drv_to_virtio(dev->dev.driver);
>  
> +	virtio_set_freeze_mode(dev, VIRTIO_PCI_FREEZE_MODE_FREEZE_S3);
> +
>  	virtio_config_disable(dev);
>  
>  	dev->failed = dev->config->get_status(dev) & VIRTIO_CONFIG_S_FAILED;
> @@ -544,6 +555,8 @@ int virtio_device_restore(struct virtio_device *dev)
>  
>  	virtio_config_enable(dev);
>  
> +	virtio_set_freeze_mode(dev, VIRTIO_PCI_FREEZE_MODE_UNFREEZE);
> +
>  	return 0;
>  
>  err:
> diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c
> index d6bb68ba84e5..846b70919cbd 100644
> --- a/drivers/virtio/virtio_pci_modern.c
> +++ b/drivers/virtio/virtio_pci_modern.c
> @@ -491,6 +491,13 @@ static bool vp_get_shm_region(struct virtio_device *vdev,
>  	return true;
>  }
>  
> +static void vp_set_freeze_mode(struct virtio_device *vdev, u16 mode)
> +{
> +	struct virtio_pci_device *vp_dev = to_vp_device(vdev);
> +
> +	vp_modern_set_freeze_mode(&vp_dev->mdev, mode);
> +}
> +
>  static const struct virtio_config_ops virtio_pci_config_nodev_ops = {
>  	.get		= NULL,
>  	.set		= NULL,
> @@ -509,6 +516,7 @@ static const struct virtio_config_ops virtio_pci_config_nodev_ops = {
>  	.get_shm_region  = vp_get_shm_region,
>  	.disable_vq_and_reset = vp_modern_disable_vq_and_reset,
>  	.enable_vq_after_reset = vp_modern_enable_vq_after_reset,
> +	.set_freeze_mode = vp_set_freeze_mode,
>  };
>  
>  static const struct virtio_config_ops virtio_pci_config_ops = {
> @@ -529,6 +537,7 @@ static const struct virtio_config_ops virtio_pci_config_ops = {
>  	.get_shm_region  = vp_get_shm_region,
>  	.disable_vq_and_reset = vp_modern_disable_vq_and_reset,
>  	.enable_vq_after_reset = vp_modern_enable_vq_after_reset,
> +	.set_freeze_mode = vp_set_freeze_mode,
>  };
>  
>  /* the PCI probing function */
> diff --git a/drivers/virtio/virtio_pci_modern_dev.c b/drivers/virtio/virtio_pci_modern_dev.c
> index aad7d9296e77..4a6f7d130b6e 100644
> --- a/drivers/virtio/virtio_pci_modern_dev.c
> +++ b/drivers/virtio/virtio_pci_modern_dev.c
> @@ -203,6 +203,8 @@ static inline void check_offsets(void)
>  		     offsetof(struct virtio_pci_common_cfg, queue_used_lo));
>  	BUILD_BUG_ON(VIRTIO_PCI_COMMON_Q_USEDHI !=
>  		     offsetof(struct virtio_pci_common_cfg, queue_used_hi));
> +	BUILD_BUG_ON(VIRTIO_PCI_COMMON_F_MODE !=
> +		     offsetof(struct virtio_pci_common_cfg, freeze_mode));
>  }
>  
>  /*
> @@ -714,6 +716,20 @@ void __iomem *vp_modern_map_vq_notify(struct virtio_pci_modern_device *mdev,
>  }
>  EXPORT_SYMBOL_GPL(vp_modern_map_vq_notify);
>  
> +/*
> + * vp_modern_set_freeze_mode - set freeze mode to device
> + * @mdev: the modern virtio-pci device
> + * @mode: the mode set to device
> + */
> +void vp_modern_set_freeze_mode(struct virtio_pci_modern_device *mdev,
> +				 u16 mode)
> +{
> +	struct virtio_pci_common_cfg __iomem *cfg = mdev->common;
> +
> +	vp_iowrite16(mode, &cfg->freeze_mode);
> +}
> +EXPORT_SYMBOL_GPL(vp_modern_set_freeze_mode);
> +
>  MODULE_VERSION("0.1");
>  MODULE_DESCRIPTION("Modern Virtio PCI Device");
>  MODULE_AUTHOR("Jason Wang <jasowang@redhat.com>");
> diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
> index 2b3438de2c4d..2a7443ff7f12 100644
> --- a/include/linux/virtio_config.h
> +++ b/include/linux/virtio_config.h
> @@ -120,6 +120,7 @@ struct virtio_config_ops {
>  			       struct virtio_shm_region *region, u8 id);
>  	int (*disable_vq_and_reset)(struct virtqueue *vq);
>  	int (*enable_vq_after_reset)(struct virtqueue *vq);
> +	void (*set_freeze_mode)(struct virtio_device *vdev, u16 mode);
>  };
>  
>  /* If driver didn't advertise the feature, it will never appear. */
> diff --git a/include/linux/virtio_pci_modern.h b/include/linux/virtio_pci_modern.h
> index 067ac1d789bc..ba6eed216ded 100644
> --- a/include/linux/virtio_pci_modern.h
> +++ b/include/linux/virtio_pci_modern.h
> @@ -121,4 +121,6 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev);
>  void vp_modern_remove(struct virtio_pci_modern_device *mdev);
>  int vp_modern_get_queue_reset(struct virtio_pci_modern_device *mdev, u16 index);
>  void vp_modern_set_queue_reset(struct virtio_pci_modern_device *mdev, u16 index);
> +void vp_modern_set_freeze_mode(struct virtio_pci_modern_device *mdev,
> +		   u16 mode);
>  #endif
> diff --git a/include/uapi/linux/virtio_pci.h b/include/uapi/linux/virtio_pci.h
> index f703afc7ad31..725ace458a1b 100644
> --- a/include/uapi/linux/virtio_pci.h
> +++ b/include/uapi/linux/virtio_pci.h
> @@ -140,6 +140,15 @@ struct virtio_pci_notify_cap {
>  	__le32 notify_off_multiplier;	/* Multiplier for queue_notify_off. */
>  };
>  
> +typedef enum {
> +       VIRTIO_PCI_FREEZE_MODE_UNFREEZE = 0,
> +       VIRTIO_PCI_FREEZE_MODE_FREEZE_S3 = 3,
> +} virtio_pci_freeze_mode_t;

we don't normally do typedefs.

> +
> +#define VIRTIO_PCI_FREEZE_MODE_MASK \
> +	((1 << VIRTIO_PCI_FREEZE_MODE_UNFREEZE) | \
> +	(1 << VIRTIO_PCI_FREEZE_MODE_FREEZE_S3))
> +

not sure why is this useful generally.

>  /* Fields in VIRTIO_PCI_CAP_COMMON_CFG: */
>  struct virtio_pci_common_cfg {
>  	/* About the whole device. */
> @@ -164,6 +173,8 @@ struct virtio_pci_common_cfg {
>  	__le32 queue_avail_hi;		/* read-write */
>  	__le32 queue_used_lo;		/* read-write */
>  	__le32 queue_used_hi;		/* read-write */
> +
> +	__le16 freeze_mode;		/* read-write */
>  };
>

Your patch will likely break uses of sizeof(struct virtio_pci_common_cfg)
on existing devices.
  
>  /* Fields in VIRTIO_PCI_CAP_PCI_CFG: */
> @@ -202,8 +213,9 @@ struct virtio_pci_cfg_cap {
>  #define VIRTIO_PCI_COMMON_Q_AVAILHI	44
>  #define VIRTIO_PCI_COMMON_Q_USEDLO	48
>  #define VIRTIO_PCI_COMMON_Q_USEDHI	52
> -#define VIRTIO_PCI_COMMON_Q_NDATA	56
> -#define VIRTIO_PCI_COMMON_Q_RESET	58
> +#define VIRTIO_PCI_COMMON_F_MODE	56


F_ here stands for freeze? Please don't abbreviate.
Q for queue is a pun that works, F for freeze doesn't.


> +#define VIRTIO_PCI_COMMON_Q_NDATA	58
> +#define VIRTIO_PCI_COMMON_Q_RESET	60
>  
>  #endif /* VIRTIO_PCI_NO_MODERN */
>  
> -- 
> 2.34.1