[PATCH V3 0/9] rework on the IRQ hardening of virtio

[PATCH V3 0/9] rework on the IRQ hardening of virtio

[Jason Wang]

Hi All:

This is a rework on the IRQ hardening for virtio which is done
previously by the following commits are reverted:

9e35276a5344 ("virtio_pci: harden MSI-X interrupts")
080cd7c3ac87 ("virtio-pci: harden INTX interrupts")

The reason is that it depends on the IRQF_NO_AUTOEN which may conflict
with the assumption of the affinity managed IRQ that is used by some
virtio drivers. And what's more, it is only done for virtio-pci but
not other transports.

In this rework, I try to implement a general virtio solution which
borrows the idea of the INTX hardening by re-using per virtqueue
boolean vq->broken and toggle it in virtio_device_ready() and
virtio_reset_device(). Then we can simply reuse the existing checks in
the vring_interrupt() and return early if the driver is not ready.

Note that, I only did compile test on ccw and MMIO transport.

Please review.

Changes since v1:

- Use transport specific irq synchronization method when possible
- Drop the module parameter and enable the hardening unconditonally
- Tweak the barrier/ordering facilities used in the code
- Reanme irq_soft_enabled to driver_ready
- Avoid unnecssary IRQ synchornization (e.g during boot)

Changes since V2:

- add ccw and MMIO support
- rename synchronize_vqs() to synchronize_cbs()
- switch to re-use vq->broken instead of introducing new device
  attributes for the future virtqueue reset support
- remove unnecssary READ_ONCE()/WRITE_ONCE()
- a new patch to remove device triggerable BUG_ON()
- more tweaks on the comments

Jason Wang (8):
  virtio: use virtio_reset_device() when possible
  virtio: introduce config op to synchronize vring callbacks
  virtio-pci: implement synchronize_cbs()
  virtio-mmio: implement synchronize_cbs()
  virtio-ccw: implement synchronize_cbs()
  virtio: allow to unbreak virtqueue
  virtio: harden vring IRQ
  virtio: use WARN_ON() to warning illegal status value

Stefano Garzarella (1):
  virtio: use virtio_device_ready() in virtio_device_restore()

 drivers/char/virtio_console.c              |  2 +-
 drivers/crypto/virtio/virtio_crypto_core.c |  2 +-
 drivers/s390/virtio/virtio_ccw.c           | 31 ++++++++++++++++--
 drivers/virtio/virtio.c                    | 24 ++++++++++----
 drivers/virtio/virtio_mmio.c               |  9 +++++
 drivers/virtio/virtio_pci_common.c         |  2 +-
 drivers/virtio/virtio_pci_common.h         |  2 ++
 drivers/virtio/virtio_pci_legacy.c         |  1 +
 drivers/virtio/virtio_pci_modern.c         |  2 ++
 drivers/virtio/virtio_ring.c               | 15 +++++----
 include/linux/virtio.h                     |  2 +-
 include/linux/virtio_config.h              | 38 +++++++++++++++++++++-
 12 files changed, 110 insertions(+), 20 deletions(-)

-- 
2.25.1

[PATCH V3 0/9] rework on the IRQ hardening of virtio

[Jason Wang]

Hi All:

This is a rework on the IRQ hardening for virtio which is done
previously by the following commits are reverted:

9e35276a5344 ("virtio_pci: harden MSI-X interrupts")
080cd7c3ac87 ("virtio-pci: harden INTX interrupts")

The reason is that it depends on the IRQF_NO_AUTOEN which may conflict
with the assumption of the affinity managed IRQ that is used by some
virtio drivers. And what's more, it is only done for virtio-pci but
not other transports.

In this rework, I try to implement a general virtio solution which
borrows the idea of the INTX hardening by re-using per virtqueue
boolean vq->broken and toggle it in virtio_device_ready() and
virtio_reset_device(). Then we can simply reuse the existing checks in
the vring_interrupt() and return early if the driver is not ready.

Note that, I only did compile test on ccw and MMIO transport.

Please review.

Changes since v1:

- Use transport specific irq synchronization method when possible
- Drop the module parameter and enable the hardening unconditonally
- Tweak the barrier/ordering facilities used in the code
- Reanme irq_soft_enabled to driver_ready
- Avoid unnecssary IRQ synchornization (e.g during boot)

Changes since V2:

- add ccw and MMIO support
- rename synchronize_vqs() to synchronize_cbs()
- switch to re-use vq->broken instead of introducing new device
  attributes for the future virtqueue reset support
- remove unnecssary READ_ONCE()/WRITE_ONCE()
- a new patch to remove device triggerable BUG_ON()
- more tweaks on the comments

Jason Wang (8):
  virtio: use virtio_reset_device() when possible
  virtio: introduce config op to synchronize vring callbacks
  virtio-pci: implement synchronize_cbs()
  virtio-mmio: implement synchronize_cbs()
  virtio-ccw: implement synchronize_cbs()
  virtio: allow to unbreak virtqueue
  virtio: harden vring IRQ
  virtio: use WARN_ON() to warning illegal status value

Stefano Garzarella (1):
  virtio: use virtio_device_ready() in virtio_device_restore()

 drivers/char/virtio_console.c              |  2 +-
 drivers/crypto/virtio/virtio_crypto_core.c |  2 +-
 drivers/s390/virtio/virtio_ccw.c           | 31 ++++++++++++++++--
 drivers/virtio/virtio.c                    | 24 ++++++++++----
 drivers/virtio/virtio_mmio.c               |  9 +++++
 drivers/virtio/virtio_pci_common.c         |  2 +-
 drivers/virtio/virtio_pci_common.h         |  2 ++
 drivers/virtio/virtio_pci_legacy.c         |  1 +
 drivers/virtio/virtio_pci_modern.c         |  2 ++
 drivers/virtio/virtio_ring.c               | 15 +++++----
 include/linux/virtio.h                     |  2 +-
 include/linux/virtio_config.h              | 38 +++++++++++++++++++++-
 12 files changed, 110 insertions(+), 20 deletions(-)

-- 
2.25.1


_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH V3 1/9] virtio: use virtio_device_ready() in virtio_device_restore()

[Jason Wang]

From: Stefano Garzarella <sgarzare@redhat.com>

It will allow us to do extension on virtio_device_ready() without
duplicating code.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/virtio/virtio.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index 22f15f444f75..75c8d560bbd3 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -526,8 +526,9 @@ int virtio_device_restore(struct virtio_device *dev)
 			goto err;
 	}
 
-	/* Finally, tell the device we're all set */
-	virtio_add_status(dev, VIRTIO_CONFIG_S_DRIVER_OK);
+	/* If restore didn't do it, mark device DRIVER_OK ourselves. */
+	if (!(dev->config->get_status(dev) & VIRTIO_CONFIG_S_DRIVER_OK))
+		virtio_device_ready(dev);
 
 	virtio_config_enable(dev);
 
-- 
2.25.1

[PATCH V3 1/9] virtio: use virtio_device_ready() in virtio_device_restore()

[Jason Wang]

From: Stefano Garzarella <sgarzare@redhat.com>

It will allow us to do extension on virtio_device_ready() without
duplicating code.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/virtio/virtio.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index 22f15f444f75..75c8d560bbd3 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -526,8 +526,9 @@ int virtio_device_restore(struct virtio_device *dev)
 			goto err;
 	}
 
-	/* Finally, tell the device we're all set */
-	virtio_add_status(dev, VIRTIO_CONFIG_S_DRIVER_OK);
+	/* If restore didn't do it, mark device DRIVER_OK ourselves. */
+	if (!(dev->config->get_status(dev) & VIRTIO_CONFIG_S_DRIVER_OK))
+		virtio_device_ready(dev);
 
 	virtio_config_enable(dev);
 
-- 
2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 1/9] virtio: use virtio_device_ready() in virtio_device_restore()

[Cornelia Huck]

On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:

> From: Stefano Garzarella <sgarzare@redhat.com>
>
> It will allow us to do extension on virtio_device_ready() without
> duplicating code.
>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Halil Pasic <pasic@linux.ibm.com>
> Cc: Cornelia Huck <cohuck@redhat.com>
> Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  drivers/virtio/virtio.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [PATCH V3 1/9] virtio: use virtio_device_ready() in virtio_device_restore()

[Cornelia Huck]

On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:

> From: Stefano Garzarella <sgarzare@redhat.com>
>
> It will allow us to do extension on virtio_device_ready() without
> duplicating code.
>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Halil Pasic <pasic@linux.ibm.com>
> Cc: Cornelia Huck <cohuck@redhat.com>
> Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  drivers/virtio/virtio.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH V3 2/9] virtio: use virtio_reset_device() when possible

[Jason Wang]

This allows us to do common extension without duplicating code.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/virtio/virtio.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index 75c8d560bbd3..8dde44ea044a 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -430,7 +430,7 @@ int register_virtio_device(struct virtio_device *dev)
 
 	/* We always start by resetting the device, in case a previous
 	 * driver messed it up.  This also tests that code path a little. */
-	dev->config->reset(dev);
+	virtio_reset_device(dev);
 
 	/* Acknowledge that we've seen the device. */
 	virtio_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE);
@@ -496,7 +496,7 @@ int virtio_device_restore(struct virtio_device *dev)
 
 	/* We always start by resetting the device, in case a previous
 	 * driver messed it up. */
-	dev->config->reset(dev);
+	virtio_reset_device(dev);
 
 	/* Acknowledge that we've seen the device. */
 	virtio_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE);
-- 
2.25.1

[PATCH V3 2/9] virtio: use virtio_reset_device() when possible

[Jason Wang]

This allows us to do common extension without duplicating code.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/virtio/virtio.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index 75c8d560bbd3..8dde44ea044a 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -430,7 +430,7 @@ int register_virtio_device(struct virtio_device *dev)
 
 	/* We always start by resetting the device, in case a previous
 	 * driver messed it up.  This also tests that code path a little. */
-	dev->config->reset(dev);
+	virtio_reset_device(dev);
 
 	/* Acknowledge that we've seen the device. */
 	virtio_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE);
@@ -496,7 +496,7 @@ int virtio_device_restore(struct virtio_device *dev)
 
 	/* We always start by resetting the device, in case a previous
 	 * driver messed it up. */
-	dev->config->reset(dev);
+	virtio_reset_device(dev);
 
 	/* Acknowledge that we've seen the device. */
 	virtio_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE);
-- 
2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 2/9] virtio: use virtio_reset_device() when possible

[Cornelia Huck]

On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:

> This allows us to do common extension without duplicating code.
>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Halil Pasic <pasic@linux.ibm.com>
> Cc: Cornelia Huck <cohuck@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  drivers/virtio/virtio.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 2/9] virtio: use virtio_reset_device() when possible

[Cornelia Huck]

On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:

> This allows us to do common extension without duplicating code.
>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Halil Pasic <pasic@linux.ibm.com>
> Cc: Cornelia Huck <cohuck@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  drivers/virtio/virtio.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

[PATCH V3 3/9] virtio: introduce config op to synchronize vring callbacks

[Jason Wang]

This patch introduces new virtio config op to vring
callbacks. Transport specific method is required to make sure the
write before this function is visible to the vring_interrupt() that is
called after the return of this function. For the transport that
doesn't provide synchronize_vqs(), use synchornize_rcu() which
synchronize with IRQ implicitly as a fallback.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 include/linux/virtio_config.h | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
index b341dd62aa4d..14fe89ff99c7 100644
--- a/include/linux/virtio_config.h
+++ b/include/linux/virtio_config.h
@@ -57,6 +57,10 @@ struct virtio_shm_region {
  *		include a NULL entry for vqs unused by driver
  *	Returns 0 on success or error status
  * @del_vqs: free virtqueues found by find_vqs().
+ * @synchronize_cbs: synchronize with the virtqueue callbacks (optional)
+ *      Make sure the writes commited before this method is visible to
+ *      vring_interrupt() which is called after this method.
+ *      vdev: the virtio_device
  * @get_features: get the array of feature bits for this device.
  *	vdev: the virtio_device
  *	Returns the first 64 feature bits (all we currently need).
@@ -89,6 +93,7 @@ struct virtio_config_ops {
 			const char * const names[], const bool *ctx,
 			struct irq_affinity *desc);
 	void (*del_vqs)(struct virtio_device *);
+	void (*synchronize_cbs)(struct virtio_device *);
 	u64 (*get_features)(struct virtio_device *vdev);
 	int (*finalize_features)(struct virtio_device *vdev);
 	const char *(*bus_name)(struct virtio_device *vdev);
@@ -217,6 +222,25 @@ int virtio_find_vqs_ctx(struct virtio_device *vdev, unsigned nvqs,
 				      desc);
 }
 
+/**
+ * virtio_synchronize_cbs - synchronize with virtqueue callbacks
+ * @vdev: the device
+ */
+static inline
+void virtio_synchronize_cbs(struct virtio_device *dev)
+{
+	if (dev->config->synchronize_cbs) {
+		dev->config->synchronize_cbs(dev);
+	} else {
+		/*
+		 * A best effort fallback to synchronize with
+		 * interrupts, preemption and softirq. See comment
+		 * above synchronize_rcu().
+		 */
+		synchronize_rcu();
+	}
+}
+
 /**
  * virtio_device_ready - enable vq use in probe function
  * @vdev: the device
-- 
2.25.1

[PATCH V3 3/9] virtio: introduce config op to synchronize vring callbacks

[Jason Wang]

This patch introduces new virtio config op to vring
callbacks. Transport specific method is required to make sure the
write before this function is visible to the vring_interrupt() that is
called after the return of this function. For the transport that
doesn't provide synchronize_vqs(), use synchornize_rcu() which
synchronize with IRQ implicitly as a fallback.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 include/linux/virtio_config.h | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
index b341dd62aa4d..14fe89ff99c7 100644
--- a/include/linux/virtio_config.h
+++ b/include/linux/virtio_config.h
@@ -57,6 +57,10 @@ struct virtio_shm_region {
  *		include a NULL entry for vqs unused by driver
  *	Returns 0 on success or error status
  * @del_vqs: free virtqueues found by find_vqs().
+ * @synchronize_cbs: synchronize with the virtqueue callbacks (optional)
+ *      Make sure the writes commited before this method is visible to
+ *      vring_interrupt() which is called after this method.
+ *      vdev: the virtio_device
  * @get_features: get the array of feature bits for this device.
  *	vdev: the virtio_device
  *	Returns the first 64 feature bits (all we currently need).
@@ -89,6 +93,7 @@ struct virtio_config_ops {
 			const char * const names[], const bool *ctx,
 			struct irq_affinity *desc);
 	void (*del_vqs)(struct virtio_device *);
+	void (*synchronize_cbs)(struct virtio_device *);
 	u64 (*get_features)(struct virtio_device *vdev);
 	int (*finalize_features)(struct virtio_device *vdev);
 	const char *(*bus_name)(struct virtio_device *vdev);
@@ -217,6 +222,25 @@ int virtio_find_vqs_ctx(struct virtio_device *vdev, unsigned nvqs,
 				      desc);
 }
 
+/**
+ * virtio_synchronize_cbs - synchronize with virtqueue callbacks
+ * @vdev: the device
+ */
+static inline
+void virtio_synchronize_cbs(struct virtio_device *dev)
+{
+	if (dev->config->synchronize_cbs) {
+		dev->config->synchronize_cbs(dev);
+	} else {
+		/*
+		 * A best effort fallback to synchronize with
+		 * interrupts, preemption and softirq. See comment
+		 * above synchronize_rcu().
+		 */
+		synchronize_rcu();
+	}
+}
+
 /**
  * virtio_device_ready - enable vq use in probe function
  * @vdev: the device
-- 
2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 3/9] virtio: introduce config op to synchronize vring callbacks

[Cornelia Huck]

On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:

> This patch introduces new virtio config op to vring
> callbacks. Transport specific method is required to make sure the
> write before this function is visible to the vring_interrupt() that is

Which kind of writes? I.e., what is the scope?

> called after the return of this function. For the transport that
> doesn't provide synchronize_vqs(), use synchornize_rcu() which

Typo: synchronize_rcu()

> synchronize with IRQ implicitly as a fallback.
>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Halil Pasic <pasic@linux.ibm.com>
> Cc: Cornelia Huck <cohuck@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  include/linux/virtio_config.h | 24 ++++++++++++++++++++++++
>  1 file changed, 24 insertions(+)
>
> diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
> index b341dd62aa4d..14fe89ff99c7 100644
> --- a/include/linux/virtio_config.h
> +++ b/include/linux/virtio_config.h
> @@ -57,6 +57,10 @@ struct virtio_shm_region {
>   *		include a NULL entry for vqs unused by driver
>   *	Returns 0 on success or error status
>   * @del_vqs: free virtqueues found by find_vqs().
> + * @synchronize_cbs: synchronize with the virtqueue callbacks (optional)
> + *      Make sure the writes commited before this method is visible to
> + *      vring_interrupt() which is called after this method.

Same here, I think the description needs to be a bit more explicit about
which writes we care about here.

> + *      vdev: the virtio_device
>   * @get_features: get the array of feature bits for this device.
>   *	vdev: the virtio_device
>   *	Returns the first 64 feature bits (all we currently need).
> @@ -89,6 +93,7 @@ struct virtio_config_ops {
>  			const char * const names[], const bool *ctx,
>  			struct irq_affinity *desc);
>  	void (*del_vqs)(struct virtio_device *);
> +	void (*synchronize_cbs)(struct virtio_device *);
>  	u64 (*get_features)(struct virtio_device *vdev);
>  	int (*finalize_features)(struct virtio_device *vdev);
>  	const char *(*bus_name)(struct virtio_device *vdev);

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 3/9] virtio: introduce config op to synchronize vring callbacks

[Cornelia Huck]

On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:

> This patch introduces new virtio config op to vring
> callbacks. Transport specific method is required to make sure the
> write before this function is visible to the vring_interrupt() that is

Which kind of writes? I.e., what is the scope?

> called after the return of this function. For the transport that
> doesn't provide synchronize_vqs(), use synchornize_rcu() which

Typo: synchronize_rcu()

> synchronize with IRQ implicitly as a fallback.
>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Halil Pasic <pasic@linux.ibm.com>
> Cc: Cornelia Huck <cohuck@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  include/linux/virtio_config.h | 24 ++++++++++++++++++++++++
>  1 file changed, 24 insertions(+)
>
> diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
> index b341dd62aa4d..14fe89ff99c7 100644
> --- a/include/linux/virtio_config.h
> +++ b/include/linux/virtio_config.h
> @@ -57,6 +57,10 @@ struct virtio_shm_region {
>   *		include a NULL entry for vqs unused by driver
>   *	Returns 0 on success or error status
>   * @del_vqs: free virtqueues found by find_vqs().
> + * @synchronize_cbs: synchronize with the virtqueue callbacks (optional)
> + *      Make sure the writes commited before this method is visible to
> + *      vring_interrupt() which is called after this method.

Same here, I think the description needs to be a bit more explicit about
which writes we care about here.

> + *      vdev: the virtio_device
>   * @get_features: get the array of feature bits for this device.
>   *	vdev: the virtio_device
>   *	Returns the first 64 feature bits (all we currently need).
> @@ -89,6 +93,7 @@ struct virtio_config_ops {
>  			const char * const names[], const bool *ctx,
>  			struct irq_affinity *desc);
>  	void (*del_vqs)(struct virtio_device *);
> +	void (*synchronize_cbs)(struct virtio_device *);
>  	u64 (*get_features)(struct virtio_device *vdev);
>  	int (*finalize_features)(struct virtio_device *vdev);
>  	const char *(*bus_name)(struct virtio_device *vdev);

Re: [PATCH V3 3/9] virtio: introduce config op to synchronize vring callbacks

[Jason Wang]

On Thu, Apr 28, 2022 at 5:13 PM Cornelia Huck <cohuck@redhat.com> wrote:
>
> On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:
>
> > This patch introduces new virtio config op to vring
> > callbacks. Transport specific method is required to make sure the
> > write before this function is visible to the vring_interrupt() that is
>
> Which kind of writes? I.e., what is the scope?

Any writes before synchronize_cbs(). Is something like the following better?

The function guarantees that all memory operations before it are
visible to the vring_interrupt() that is called after it.

>
> > called after the return of this function. For the transport that
> > doesn't provide synchronize_vqs(), use synchornize_rcu() which
>
> Typo: synchronize_rcu()

Will fix it.

Thanks

>
> > synchronize with IRQ implicitly as a fallback.
> >
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > Cc: Marc Zyngier <maz@kernel.org>
> > Cc: Halil Pasic <pasic@linux.ibm.com>
> > Cc: Cornelia Huck <cohuck@redhat.com>
> > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > ---
> >  include/linux/virtio_config.h | 24 ++++++++++++++++++++++++
> >  1 file changed, 24 insertions(+)
> >
> > diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
> > index b341dd62aa4d..14fe89ff99c7 100644
> > --- a/include/linux/virtio_config.h
> > +++ b/include/linux/virtio_config.h
> > @@ -57,6 +57,10 @@ struct virtio_shm_region {
> >   *           include a NULL entry for vqs unused by driver
> >   *   Returns 0 on success or error status
> >   * @del_vqs: free virtqueues found by find_vqs().
> > + * @synchronize_cbs: synchronize with the virtqueue callbacks (optional)
> > + *      Make sure the writes commited before this method is visible to
> > + *      vring_interrupt() which is called after this method.
>
> Same here, I think the description needs to be a bit more explicit about
> which writes we care about here.
>
> > + *      vdev: the virtio_device
> >   * @get_features: get the array of feature bits for this device.
> >   *   vdev: the virtio_device
> >   *   Returns the first 64 feature bits (all we currently need).
> > @@ -89,6 +93,7 @@ struct virtio_config_ops {
> >                       const char * const names[], const bool *ctx,
> >                       struct irq_affinity *desc);
> >       void (*del_vqs)(struct virtio_device *);
> > +     void (*synchronize_cbs)(struct virtio_device *);
> >       u64 (*get_features)(struct virtio_device *vdev);
> >       int (*finalize_features)(struct virtio_device *vdev);
> >       const char *(*bus_name)(struct virtio_device *vdev);
>

Re: [PATCH V3 3/9] virtio: introduce config op to synchronize vring callbacks

[Jason Wang]

On Thu, Apr 28, 2022 at 5:13 PM Cornelia Huck <cohuck@redhat.com> wrote:
>
> On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:
>
> > This patch introduces new virtio config op to vring
> > callbacks. Transport specific method is required to make sure the
> > write before this function is visible to the vring_interrupt() that is
>
> Which kind of writes? I.e., what is the scope?

Any writes before synchronize_cbs(). Is something like the following better?

The function guarantees that all memory operations before it are
visible to the vring_interrupt() that is called after it.

>
> > called after the return of this function. For the transport that
> > doesn't provide synchronize_vqs(), use synchornize_rcu() which
>
> Typo: synchronize_rcu()

Will fix it.

Thanks

>
> > synchronize with IRQ implicitly as a fallback.
> >
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > Cc: Marc Zyngier <maz@kernel.org>
> > Cc: Halil Pasic <pasic@linux.ibm.com>
> > Cc: Cornelia Huck <cohuck@redhat.com>
> > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > ---
> >  include/linux/virtio_config.h | 24 ++++++++++++++++++++++++
> >  1 file changed, 24 insertions(+)
> >
> > diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
> > index b341dd62aa4d..14fe89ff99c7 100644
> > --- a/include/linux/virtio_config.h
> > +++ b/include/linux/virtio_config.h
> > @@ -57,6 +57,10 @@ struct virtio_shm_region {
> >   *           include a NULL entry for vqs unused by driver
> >   *   Returns 0 on success or error status
> >   * @del_vqs: free virtqueues found by find_vqs().
> > + * @synchronize_cbs: synchronize with the virtqueue callbacks (optional)
> > + *      Make sure the writes commited before this method is visible to
> > + *      vring_interrupt() which is called after this method.
>
> Same here, I think the description needs to be a bit more explicit about
> which writes we care about here.
>
> > + *      vdev: the virtio_device
> >   * @get_features: get the array of feature bits for this device.
> >   *   vdev: the virtio_device
> >   *   Returns the first 64 feature bits (all we currently need).
> > @@ -89,6 +93,7 @@ struct virtio_config_ops {
> >                       const char * const names[], const bool *ctx,
> >                       struct irq_affinity *desc);
> >       void (*del_vqs)(struct virtio_device *);
> > +     void (*synchronize_cbs)(struct virtio_device *);
> >       u64 (*get_features)(struct virtio_device *vdev);
> >       int (*finalize_features)(struct virtio_device *vdev);
> >       const char *(*bus_name)(struct virtio_device *vdev);
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 3/9] virtio: introduce config op to synchronize vring callbacks

[Cornelia Huck]

On Fri, Apr 29 2022, Jason Wang <jasowang@redhat.com> wrote:

> On Thu, Apr 28, 2022 at 5:13 PM Cornelia Huck <cohuck@redhat.com> wrote:
>>
>> On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:
>>
>> > This patch introduces new virtio config op to vring
>> > callbacks. Transport specific method is required to make sure the
>> > write before this function is visible to the vring_interrupt() that is
>>
>> Which kind of writes? I.e., what is the scope?
>
> Any writes before synchronize_cbs(). Is something like the following better?
>
> The function guarantees that all memory operations before it are
> visible to the vring_interrupt() that is called after it.

Maybe "all memory operations on the queue"?

Re: [PATCH V3 3/9] virtio: introduce config op to synchronize vring callbacks

[Cornelia Huck]

On Fri, Apr 29 2022, Jason Wang <jasowang@redhat.com> wrote:

> On Thu, Apr 28, 2022 at 5:13 PM Cornelia Huck <cohuck@redhat.com> wrote:
>>
>> On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:
>>
>> > This patch introduces new virtio config op to vring
>> > callbacks. Transport specific method is required to make sure the
>> > write before this function is visible to the vring_interrupt() that is
>>
>> Which kind of writes? I.e., what is the scope?
>
> Any writes before synchronize_cbs(). Is something like the following better?
>
> The function guarantees that all memory operations before it are
> visible to the vring_interrupt() that is called after it.

Maybe "all memory operations on the queue"?

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 3/9] virtio: introduce config op to synchronize vring callbacks

[Jason Wang]

在 2022/4/29 15:25, Cornelia Huck 写道:
> On Fri, Apr 29 2022, Jason Wang <jasowang@redhat.com> wrote:
>
>> On Thu, Apr 28, 2022 at 5:13 PM Cornelia Huck <cohuck@redhat.com> wrote:
>>> On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:
>>>
>>>> This patch introduces new virtio config op to vring
>>>> callbacks. Transport specific method is required to make sure the
>>>> write before this function is visible to the vring_interrupt() that is
>>> Which kind of writes? I.e., what is the scope?
>> Any writes before synchronize_cbs(). Is something like the following better?
>>
>> The function guarantees that all memory operations before it are
>> visible to the vring_interrupt() that is called after it.
> Maybe "all memory operations on the queue"?


That's fine. Will do.

Thanks


>

Re: [PATCH V3 3/9] virtio: introduce config op to synchronize vring callbacks

[Jason Wang]

在 2022/4/29 15:25, Cornelia Huck 写道:
> On Fri, Apr 29 2022, Jason Wang <jasowang@redhat.com> wrote:
>
>> On Thu, Apr 28, 2022 at 5:13 PM Cornelia Huck <cohuck@redhat.com> wrote:
>>> On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:
>>>
>>>> This patch introduces new virtio config op to vring
>>>> callbacks. Transport specific method is required to make sure the
>>>> write before this function is visible to the vring_interrupt() that is
>>> Which kind of writes? I.e., what is the scope?
>> Any writes before synchronize_cbs(). Is something like the following better?
>>
>> The function guarantees that all memory operations before it are
>> visible to the vring_interrupt() that is called after it.
> Maybe "all memory operations on the queue"?


That's fine. Will do.

Thanks


>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH V3 4/9] virtio-pci: implement synchronize_cbs()

[Jason Wang]

We can simply reuse vp_synchronize_vectors() for .synchronize_cbs().

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/virtio/virtio_pci_common.h | 2 ++
 drivers/virtio/virtio_pci_legacy.c | 1 +
 drivers/virtio/virtio_pci_modern.c | 2 ++
 3 files changed, 5 insertions(+)

diff --git a/drivers/virtio/virtio_pci_common.h b/drivers/virtio/virtio_pci_common.h
index eb17a29fc7ef..2b84d5c1b5bc 100644
--- a/drivers/virtio/virtio_pci_common.h
+++ b/drivers/virtio/virtio_pci_common.h
@@ -105,6 +105,8 @@ static struct virtio_pci_device *to_vp_device(struct virtio_device *vdev)
 void vp_synchronize_vectors(struct virtio_device *vdev);
 /* the notify function used when creating a virt queue */
 bool vp_notify(struct virtqueue *vq);
+/* synchronize with callbacks */
+void vp_synchronize_vqs(struct virtio_device *vdev);
 /* the config->del_vqs() implementation */
 void vp_del_vqs(struct virtio_device *vdev);
 /* the config->find_vqs() implementation */
diff --git a/drivers/virtio/virtio_pci_legacy.c b/drivers/virtio/virtio_pci_legacy.c
index 6f4e34ce96b8..207985107150 100644
--- a/drivers/virtio/virtio_pci_legacy.c
+++ b/drivers/virtio/virtio_pci_legacy.c
@@ -192,6 +192,7 @@ static const struct virtio_config_ops virtio_pci_config_ops = {
 	.reset		= vp_reset,
 	.find_vqs	= vp_find_vqs,
 	.del_vqs	= vp_del_vqs,
+	.synchronize_cbs = vp_synchronize_vectors,
 	.get_features	= vp_get_features,
 	.finalize_features = vp_finalize_features,
 	.bus_name	= vp_bus_name,
diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c
index a2671a20ef77..18c2190e3059 100644
--- a/drivers/virtio/virtio_pci_modern.c
+++ b/drivers/virtio/virtio_pci_modern.c
@@ -394,6 +394,7 @@ static const struct virtio_config_ops virtio_pci_config_nodev_ops = {
 	.reset		= vp_reset,
 	.find_vqs	= vp_modern_find_vqs,
 	.del_vqs	= vp_del_vqs,
+	.synchronize_cbs = vp_synchronize_vectors,
 	.get_features	= vp_get_features,
 	.finalize_features = vp_finalize_features,
 	.bus_name	= vp_bus_name,
@@ -411,6 +412,7 @@ static const struct virtio_config_ops virtio_pci_config_ops = {
 	.reset		= vp_reset,
 	.find_vqs	= vp_modern_find_vqs,
 	.del_vqs	= vp_del_vqs,
+	.synchronize_cbs = vp_synchronize_vectors,
 	.get_features	= vp_get_features,
 	.finalize_features = vp_finalize_features,
 	.bus_name	= vp_bus_name,
-- 
2.25.1

[PATCH V3 4/9] virtio-pci: implement synchronize_cbs()

[Jason Wang]

We can simply reuse vp_synchronize_vectors() for .synchronize_cbs().

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/virtio/virtio_pci_common.h | 2 ++
 drivers/virtio/virtio_pci_legacy.c | 1 +
 drivers/virtio/virtio_pci_modern.c | 2 ++
 3 files changed, 5 insertions(+)

diff --git a/drivers/virtio/virtio_pci_common.h b/drivers/virtio/virtio_pci_common.h
index eb17a29fc7ef..2b84d5c1b5bc 100644
--- a/drivers/virtio/virtio_pci_common.h
+++ b/drivers/virtio/virtio_pci_common.h
@@ -105,6 +105,8 @@ static struct virtio_pci_device *to_vp_device(struct virtio_device *vdev)
 void vp_synchronize_vectors(struct virtio_device *vdev);
 /* the notify function used when creating a virt queue */
 bool vp_notify(struct virtqueue *vq);
+/* synchronize with callbacks */
+void vp_synchronize_vqs(struct virtio_device *vdev);
 /* the config->del_vqs() implementation */
 void vp_del_vqs(struct virtio_device *vdev);
 /* the config->find_vqs() implementation */
diff --git a/drivers/virtio/virtio_pci_legacy.c b/drivers/virtio/virtio_pci_legacy.c
index 6f4e34ce96b8..207985107150 100644
--- a/drivers/virtio/virtio_pci_legacy.c
+++ b/drivers/virtio/virtio_pci_legacy.c
@@ -192,6 +192,7 @@ static const struct virtio_config_ops virtio_pci_config_ops = {
 	.reset		= vp_reset,
 	.find_vqs	= vp_find_vqs,
 	.del_vqs	= vp_del_vqs,
+	.synchronize_cbs = vp_synchronize_vectors,
 	.get_features	= vp_get_features,
 	.finalize_features = vp_finalize_features,
 	.bus_name	= vp_bus_name,
diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c
index a2671a20ef77..18c2190e3059 100644
--- a/drivers/virtio/virtio_pci_modern.c
+++ b/drivers/virtio/virtio_pci_modern.c
@@ -394,6 +394,7 @@ static const struct virtio_config_ops virtio_pci_config_nodev_ops = {
 	.reset		= vp_reset,
 	.find_vqs	= vp_modern_find_vqs,
 	.del_vqs	= vp_del_vqs,
+	.synchronize_cbs = vp_synchronize_vectors,
 	.get_features	= vp_get_features,
 	.finalize_features = vp_finalize_features,
 	.bus_name	= vp_bus_name,
@@ -411,6 +412,7 @@ static const struct virtio_config_ops virtio_pci_config_ops = {
 	.reset		= vp_reset,
 	.find_vqs	= vp_modern_find_vqs,
 	.del_vqs	= vp_del_vqs,
+	.synchronize_cbs = vp_synchronize_vectors,
 	.get_features	= vp_get_features,
 	.finalize_features = vp_finalize_features,
 	.bus_name	= vp_bus_name,
-- 
2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH V3 5/9] virtio-mmio: implement synchronize_cbs()

[Jason Wang]

Simply synchronize the platform irq that is used by us.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/virtio/virtio_mmio.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/virtio/virtio_mmio.c b/drivers/virtio/virtio_mmio.c
index 56128b9c46eb..4a3b66e4e198 100644
--- a/drivers/virtio/virtio_mmio.c
+++ b/drivers/virtio/virtio_mmio.c
@@ -345,6 +345,14 @@ static void vm_del_vqs(struct virtio_device *vdev)
 	free_irq(platform_get_irq(vm_dev->pdev, 0), vm_dev);
 }
 
+
+static void vm_synchronize_cbs(struct virtio_device *vdev)
+{
+	struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vdev);
+
+	synchronize_irq(platform_get_irq(vm_dev->pdev, 0));
+}
+
 static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned index,
 				  void (*callback)(struct virtqueue *vq),
 				  const char *name, bool ctx)
@@ -541,6 +549,7 @@ static const struct virtio_config_ops virtio_mmio_config_ops = {
 	.finalize_features = vm_finalize_features,
 	.bus_name	= vm_bus_name,
 	.get_shm_region = vm_get_shm_region,
+	.synchronize_cbs = vm_synchronize_cbs,
 };
 
 
-- 
2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH V3 5/9] virtio-mmio: implement synchronize_cbs()

[Jason Wang]

Simply synchronize the platform irq that is used by us.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/virtio/virtio_mmio.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/virtio/virtio_mmio.c b/drivers/virtio/virtio_mmio.c
index 56128b9c46eb..4a3b66e4e198 100644
--- a/drivers/virtio/virtio_mmio.c
+++ b/drivers/virtio/virtio_mmio.c
@@ -345,6 +345,14 @@ static void vm_del_vqs(struct virtio_device *vdev)
 	free_irq(platform_get_irq(vm_dev->pdev, 0), vm_dev);
 }
 
+
+static void vm_synchronize_cbs(struct virtio_device *vdev)
+{
+	struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vdev);
+
+	synchronize_irq(platform_get_irq(vm_dev->pdev, 0));
+}
+
 static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned index,
 				  void (*callback)(struct virtqueue *vq),
 				  const char *name, bool ctx)
@@ -541,6 +549,7 @@ static const struct virtio_config_ops virtio_mmio_config_ops = {
 	.finalize_features = vm_finalize_features,
 	.bus_name	= vm_bus_name,
 	.get_shm_region = vm_get_shm_region,
+	.synchronize_cbs = vm_synchronize_cbs,
 };
 
 
-- 
2.25.1

[PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

This patch tries to implement the synchronize_cbs() for ccw. For the
vring_interrupt() that is called via virtio_airq_handler(), the
synchronization is simply done via the airq_info's lock. For the
vring_interrupt() that is called via virtio_ccw_int_handler(), a per
device spinlock for irq is introduced ans used in the synchronization
method.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/s390/virtio/virtio_ccw.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
index d35e7a3f7067..c19f07a82d62 100644
--- a/drivers/s390/virtio/virtio_ccw.c
+++ b/drivers/s390/virtio/virtio_ccw.c
@@ -62,6 +62,7 @@ struct virtio_ccw_device {
 	unsigned int revision; /* Transport revision */
 	wait_queue_head_t wait_q;
 	spinlock_t lock;
+	spinlock_t irq_lock;
 	struct mutex io_lock; /* Serializes I/O requests */
 	struct list_head virtqueues;
 	bool is_thinint;
@@ -984,6 +985,27 @@ static const char *virtio_ccw_bus_name(struct virtio_device *vdev)
 	return dev_name(&vcdev->cdev->dev);
 }
 
+static void virtio_ccw_synchronize_cbs(struct virtio_device *vdev)
+{
+	struct virtio_ccw_device *vcdev = to_vc_device(vdev);
+	struct airq_info *info = vcdev->airq_info;
+
+	/*
+	 * Synchronize with the vring_interrupt() called by
+	 * virtio_ccw_int_handler().
+	 */
+	spin_lock(&vcdev->irq_lock);
+	spin_unlock(&vcdev->irq_lock);
+
+	if (info) {
+		/*
+		 * Synchronize with the vring_interrupt() with airq indicator
+		 */
+		write_lock(&info->lock);
+		write_unlock(&info->lock);
+	}
+}
+
 static const struct virtio_config_ops virtio_ccw_config_ops = {
 	.get_features = virtio_ccw_get_features,
 	.finalize_features = virtio_ccw_finalize_features,
@@ -995,6 +1017,7 @@ static const struct virtio_config_ops virtio_ccw_config_ops = {
 	.find_vqs = virtio_ccw_find_vqs,
 	.del_vqs = virtio_ccw_del_vqs,
 	.bus_name = virtio_ccw_bus_name,
+	.synchronize_cbs = virtio_ccw_synchronize_cbs,
 };
 
 
@@ -1079,6 +1102,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
 {
 	__u32 activity = intparm & VIRTIO_CCW_INTPARM_MASK;
 	struct virtio_ccw_device *vcdev = dev_get_drvdata(&cdev->dev);
+	unsigned long flags;
 	int i;
 	struct virtqueue *vq;
 
@@ -1106,6 +1130,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
 			vcdev->err = -EIO;
 	}
 	virtio_ccw_check_activity(vcdev, activity);
+	spin_lock_irqsave(&vcdev->irq_lock, flags);
 	for_each_set_bit(i, indicators(vcdev),
 			 sizeof(*indicators(vcdev)) * BITS_PER_BYTE) {
 		/* The bit clear must happen before the vring kick. */
@@ -1114,6 +1139,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
 		vq = virtio_ccw_vq_by_ind(vcdev, i);
 		vring_interrupt(0, vq);
 	}
+	spin_unlock_irqrestore(&vcdev->irq_lock, flags);
 	if (test_bit(0, indicators2(vcdev))) {
 		virtio_config_changed(&vcdev->vdev);
 		clear_bit(0, indicators2(vcdev));
@@ -1284,6 +1310,7 @@ static int virtio_ccw_online(struct ccw_device *cdev)
 	init_waitqueue_head(&vcdev->wait_q);
 	INIT_LIST_HEAD(&vcdev->virtqueues);
 	spin_lock_init(&vcdev->lock);
+	spin_lock_init(&vcdev->irq_lock);
 	mutex_init(&vcdev->io_lock);
 
 	spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
-- 
2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

This patch tries to implement the synchronize_cbs() for ccw. For the
vring_interrupt() that is called via virtio_airq_handler(), the
synchronization is simply done via the airq_info's lock. For the
vring_interrupt() that is called via virtio_ccw_int_handler(), a per
device spinlock for irq is introduced ans used in the synchronization
method.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/s390/virtio/virtio_ccw.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
index d35e7a3f7067..c19f07a82d62 100644
--- a/drivers/s390/virtio/virtio_ccw.c
+++ b/drivers/s390/virtio/virtio_ccw.c
@@ -62,6 +62,7 @@ struct virtio_ccw_device {
 	unsigned int revision; /* Transport revision */
 	wait_queue_head_t wait_q;
 	spinlock_t lock;
+	spinlock_t irq_lock;
 	struct mutex io_lock; /* Serializes I/O requests */
 	struct list_head virtqueues;
 	bool is_thinint;
@@ -984,6 +985,27 @@ static const char *virtio_ccw_bus_name(struct virtio_device *vdev)
 	return dev_name(&vcdev->cdev->dev);
 }
 
+static void virtio_ccw_synchronize_cbs(struct virtio_device *vdev)
+{
+	struct virtio_ccw_device *vcdev = to_vc_device(vdev);
+	struct airq_info *info = vcdev->airq_info;
+
+	/*
+	 * Synchronize with the vring_interrupt() called by
+	 * virtio_ccw_int_handler().
+	 */
+	spin_lock(&vcdev->irq_lock);
+	spin_unlock(&vcdev->irq_lock);
+
+	if (info) {
+		/*
+		 * Synchronize with the vring_interrupt() with airq indicator
+		 */
+		write_lock(&info->lock);
+		write_unlock(&info->lock);
+	}
+}
+
 static const struct virtio_config_ops virtio_ccw_config_ops = {
 	.get_features = virtio_ccw_get_features,
 	.finalize_features = virtio_ccw_finalize_features,
@@ -995,6 +1017,7 @@ static const struct virtio_config_ops virtio_ccw_config_ops = {
 	.find_vqs = virtio_ccw_find_vqs,
 	.del_vqs = virtio_ccw_del_vqs,
 	.bus_name = virtio_ccw_bus_name,
+	.synchronize_cbs = virtio_ccw_synchronize_cbs,
 };
 
 
@@ -1079,6 +1102,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
 {
 	__u32 activity = intparm & VIRTIO_CCW_INTPARM_MASK;
 	struct virtio_ccw_device *vcdev = dev_get_drvdata(&cdev->dev);
+	unsigned long flags;
 	int i;
 	struct virtqueue *vq;
 
@@ -1106,6 +1130,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
 			vcdev->err = -EIO;
 	}
 	virtio_ccw_check_activity(vcdev, activity);
+	spin_lock_irqsave(&vcdev->irq_lock, flags);
 	for_each_set_bit(i, indicators(vcdev),
 			 sizeof(*indicators(vcdev)) * BITS_PER_BYTE) {
 		/* The bit clear must happen before the vring kick. */
@@ -1114,6 +1139,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
 		vq = virtio_ccw_vq_by_ind(vcdev, i);
 		vring_interrupt(0, vq);
 	}
+	spin_unlock_irqrestore(&vcdev->irq_lock, flags);
 	if (test_bit(0, indicators2(vcdev))) {
 		virtio_config_changed(&vcdev->vdev);
 		clear_bit(0, indicators2(vcdev));
@@ -1284,6 +1310,7 @@ static int virtio_ccw_online(struct ccw_device *cdev)
 	init_waitqueue_head(&vcdev->wait_q);
 	INIT_LIST_HEAD(&vcdev->virtqueues);
 	spin_lock_init(&vcdev->lock);
+	spin_lock_init(&vcdev->irq_lock);
 	mutex_init(&vcdev->io_lock);
 
 	spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
-- 
2.25.1

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> This patch tries to implement the synchronize_cbs() for ccw. For the
> vring_interrupt() that is called via virtio_airq_handler(), the
> synchronization is simply done via the airq_info's lock. For the
> vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> device spinlock for irq is introduced ans used in the synchronization
> method.
> 
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Halil Pasic <pasic@linux.ibm.com>
> Cc: Cornelia Huck <cohuck@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>


This is the only one that is giving me pause. Halil, Cornelia,
should we be concerned about the performance impact here?
Any chance it can be tested?

> ---
>  drivers/s390/virtio/virtio_ccw.c | 27 +++++++++++++++++++++++++++
>  1 file changed, 27 insertions(+)
> 
> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
> index d35e7a3f7067..c19f07a82d62 100644
> --- a/drivers/s390/virtio/virtio_ccw.c
> +++ b/drivers/s390/virtio/virtio_ccw.c
> @@ -62,6 +62,7 @@ struct virtio_ccw_device {
>  	unsigned int revision; /* Transport revision */
>  	wait_queue_head_t wait_q;
>  	spinlock_t lock;
> +	spinlock_t irq_lock;
>  	struct mutex io_lock; /* Serializes I/O requests */
>  	struct list_head virtqueues;
>  	bool is_thinint;
> @@ -984,6 +985,27 @@ static const char *virtio_ccw_bus_name(struct virtio_device *vdev)
>  	return dev_name(&vcdev->cdev->dev);
>  }
>  
> +static void virtio_ccw_synchronize_cbs(struct virtio_device *vdev)
> +{
> +	struct virtio_ccw_device *vcdev = to_vc_device(vdev);
> +	struct airq_info *info = vcdev->airq_info;
> +
> +	/*
> +	 * Synchronize with the vring_interrupt() called by
> +	 * virtio_ccw_int_handler().
> +	 */
> +	spin_lock(&vcdev->irq_lock);
> +	spin_unlock(&vcdev->irq_lock);
> +
> +	if (info) {
> +		/*
> +		 * Synchronize with the vring_interrupt() with airq indicator
> +		 */
> +		write_lock(&info->lock);
> +		write_unlock(&info->lock);
> +	}
> +}
> +
>  static const struct virtio_config_ops virtio_ccw_config_ops = {
>  	.get_features = virtio_ccw_get_features,
>  	.finalize_features = virtio_ccw_finalize_features,
> @@ -995,6 +1017,7 @@ static const struct virtio_config_ops virtio_ccw_config_ops = {
>  	.find_vqs = virtio_ccw_find_vqs,
>  	.del_vqs = virtio_ccw_del_vqs,
>  	.bus_name = virtio_ccw_bus_name,
> +	.synchronize_cbs = virtio_ccw_synchronize_cbs,
>  };
>  
>  
> @@ -1079,6 +1102,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
>  {
>  	__u32 activity = intparm & VIRTIO_CCW_INTPARM_MASK;
>  	struct virtio_ccw_device *vcdev = dev_get_drvdata(&cdev->dev);
> +	unsigned long flags;
>  	int i;
>  	struct virtqueue *vq;
>  
> @@ -1106,6 +1130,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
>  			vcdev->err = -EIO;
>  	}
>  	virtio_ccw_check_activity(vcdev, activity);
> +	spin_lock_irqsave(&vcdev->irq_lock, flags);
>  	for_each_set_bit(i, indicators(vcdev),
>  			 sizeof(*indicators(vcdev)) * BITS_PER_BYTE) {
>  		/* The bit clear must happen before the vring kick. */
> @@ -1114,6 +1139,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
>  		vq = virtio_ccw_vq_by_ind(vcdev, i);
>  		vring_interrupt(0, vq);
>  	}
> +	spin_unlock_irqrestore(&vcdev->irq_lock, flags);
>  	if (test_bit(0, indicators2(vcdev))) {
>  		virtio_config_changed(&vcdev->vdev);
>  		clear_bit(0, indicators2(vcdev));
> @@ -1284,6 +1310,7 @@ static int virtio_ccw_online(struct ccw_device *cdev)
>  	init_waitqueue_head(&vcdev->wait_q);
>  	INIT_LIST_HEAD(&vcdev->virtqueues);
>  	spin_lock_init(&vcdev->lock);
> +	spin_lock_init(&vcdev->irq_lock);
>  	mutex_init(&vcdev->io_lock);
>  
>  	spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
> -- 
> 2.25.1

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> This patch tries to implement the synchronize_cbs() for ccw. For the
> vring_interrupt() that is called via virtio_airq_handler(), the
> synchronization is simply done via the airq_info's lock. For the
> vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> device spinlock for irq is introduced ans used in the synchronization
> method.
> 
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Halil Pasic <pasic@linux.ibm.com>
> Cc: Cornelia Huck <cohuck@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>


This is the only one that is giving me pause. Halil, Cornelia,
should we be concerned about the performance impact here?
Any chance it can be tested?

> ---
>  drivers/s390/virtio/virtio_ccw.c | 27 +++++++++++++++++++++++++++
>  1 file changed, 27 insertions(+)
> 
> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
> index d35e7a3f7067..c19f07a82d62 100644
> --- a/drivers/s390/virtio/virtio_ccw.c
> +++ b/drivers/s390/virtio/virtio_ccw.c
> @@ -62,6 +62,7 @@ struct virtio_ccw_device {
>  	unsigned int revision; /* Transport revision */
>  	wait_queue_head_t wait_q;
>  	spinlock_t lock;
> +	spinlock_t irq_lock;
>  	struct mutex io_lock; /* Serializes I/O requests */
>  	struct list_head virtqueues;
>  	bool is_thinint;
> @@ -984,6 +985,27 @@ static const char *virtio_ccw_bus_name(struct virtio_device *vdev)
>  	return dev_name(&vcdev->cdev->dev);
>  }
>  
> +static void virtio_ccw_synchronize_cbs(struct virtio_device *vdev)
> +{
> +	struct virtio_ccw_device *vcdev = to_vc_device(vdev);
> +	struct airq_info *info = vcdev->airq_info;
> +
> +	/*
> +	 * Synchronize with the vring_interrupt() called by
> +	 * virtio_ccw_int_handler().
> +	 */
> +	spin_lock(&vcdev->irq_lock);
> +	spin_unlock(&vcdev->irq_lock);
> +
> +	if (info) {
> +		/*
> +		 * Synchronize with the vring_interrupt() with airq indicator
> +		 */
> +		write_lock(&info->lock);
> +		write_unlock(&info->lock);
> +	}
> +}
> +
>  static const struct virtio_config_ops virtio_ccw_config_ops = {
>  	.get_features = virtio_ccw_get_features,
>  	.finalize_features = virtio_ccw_finalize_features,
> @@ -995,6 +1017,7 @@ static const struct virtio_config_ops virtio_ccw_config_ops = {
>  	.find_vqs = virtio_ccw_find_vqs,
>  	.del_vqs = virtio_ccw_del_vqs,
>  	.bus_name = virtio_ccw_bus_name,
> +	.synchronize_cbs = virtio_ccw_synchronize_cbs,
>  };
>  
>  
> @@ -1079,6 +1102,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
>  {
>  	__u32 activity = intparm & VIRTIO_CCW_INTPARM_MASK;
>  	struct virtio_ccw_device *vcdev = dev_get_drvdata(&cdev->dev);
> +	unsigned long flags;
>  	int i;
>  	struct virtqueue *vq;
>  
> @@ -1106,6 +1130,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
>  			vcdev->err = -EIO;
>  	}
>  	virtio_ccw_check_activity(vcdev, activity);
> +	spin_lock_irqsave(&vcdev->irq_lock, flags);
>  	for_each_set_bit(i, indicators(vcdev),
>  			 sizeof(*indicators(vcdev)) * BITS_PER_BYTE) {
>  		/* The bit clear must happen before the vring kick. */
> @@ -1114,6 +1139,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
>  		vq = virtio_ccw_vq_by_ind(vcdev, i);
>  		vring_interrupt(0, vq);
>  	}
> +	spin_unlock_irqrestore(&vcdev->irq_lock, flags);
>  	if (test_bit(0, indicators2(vcdev))) {
>  		virtio_config_changed(&vcdev->vdev);
>  		clear_bit(0, indicators2(vcdev));
> @@ -1284,6 +1310,7 @@ static int virtio_ccw_online(struct ccw_device *cdev)
>  	init_waitqueue_head(&vcdev->wait_q);
>  	INIT_LIST_HEAD(&vcdev->virtqueues);
>  	spin_lock_init(&vcdev->lock);
> +	spin_lock_init(&vcdev->irq_lock);
>  	mutex_init(&vcdev->io_lock);
>  
>  	spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
> -- 
> 2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Cornelia Huck]

On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
>> This patch tries to implement the synchronize_cbs() for ccw. For the
>> vring_interrupt() that is called via virtio_airq_handler(), the
>> synchronization is simply done via the airq_info's lock. For the
>> vring_interrupt() that is called via virtio_ccw_int_handler(), a per
>> device spinlock for irq is introduced ans used in the synchronization
>> method.
>> 
>> Cc: Thomas Gleixner <tglx@linutronix.de>
>> Cc: Peter Zijlstra <peterz@infradead.org>
>> Cc: "Paul E. McKenney" <paulmck@kernel.org>
>> Cc: Marc Zyngier <maz@kernel.org>
>> Cc: Halil Pasic <pasic@linux.ibm.com>
>> Cc: Cornelia Huck <cohuck@redhat.com>
>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>
>
> This is the only one that is giving me pause. Halil, Cornelia,
> should we be concerned about the performance impact here?
> Any chance it can be tested?

We can have a bunch of devices using the same airq structure, and the
sync cb creates a choke point, same as registering/unregistering. If
invoking the sync cb is a rare operation (same as (un)registering), it
should not affect interrupt processing for other devices too much, but
it really should be rare.

For testing, you would probably want to use a setup with many devices
that share the same airq area (you can fit a lot of devices if they have
few queues), generate traffic on the queues, and then do something that
triggers the callback (adding/removing a new device in a loop?)

I currently don't have such a setup handy; Halil, would you be able to
test that?

>
>> ---
>>  drivers/s390/virtio/virtio_ccw.c | 27 +++++++++++++++++++++++++++
>>  1 file changed, 27 insertions(+)
>> 
>> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
>> index d35e7a3f7067..c19f07a82d62 100644
>> --- a/drivers/s390/virtio/virtio_ccw.c
>> +++ b/drivers/s390/virtio/virtio_ccw.c
>> @@ -62,6 +62,7 @@ struct virtio_ccw_device {
>>  	unsigned int revision; /* Transport revision */
>>  	wait_queue_head_t wait_q;
>>  	spinlock_t lock;
>> +	spinlock_t irq_lock;
>>  	struct mutex io_lock; /* Serializes I/O requests */
>>  	struct list_head virtqueues;
>>  	bool is_thinint;
>> @@ -984,6 +985,27 @@ static const char *virtio_ccw_bus_name(struct virtio_device *vdev)
>>  	return dev_name(&vcdev->cdev->dev);
>>  }
>>  
>> +static void virtio_ccw_synchronize_cbs(struct virtio_device *vdev)
>> +{
>> +	struct virtio_ccw_device *vcdev = to_vc_device(vdev);
>> +	struct airq_info *info = vcdev->airq_info;
>> +
>> +	/*
>> +	 * Synchronize with the vring_interrupt() called by
>> +	 * virtio_ccw_int_handler().
>> +	 */
>> +	spin_lock(&vcdev->irq_lock);
>> +	spin_unlock(&vcdev->irq_lock);
>> +
>> +	if (info) {
>> +		/*
>> +		 * Synchronize with the vring_interrupt() with airq indicator
>> +		 */
>> +		write_lock(&info->lock);
>> +		write_unlock(&info->lock);
>> +	}

I think we can make this an either/or operation (devices will either use
classic interrupts or adapter interrupts)?

>> +}
>> +
>>  static const struct virtio_config_ops virtio_ccw_config_ops = {
>>  	.get_features = virtio_ccw_get_features,
>>  	.finalize_features = virtio_ccw_finalize_features,

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Cornelia Huck]

On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
>> This patch tries to implement the synchronize_cbs() for ccw. For the
>> vring_interrupt() that is called via virtio_airq_handler(), the
>> synchronization is simply done via the airq_info's lock. For the
>> vring_interrupt() that is called via virtio_ccw_int_handler(), a per
>> device spinlock for irq is introduced ans used in the synchronization
>> method.
>> 
>> Cc: Thomas Gleixner <tglx@linutronix.de>
>> Cc: Peter Zijlstra <peterz@infradead.org>
>> Cc: "Paul E. McKenney" <paulmck@kernel.org>
>> Cc: Marc Zyngier <maz@kernel.org>
>> Cc: Halil Pasic <pasic@linux.ibm.com>
>> Cc: Cornelia Huck <cohuck@redhat.com>
>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>
>
> This is the only one that is giving me pause. Halil, Cornelia,
> should we be concerned about the performance impact here?
> Any chance it can be tested?

We can have a bunch of devices using the same airq structure, and the
sync cb creates a choke point, same as registering/unregistering. If
invoking the sync cb is a rare operation (same as (un)registering), it
should not affect interrupt processing for other devices too much, but
it really should be rare.

For testing, you would probably want to use a setup with many devices
that share the same airq area (you can fit a lot of devices if they have
few queues), generate traffic on the queues, and then do something that
triggers the callback (adding/removing a new device in a loop?)

I currently don't have such a setup handy; Halil, would you be able to
test that?

>
>> ---
>>  drivers/s390/virtio/virtio_ccw.c | 27 +++++++++++++++++++++++++++
>>  1 file changed, 27 insertions(+)
>> 
>> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
>> index d35e7a3f7067..c19f07a82d62 100644
>> --- a/drivers/s390/virtio/virtio_ccw.c
>> +++ b/drivers/s390/virtio/virtio_ccw.c
>> @@ -62,6 +62,7 @@ struct virtio_ccw_device {
>>  	unsigned int revision; /* Transport revision */
>>  	wait_queue_head_t wait_q;
>>  	spinlock_t lock;
>> +	spinlock_t irq_lock;
>>  	struct mutex io_lock; /* Serializes I/O requests */
>>  	struct list_head virtqueues;
>>  	bool is_thinint;
>> @@ -984,6 +985,27 @@ static const char *virtio_ccw_bus_name(struct virtio_device *vdev)
>>  	return dev_name(&vcdev->cdev->dev);
>>  }
>>  
>> +static void virtio_ccw_synchronize_cbs(struct virtio_device *vdev)
>> +{
>> +	struct virtio_ccw_device *vcdev = to_vc_device(vdev);
>> +	struct airq_info *info = vcdev->airq_info;
>> +
>> +	/*
>> +	 * Synchronize with the vring_interrupt() called by
>> +	 * virtio_ccw_int_handler().
>> +	 */
>> +	spin_lock(&vcdev->irq_lock);
>> +	spin_unlock(&vcdev->irq_lock);
>> +
>> +	if (info) {
>> +		/*
>> +		 * Synchronize with the vring_interrupt() with airq indicator
>> +		 */
>> +		write_lock(&info->lock);
>> +		write_unlock(&info->lock);
>> +	}

I think we can make this an either/or operation (devices will either use
classic interrupts or adapter interrupts)?

>> +}
>> +
>>  static const struct virtio_config_ops virtio_ccw_config_ops = {
>>  	.get_features = virtio_ccw_get_features,
>>  	.finalize_features = virtio_ccw_finalize_features,

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> >> This patch tries to implement the synchronize_cbs() for ccw. For the
> >> vring_interrupt() that is called via virtio_airq_handler(), the
> >> synchronization is simply done via the airq_info's lock. For the
> >> vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> >> device spinlock for irq is introduced ans used in the synchronization
> >> method.
> >> 
> >> Cc: Thomas Gleixner <tglx@linutronix.de>
> >> Cc: Peter Zijlstra <peterz@infradead.org>
> >> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> >> Cc: Marc Zyngier <maz@kernel.org>
> >> Cc: Halil Pasic <pasic@linux.ibm.com>
> >> Cc: Cornelia Huck <cohuck@redhat.com>
> >> Signed-off-by: Jason Wang <jasowang@redhat.com>
> >
> >
> > This is the only one that is giving me pause. Halil, Cornelia,
> > should we be concerned about the performance impact here?
> > Any chance it can be tested?
> 
> We can have a bunch of devices using the same airq structure, and the
> sync cb creates a choke point, same as registering/unregistering.

BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
this patch serializes them on a spinlock.

> If
> invoking the sync cb is a rare operation (same as (un)registering), it
> should not affect interrupt processing for other devices too much, but
> it really should be rare.
> 
> For testing, you would probably want to use a setup with many devices
> that share the same airq area (you can fit a lot of devices if they have
> few queues), generate traffic on the queues, and then do something that
> triggers the callback (adding/removing a new device in a loop?)
> 
> I currently don't have such a setup handy; Halil, would you be able to
> test that?
> 
> >
> >> ---
> >>  drivers/s390/virtio/virtio_ccw.c | 27 +++++++++++++++++++++++++++
> >>  1 file changed, 27 insertions(+)
> >> 
> >> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
> >> index d35e7a3f7067..c19f07a82d62 100644
> >> --- a/drivers/s390/virtio/virtio_ccw.c
> >> +++ b/drivers/s390/virtio/virtio_ccw.c
> >> @@ -62,6 +62,7 @@ struct virtio_ccw_device {
> >>  	unsigned int revision; /* Transport revision */
> >>  	wait_queue_head_t wait_q;
> >>  	spinlock_t lock;
> >> +	spinlock_t irq_lock;
> >>  	struct mutex io_lock; /* Serializes I/O requests */
> >>  	struct list_head virtqueues;
> >>  	bool is_thinint;
> >> @@ -984,6 +985,27 @@ static const char *virtio_ccw_bus_name(struct virtio_device *vdev)
> >>  	return dev_name(&vcdev->cdev->dev);
> >>  }
> >>  
> >> +static void virtio_ccw_synchronize_cbs(struct virtio_device *vdev)
> >> +{
> >> +	struct virtio_ccw_device *vcdev = to_vc_device(vdev);
> >> +	struct airq_info *info = vcdev->airq_info;
> >> +
> >> +	/*
> >> +	 * Synchronize with the vring_interrupt() called by
> >> +	 * virtio_ccw_int_handler().
> >> +	 */
> >> +	spin_lock(&vcdev->irq_lock);
> >> +	spin_unlock(&vcdev->irq_lock);
> >> +
> >> +	if (info) {
> >> +		/*
> >> +		 * Synchronize with the vring_interrupt() with airq indicator
> >> +		 */
> >> +		write_lock(&info->lock);
> >> +		write_unlock(&info->lock);
> >> +	}
> 
> I think we can make this an either/or operation (devices will either use
> classic interrupts or adapter interrupts)?
> 
> >> +}
> >> +
> >>  static const struct virtio_config_ops virtio_ccw_config_ops = {
> >>  	.get_features = virtio_ccw_get_features,
> >>  	.finalize_features = virtio_ccw_finalize_features,

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> >> This patch tries to implement the synchronize_cbs() for ccw. For the
> >> vring_interrupt() that is called via virtio_airq_handler(), the
> >> synchronization is simply done via the airq_info's lock. For the
> >> vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> >> device spinlock for irq is introduced ans used in the synchronization
> >> method.
> >> 
> >> Cc: Thomas Gleixner <tglx@linutronix.de>
> >> Cc: Peter Zijlstra <peterz@infradead.org>
> >> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> >> Cc: Marc Zyngier <maz@kernel.org>
> >> Cc: Halil Pasic <pasic@linux.ibm.com>
> >> Cc: Cornelia Huck <cohuck@redhat.com>
> >> Signed-off-by: Jason Wang <jasowang@redhat.com>
> >
> >
> > This is the only one that is giving me pause. Halil, Cornelia,
> > should we be concerned about the performance impact here?
> > Any chance it can be tested?
> 
> We can have a bunch of devices using the same airq structure, and the
> sync cb creates a choke point, same as registering/unregistering.

BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
this patch serializes them on a spinlock.

> If
> invoking the sync cb is a rare operation (same as (un)registering), it
> should not affect interrupt processing for other devices too much, but
> it really should be rare.
> 
> For testing, you would probably want to use a setup with many devices
> that share the same airq area (you can fit a lot of devices if they have
> few queues), generate traffic on the queues, and then do something that
> triggers the callback (adding/removing a new device in a loop?)
> 
> I currently don't have such a setup handy; Halil, would you be able to
> test that?
> 
> >
> >> ---
> >>  drivers/s390/virtio/virtio_ccw.c | 27 +++++++++++++++++++++++++++
> >>  1 file changed, 27 insertions(+)
> >> 
> >> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
> >> index d35e7a3f7067..c19f07a82d62 100644
> >> --- a/drivers/s390/virtio/virtio_ccw.c
> >> +++ b/drivers/s390/virtio/virtio_ccw.c
> >> @@ -62,6 +62,7 @@ struct virtio_ccw_device {
> >>  	unsigned int revision; /* Transport revision */
> >>  	wait_queue_head_t wait_q;
> >>  	spinlock_t lock;
> >> +	spinlock_t irq_lock;
> >>  	struct mutex io_lock; /* Serializes I/O requests */
> >>  	struct list_head virtqueues;
> >>  	bool is_thinint;
> >> @@ -984,6 +985,27 @@ static const char *virtio_ccw_bus_name(struct virtio_device *vdev)
> >>  	return dev_name(&vcdev->cdev->dev);
> >>  }
> >>  
> >> +static void virtio_ccw_synchronize_cbs(struct virtio_device *vdev)
> >> +{
> >> +	struct virtio_ccw_device *vcdev = to_vc_device(vdev);
> >> +	struct airq_info *info = vcdev->airq_info;
> >> +
> >> +	/*
> >> +	 * Synchronize with the vring_interrupt() called by
> >> +	 * virtio_ccw_int_handler().
> >> +	 */
> >> +	spin_lock(&vcdev->irq_lock);
> >> +	spin_unlock(&vcdev->irq_lock);
> >> +
> >> +	if (info) {
> >> +		/*
> >> +		 * Synchronize with the vring_interrupt() with airq indicator
> >> +		 */
> >> +		write_lock(&info->lock);
> >> +		write_unlock(&info->lock);
> >> +	}
> 
> I think we can make this an either/or operation (devices will either use
> classic interrupts or adapter interrupts)?
> 
> >> +}
> >> +
> >>  static const struct virtio_config_ops virtio_ccw_config_ops = {
> >>  	.get_features = virtio_ccw_get_features,
> >>  	.finalize_features = virtio_ccw_finalize_features,

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Halil Pasic]

On Mon, 25 Apr 2022 09:59:55 -0400
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >   
> > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:  
> > >> This patch tries to implement the synchronize_cbs() for ccw. For the
> > >> vring_interrupt() that is called via virtio_airq_handler(), the
> > >> synchronization is simply done via the airq_info's lock. For the
> > >> vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > >> device spinlock for irq is introduced ans used in the synchronization
> > >> method.
> > >> 
> > >> Cc: Thomas Gleixner <tglx@linutronix.de>
> > >> Cc: Peter Zijlstra <peterz@infradead.org>
> > >> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > >> Cc: Marc Zyngier <maz@kernel.org>
> > >> Cc: Halil Pasic <pasic@linux.ibm.com>
> > >> Cc: Cornelia Huck <cohuck@redhat.com>
> > >> Signed-off-by: Jason Wang <jasowang@redhat.com>  
> > >
> > >
> > > This is the only one that is giving me pause. Halil, Cornelia,
> > > should we be concerned about the performance impact here?
> > > Any chance it can be tested?  
> > 
> > We can have a bunch of devices using the same airq structure, and the
> > sync cb creates a choke point, same as registering/unregistering.  
> 
> BTW can callbacks for multiple VQs run on multiple CPUs at the moment?

I'm not sure I understand the question.

I do think we can have multiple CPUs that are executing some portion of
virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?

On the other hand we could also end up serializing synchronize_cbs()
calls for different devices if they happen to use the same airq_info. But
this probably was not your question

> this patch serializes them on a spinlock.
>

Those could then pile up on the newly introduced spinlock.

Regards,
Halil

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Halil Pasic]

On Mon, 25 Apr 2022 09:59:55 -0400
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >   
> > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:  
> > >> This patch tries to implement the synchronize_cbs() for ccw. For the
> > >> vring_interrupt() that is called via virtio_airq_handler(), the
> > >> synchronization is simply done via the airq_info's lock. For the
> > >> vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > >> device spinlock for irq is introduced ans used in the synchronization
> > >> method.
> > >> 
> > >> Cc: Thomas Gleixner <tglx@linutronix.de>
> > >> Cc: Peter Zijlstra <peterz@infradead.org>
> > >> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > >> Cc: Marc Zyngier <maz@kernel.org>
> > >> Cc: Halil Pasic <pasic@linux.ibm.com>
> > >> Cc: Cornelia Huck <cohuck@redhat.com>
> > >> Signed-off-by: Jason Wang <jasowang@redhat.com>  
> > >
> > >
> > > This is the only one that is giving me pause. Halil, Cornelia,
> > > should we be concerned about the performance impact here?
> > > Any chance it can be tested?  
> > 
> > We can have a bunch of devices using the same airq structure, and the
> > sync cb creates a choke point, same as registering/unregistering.  
> 
> BTW can callbacks for multiple VQs run on multiple CPUs at the moment?

I'm not sure I understand the question.

I do think we can have multiple CPUs that are executing some portion of
virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?

On the other hand we could also end up serializing synchronize_cbs()
calls for different devices if they happen to use the same airq_info. But
this probably was not your question

> this patch serializes them on a spinlock.
>

Those could then pile up on the newly introduced spinlock.

Regards,
Halil

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> On Mon, 25 Apr 2022 09:59:55 -0400
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >   
> > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:  
> > > >> This patch tries to implement the synchronize_cbs() for ccw. For the
> > > >> vring_interrupt() that is called via virtio_airq_handler(), the
> > > >> synchronization is simply done via the airq_info's lock. For the
> > > >> vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > >> device spinlock for irq is introduced ans used in the synchronization
> > > >> method.
> > > >> 
> > > >> Cc: Thomas Gleixner <tglx@linutronix.de>
> > > >> Cc: Peter Zijlstra <peterz@infradead.org>
> > > >> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > >> Cc: Marc Zyngier <maz@kernel.org>
> > > >> Cc: Halil Pasic <pasic@linux.ibm.com>
> > > >> Cc: Cornelia Huck <cohuck@redhat.com>
> > > >> Signed-off-by: Jason Wang <jasowang@redhat.com>  
> > > >
> > > >
> > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > should we be concerned about the performance impact here?
> > > > Any chance it can be tested?  
> > > 
> > > We can have a bunch of devices using the same airq structure, and the
> > > sync cb creates a choke point, same as registering/unregistering.  
> > 
> > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> 
> I'm not sure I understand the question.
> 
> I do think we can have multiple CPUs that are executing some portion of
> virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> 
> On the other hand we could also end up serializing synchronize_cbs()
> calls for different devices if they happen to use the same airq_info. But
> this probably was not your question


I am less concerned about  synchronize_cbs being slow and more about
the slowdown in interrupt processing itself.

> > this patch serializes them on a spinlock.
> >
> 
> Those could then pile up on the newly introduced spinlock.
> 
> Regards,
> Halil

Hmm yea ... not good.

-- 
MST

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> On Mon, 25 Apr 2022 09:59:55 -0400
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >   
> > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:  
> > > >> This patch tries to implement the synchronize_cbs() for ccw. For the
> > > >> vring_interrupt() that is called via virtio_airq_handler(), the
> > > >> synchronization is simply done via the airq_info's lock. For the
> > > >> vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > >> device spinlock for irq is introduced ans used in the synchronization
> > > >> method.
> > > >> 
> > > >> Cc: Thomas Gleixner <tglx@linutronix.de>
> > > >> Cc: Peter Zijlstra <peterz@infradead.org>
> > > >> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > >> Cc: Marc Zyngier <maz@kernel.org>
> > > >> Cc: Halil Pasic <pasic@linux.ibm.com>
> > > >> Cc: Cornelia Huck <cohuck@redhat.com>
> > > >> Signed-off-by: Jason Wang <jasowang@redhat.com>  
> > > >
> > > >
> > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > should we be concerned about the performance impact here?
> > > > Any chance it can be tested?  
> > > 
> > > We can have a bunch of devices using the same airq structure, and the
> > > sync cb creates a choke point, same as registering/unregistering.  
> > 
> > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> 
> I'm not sure I understand the question.
> 
> I do think we can have multiple CPUs that are executing some portion of
> virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> 
> On the other hand we could also end up serializing synchronize_cbs()
> calls for different devices if they happen to use the same airq_info. But
> this probably was not your question


I am less concerned about  synchronize_cbs being slow and more about
the slowdown in interrupt processing itself.

> > this patch serializes them on a spinlock.
> >
> 
> Those could then pile up on the newly introduced spinlock.
> 
> Regards,
> Halil

Hmm yea ... not good.

-- 
MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > On Mon, 25 Apr 2022 09:59:55 -0400
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > 
> > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > >   
> > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:  
> > > > >> This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > >> vring_interrupt() that is called via virtio_airq_handler(), the
> > > > >> synchronization is simply done via the airq_info's lock. For the
> > > > >> vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > >> device spinlock for irq is introduced ans used in the synchronization
> > > > >> method.
> > > > >> 
> > > > >> Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > >> Cc: Peter Zijlstra <peterz@infradead.org>
> > > > >> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > >> Cc: Marc Zyngier <maz@kernel.org>
> > > > >> Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > >> Cc: Cornelia Huck <cohuck@redhat.com>
> > > > >> Signed-off-by: Jason Wang <jasowang@redhat.com>  
> > > > >
> > > > >
> > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > should we be concerned about the performance impact here?
> > > > > Any chance it can be tested?  
> > > > 
> > > > We can have a bunch of devices using the same airq structure, and the
> > > > sync cb creates a choke point, same as registering/unregistering.  
> > > 
> > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > 
> > I'm not sure I understand the question.
> > 
> > I do think we can have multiple CPUs that are executing some portion of
> > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > 
> > On the other hand we could also end up serializing synchronize_cbs()
> > calls for different devices if they happen to use the same airq_info. But
> > this probably was not your question
> 
> 
> I am less concerned about  synchronize_cbs being slow and more about
> the slowdown in interrupt processing itself.
> 
> > > this patch serializes them on a spinlock.
> > >
> > 
> > Those could then pile up on the newly introduced spinlock.
> > 
> > Regards,
> > Halil
> 
> Hmm yea ... not good.

Is there any other way to synchronize with all callbacks?

> -- 
> MST

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > On Mon, 25 Apr 2022 09:59:55 -0400
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > 
> > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > >   
> > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:  
> > > > >> This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > >> vring_interrupt() that is called via virtio_airq_handler(), the
> > > > >> synchronization is simply done via the airq_info's lock. For the
> > > > >> vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > >> device spinlock for irq is introduced ans used in the synchronization
> > > > >> method.
> > > > >> 
> > > > >> Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > >> Cc: Peter Zijlstra <peterz@infradead.org>
> > > > >> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > >> Cc: Marc Zyngier <maz@kernel.org>
> > > > >> Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > >> Cc: Cornelia Huck <cohuck@redhat.com>
> > > > >> Signed-off-by: Jason Wang <jasowang@redhat.com>  
> > > > >
> > > > >
> > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > should we be concerned about the performance impact here?
> > > > > Any chance it can be tested?  
> > > > 
> > > > We can have a bunch of devices using the same airq structure, and the
> > > > sync cb creates a choke point, same as registering/unregistering.  
> > > 
> > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > 
> > I'm not sure I understand the question.
> > 
> > I do think we can have multiple CPUs that are executing some portion of
> > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > 
> > On the other hand we could also end up serializing synchronize_cbs()
> > calls for different devices if they happen to use the same airq_info. But
> > this probably was not your question
> 
> 
> I am less concerned about  synchronize_cbs being slow and more about
> the slowdown in interrupt processing itself.
> 
> > > this patch serializes them on a spinlock.
> > >
> > 
> > Those could then pile up on the newly introduced spinlock.
> > 
> > Regards,
> > Halil
> 
> Hmm yea ... not good.

Is there any other way to synchronize with all callbacks?

> -- 
> MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
>> On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
>>> On Mon, 25 Apr 2022 09:59:55 -0400
>>> "Michael S. Tsirkin" <mst@redhat.com> wrote:
>>>
>>>> On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
>>>>> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>>>>>    
>>>>>> On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
>>>>>>> This patch tries to implement the synchronize_cbs() for ccw. For the
>>>>>>> vring_interrupt() that is called via virtio_airq_handler(), the
>>>>>>> synchronization is simply done via the airq_info's lock. For the
>>>>>>> vring_interrupt() that is called via virtio_ccw_int_handler(), a per
>>>>>>> device spinlock for irq is introduced ans used in the synchronization
>>>>>>> method.
>>>>>>>
>>>>>>> Cc: Thomas Gleixner <tglx@linutronix.de>
>>>>>>> Cc: Peter Zijlstra <peterz@infradead.org>
>>>>>>> Cc: "Paul E. McKenney" <paulmck@kernel.org>
>>>>>>> Cc: Marc Zyngier <maz@kernel.org>
>>>>>>> Cc: Halil Pasic <pasic@linux.ibm.com>
>>>>>>> Cc: Cornelia Huck <cohuck@redhat.com>
>>>>>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>>>>>
>>>>>> This is the only one that is giving me pause. Halil, Cornelia,
>>>>>> should we be concerned about the performance impact here?
>>>>>> Any chance it can be tested?
>>>>> We can have a bunch of devices using the same airq structure, and the
>>>>> sync cb creates a choke point, same as registering/unregistering.
>>>> BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
>>> I'm not sure I understand the question.
>>>
>>> I do think we can have multiple CPUs that are executing some portion of
>>> virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
>>>
>>> On the other hand we could also end up serializing synchronize_cbs()
>>> calls for different devices if they happen to use the same airq_info. But
>>> this probably was not your question
>>
>> I am less concerned about  synchronize_cbs being slow and more about
>> the slowdown in interrupt processing itself.
>>
>>>> this patch serializes them on a spinlock.
>>>>
>>> Those could then pile up on the newly introduced spinlock.
>>>
>>> Regards,
>>> Halil
>> Hmm yea ... not good.
> Is there any other way to synchronize with all callbacks?


Maybe using rwlock as airq handler?

Thanks


>
>> -- 
>> MST

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
>> On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
>>> On Mon, 25 Apr 2022 09:59:55 -0400
>>> "Michael S. Tsirkin" <mst@redhat.com> wrote:
>>>
>>>> On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
>>>>> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>>>>>    
>>>>>> On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
>>>>>>> This patch tries to implement the synchronize_cbs() for ccw. For the
>>>>>>> vring_interrupt() that is called via virtio_airq_handler(), the
>>>>>>> synchronization is simply done via the airq_info's lock. For the
>>>>>>> vring_interrupt() that is called via virtio_ccw_int_handler(), a per
>>>>>>> device spinlock for irq is introduced ans used in the synchronization
>>>>>>> method.
>>>>>>>
>>>>>>> Cc: Thomas Gleixner <tglx@linutronix.de>
>>>>>>> Cc: Peter Zijlstra <peterz@infradead.org>
>>>>>>> Cc: "Paul E. McKenney" <paulmck@kernel.org>
>>>>>>> Cc: Marc Zyngier <maz@kernel.org>
>>>>>>> Cc: Halil Pasic <pasic@linux.ibm.com>
>>>>>>> Cc: Cornelia Huck <cohuck@redhat.com>
>>>>>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>>>>>
>>>>>> This is the only one that is giving me pause. Halil, Cornelia,
>>>>>> should we be concerned about the performance impact here?
>>>>>> Any chance it can be tested?
>>>>> We can have a bunch of devices using the same airq structure, and the
>>>>> sync cb creates a choke point, same as registering/unregistering.
>>>> BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
>>> I'm not sure I understand the question.
>>>
>>> I do think we can have multiple CPUs that are executing some portion of
>>> virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
>>>
>>> On the other hand we could also end up serializing synchronize_cbs()
>>> calls for different devices if they happen to use the same airq_info. But
>>> this probably was not your question
>>
>> I am less concerned about  synchronize_cbs being slow and more about
>> the slowdown in interrupt processing itself.
>>
>>>> this patch serializes them on a spinlock.
>>>>
>>> Those could then pile up on the newly introduced spinlock.
>>>
>>> Regards,
>>> Halil
>> Hmm yea ... not good.
> Is there any other way to synchronize with all callbacks?


Maybe using rwlock as airq handler?

Thanks


>
>> -- 
>> MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> 
> 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > 
> > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > method.
> > > > > > > > 
> > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > 
> > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > should we be concerned about the performance impact here?
> > > > > > > Any chance it can be tested?
> > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > I'm not sure I understand the question.
> > > > 
> > > > I do think we can have multiple CPUs that are executing some portion of
> > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > 
> > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > calls for different devices if they happen to use the same airq_info. But
> > > > this probably was not your question
> > > 
> > > I am less concerned about  synchronize_cbs being slow and more about
> > > the slowdown in interrupt processing itself.
> > > 
> > > > > this patch serializes them on a spinlock.
> > > > > 
> > > > Those could then pile up on the newly introduced spinlock.
> > > > 
> > > > Regards,
> > > > Halil
> > > Hmm yea ... not good.
> > Is there any other way to synchronize with all callbacks?
> 
> 
> Maybe using rwlock as airq handler?
> 
> Thanks
> 

rwlock is still a shared cacheline bouncing between CPUs and
a bunch of ordering instructions.
Maybe something per-cpu + some IPIs to run things on all CPUs instead?

> > 
> > > -- 
> > > MST

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> 
> 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > 
> > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > method.
> > > > > > > > 
> > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > 
> > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > should we be concerned about the performance impact here?
> > > > > > > Any chance it can be tested?
> > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > I'm not sure I understand the question.
> > > > 
> > > > I do think we can have multiple CPUs that are executing some portion of
> > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > 
> > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > calls for different devices if they happen to use the same airq_info. But
> > > > this probably was not your question
> > > 
> > > I am less concerned about  synchronize_cbs being slow and more about
> > > the slowdown in interrupt processing itself.
> > > 
> > > > > this patch serializes them on a spinlock.
> > > > > 
> > > > Those could then pile up on the newly introduced spinlock.
> > > > 
> > > > Regards,
> > > > Halil
> > > Hmm yea ... not good.
> > Is there any other way to synchronize with all callbacks?
> 
> 
> Maybe using rwlock as airq handler?
> 
> Thanks
> 

rwlock is still a shared cacheline bouncing between CPUs and
a bunch of ordering instructions.
Maybe something per-cpu + some IPIs to run things on all CPUs instead?

> > 
> > > -- 
> > > MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > 
> > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > 
> > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > > method.
> > > > > > > > > 
> > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > > 
> > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > > should we be concerned about the performance impact here?
> > > > > > > > Any chance it can be tested?
> > > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > > I'm not sure I understand the question.
> > > > > 
> > > > > I do think we can have multiple CPUs that are executing some portion of
> > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > > 
> > > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > > calls for different devices if they happen to use the same airq_info. But
> > > > > this probably was not your question
> > > > 
> > > > I am less concerned about  synchronize_cbs being slow and more about
> > > > the slowdown in interrupt processing itself.
> > > > 
> > > > > > this patch serializes them on a spinlock.
> > > > > > 
> > > > > Those could then pile up on the newly introduced spinlock.
> > > > > 
> > > > > Regards,
> > > > > Halil
> > > > Hmm yea ... not good.
> > > Is there any other way to synchronize with all callbacks?
> > 
> > 
> > Maybe using rwlock as airq handler?
> > 
> > Thanks
> > 
> 
> rwlock is still a shared cacheline bouncing between CPUs and
> a bunch of ordering instructions.
> Maybe something per-cpu + some IPIs to run things on all CPUs instead?

... and I think classic and device interrupts are different enough
here ...

> > > 
> > > > -- 
> > > > MST

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > 
> > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > 
> > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > > method.
> > > > > > > > > 
> > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > > 
> > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > > should we be concerned about the performance impact here?
> > > > > > > > Any chance it can be tested?
> > > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > > I'm not sure I understand the question.
> > > > > 
> > > > > I do think we can have multiple CPUs that are executing some portion of
> > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > > 
> > > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > > calls for different devices if they happen to use the same airq_info. But
> > > > > this probably was not your question
> > > > 
> > > > I am less concerned about  synchronize_cbs being slow and more about
> > > > the slowdown in interrupt processing itself.
> > > > 
> > > > > > this patch serializes them on a spinlock.
> > > > > > 
> > > > > Those could then pile up on the newly introduced spinlock.
> > > > > 
> > > > > Regards,
> > > > > Halil
> > > > Hmm yea ... not good.
> > > Is there any other way to synchronize with all callbacks?
> > 
> > 
> > Maybe using rwlock as airq handler?
> > 
> > Thanks
> > 
> 
> rwlock is still a shared cacheline bouncing between CPUs and
> a bunch of ordering instructions.
> Maybe something per-cpu + some IPIs to run things on all CPUs instead?

... and I think classic and device interrupts are different enough
here ...

> > > 
> > > > -- 
> > > > MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

On Tue, Apr 26, 2022 at 11:55 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> > On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > >
> > > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > >
> > > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > > > method.
> > > > > > > > > >
> > > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > > >
> > > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > > > should we be concerned about the performance impact here?
> > > > > > > > > Any chance it can be tested?
> > > > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > > > I'm not sure I understand the question.
> > > > > >
> > > > > > I do think we can have multiple CPUs that are executing some portion of
> > > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > > >
> > > > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > > > calls for different devices if they happen to use the same airq_info. But
> > > > > > this probably was not your question
> > > > >
> > > > > I am less concerned about  synchronize_cbs being slow and more about
> > > > > the slowdown in interrupt processing itself.
> > > > >
> > > > > > > this patch serializes them on a spinlock.
> > > > > > >
> > > > > > Those could then pile up on the newly introduced spinlock.
> > > > > >
> > > > > > Regards,
> > > > > > Halil
> > > > > Hmm yea ... not good.
> > > > Is there any other way to synchronize with all callbacks?
> > >
> > >
> > > Maybe using rwlock as airq handler?
> > >
> > > Thanks
> > >
> >
> > rwlock is still a shared cacheline bouncing between CPUs and
> > a bunch of ordering instructions.

Yes, but it should be faster than spinlocks anyhow.

> > Maybe something per-cpu + some IPIs to run things on all CPUs instead?

Is this something like a customized version of synchronzie_rcu_expedited()?

>
> ... and I think classic and device interrupts are different enough
> here ...

Yes.

Thanks

>
> > > >
> > > > > --
> > > > > MST
>

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

On Tue, Apr 26, 2022 at 11:55 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> > On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > >
> > > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > >
> > > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > > > method.
> > > > > > > > > >
> > > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > > >
> > > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > > > should we be concerned about the performance impact here?
> > > > > > > > > Any chance it can be tested?
> > > > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > > > I'm not sure I understand the question.
> > > > > >
> > > > > > I do think we can have multiple CPUs that are executing some portion of
> > > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > > >
> > > > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > > > calls for different devices if they happen to use the same airq_info. But
> > > > > > this probably was not your question
> > > > >
> > > > > I am less concerned about  synchronize_cbs being slow and more about
> > > > > the slowdown in interrupt processing itself.
> > > > >
> > > > > > > this patch serializes them on a spinlock.
> > > > > > >
> > > > > > Those could then pile up on the newly introduced spinlock.
> > > > > >
> > > > > > Regards,
> > > > > > Halil
> > > > > Hmm yea ... not good.
> > > > Is there any other way to synchronize with all callbacks?
> > >
> > >
> > > Maybe using rwlock as airq handler?
> > >
> > > Thanks
> > >
> >
> > rwlock is still a shared cacheline bouncing between CPUs and
> > a bunch of ordering instructions.

Yes, but it should be faster than spinlocks anyhow.

> > Maybe something per-cpu + some IPIs to run things on all CPUs instead?

Is this something like a customized version of synchronzie_rcu_expedited()?

>
> ... and I think classic and device interrupts are different enough
> here ...

Yes.

Thanks

>
> > > >
> > > > > --
> > > > > MST
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Tue, Apr 26, 2022 at 12:07:39PM +0800, Jason Wang wrote:
> On Tue, Apr 26, 2022 at 11:55 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> > > On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > > >
> > > > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > > > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > >
> > > > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > > > > method.
> > > > > > > > > > >
> > > > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > > > >
> > > > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > > > > should we be concerned about the performance impact here?
> > > > > > > > > > Any chance it can be tested?
> > > > > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > > > > I'm not sure I understand the question.
> > > > > > >
> > > > > > > I do think we can have multiple CPUs that are executing some portion of
> > > > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > > > >
> > > > > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > > > > calls for different devices if they happen to use the same airq_info. But
> > > > > > > this probably was not your question
> > > > > >
> > > > > > I am less concerned about  synchronize_cbs being slow and more about
> > > > > > the slowdown in interrupt processing itself.
> > > > > >
> > > > > > > > this patch serializes them on a spinlock.
> > > > > > > >
> > > > > > > Those could then pile up on the newly introduced spinlock.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Halil
> > > > > > Hmm yea ... not good.
> > > > > Is there any other way to synchronize with all callbacks?
> > > >
> > > >
> > > > Maybe using rwlock as airq handler?
> > > >
> > > > Thanks
> > > >
> > >
> > > rwlock is still a shared cacheline bouncing between CPUs and
> > > a bunch of ordering instructions.
> 
> Yes, but it should be faster than spinlocks anyhow.
> 
> > > Maybe something per-cpu + some IPIs to run things on all CPUs instead?
> 
> Is this something like a customized version of synchronzie_rcu_expedited()?

With interrupts running in an RCU read size critical section?
Quite possibly that is also an option.
This will need a bunch of documentation since this is not
a standard use of RCU, and probably get a confirmation
from RCU maintainers that whatever assumptions we make
are guaranteed to hold down the road.

> >
> > ... and I think classic and device interrupts are different enough
> > here ...
> 
> Yes.
> 
> Thanks
> 
> >
> > > > >
> > > > > > --
> > > > > > MST
> >

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Tue, Apr 26, 2022 at 12:07:39PM +0800, Jason Wang wrote:
> On Tue, Apr 26, 2022 at 11:55 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> > > On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > > >
> > > > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > > > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > >
> > > > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > > > > method.
> > > > > > > > > > >
> > > > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > > > >
> > > > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > > > > should we be concerned about the performance impact here?
> > > > > > > > > > Any chance it can be tested?
> > > > > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > > > > I'm not sure I understand the question.
> > > > > > >
> > > > > > > I do think we can have multiple CPUs that are executing some portion of
> > > > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > > > >
> > > > > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > > > > calls for different devices if they happen to use the same airq_info. But
> > > > > > > this probably was not your question
> > > > > >
> > > > > > I am less concerned about  synchronize_cbs being slow and more about
> > > > > > the slowdown in interrupt processing itself.
> > > > > >
> > > > > > > > this patch serializes them on a spinlock.
> > > > > > > >
> > > > > > > Those could then pile up on the newly introduced spinlock.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Halil
> > > > > > Hmm yea ... not good.
> > > > > Is there any other way to synchronize with all callbacks?
> > > >
> > > >
> > > > Maybe using rwlock as airq handler?
> > > >
> > > > Thanks
> > > >
> > >
> > > rwlock is still a shared cacheline bouncing between CPUs and
> > > a bunch of ordering instructions.
> 
> Yes, but it should be faster than spinlocks anyhow.
> 
> > > Maybe something per-cpu + some IPIs to run things on all CPUs instead?
> 
> Is this something like a customized version of synchronzie_rcu_expedited()?

With interrupts running in an RCU read size critical section?
Quite possibly that is also an option.
This will need a bunch of documentation since this is not
a standard use of RCU, and probably get a confirmation
from RCU maintainers that whatever assumptions we make
are guaranteed to hold down the road.

> >
> > ... and I think classic and device interrupts are different enough
> > here ...
> 
> Yes.
> 
> Thanks
> 
> >
> > > > >
> > > > > > --
> > > > > > MST
> >

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

On Tue, Apr 26, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Apr 26, 2022 at 12:07:39PM +0800, Jason Wang wrote:
> > On Tue, Apr 26, 2022 at 11:55 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> > > > On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > > > >
> > > > > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > > > > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > > > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > >
> > > > > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > > > > > method.
> > > > > > > > > > > >
> > > > > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > > > > >
> > > > > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > > > > > should we be concerned about the performance impact here?
> > > > > > > > > > > Any chance it can be tested?
> > > > > > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > > > > > I'm not sure I understand the question.
> > > > > > > >
> > > > > > > > I do think we can have multiple CPUs that are executing some portion of
> > > > > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > > > > >
> > > > > > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > > > > > calls for different devices if they happen to use the same airq_info. But
> > > > > > > > this probably was not your question
> > > > > > >
> > > > > > > I am less concerned about  synchronize_cbs being slow and more about
> > > > > > > the slowdown in interrupt processing itself.
> > > > > > >
> > > > > > > > > this patch serializes them on a spinlock.
> > > > > > > > >
> > > > > > > > Those could then pile up on the newly introduced spinlock.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Halil
> > > > > > > Hmm yea ... not good.
> > > > > > Is there any other way to synchronize with all callbacks?
> > > > >
> > > > >
> > > > > Maybe using rwlock as airq handler?
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > > > rwlock is still a shared cacheline bouncing between CPUs and
> > > > a bunch of ordering instructions.
> >
> > Yes, but it should be faster than spinlocks anyhow.
> >
> > > > Maybe something per-cpu + some IPIs to run things on all CPUs instead?
> >
> > Is this something like a customized version of synchronzie_rcu_expedited()?
>
> With interrupts running in an RCU read size critical section?

For vring_interrupt(), yes.


> Quite possibly that is also an option.
> This will need a bunch of documentation since this is not
> a standard use of RCU,

According to Documentation/RCU/requirements.rst, it looks like a legal case:

"
The Linux kernel has interrupts, and RCU read-side critical sections are
legal within interrupt handlers and within interrupt-disabled regions of
code, as are invocations of call_rcu().
"

And as discussed, synchronize_rcu_expedited() is not friendly to real
time workload. I think we can simply

1) protect vring_interrupt() with rcu_read_lock()
2) use synchronize_rcu() in synchronize_cbs for ccw

And if we care about the long delay we can use per device srcu to reduce that?

Thanks

> and probably get a confirmation
> from RCU maintainers that whatever assumptions we make
> are guaranteed to hold down the road.
>
> > >
> > > ... and I think classic and device interrupts are different enough
> > > here ...
> >
> > Yes.
> >
> > Thanks
> >
> > >
> > > > > >
> > > > > > > --
> > > > > > > MST
> > >
>

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

On Tue, Apr 26, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Apr 26, 2022 at 12:07:39PM +0800, Jason Wang wrote:
> > On Tue, Apr 26, 2022 at 11:55 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> > > > On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > > > >
> > > > > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > > > > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > > > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > >
> > > > > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > > > > > method.
> > > > > > > > > > > >
> > > > > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > > > > >
> > > > > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > > > > > should we be concerned about the performance impact here?
> > > > > > > > > > > Any chance it can be tested?
> > > > > > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > > > > > I'm not sure I understand the question.
> > > > > > > >
> > > > > > > > I do think we can have multiple CPUs that are executing some portion of
> > > > > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > > > > >
> > > > > > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > > > > > calls for different devices if they happen to use the same airq_info. But
> > > > > > > > this probably was not your question
> > > > > > >
> > > > > > > I am less concerned about  synchronize_cbs being slow and more about
> > > > > > > the slowdown in interrupt processing itself.
> > > > > > >
> > > > > > > > > this patch serializes them on a spinlock.
> > > > > > > > >
> > > > > > > > Those could then pile up on the newly introduced spinlock.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Halil
> > > > > > > Hmm yea ... not good.
> > > > > > Is there any other way to synchronize with all callbacks?
> > > > >
> > > > >
> > > > > Maybe using rwlock as airq handler?
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > > > rwlock is still a shared cacheline bouncing between CPUs and
> > > > a bunch of ordering instructions.
> >
> > Yes, but it should be faster than spinlocks anyhow.
> >
> > > > Maybe something per-cpu + some IPIs to run things on all CPUs instead?
> >
> > Is this something like a customized version of synchronzie_rcu_expedited()?
>
> With interrupts running in an RCU read size critical section?

For vring_interrupt(), yes.


> Quite possibly that is also an option.
> This will need a bunch of documentation since this is not
> a standard use of RCU,

According to Documentation/RCU/requirements.rst, it looks like a legal case:

"
The Linux kernel has interrupts, and RCU read-side critical sections are
legal within interrupt handlers and within interrupt-disabled regions of
code, as are invocations of call_rcu().
"

And as discussed, synchronize_rcu_expedited() is not friendly to real
time workload. I think we can simply

1) protect vring_interrupt() with rcu_read_lock()
2) use synchronize_rcu() in synchronize_cbs for ccw

And if we care about the long delay we can use per device srcu to reduce that?

Thanks

> and probably get a confirmation
> from RCU maintainers that whatever assumptions we make
> are guaranteed to hold down the road.
>
> > >
> > > ... and I think classic and device interrupts are different enough
> > > here ...
> >
> > Yes.
> >
> > Thanks
> >
> > >
> > > > > >
> > > > > > > --
> > > > > > > MST
> > >
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Wed, Apr 27, 2022 at 11:53:25AM +0800, Jason Wang wrote:
> On Tue, Apr 26, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Apr 26, 2022 at 12:07:39PM +0800, Jason Wang wrote:
> > > On Tue, Apr 26, 2022 at 11:55 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> > > > > On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > > > > >
> > > > > > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > > > > > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > > > > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > > > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > > > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > > > > > > method.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > > > > > >
> > > > > > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > > > > > > should we be concerned about the performance impact here?
> > > > > > > > > > > > Any chance it can be tested?
> > > > > > > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > > > > > > I'm not sure I understand the question.
> > > > > > > > >
> > > > > > > > > I do think we can have multiple CPUs that are executing some portion of
> > > > > > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > > > > > >
> > > > > > > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > > > > > > calls for different devices if they happen to use the same airq_info. But
> > > > > > > > > this probably was not your question
> > > > > > > >
> > > > > > > > I am less concerned about  synchronize_cbs being slow and more about
> > > > > > > > the slowdown in interrupt processing itself.
> > > > > > > >
> > > > > > > > > > this patch serializes them on a spinlock.
> > > > > > > > > >
> > > > > > > > > Those could then pile up on the newly introduced spinlock.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Halil
> > > > > > > > Hmm yea ... not good.
> > > > > > > Is there any other way to synchronize with all callbacks?
> > > > > >
> > > > > >
> > > > > > Maybe using rwlock as airq handler?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > >
> > > > > rwlock is still a shared cacheline bouncing between CPUs and
> > > > > a bunch of ordering instructions.
> > >
> > > Yes, but it should be faster than spinlocks anyhow.
> > >
> > > > > Maybe something per-cpu + some IPIs to run things on all CPUs instead?
> > >
> > > Is this something like a customized version of synchronzie_rcu_expedited()?
> >
> > With interrupts running in an RCU read size critical section?
> 
> For vring_interrupt(), yes.
> 
> 
> > Quite possibly that is also an option.
> > This will need a bunch of documentation since this is not
> > a standard use of RCU,
> 
> According to Documentation/RCU/requirements.rst, it looks like a legal case:
> 
> "
> The Linux kernel has interrupts, and RCU read-side critical sections are
> legal within interrupt handlers and within interrupt-disabled regions of
> code, as are invocations of call_rcu().
> "
> 
> And as discussed, synchronize_rcu_expedited() is not friendly to real
> time workload.

I am not sure hotplug removal is relevant for realtime anyway though.

> I think we can simply
> 
> 1) protect vring_interrupt() with rcu_read_lock()
> 2) use synchronize_rcu() in synchronize_cbs for ccw
> 
> And if we care about the long delay we can use per device srcu to reduce that?
> 
> Thanks

Maybe ...

> > and probably get a confirmation
> > from RCU maintainers that whatever assumptions we make
> > are guaranteed to hold down the road.
> >
> > > >
> > > > ... and I think classic and device interrupts are different enough
> > > > here ...
> > >
> > > Yes.
> > >
> > > Thanks
> > >
> > > >
> > > > > > >
> > > > > > > > --
> > > > > > > > MST
> > > >
> >

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Wed, Apr 27, 2022 at 11:53:25AM +0800, Jason Wang wrote:
> On Tue, Apr 26, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Apr 26, 2022 at 12:07:39PM +0800, Jason Wang wrote:
> > > On Tue, Apr 26, 2022 at 11:55 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> > > > > On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > > > > >
> > > > > > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > > > > > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > > > > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > > > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > > > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > > > > > > method.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > > > > > >
> > > > > > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > > > > > > should we be concerned about the performance impact here?
> > > > > > > > > > > > Any chance it can be tested?
> > > > > > > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > > > > > > I'm not sure I understand the question.
> > > > > > > > >
> > > > > > > > > I do think we can have multiple CPUs that are executing some portion of
> > > > > > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > > > > > >
> > > > > > > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > > > > > > calls for different devices if they happen to use the same airq_info. But
> > > > > > > > > this probably was not your question
> > > > > > > >
> > > > > > > > I am less concerned about  synchronize_cbs being slow and more about
> > > > > > > > the slowdown in interrupt processing itself.
> > > > > > > >
> > > > > > > > > > this patch serializes them on a spinlock.
> > > > > > > > > >
> > > > > > > > > Those could then pile up on the newly introduced spinlock.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Halil
> > > > > > > > Hmm yea ... not good.
> > > > > > > Is there any other way to synchronize with all callbacks?
> > > > > >
> > > > > >
> > > > > > Maybe using rwlock as airq handler?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > >
> > > > > rwlock is still a shared cacheline bouncing between CPUs and
> > > > > a bunch of ordering instructions.
> > >
> > > Yes, but it should be faster than spinlocks anyhow.
> > >
> > > > > Maybe something per-cpu + some IPIs to run things on all CPUs instead?
> > >
> > > Is this something like a customized version of synchronzie_rcu_expedited()?
> >
> > With interrupts running in an RCU read size critical section?
> 
> For vring_interrupt(), yes.
> 
> 
> > Quite possibly that is also an option.
> > This will need a bunch of documentation since this is not
> > a standard use of RCU,
> 
> According to Documentation/RCU/requirements.rst, it looks like a legal case:
> 
> "
> The Linux kernel has interrupts, and RCU read-side critical sections are
> legal within interrupt handlers and within interrupt-disabled regions of
> code, as are invocations of call_rcu().
> "
> 
> And as discussed, synchronize_rcu_expedited() is not friendly to real
> time workload.

I am not sure hotplug removal is relevant for realtime anyway though.

> I think we can simply
> 
> 1) protect vring_interrupt() with rcu_read_lock()
> 2) use synchronize_rcu() in synchronize_cbs for ccw
> 
> And if we care about the long delay we can use per device srcu to reduce that?
> 
> Thanks

Maybe ...

> > and probably get a confirmation
> > from RCU maintainers that whatever assumptions we make
> > are guaranteed to hold down the road.
> >
> > > >
> > > > ... and I think classic and device interrupts are different enough
> > > > here ...
> > >
> > > Yes.
> > >
> > > Thanks
> > >
> > > >
> > > > > > >
> > > > > > > > --
> > > > > > > > MST
> > > >
> >

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Wed, Apr 27, 2022 at 11:53:25AM +0800, Jason Wang wrote:
> On Tue, Apr 26, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Apr 26, 2022 at 12:07:39PM +0800, Jason Wang wrote:
> > > On Tue, Apr 26, 2022 at 11:55 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> > > > > On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > > > > >
> > > > > > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > > > > > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > > > > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > > > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > > > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > > > > > > method.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > > > > > >
> > > > > > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > > > > > > should we be concerned about the performance impact here?
> > > > > > > > > > > > Any chance it can be tested?
> > > > > > > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > > > > > > I'm not sure I understand the question.
> > > > > > > > >
> > > > > > > > > I do think we can have multiple CPUs that are executing some portion of
> > > > > > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > > > > > >
> > > > > > > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > > > > > > calls for different devices if they happen to use the same airq_info. But
> > > > > > > > > this probably was not your question
> > > > > > > >
> > > > > > > > I am less concerned about  synchronize_cbs being slow and more about
> > > > > > > > the slowdown in interrupt processing itself.
> > > > > > > >
> > > > > > > > > > this patch serializes them on a spinlock.
> > > > > > > > > >
> > > > > > > > > Those could then pile up on the newly introduced spinlock.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Halil
> > > > > > > > Hmm yea ... not good.
> > > > > > > Is there any other way to synchronize with all callbacks?
> > > > > >
> > > > > >
> > > > > > Maybe using rwlock as airq handler?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > >
> > > > > rwlock is still a shared cacheline bouncing between CPUs and
> > > > > a bunch of ordering instructions.
> > >
> > > Yes, but it should be faster than spinlocks anyhow.
> > >
> > > > > Maybe something per-cpu + some IPIs to run things on all CPUs instead?
> > >
> > > Is this something like a customized version of synchronzie_rcu_expedited()?
> >
> > With interrupts running in an RCU read size critical section?
> 
> For vring_interrupt(), yes.
> 
> 
> > Quite possibly that is also an option.
> > This will need a bunch of documentation since this is not
> > a standard use of RCU,
> 
> According to Documentation/RCU/requirements.rst, it looks like a legal case:
> 
> "
> The Linux kernel has interrupts, and RCU read-side critical sections are
> legal within interrupt handlers and within interrupt-disabled regions of
> code, as are invocations of call_rcu().
> "

My problem is it is not clear what data is protected by rcu here.
Nothing is tagged with __rcu or uses rcu_dereference.
We need at least an ack from rcu maintainers that this is
a valid use of rcu and not an undocumented side effect.

> And as discussed, synchronize_rcu_expedited() is not friendly to real
> time workload. I think we can simply
> 
> 1) protect vring_interrupt() with rcu_read_lock()
> 2) use synchronize_rcu() in synchronize_cbs for ccw
> 
> And if we care about the long delay we can use per device srcu to reduce that?
> 
> Thanks
> 
> > and probably get a confirmation
> > from RCU maintainers that whatever assumptions we make
> > are guaranteed to hold down the road.
> >
> > > >
> > > > ... and I think classic and device interrupts are different enough
> > > > here ...
> > >
> > > Yes.
> > >
> > > Thanks
> > >
> > > >
> > > > > > >
> > > > > > > > --
> > > > > > > > MST
> > > >
> >

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Wed, Apr 27, 2022 at 11:53:25AM +0800, Jason Wang wrote:
> On Tue, Apr 26, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Apr 26, 2022 at 12:07:39PM +0800, Jason Wang wrote:
> > > On Tue, Apr 26, 2022 at 11:55 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> > > > > On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > > > > >
> > > > > > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > > > > > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > > > > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > > > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > > > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > > > > > > method.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > > > > > >
> > > > > > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > > > > > > should we be concerned about the performance impact here?
> > > > > > > > > > > > Any chance it can be tested?
> > > > > > > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > > > > > > I'm not sure I understand the question.
> > > > > > > > >
> > > > > > > > > I do think we can have multiple CPUs that are executing some portion of
> > > > > > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > > > > > >
> > > > > > > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > > > > > > calls for different devices if they happen to use the same airq_info. But
> > > > > > > > > this probably was not your question
> > > > > > > >
> > > > > > > > I am less concerned about  synchronize_cbs being slow and more about
> > > > > > > > the slowdown in interrupt processing itself.
> > > > > > > >
> > > > > > > > > > this patch serializes them on a spinlock.
> > > > > > > > > >
> > > > > > > > > Those could then pile up on the newly introduced spinlock.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Halil
> > > > > > > > Hmm yea ... not good.
> > > > > > > Is there any other way to synchronize with all callbacks?
> > > > > >
> > > > > >
> > > > > > Maybe using rwlock as airq handler?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > >
> > > > > rwlock is still a shared cacheline bouncing between CPUs and
> > > > > a bunch of ordering instructions.
> > >
> > > Yes, but it should be faster than spinlocks anyhow.
> > >
> > > > > Maybe something per-cpu + some IPIs to run things on all CPUs instead?
> > >
> > > Is this something like a customized version of synchronzie_rcu_expedited()?
> >
> > With interrupts running in an RCU read size critical section?
> 
> For vring_interrupt(), yes.
> 
> 
> > Quite possibly that is also an option.
> > This will need a bunch of documentation since this is not
> > a standard use of RCU,
> 
> According to Documentation/RCU/requirements.rst, it looks like a legal case:
> 
> "
> The Linux kernel has interrupts, and RCU read-side critical sections are
> legal within interrupt handlers and within interrupt-disabled regions of
> code, as are invocations of call_rcu().
> "

My problem is it is not clear what data is protected by rcu here.
Nothing is tagged with __rcu or uses rcu_dereference.
We need at least an ack from rcu maintainers that this is
a valid use of rcu and not an undocumented side effect.

> And as discussed, synchronize_rcu_expedited() is not friendly to real
> time workload. I think we can simply
> 
> 1) protect vring_interrupt() with rcu_read_lock()
> 2) use synchronize_rcu() in synchronize_cbs for ccw
> 
> And if we care about the long delay we can use per device srcu to reduce that?
> 
> Thanks
> 
> > and probably get a confirmation
> > from RCU maintainers that whatever assumptions we make
> > are guaranteed to hold down the road.
> >
> > > >
> > > > ... and I think classic and device interrupts are different enough
> > > > here ...
> > >
> > > Yes.
> > >
> > > Thanks
> > >
> > > >
> > > > > > >
> > > > > > > > --
> > > > > > > > MST
> > > >
> >

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

On Wed, Apr 27, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Wed, Apr 27, 2022 at 11:53:25AM +0800, Jason Wang wrote:
> > On Tue, Apr 26, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Tue, Apr 26, 2022 at 12:07:39PM +0800, Jason Wang wrote:
> > > > On Tue, Apr 26, 2022 at 11:55 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> > > > > > On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > > > > > >
> > > > > > > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > > > > > > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > > > > > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > > > > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > > > > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > > > > > > > method.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > > > > > > > should we be concerned about the performance impact here?
> > > > > > > > > > > > > Any chance it can be tested?
> > > > > > > > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > > > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > > > > > > > I'm not sure I understand the question.
> > > > > > > > > >
> > > > > > > > > > I do think we can have multiple CPUs that are executing some portion of
> > > > > > > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > > > > > > >
> > > > > > > > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > > > > > > > calls for different devices if they happen to use the same airq_info. But
> > > > > > > > > > this probably was not your question
> > > > > > > > >
> > > > > > > > > I am less concerned about  synchronize_cbs being slow and more about
> > > > > > > > > the slowdown in interrupt processing itself.
> > > > > > > > >
> > > > > > > > > > > this patch serializes them on a spinlock.
> > > > > > > > > > >
> > > > > > > > > > Those could then pile up on the newly introduced spinlock.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Halil
> > > > > > > > > Hmm yea ... not good.
> > > > > > > > Is there any other way to synchronize with all callbacks?
> > > > > > >
> > > > > > >
> > > > > > > Maybe using rwlock as airq handler?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > >
> > > > > > rwlock is still a shared cacheline bouncing between CPUs and
> > > > > > a bunch of ordering instructions.
> > > >
> > > > Yes, but it should be faster than spinlocks anyhow.
> > > >
> > > > > > Maybe something per-cpu + some IPIs to run things on all CPUs instead?
> > > >
> > > > Is this something like a customized version of synchronzie_rcu_expedited()?
> > >
> > > With interrupts running in an RCU read size critical section?
> >
> > For vring_interrupt(), yes.
> >
> >
> > > Quite possibly that is also an option.
> > > This will need a bunch of documentation since this is not
> > > a standard use of RCU,
> >
> > According to Documentation/RCU/requirements.rst, it looks like a legal case:
> >
> > "
> > The Linux kernel has interrupts, and RCU read-side critical sections are
> > legal within interrupt handlers and within interrupt-disabled regions of
> > code, as are invocations of call_rcu().
> > "
>
> My problem is it is not clear what data is protected by rcu here.
> Nothing is tagged with __rcu or uses rcu_dereference.

It should be the vq->broken here. We can rcuify it.


> We need at least an ack from rcu maintainers that this is
> a valid use of rcu and not an undocumented side effect.

Yes.

Thanks


>
> > And as discussed, synchronize_rcu_expedited() is not friendly to real
> > time workload. I think we can simply
> >
> > 1) protect vring_interrupt() with rcu_read_lock()
> > 2) use synchronize_rcu() in synchronize_cbs for ccw
> >
> > And if we care about the long delay we can use per device srcu to reduce that?
> >
> > Thanks
> >
> > > and probably get a confirmation
> > > from RCU maintainers that whatever assumptions we make
> > > are guaranteed to hold down the road.
> > >
> > > > >
> > > > > ... and I think classic and device interrupts are different enough
> > > > > here ...
> > > >
> > > > Yes.
> > > >
> > > > Thanks
> > > >
> > > > >
> > > > > > > >
> > > > > > > > > --
> > > > > > > > > MST
> > > > >
> > >
>

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

On Wed, Apr 27, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Wed, Apr 27, 2022 at 11:53:25AM +0800, Jason Wang wrote:
> > On Tue, Apr 26, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Tue, Apr 26, 2022 at 12:07:39PM +0800, Jason Wang wrote:
> > > > On Tue, Apr 26, 2022 at 11:55 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> > > > > > On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > > > > > >
> > > > > > > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > > > > > > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > > > > > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > > > > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > > > > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > > > > > > > method.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > > > > > > > should we be concerned about the performance impact here?
> > > > > > > > > > > > > Any chance it can be tested?
> > > > > > > > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > > > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > > > > > > > I'm not sure I understand the question.
> > > > > > > > > >
> > > > > > > > > > I do think we can have multiple CPUs that are executing some portion of
> > > > > > > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > > > > > > >
> > > > > > > > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > > > > > > > calls for different devices if they happen to use the same airq_info. But
> > > > > > > > > > this probably was not your question
> > > > > > > > >
> > > > > > > > > I am less concerned about  synchronize_cbs being slow and more about
> > > > > > > > > the slowdown in interrupt processing itself.
> > > > > > > > >
> > > > > > > > > > > this patch serializes them on a spinlock.
> > > > > > > > > > >
> > > > > > > > > > Those could then pile up on the newly introduced spinlock.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Halil
> > > > > > > > > Hmm yea ... not good.
> > > > > > > > Is there any other way to synchronize with all callbacks?
> > > > > > >
> > > > > > >
> > > > > > > Maybe using rwlock as airq handler?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > >
> > > > > > rwlock is still a shared cacheline bouncing between CPUs and
> > > > > > a bunch of ordering instructions.
> > > >
> > > > Yes, but it should be faster than spinlocks anyhow.
> > > >
> > > > > > Maybe something per-cpu + some IPIs to run things on all CPUs instead?
> > > >
> > > > Is this something like a customized version of synchronzie_rcu_expedited()?
> > >
> > > With interrupts running in an RCU read size critical section?
> >
> > For vring_interrupt(), yes.
> >
> >
> > > Quite possibly that is also an option.
> > > This will need a bunch of documentation since this is not
> > > a standard use of RCU,
> >
> > According to Documentation/RCU/requirements.rst, it looks like a legal case:
> >
> > "
> > The Linux kernel has interrupts, and RCU read-side critical sections are
> > legal within interrupt handlers and within interrupt-disabled regions of
> > code, as are invocations of call_rcu().
> > "
>
> My problem is it is not clear what data is protected by rcu here.
> Nothing is tagged with __rcu or uses rcu_dereference.

It should be the vq->broken here. We can rcuify it.


> We need at least an ack from rcu maintainers that this is
> a valid use of rcu and not an undocumented side effect.

Yes.

Thanks


>
> > And as discussed, synchronize_rcu_expedited() is not friendly to real
> > time workload. I think we can simply
> >
> > 1) protect vring_interrupt() with rcu_read_lock()
> > 2) use synchronize_rcu() in synchronize_cbs for ccw
> >
> > And if we care about the long delay we can use per device srcu to reduce that?
> >
> > Thanks
> >
> > > and probably get a confirmation
> > > from RCU maintainers that whatever assumptions we make
> > > are guaranteed to hold down the road.
> > >
> > > > >
> > > > > ... and I think classic and device interrupts are different enough
> > > > > here ...
> > > >
> > > > Yes.
> > > >
> > > > Thanks
> > > >
> > > > >
> > > > > > > >
> > > > > > > > > --
> > > > > > > > > MST
> > > > >
> > >
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Wed, Apr 27, 2022 at 03:57:57PM +0800, Jason Wang wrote:
> On Wed, Apr 27, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Wed, Apr 27, 2022 at 11:53:25AM +0800, Jason Wang wrote:
> > > On Tue, Apr 26, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Tue, Apr 26, 2022 at 12:07:39PM +0800, Jason Wang wrote:
> > > > > On Tue, Apr 26, 2022 at 11:55 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > >
> > > > > > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> > > > > > > On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > > > > > > >
> > > > > > > > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > > > > > > > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > > > > > > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > > > > > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > > > > > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > > > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > > > > > > > > method.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > > > > > > > > should we be concerned about the performance impact here?
> > > > > > > > > > > > > > Any chance it can be tested?
> > > > > > > > > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > > > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > > > > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > > > > > > > > I'm not sure I understand the question.
> > > > > > > > > > >
> > > > > > > > > > > I do think we can have multiple CPUs that are executing some portion of
> > > > > > > > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > > > > > > > >
> > > > > > > > > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > > > > > > > > calls for different devices if they happen to use the same airq_info. But
> > > > > > > > > > > this probably was not your question
> > > > > > > > > >
> > > > > > > > > > I am less concerned about  synchronize_cbs being slow and more about
> > > > > > > > > > the slowdown in interrupt processing itself.
> > > > > > > > > >
> > > > > > > > > > > > this patch serializes them on a spinlock.
> > > > > > > > > > > >
> > > > > > > > > > > Those could then pile up on the newly introduced spinlock.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Halil
> > > > > > > > > > Hmm yea ... not good.
> > > > > > > > > Is there any other way to synchronize with all callbacks?
> > > > > > > >
> > > > > > > >
> > > > > > > > Maybe using rwlock as airq handler?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > >
> > > > > > > rwlock is still a shared cacheline bouncing between CPUs and
> > > > > > > a bunch of ordering instructions.
> > > > >
> > > > > Yes, but it should be faster than spinlocks anyhow.
> > > > >
> > > > > > > Maybe something per-cpu + some IPIs to run things on all CPUs instead?
> > > > >
> > > > > Is this something like a customized version of synchronzie_rcu_expedited()?
> > > >
> > > > With interrupts running in an RCU read size critical section?
> > >
> > > For vring_interrupt(), yes.
> > >
> > >
> > > > Quite possibly that is also an option.
> > > > This will need a bunch of documentation since this is not
> > > > a standard use of RCU,
> > >
> > > According to Documentation/RCU/requirements.rst, it looks like a legal case:
> > >
> > > "
> > > The Linux kernel has interrupts, and RCU read-side critical sections are
> > > legal within interrupt handlers and within interrupt-disabled regions of
> > > code, as are invocations of call_rcu().
> > > "
> >
> > My problem is it is not clear what data is protected by rcu here.
> > Nothing is tagged with __rcu or uses rcu_dereference.
> 
> It should be the vq->broken here. We can rcuify it.


I don't exactly exactly see how - rcu_dereference works on pointers.
Feel free to send a patch.


> 
> > We need at least an ack from rcu maintainers that this is
> > a valid use of rcu and not an undocumented side effect.
> 
> Yes.
> 
> Thanks
> 
> 
> >
> > > And as discussed, synchronize_rcu_expedited() is not friendly to real
> > > time workload. I think we can simply
> > >
> > > 1) protect vring_interrupt() with rcu_read_lock()
> > > 2) use synchronize_rcu() in synchronize_cbs for ccw
> > >
> > > And if we care about the long delay we can use per device srcu to reduce that?
> > >
> > > Thanks
> > >
> > > > and probably get a confirmation
> > > > from RCU maintainers that whatever assumptions we make
> > > > are guaranteed to hold down the road.
> > > >
> > > > > >
> > > > > > ... and I think classic and device interrupts are different enough
> > > > > > here ...
> > > > >
> > > > > Yes.
> > > > >
> > > > > Thanks
> > > > >
> > > > > >
> > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > MST
> > > > > >
> > > >
> >

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Wed, Apr 27, 2022 at 03:57:57PM +0800, Jason Wang wrote:
> On Wed, Apr 27, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Wed, Apr 27, 2022 at 11:53:25AM +0800, Jason Wang wrote:
> > > On Tue, Apr 26, 2022 at 2:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Tue, Apr 26, 2022 at 12:07:39PM +0800, Jason Wang wrote:
> > > > > On Tue, Apr 26, 2022 at 11:55 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > >
> > > > > > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> > > > > > > On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > > > > > > >
> > > > > > > > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > > > > > > > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > > > > > > > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > > > > > > > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > > > > > > > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > > > > > > > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > > > > > > > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > > > > > > > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > > > > > > > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > > > > > > > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > > > > > > > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > > > > > > > > > > > > > > method.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > > > > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > > > > > > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > > > > > > > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > > > > > > > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > > > > > > > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > > > > > > > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > > > > > > > > > > > > > should we be concerned about the performance impact here?
> > > > > > > > > > > > > > Any chance it can be tested?
> > > > > > > > > > > > > We can have a bunch of devices using the same airq structure, and the
> > > > > > > > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > > > > > > > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > > > > > > > > > > I'm not sure I understand the question.
> > > > > > > > > > >
> > > > > > > > > > > I do think we can have multiple CPUs that are executing some portion of
> > > > > > > > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > > > > > > > > > >
> > > > > > > > > > > On the other hand we could also end up serializing synchronize_cbs()
> > > > > > > > > > > calls for different devices if they happen to use the same airq_info. But
> > > > > > > > > > > this probably was not your question
> > > > > > > > > >
> > > > > > > > > > I am less concerned about  synchronize_cbs being slow and more about
> > > > > > > > > > the slowdown in interrupt processing itself.
> > > > > > > > > >
> > > > > > > > > > > > this patch serializes them on a spinlock.
> > > > > > > > > > > >
> > > > > > > > > > > Those could then pile up on the newly introduced spinlock.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Halil
> > > > > > > > > > Hmm yea ... not good.
> > > > > > > > > Is there any other way to synchronize with all callbacks?
> > > > > > > >
> > > > > > > >
> > > > > > > > Maybe using rwlock as airq handler?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > >
> > > > > > > rwlock is still a shared cacheline bouncing between CPUs and
> > > > > > > a bunch of ordering instructions.
> > > > >
> > > > > Yes, but it should be faster than spinlocks anyhow.
> > > > >
> > > > > > > Maybe something per-cpu + some IPIs to run things on all CPUs instead?
> > > > >
> > > > > Is this something like a customized version of synchronzie_rcu_expedited()?
> > > >
> > > > With interrupts running in an RCU read size critical section?
> > >
> > > For vring_interrupt(), yes.
> > >
> > >
> > > > Quite possibly that is also an option.
> > > > This will need a bunch of documentation since this is not
> > > > a standard use of RCU,
> > >
> > > According to Documentation/RCU/requirements.rst, it looks like a legal case:
> > >
> > > "
> > > The Linux kernel has interrupts, and RCU read-side critical sections are
> > > legal within interrupt handlers and within interrupt-disabled regions of
> > > code, as are invocations of call_rcu().
> > > "
> >
> > My problem is it is not clear what data is protected by rcu here.
> > Nothing is tagged with __rcu or uses rcu_dereference.
> 
> It should be the vq->broken here. We can rcuify it.


I don't exactly exactly see how - rcu_dereference works on pointers.
Feel free to send a patch.


> 
> > We need at least an ack from rcu maintainers that this is
> > a valid use of rcu and not an undocumented side effect.
> 
> Yes.
> 
> Thanks
> 
> 
> >
> > > And as discussed, synchronize_rcu_expedited() is not friendly to real
> > > time workload. I think we can simply
> > >
> > > 1) protect vring_interrupt() with rcu_read_lock()
> > > 2) use synchronize_rcu() in synchronize_cbs for ccw
> > >
> > > And if we care about the long delay we can use per device srcu to reduce that?
> > >
> > > Thanks
> > >
> > > > and probably get a confirmation
> > > > from RCU maintainers that whatever assumptions we make
> > > > are guaranteed to hold down the road.
> > > >
> > > > > >
> > > > > > ... and I think classic and device interrupts are different enough
> > > > > > here ...
> > > > >
> > > > > Yes.
> > > > >
> > > > > Thanks
> > > > >
> > > > > >
> > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > MST
> > > > > >
> > > >
> >

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Cornelia Huck]

On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
>> On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
>> > 
>> > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
>> > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
>> > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
>> > > > > On Mon, 25 Apr 2022 09:59:55 -0400
>> > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> > > > > 
>> > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
>> > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
>> > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
>> > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
>> > > > > > > > > synchronization is simply done via the airq_info's lock. For the
>> > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
>> > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
>> > > > > > > > > method.
>> > > > > > > > > 
>> > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
>> > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
>> > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
>> > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
>> > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
>> > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
>> > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
>> > > > > > > > 
>> > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
>> > > > > > > > should we be concerned about the performance impact here?
>> > > > > > > > Any chance it can be tested?
>> > > > > > > We can have a bunch of devices using the same airq structure, and the
>> > > > > > > sync cb creates a choke point, same as registering/unregistering.
>> > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
>> > > > > I'm not sure I understand the question.
>> > > > > 
>> > > > > I do think we can have multiple CPUs that are executing some portion of
>> > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
>> > > > > 
>> > > > > On the other hand we could also end up serializing synchronize_cbs()
>> > > > > calls for different devices if they happen to use the same airq_info. But
>> > > > > this probably was not your question
>> > > > 
>> > > > I am less concerned about  synchronize_cbs being slow and more about
>> > > > the slowdown in interrupt processing itself.
>> > > > 
>> > > > > > this patch serializes them on a spinlock.
>> > > > > > 
>> > > > > Those could then pile up on the newly introduced spinlock.

How bad would that be in practice? IIUC, we hit on the spinlock when
- doing synchronize_cbs (should be rare)
- processing queue interrupts for devices using per-device indicators
  (which is the non-preferred path, which I would basically only expect
  when running on an ancient or non-standard hypervisor)
- configuration change interrupts (should be rare)
- during setup, reset, etc. (should not be a concern)

>> > > > > 
>> > > > > Regards,
>> > > > > Halil
>> > > > Hmm yea ... not good.
>> > > Is there any other way to synchronize with all callbacks?
>> > 
>> > 
>> > Maybe using rwlock as airq handler?
>> > 
>> > Thanks
>> > 
>> 
>> rwlock is still a shared cacheline bouncing between CPUs and
>> a bunch of ordering instructions.
>> Maybe something per-cpu + some IPIs to run things on all CPUs instead?
>
> ... and I think classic and device interrupts are different enough
> here ...

You mean classic (per-device) and adapter interrupts, right?

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Cornelia Huck]

On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
>> On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
>> > 
>> > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
>> > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
>> > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
>> > > > > On Mon, 25 Apr 2022 09:59:55 -0400
>> > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> > > > > 
>> > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
>> > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
>> > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
>> > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
>> > > > > > > > > synchronization is simply done via the airq_info's lock. For the
>> > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
>> > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
>> > > > > > > > > method.
>> > > > > > > > > 
>> > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
>> > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
>> > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
>> > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
>> > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
>> > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
>> > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
>> > > > > > > > 
>> > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
>> > > > > > > > should we be concerned about the performance impact here?
>> > > > > > > > Any chance it can be tested?
>> > > > > > > We can have a bunch of devices using the same airq structure, and the
>> > > > > > > sync cb creates a choke point, same as registering/unregistering.
>> > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
>> > > > > I'm not sure I understand the question.
>> > > > > 
>> > > > > I do think we can have multiple CPUs that are executing some portion of
>> > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
>> > > > > 
>> > > > > On the other hand we could also end up serializing synchronize_cbs()
>> > > > > calls for different devices if they happen to use the same airq_info. But
>> > > > > this probably was not your question
>> > > > 
>> > > > I am less concerned about  synchronize_cbs being slow and more about
>> > > > the slowdown in interrupt processing itself.
>> > > > 
>> > > > > > this patch serializes them on a spinlock.
>> > > > > > 
>> > > > > Those could then pile up on the newly introduced spinlock.

How bad would that be in practice? IIUC, we hit on the spinlock when
- doing synchronize_cbs (should be rare)
- processing queue interrupts for devices using per-device indicators
  (which is the non-preferred path, which I would basically only expect
  when running on an ancient or non-standard hypervisor)
- configuration change interrupts (should be rare)
- during setup, reset, etc. (should not be a concern)

>> > > > > 
>> > > > > Regards,
>> > > > > Halil
>> > > > Hmm yea ... not good.
>> > > Is there any other way to synchronize with all callbacks?
>> > 
>> > 
>> > Maybe using rwlock as airq handler?
>> > 
>> > Thanks
>> > 
>> 
>> rwlock is still a shared cacheline bouncing between CPUs and
>> a bunch of ordering instructions.
>> Maybe something per-cpu + some IPIs to run things on all CPUs instead?
>
> ... and I think classic and device interrupts are different enough
> here ...

You mean classic (per-device) and adapter interrupts, right?

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Tue, Apr 26, 2022 at 05:47:17PM +0200, Cornelia Huck wrote:
> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> >> On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> >> > 
> >> > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> >> > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> >> > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> >> > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> >> > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> > > > > 
> >> > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> >> > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> >> > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> >> > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> >> > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> >> > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> >> > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> >> > > > > > > > > method.
> >> > > > > > > > > 
> >> > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> >> > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> >> > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> >> > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> >> > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> >> > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> >> > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> >> > > > > > > > 
> >> > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> >> > > > > > > > should we be concerned about the performance impact here?
> >> > > > > > > > Any chance it can be tested?
> >> > > > > > > We can have a bunch of devices using the same airq structure, and the
> >> > > > > > > sync cb creates a choke point, same as registering/unregistering.
> >> > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> >> > > > > I'm not sure I understand the question.
> >> > > > > 
> >> > > > > I do think we can have multiple CPUs that are executing some portion of
> >> > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> >> > > > > 
> >> > > > > On the other hand we could also end up serializing synchronize_cbs()
> >> > > > > calls for different devices if they happen to use the same airq_info. But
> >> > > > > this probably was not your question
> >> > > > 
> >> > > > I am less concerned about  synchronize_cbs being slow and more about
> >> > > > the slowdown in interrupt processing itself.
> >> > > > 
> >> > > > > > this patch serializes them on a spinlock.
> >> > > > > > 
> >> > > > > Those could then pile up on the newly introduced spinlock.
> 
> How bad would that be in practice? IIUC, we hit on the spinlock when
> - doing synchronize_cbs (should be rare)
> - processing queue interrupts for devices using per-device indicators
>   (which is the non-preferred path, which I would basically only expect
>   when running on an ancient or non-standard hypervisor)

this one is my concern. I am worried serializing everything on a single lock
will drastically regress performance here.


> - configuration change interrupts (should be rare)
> - during setup, reset, etc. (should not be a concern)
> 
> >> > > > > 
> >> > > > > Regards,
> >> > > > > Halil
> >> > > > Hmm yea ... not good.
> >> > > Is there any other way to synchronize with all callbacks?
> >> > 
> >> > 
> >> > Maybe using rwlock as airq handler?
> >> > 
> >> > Thanks
> >> > 
> >> 
> >> rwlock is still a shared cacheline bouncing between CPUs and
> >> a bunch of ordering instructions.
> >> Maybe something per-cpu + some IPIs to run things on all CPUs instead?
> >
> > ... and I think classic and device interrupts are different enough
> > here ...
> 
> You mean classic (per-device) and adapter interrupts, right?

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Tue, Apr 26, 2022 at 05:47:17PM +0200, Cornelia Huck wrote:
> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> >> On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> >> > 
> >> > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> >> > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> >> > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> >> > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> >> > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> > > > > 
> >> > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> >> > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> >> > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> >> > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> >> > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> >> > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> >> > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> >> > > > > > > > > method.
> >> > > > > > > > > 
> >> > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> >> > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> >> > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> >> > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> >> > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> >> > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> >> > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> >> > > > > > > > 
> >> > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> >> > > > > > > > should we be concerned about the performance impact here?
> >> > > > > > > > Any chance it can be tested?
> >> > > > > > > We can have a bunch of devices using the same airq structure, and the
> >> > > > > > > sync cb creates a choke point, same as registering/unregistering.
> >> > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> >> > > > > I'm not sure I understand the question.
> >> > > > > 
> >> > > > > I do think we can have multiple CPUs that are executing some portion of
> >> > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> >> > > > > 
> >> > > > > On the other hand we could also end up serializing synchronize_cbs()
> >> > > > > calls for different devices if they happen to use the same airq_info. But
> >> > > > > this probably was not your question
> >> > > > 
> >> > > > I am less concerned about  synchronize_cbs being slow and more about
> >> > > > the slowdown in interrupt processing itself.
> >> > > > 
> >> > > > > > this patch serializes them on a spinlock.
> >> > > > > > 
> >> > > > > Those could then pile up on the newly introduced spinlock.
> 
> How bad would that be in practice? IIUC, we hit on the spinlock when
> - doing synchronize_cbs (should be rare)
> - processing queue interrupts for devices using per-device indicators
>   (which is the non-preferred path, which I would basically only expect
>   when running on an ancient or non-standard hypervisor)

this one is my concern. I am worried serializing everything on a single lock
will drastically regress performance here.


> - configuration change interrupts (should be rare)
> - during setup, reset, etc. (should not be a concern)
> 
> >> > > > > 
> >> > > > > Regards,
> >> > > > > Halil
> >> > > > Hmm yea ... not good.
> >> > > Is there any other way to synchronize with all callbacks?
> >> > 
> >> > 
> >> > Maybe using rwlock as airq handler?
> >> > 
> >> > Thanks
> >> > 
> >> 
> >> rwlock is still a shared cacheline bouncing between CPUs and
> >> a bunch of ordering instructions.
> >> Maybe something per-cpu + some IPIs to run things on all CPUs instead?
> >
> > ... and I think classic and device interrupts are different enough
> > here ...
> 
> You mean classic (per-device) and adapter interrupts, right?

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Cornelia Huck]

On Tue, Apr 26 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Tue, Apr 26, 2022 at 05:47:17PM +0200, Cornelia Huck wrote:
>> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> 
>> > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
>> >> On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
>> >> > 
>> >> > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
>> >> > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
>> >> > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
>> >> > > > > On Mon, 25 Apr 2022 09:59:55 -0400
>> >> > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> >> > > > > 
>> >> > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
>> >> > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> >> > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
>> >> > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
>> >> > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
>> >> > > > > > > > > synchronization is simply done via the airq_info's lock. For the
>> >> > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
>> >> > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
>> >> > > > > > > > > method.
>> >> > > > > > > > > 
>> >> > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
>> >> > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
>> >> > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
>> >> > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
>> >> > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
>> >> > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
>> >> > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
>> >> > > > > > > > 
>> >> > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
>> >> > > > > > > > should we be concerned about the performance impact here?
>> >> > > > > > > > Any chance it can be tested?
>> >> > > > > > > We can have a bunch of devices using the same airq structure, and the
>> >> > > > > > > sync cb creates a choke point, same as registering/unregistering.
>> >> > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
>> >> > > > > I'm not sure I understand the question.
>> >> > > > > 
>> >> > > > > I do think we can have multiple CPUs that are executing some portion of
>> >> > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
>> >> > > > > 
>> >> > > > > On the other hand we could also end up serializing synchronize_cbs()
>> >> > > > > calls for different devices if they happen to use the same airq_info. But
>> >> > > > > this probably was not your question
>> >> > > > 
>> >> > > > I am less concerned about  synchronize_cbs being slow and more about
>> >> > > > the slowdown in interrupt processing itself.
>> >> > > > 
>> >> > > > > > this patch serializes them on a spinlock.
>> >> > > > > > 
>> >> > > > > Those could then pile up on the newly introduced spinlock.
>> 
>> How bad would that be in practice? IIUC, we hit on the spinlock when
>> - doing synchronize_cbs (should be rare)
>> - processing queue interrupts for devices using per-device indicators
>>   (which is the non-preferred path, which I would basically only expect
>>   when running on an ancient or non-standard hypervisor)
>
> this one is my concern. I am worried serializing everything on a single lock
> will drastically regress performance here.

Yeah, that case could get much worse. OTOH, how likely is it that any
setup that runs a recent kernel will actually end up with devices using
per-device indicators? Anything running under a QEMU released in the
last couple of years is unlikely to not use airqs, I think. Halil, do
you think that the classic indicator setup would be more common on any
non-QEMU hypervisors?

IOW, how much effort is it worth spending on optimizing this case? We
certainly should explore any simple solutions, but I don't think we need
to twist ourselves into pretzels to solve it.

>
>
>> - configuration change interrupts (should be rare)
>> - during setup, reset, etc. (should not be a concern)

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Cornelia Huck]

On Tue, Apr 26 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Tue, Apr 26, 2022 at 05:47:17PM +0200, Cornelia Huck wrote:
>> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> 
>> > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
>> >> On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
>> >> > 
>> >> > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
>> >> > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
>> >> > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
>> >> > > > > On Mon, 25 Apr 2022 09:59:55 -0400
>> >> > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> >> > > > > 
>> >> > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
>> >> > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> >> > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
>> >> > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
>> >> > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
>> >> > > > > > > > > synchronization is simply done via the airq_info's lock. For the
>> >> > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
>> >> > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
>> >> > > > > > > > > method.
>> >> > > > > > > > > 
>> >> > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
>> >> > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
>> >> > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
>> >> > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
>> >> > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
>> >> > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
>> >> > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
>> >> > > > > > > > 
>> >> > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
>> >> > > > > > > > should we be concerned about the performance impact here?
>> >> > > > > > > > Any chance it can be tested?
>> >> > > > > > > We can have a bunch of devices using the same airq structure, and the
>> >> > > > > > > sync cb creates a choke point, same as registering/unregistering.
>> >> > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
>> >> > > > > I'm not sure I understand the question.
>> >> > > > > 
>> >> > > > > I do think we can have multiple CPUs that are executing some portion of
>> >> > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
>> >> > > > > 
>> >> > > > > On the other hand we could also end up serializing synchronize_cbs()
>> >> > > > > calls for different devices if they happen to use the same airq_info. But
>> >> > > > > this probably was not your question
>> >> > > > 
>> >> > > > I am less concerned about  synchronize_cbs being slow and more about
>> >> > > > the slowdown in interrupt processing itself.
>> >> > > > 
>> >> > > > > > this patch serializes them on a spinlock.
>> >> > > > > > 
>> >> > > > > Those could then pile up on the newly introduced spinlock.
>> 
>> How bad would that be in practice? IIUC, we hit on the spinlock when
>> - doing synchronize_cbs (should be rare)
>> - processing queue interrupts for devices using per-device indicators
>>   (which is the non-preferred path, which I would basically only expect
>>   when running on an ancient or non-standard hypervisor)
>
> this one is my concern. I am worried serializing everything on a single lock
> will drastically regress performance here.

Yeah, that case could get much worse. OTOH, how likely is it that any
setup that runs a recent kernel will actually end up with devices using
per-device indicators? Anything running under a QEMU released in the
last couple of years is unlikely to not use airqs, I think. Halil, do
you think that the classic indicator setup would be more common on any
non-QEMU hypervisors?

IOW, how much effort is it worth spending on optimizing this case? We
certainly should explore any simple solutions, but I don't think we need
to twist ourselves into pretzels to solve it.

>
>
>> - configuration change interrupts (should be rare)
>> - during setup, reset, etc. (should not be a concern)

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Halil Pasic]

On Wed, 27 Apr 2022 11:27:03 +0200
Cornelia Huck <cohuck@redhat.com> wrote:

> On Tue, Apr 26 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Tue, Apr 26, 2022 at 05:47:17PM +0200, Cornelia Huck wrote:  
> >> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >>   
> >> > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:  
> >> >> On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:  
> >> >> > 
> >> >> > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:  
> >> >> > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:  
> >> >> > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:  
> >> >> > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> >> >> > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> >> > > > >   
> >> >> > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:  
> >> >> > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:  
> >> >> > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:  
> >> >> > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> >> >> > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> >> >> > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> >> >> > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> >> >> > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> >> >> > > > > > > > > method.
> >> >> > > > > > > > > 
> >> >> > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> >> >> > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> >> >> > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> >> >> > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> >> >> > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> >> >> > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> >> >> > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>  
> >> >> > > > > > > > 
> >> >> > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> >> >> > > > > > > > should we be concerned about the performance impact here?
> >> >> > > > > > > > Any chance it can be tested?  
> >> >> > > > > > > We can have a bunch of devices using the same airq structure, and the
> >> >> > > > > > > sync cb creates a choke point, same as registering/unregistering.  
> >> >> > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?  
> >> >> > > > > I'm not sure I understand the question.
> >> >> > > > > 
> >> >> > > > > I do think we can have multiple CPUs that are executing some portion of
> >> >> > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> >> >> > > > > 
> >> >> > > > > On the other hand we could also end up serializing synchronize_cbs()
> >> >> > > > > calls for different devices if they happen to use the same airq_info. But
> >> >> > > > > this probably was not your question  
> >> >> > > > 
> >> >> > > > I am less concerned about  synchronize_cbs being slow and more about
> >> >> > > > the slowdown in interrupt processing itself.
> >> >> > > >   
> >> >> > > > > > this patch serializes them on a spinlock.
> >> >> > > > > >   
> >> >> > > > > Those could then pile up on the newly introduced spinlock.  
> >> 
> >> How bad would that be in practice? IIUC, we hit on the spinlock when
> >> - doing synchronize_cbs (should be rare)
> >> - processing queue interrupts for devices using per-device indicators
> >>   (which is the non-preferred path, which I would basically only expect
> >>   when running on an ancient or non-standard hypervisor)  
> >
> > this one is my concern. I am worried serializing everything on a single lock
> > will drastically regress performance here.  
> 
> Yeah, that case could get much worse. OTOH, how likely is it that any
> setup that runs a recent kernel will actually end up with devices using
> per-device indicators? Anything running under a QEMU released in the
> last couple of years is unlikely to not use airqs, I think. Halil, do
> you think that the classic indicator setup would be more common on any
> non-QEMU hypervisors?
> 

I really don't know. My opinion is that, two stages indicators are kind
of recommended for anybody who cares about notifications performance.

> IOW, how much effort is it worth spending on optimizing this case? We
> certainly should explore any simple solutions, but I don't think we need
> to twist ourselves into pretzels to solve it.
> 

Frankly, I would be fine with an rwlock based solution as proposed by
Jason. My rationale is: we recommend two stage indicators, and the two
stage indicators are already encumbered by an rwlock on the interrupt
path. Yes, the coalescence of adapter interrupts is architecturally
different, and so it is with GISA (without GISA, I'm not even sure), so
this rwlock end up being worse than the one for 2 stage. But my feeling
is, that it should be fine. On the other hand, I don't feel comfortable
with plain spinlock, and I am curious about a more advanced solution.
But my guess is that rwlock + some testing for the legacy indicator case
just to double check if there is a heavy regression despite of our
expectations to see none should do the trick.

Regards,
Halil

> >
> >  
> >> - configuration change interrupts (should be rare)
> >> - during setup, reset, etc. (should not be a concern)  
> 

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Halil Pasic]

On Wed, 27 Apr 2022 11:27:03 +0200
Cornelia Huck <cohuck@redhat.com> wrote:

> On Tue, Apr 26 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Tue, Apr 26, 2022 at 05:47:17PM +0200, Cornelia Huck wrote:  
> >> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >>   
> >> > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:  
> >> >> On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:  
> >> >> > 
> >> >> > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:  
> >> >> > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:  
> >> >> > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:  
> >> >> > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> >> >> > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> >> > > > >   
> >> >> > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:  
> >> >> > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:  
> >> >> > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:  
> >> >> > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> >> >> > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> >> >> > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> >> >> > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> >> >> > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> >> >> > > > > > > > > method.
> >> >> > > > > > > > > 
> >> >> > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> >> >> > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> >> >> > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> >> >> > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> >> >> > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> >> >> > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> >> >> > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>  
> >> >> > > > > > > > 
> >> >> > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> >> >> > > > > > > > should we be concerned about the performance impact here?
> >> >> > > > > > > > Any chance it can be tested?  
> >> >> > > > > > > We can have a bunch of devices using the same airq structure, and the
> >> >> > > > > > > sync cb creates a choke point, same as registering/unregistering.  
> >> >> > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?  
> >> >> > > > > I'm not sure I understand the question.
> >> >> > > > > 
> >> >> > > > > I do think we can have multiple CPUs that are executing some portion of
> >> >> > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> >> >> > > > > 
> >> >> > > > > On the other hand we could also end up serializing synchronize_cbs()
> >> >> > > > > calls for different devices if they happen to use the same airq_info. But
> >> >> > > > > this probably was not your question  
> >> >> > > > 
> >> >> > > > I am less concerned about  synchronize_cbs being slow and more about
> >> >> > > > the slowdown in interrupt processing itself.
> >> >> > > >   
> >> >> > > > > > this patch serializes them on a spinlock.
> >> >> > > > > >   
> >> >> > > > > Those could then pile up on the newly introduced spinlock.  
> >> 
> >> How bad would that be in practice? IIUC, we hit on the spinlock when
> >> - doing synchronize_cbs (should be rare)
> >> - processing queue interrupts for devices using per-device indicators
> >>   (which is the non-preferred path, which I would basically only expect
> >>   when running on an ancient or non-standard hypervisor)  
> >
> > this one is my concern. I am worried serializing everything on a single lock
> > will drastically regress performance here.  
> 
> Yeah, that case could get much worse. OTOH, how likely is it that any
> setup that runs a recent kernel will actually end up with devices using
> per-device indicators? Anything running under a QEMU released in the
> last couple of years is unlikely to not use airqs, I think. Halil, do
> you think that the classic indicator setup would be more common on any
> non-QEMU hypervisors?
> 

I really don't know. My opinion is that, two stages indicators are kind
of recommended for anybody who cares about notifications performance.

> IOW, how much effort is it worth spending on optimizing this case? We
> certainly should explore any simple solutions, but I don't think we need
> to twist ourselves into pretzels to solve it.
> 

Frankly, I would be fine with an rwlock based solution as proposed by
Jason. My rationale is: we recommend two stage indicators, and the two
stage indicators are already encumbered by an rwlock on the interrupt
path. Yes, the coalescence of adapter interrupts is architecturally
different, and so it is with GISA (without GISA, I'm not even sure), so
this rwlock end up being worse than the one for 2 stage. But my feeling
is, that it should be fine. On the other hand, I don't feel comfortable
with plain spinlock, and I am curious about a more advanced solution.
But my guess is that rwlock + some testing for the legacy indicator case
just to double check if there is a heavy regression despite of our
expectations to see none should do the trick.

Regards,
Halil

> >
> >  
> >> - configuration change interrupts (should be rare)
> >> - during setup, reset, etc. (should not be a concern)  
> 

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

On Thu, Apr 28, 2022 at 10:43 AM Halil Pasic <pasic@linux.ibm.com> wrote:
>
> On Wed, 27 Apr 2022 11:27:03 +0200
> Cornelia Huck <cohuck@redhat.com> wrote:
>
> > On Tue, Apr 26 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >
> > > On Tue, Apr 26, 2022 at 05:47:17PM +0200, Cornelia Huck wrote:
> > >> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >>
> > >> > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> > >> >> On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > >> >> >
> > >> >> > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > >> >> > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > >> >> > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > >> >> > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > >> >> > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >> >> > > > >
> > >> >> > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > >> >> > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >> >> > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > >> >> > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > >> >> > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > >> >> > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > >> >> > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > >> >> > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > >> >> > > > > > > > > method.
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > >> >> > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > >> >> > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > >> >> > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > >> >> > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > >> >> > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > >> >> > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > >> >> > > > > > > >
> > >> >> > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > >> >> > > > > > > > should we be concerned about the performance impact here?
> > >> >> > > > > > > > Any chance it can be tested?
> > >> >> > > > > > > We can have a bunch of devices using the same airq structure, and the
> > >> >> > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > >> >> > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > >> >> > > > > I'm not sure I understand the question.
> > >> >> > > > >
> > >> >> > > > > I do think we can have multiple CPUs that are executing some portion of
> > >> >> > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > >> >> > > > >
> > >> >> > > > > On the other hand we could also end up serializing synchronize_cbs()
> > >> >> > > > > calls for different devices if they happen to use the same airq_info. But
> > >> >> > > > > this probably was not your question
> > >> >> > > >
> > >> >> > > > I am less concerned about  synchronize_cbs being slow and more about
> > >> >> > > > the slowdown in interrupt processing itself.
> > >> >> > > >
> > >> >> > > > > > this patch serializes them on a spinlock.
> > >> >> > > > > >
> > >> >> > > > > Those could then pile up on the newly introduced spinlock.
> > >>
> > >> How bad would that be in practice? IIUC, we hit on the spinlock when
> > >> - doing synchronize_cbs (should be rare)
> > >> - processing queue interrupts for devices using per-device indicators
> > >>   (which is the non-preferred path, which I would basically only expect
> > >>   when running on an ancient or non-standard hypervisor)
> > >
> > > this one is my concern. I am worried serializing everything on a single lock
> > > will drastically regress performance here.
> >
> > Yeah, that case could get much worse. OTOH, how likely is it that any
> > setup that runs a recent kernel will actually end up with devices using
> > per-device indicators? Anything running under a QEMU released in the
> > last couple of years is unlikely to not use airqs, I think. Halil, do
> > you think that the classic indicator setup would be more common on any
> > non-QEMU hypervisors?
> >
>
> I really don't know. My opinion is that, two stages indicators are kind
> of recommended for anybody who cares about notifications performance.
>
> > IOW, how much effort is it worth spending on optimizing this case? We
> > certainly should explore any simple solutions, but I don't think we need
> > to twist ourselves into pretzels to solve it.
> >
>
> Frankly, I would be fine with an rwlock based solution as proposed by
> Jason. My rationale is: we recommend two stage indicators, and the two
> stage indicators are already encumbered by an rwlock on the interrupt
> path. Yes, the coalescence of adapter interrupts is architecturally
> different, and so it is with GISA (without GISA, I'm not even sure), so
> this rwlock end up being worse than the one for 2 stage. But my feeling
> is, that it should be fine. On the other hand, I don't feel comfortable
> with plain spinlock, and I am curious about a more advanced solution.

Yes, I'm trying to use (S)RCU, let's see if it works.

> But my guess is that rwlock + some testing for the legacy indicator case
> just to double check if there is a heavy regression despite of our
> expectations to see none should do the trick.

I suggest this, rwlock (for not airq) seems better than spinlock, but
at worst case it will cause cache line bouncing. But I wonder if it's
noticeable (anyhow it has been used for airq).

Thanks

>
> Regards,
> Halil
>
> > >
> > >
> > >> - configuration change interrupts (should be rare)
> > >> - during setup, reset, etc. (should not be a concern)
> >
>

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

On Thu, Apr 28, 2022 at 10:43 AM Halil Pasic <pasic@linux.ibm.com> wrote:
>
> On Wed, 27 Apr 2022 11:27:03 +0200
> Cornelia Huck <cohuck@redhat.com> wrote:
>
> > On Tue, Apr 26 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >
> > > On Tue, Apr 26, 2022 at 05:47:17PM +0200, Cornelia Huck wrote:
> > >> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >>
> > >> > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:
> > >> >> On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:
> > >> >> >
> > >> >> > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:
> > >> >> > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:
> > >> >> > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:
> > >> >> > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > >> >> > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >> >> > > > >
> > >> >> > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:
> > >> >> > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >> >> > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:
> > >> >> > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > >> >> > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > >> >> > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > >> >> > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > >> >> > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > >> >> > > > > > > > > method.
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > >> >> > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > >> >> > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > >> >> > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > >> >> > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > >> >> > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > >> >> > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > >> >> > > > > > > >
> > >> >> > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > >> >> > > > > > > > should we be concerned about the performance impact here?
> > >> >> > > > > > > > Any chance it can be tested?
> > >> >> > > > > > > We can have a bunch of devices using the same airq structure, and the
> > >> >> > > > > > > sync cb creates a choke point, same as registering/unregistering.
> > >> >> > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?
> > >> >> > > > > I'm not sure I understand the question.
> > >> >> > > > >
> > >> >> > > > > I do think we can have multiple CPUs that are executing some portion of
> > >> >> > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > >> >> > > > >
> > >> >> > > > > On the other hand we could also end up serializing synchronize_cbs()
> > >> >> > > > > calls for different devices if they happen to use the same airq_info. But
> > >> >> > > > > this probably was not your question
> > >> >> > > >
> > >> >> > > > I am less concerned about  synchronize_cbs being slow and more about
> > >> >> > > > the slowdown in interrupt processing itself.
> > >> >> > > >
> > >> >> > > > > > this patch serializes them on a spinlock.
> > >> >> > > > > >
> > >> >> > > > > Those could then pile up on the newly introduced spinlock.
> > >>
> > >> How bad would that be in practice? IIUC, we hit on the spinlock when
> > >> - doing synchronize_cbs (should be rare)
> > >> - processing queue interrupts for devices using per-device indicators
> > >>   (which is the non-preferred path, which I would basically only expect
> > >>   when running on an ancient or non-standard hypervisor)
> > >
> > > this one is my concern. I am worried serializing everything on a single lock
> > > will drastically regress performance here.
> >
> > Yeah, that case could get much worse. OTOH, how likely is it that any
> > setup that runs a recent kernel will actually end up with devices using
> > per-device indicators? Anything running under a QEMU released in the
> > last couple of years is unlikely to not use airqs, I think. Halil, do
> > you think that the classic indicator setup would be more common on any
> > non-QEMU hypervisors?
> >
>
> I really don't know. My opinion is that, two stages indicators are kind
> of recommended for anybody who cares about notifications performance.
>
> > IOW, how much effort is it worth spending on optimizing this case? We
> > certainly should explore any simple solutions, but I don't think we need
> > to twist ourselves into pretzels to solve it.
> >
>
> Frankly, I would be fine with an rwlock based solution as proposed by
> Jason. My rationale is: we recommend two stage indicators, and the two
> stage indicators are already encumbered by an rwlock on the interrupt
> path. Yes, the coalescence of adapter interrupts is architecturally
> different, and so it is with GISA (without GISA, I'm not even sure), so
> this rwlock end up being worse than the one for 2 stage. But my feeling
> is, that it should be fine. On the other hand, I don't feel comfortable
> with plain spinlock, and I am curious about a more advanced solution.

Yes, I'm trying to use (S)RCU, let's see if it works.

> But my guess is that rwlock + some testing for the legacy indicator case
> just to double check if there is a heavy regression despite of our
> expectations to see none should do the trick.

I suggest this, rwlock (for not airq) seems better than spinlock, but
at worst case it will cause cache line bouncing. But I wonder if it's
noticeable (anyhow it has been used for airq).

Thanks

>
> Regards,
> Halil
>
> > >
> > >
> > >> - configuration change interrupts (should be rare)
> > >> - during setup, reset, etc. (should not be a concern)
> >
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Thu, Apr 28, 2022 at 11:04:41AM +0800, Jason Wang wrote:
> > But my guess is that rwlock + some testing for the legacy indicator case
> > just to double check if there is a heavy regression despite of our
> > expectations to see none should do the trick.
> 
> I suggest this, rwlock (for not airq) seems better than spinlock, but
> at worst case it will cause cache line bouncing. But I wonder if it's
> noticeable (anyhow it has been used for airq).
> 
> Thanks

Which existing rwlock does airq use right now? Can we take it to sync?

-- 
MST

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Thu, Apr 28, 2022 at 11:04:41AM +0800, Jason Wang wrote:
> > But my guess is that rwlock + some testing for the legacy indicator case
> > just to double check if there is a heavy regression despite of our
> > expectations to see none should do the trick.
> 
> I suggest this, rwlock (for not airq) seems better than spinlock, but
> at worst case it will cause cache line bouncing. But I wonder if it's
> noticeable (anyhow it has been used for airq).
> 
> Thanks

Which existing rwlock does airq use right now? Can we take it to sync?

-- 
MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

On Thu, Apr 28, 2022 at 1:24 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Thu, Apr 28, 2022 at 11:04:41AM +0800, Jason Wang wrote:
> > > But my guess is that rwlock + some testing for the legacy indicator case
> > > just to double check if there is a heavy regression despite of our
> > > expectations to see none should do the trick.
> >
> > I suggest this, rwlock (for not airq) seems better than spinlock, but
> > at worst case it will cause cache line bouncing. But I wonder if it's
> > noticeable (anyhow it has been used for airq).
> >
> > Thanks
>
> Which existing rwlock does airq use right now? Can we take it to sync?

It's the rwlock in airq_info, it has already been used in this patch.

                write_lock(&info->lock);
                write_unlock(&info->lock);

But the problem is, it looks to me there could be a case that airq is
not used, (virtio_ccw_int_hander()). That's why the patch use a
spinlock, it could be optimized with using a rwlock as well.

Thanks

>
> --
> MST
>

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

On Thu, Apr 28, 2022 at 1:24 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Thu, Apr 28, 2022 at 11:04:41AM +0800, Jason Wang wrote:
> > > But my guess is that rwlock + some testing for the legacy indicator case
> > > just to double check if there is a heavy regression despite of our
> > > expectations to see none should do the trick.
> >
> > I suggest this, rwlock (for not airq) seems better than spinlock, but
> > at worst case it will cause cache line bouncing. But I wonder if it's
> > noticeable (anyhow it has been used for airq).
> >
> > Thanks
>
> Which existing rwlock does airq use right now? Can we take it to sync?

It's the rwlock in airq_info, it has already been used in this patch.

                write_lock(&info->lock);
                write_unlock(&info->lock);

But the problem is, it looks to me there could be a case that airq is
not used, (virtio_ccw_int_hander()). That's why the patch use a
spinlock, it could be optimized with using a rwlock as well.

Thanks

>
> --
> MST
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Thu, Apr 28, 2022 at 01:51:59PM +0800, Jason Wang wrote:
> On Thu, Apr 28, 2022 at 1:24 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Thu, Apr 28, 2022 at 11:04:41AM +0800, Jason Wang wrote:
> > > > But my guess is that rwlock + some testing for the legacy indicator case
> > > > just to double check if there is a heavy regression despite of our
> > > > expectations to see none should do the trick.
> > >
> > > I suggest this, rwlock (for not airq) seems better than spinlock, but
> > > at worst case it will cause cache line bouncing. But I wonder if it's
> > > noticeable (anyhow it has been used for airq).
> > >
> > > Thanks
> >
> > Which existing rwlock does airq use right now? Can we take it to sync?
> 
> It's the rwlock in airq_info, it has already been used in this patch.
> 
>                 write_lock(&info->lock);
>                 write_unlock(&info->lock);
> 
> But the problem is, it looks to me there could be a case that airq is
> not used, (virtio_ccw_int_hander()). That's why the patch use a
> spinlock, it could be optimized with using a rwlock as well.
> 
> Thanks

Ah, right. So let's take that on the legacy path too and Halil promises
to test to make sure performance isn't impacted too badly?

> >
> > --
> > MST
> >

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Thu, Apr 28, 2022 at 01:51:59PM +0800, Jason Wang wrote:
> On Thu, Apr 28, 2022 at 1:24 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Thu, Apr 28, 2022 at 11:04:41AM +0800, Jason Wang wrote:
> > > > But my guess is that rwlock + some testing for the legacy indicator case
> > > > just to double check if there is a heavy regression despite of our
> > > > expectations to see none should do the trick.
> > >
> > > I suggest this, rwlock (for not airq) seems better than spinlock, but
> > > at worst case it will cause cache line bouncing. But I wonder if it's
> > > noticeable (anyhow it has been used for airq).
> > >
> > > Thanks
> >
> > Which existing rwlock does airq use right now? Can we take it to sync?
> 
> It's the rwlock in airq_info, it has already been used in this patch.
> 
>                 write_lock(&info->lock);
>                 write_unlock(&info->lock);
> 
> But the problem is, it looks to me there could be a case that airq is
> not used, (virtio_ccw_int_hander()). That's why the patch use a
> spinlock, it could be optimized with using a rwlock as well.
> 
> Thanks

Ah, right. So let's take that on the legacy path too and Halil promises
to test to make sure performance isn't impacted too badly?

> >
> > --
> > MST
> >

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

On Thu, Apr 28, 2022 at 1:55 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Thu, Apr 28, 2022 at 01:51:59PM +0800, Jason Wang wrote:
> > On Thu, Apr 28, 2022 at 1:24 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Thu, Apr 28, 2022 at 11:04:41AM +0800, Jason Wang wrote:
> > > > > But my guess is that rwlock + some testing for the legacy indicator case
> > > > > just to double check if there is a heavy regression despite of our
> > > > > expectations to see none should do the trick.
> > > >
> > > > I suggest this, rwlock (for not airq) seems better than spinlock, but
> > > > at worst case it will cause cache line bouncing. But I wonder if it's
> > > > noticeable (anyhow it has been used for airq).
> > > >
> > > > Thanks
> > >
> > > Which existing rwlock does airq use right now? Can we take it to sync?
> >
> > It's the rwlock in airq_info, it has already been used in this patch.
> >
> >                 write_lock(&info->lock);
> >                 write_unlock(&info->lock);
> >
> > But the problem is, it looks to me there could be a case that airq is
> > not used, (virtio_ccw_int_hander()). That's why the patch use a
> > spinlock, it could be optimized with using a rwlock as well.
> >
> > Thanks
>
> Ah, right. So let's take that on the legacy path too and Halil promises
> to test to make sure performance isn't impacted too badly?

I think what you meant is using a dedicated rwlock instead of trying
to reuse one of the airq_info locks.

If this is true, it should be fine.

Thanks

>
> > >
> > > --
> > > MST
> > >
>

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

On Thu, Apr 28, 2022 at 1:55 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Thu, Apr 28, 2022 at 01:51:59PM +0800, Jason Wang wrote:
> > On Thu, Apr 28, 2022 at 1:24 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Thu, Apr 28, 2022 at 11:04:41AM +0800, Jason Wang wrote:
> > > > > But my guess is that rwlock + some testing for the legacy indicator case
> > > > > just to double check if there is a heavy regression despite of our
> > > > > expectations to see none should do the trick.
> > > >
> > > > I suggest this, rwlock (for not airq) seems better than spinlock, but
> > > > at worst case it will cause cache line bouncing. But I wonder if it's
> > > > noticeable (anyhow it has been used for airq).
> > > >
> > > > Thanks
> > >
> > > Which existing rwlock does airq use right now? Can we take it to sync?
> >
> > It's the rwlock in airq_info, it has already been used in this patch.
> >
> >                 write_lock(&info->lock);
> >                 write_unlock(&info->lock);
> >
> > But the problem is, it looks to me there could be a case that airq is
> > not used, (virtio_ccw_int_hander()). That's why the patch use a
> > spinlock, it could be optimized with using a rwlock as well.
> >
> > Thanks
>
> Ah, right. So let's take that on the legacy path too and Halil promises
> to test to make sure performance isn't impacted too badly?

I think what you meant is using a dedicated rwlock instead of trying
to reuse one of the airq_info locks.

If this is true, it should be fine.

Thanks

>
> > >
> > > --
> > > MST
> > >
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Thu, Apr 28, 2022 at 02:02:16PM +0800, Jason Wang wrote:
> On Thu, Apr 28, 2022 at 1:55 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Thu, Apr 28, 2022 at 01:51:59PM +0800, Jason Wang wrote:
> > > On Thu, Apr 28, 2022 at 1:24 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Thu, Apr 28, 2022 at 11:04:41AM +0800, Jason Wang wrote:
> > > > > > But my guess is that rwlock + some testing for the legacy indicator case
> > > > > > just to double check if there is a heavy regression despite of our
> > > > > > expectations to see none should do the trick.
> > > > >
> > > > > I suggest this, rwlock (for not airq) seems better than spinlock, but
> > > > > at worst case it will cause cache line bouncing. But I wonder if it's
> > > > > noticeable (anyhow it has been used for airq).
> > > > >
> > > > > Thanks
> > > >
> > > > Which existing rwlock does airq use right now? Can we take it to sync?
> > >
> > > It's the rwlock in airq_info, it has already been used in this patch.
> > >
> > >                 write_lock(&info->lock);
> > >                 write_unlock(&info->lock);
> > >
> > > But the problem is, it looks to me there could be a case that airq is
> > > not used, (virtio_ccw_int_hander()). That's why the patch use a
> > > spinlock, it could be optimized with using a rwlock as well.
> > >
> > > Thanks
> >
> > Ah, right. So let's take that on the legacy path too and Halil promises
> > to test to make sure performance isn't impacted too badly?
> 
> I think what you meant is using a dedicated rwlock instead of trying
> to reuse one of the airq_info locks.
> 
> If this is true, it should be fine.
> 
> Thanks

yes

> >
> > > >
> > > > --
> > > > MST
> > > >
> >

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Thu, Apr 28, 2022 at 02:02:16PM +0800, Jason Wang wrote:
> On Thu, Apr 28, 2022 at 1:55 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Thu, Apr 28, 2022 at 01:51:59PM +0800, Jason Wang wrote:
> > > On Thu, Apr 28, 2022 at 1:24 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Thu, Apr 28, 2022 at 11:04:41AM +0800, Jason Wang wrote:
> > > > > > But my guess is that rwlock + some testing for the legacy indicator case
> > > > > > just to double check if there is a heavy regression despite of our
> > > > > > expectations to see none should do the trick.
> > > > >
> > > > > I suggest this, rwlock (for not airq) seems better than spinlock, but
> > > > > at worst case it will cause cache line bouncing. But I wonder if it's
> > > > > noticeable (anyhow it has been used for airq).
> > > > >
> > > > > Thanks
> > > >
> > > > Which existing rwlock does airq use right now? Can we take it to sync?
> > >
> > > It's the rwlock in airq_info, it has already been used in this patch.
> > >
> > >                 write_lock(&info->lock);
> > >                 write_unlock(&info->lock);
> > >
> > > But the problem is, it looks to me there could be a case that airq is
> > > not used, (virtio_ccw_int_hander()). That's why the patch use a
> > > spinlock, it could be optimized with using a rwlock as well.
> > >
> > > Thanks
> >
> > Ah, right. So let's take that on the legacy path too and Halil promises
> > to test to make sure performance isn't impacted too badly?
> 
> I think what you meant is using a dedicated rwlock instead of trying
> to reuse one of the airq_info locks.
> 
> If this is true, it should be fine.
> 
> Thanks

yes

> >
> > > >
> > > > --
> > > > MST
> > > >
> >

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Cornelia Huck]

On Thu, Apr 28 2022, Jason Wang <jasowang@redhat.com> wrote:

> On Thu, Apr 28, 2022 at 1:55 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>
>> On Thu, Apr 28, 2022 at 01:51:59PM +0800, Jason Wang wrote:
>> > On Thu, Apr 28, 2022 at 1:24 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>> > >
>> > > On Thu, Apr 28, 2022 at 11:04:41AM +0800, Jason Wang wrote:
>> > > > > But my guess is that rwlock + some testing for the legacy indicator case
>> > > > > just to double check if there is a heavy regression despite of our
>> > > > > expectations to see none should do the trick.
>> > > >
>> > > > I suggest this, rwlock (for not airq) seems better than spinlock, but
>> > > > at worst case it will cause cache line bouncing. But I wonder if it's
>> > > > noticeable (anyhow it has been used for airq).
>> > > >
>> > > > Thanks
>> > >
>> > > Which existing rwlock does airq use right now? Can we take it to sync?
>> >
>> > It's the rwlock in airq_info, it has already been used in this patch.
>> >
>> >                 write_lock(&info->lock);
>> >                 write_unlock(&info->lock);
>> >
>> > But the problem is, it looks to me there could be a case that airq is
>> > not used, (virtio_ccw_int_hander()). That's why the patch use a
>> > spinlock, it could be optimized with using a rwlock as well.
>> >
>> > Thanks
>>
>> Ah, right. So let's take that on the legacy path too and Halil promises
>> to test to make sure performance isn't impacted too badly?
>
> I think what you meant is using a dedicated rwlock instead of trying
> to reuse one of the airq_info locks.
>
> If this is true, it should be fine.

FWIW, that approach makes sense to me as well.

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Cornelia Huck]

On Thu, Apr 28 2022, Jason Wang <jasowang@redhat.com> wrote:

> On Thu, Apr 28, 2022 at 1:55 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>
>> On Thu, Apr 28, 2022 at 01:51:59PM +0800, Jason Wang wrote:
>> > On Thu, Apr 28, 2022 at 1:24 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>> > >
>> > > On Thu, Apr 28, 2022 at 11:04:41AM +0800, Jason Wang wrote:
>> > > > > But my guess is that rwlock + some testing for the legacy indicator case
>> > > > > just to double check if there is a heavy regression despite of our
>> > > > > expectations to see none should do the trick.
>> > > >
>> > > > I suggest this, rwlock (for not airq) seems better than spinlock, but
>> > > > at worst case it will cause cache line bouncing. But I wonder if it's
>> > > > noticeable (anyhow it has been used for airq).
>> > > >
>> > > > Thanks
>> > >
>> > > Which existing rwlock does airq use right now? Can we take it to sync?
>> >
>> > It's the rwlock in airq_info, it has already been used in this patch.
>> >
>> >                 write_lock(&info->lock);
>> >                 write_unlock(&info->lock);
>> >
>> > But the problem is, it looks to me there could be a case that airq is
>> > not used, (virtio_ccw_int_hander()). That's why the patch use a
>> > spinlock, it could be optimized with using a rwlock as well.
>> >
>> > Thanks
>>
>> Ah, right. So let's take that on the legacy path too and Halil promises
>> to test to make sure performance isn't impacted too badly?
>
> I think what you meant is using a dedicated rwlock instead of trying
> to reuse one of the airq_info locks.
>
> If this is true, it should be fine.

FWIW, that approach makes sense to me as well.

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

On Thu, Apr 28, 2022 at 3:42 PM Cornelia Huck <cohuck@redhat.com> wrote:
>
> On Thu, Apr 28 2022, Jason Wang <jasowang@redhat.com> wrote:
>
> > On Thu, Apr 28, 2022 at 1:55 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >>
> >> On Thu, Apr 28, 2022 at 01:51:59PM +0800, Jason Wang wrote:
> >> > On Thu, Apr 28, 2022 at 1:24 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >> > >
> >> > > On Thu, Apr 28, 2022 at 11:04:41AM +0800, Jason Wang wrote:
> >> > > > > But my guess is that rwlock + some testing for the legacy indicator case
> >> > > > > just to double check if there is a heavy regression despite of our
> >> > > > > expectations to see none should do the trick.
> >> > > >
> >> > > > I suggest this, rwlock (for not airq) seems better than spinlock, but
> >> > > > at worst case it will cause cache line bouncing. But I wonder if it's
> >> > > > noticeable (anyhow it has been used for airq).
> >> > > >
> >> > > > Thanks
> >> > >
> >> > > Which existing rwlock does airq use right now? Can we take it to sync?
> >> >
> >> > It's the rwlock in airq_info, it has already been used in this patch.
> >> >
> >> >                 write_lock(&info->lock);
> >> >                 write_unlock(&info->lock);
> >> >
> >> > But the problem is, it looks to me there could be a case that airq is
> >> > not used, (virtio_ccw_int_hander()). That's why the patch use a
> >> > spinlock, it could be optimized with using a rwlock as well.
> >> >
> >> > Thanks
> >>
> >> Ah, right. So let's take that on the legacy path too and Halil promises
> >> to test to make sure performance isn't impacted too badly?
> >
> > I think what you meant is using a dedicated rwlock instead of trying
> > to reuse one of the airq_info locks.
> >
> > If this is true, it should be fine.
>
> FWIW, that approach makes sense to me as well.
>

Good to know that. Let me post a new version.

Thanks

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

On Thu, Apr 28, 2022 at 3:42 PM Cornelia Huck <cohuck@redhat.com> wrote:
>
> On Thu, Apr 28 2022, Jason Wang <jasowang@redhat.com> wrote:
>
> > On Thu, Apr 28, 2022 at 1:55 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >>
> >> On Thu, Apr 28, 2022 at 01:51:59PM +0800, Jason Wang wrote:
> >> > On Thu, Apr 28, 2022 at 1:24 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >> > >
> >> > > On Thu, Apr 28, 2022 at 11:04:41AM +0800, Jason Wang wrote:
> >> > > > > But my guess is that rwlock + some testing for the legacy indicator case
> >> > > > > just to double check if there is a heavy regression despite of our
> >> > > > > expectations to see none should do the trick.
> >> > > >
> >> > > > I suggest this, rwlock (for not airq) seems better than spinlock, but
> >> > > > at worst case it will cause cache line bouncing. But I wonder if it's
> >> > > > noticeable (anyhow it has been used for airq).
> >> > > >
> >> > > > Thanks
> >> > >
> >> > > Which existing rwlock does airq use right now? Can we take it to sync?
> >> >
> >> > It's the rwlock in airq_info, it has already been used in this patch.
> >> >
> >> >                 write_lock(&info->lock);
> >> >                 write_unlock(&info->lock);
> >> >
> >> > But the problem is, it looks to me there could be a case that airq is
> >> > not used, (virtio_ccw_int_hander()). That's why the patch use a
> >> > spinlock, it could be optimized with using a rwlock as well.
> >> >
> >> > Thanks
> >>
> >> Ah, right. So let's take that on the legacy path too and Halil promises
> >> to test to make sure performance isn't impacted too badly?
> >
> > I think what you meant is using a dedicated rwlock instead of trying
> > to reuse one of the airq_info locks.
> >
> > If this is true, it should be fine.
>
> FWIW, that approach makes sense to me as well.
>

Good to know that. Let me post a new version.

Thanks

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Thu, Apr 28, 2022 at 04:43:15AM +0200, Halil Pasic wrote:
> On Wed, 27 Apr 2022 11:27:03 +0200
> Cornelia Huck <cohuck@redhat.com> wrote:
> 
> > On Tue, Apr 26 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > 
> > > On Tue, Apr 26, 2022 at 05:47:17PM +0200, Cornelia Huck wrote:  
> > >> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >>   
> > >> > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:  
> > >> >> On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:  
> > >> >> > 
> > >> >> > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:  
> > >> >> > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:  
> > >> >> > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:  
> > >> >> > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > >> >> > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >> >> > > > >   
> > >> >> > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:  
> > >> >> > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:  
> > >> >> > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:  
> > >> >> > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > >> >> > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > >> >> > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > >> >> > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > >> >> > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > >> >> > > > > > > > > method.
> > >> >> > > > > > > > > 
> > >> >> > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > >> >> > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > >> >> > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > >> >> > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > >> >> > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > >> >> > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > >> >> > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>  
> > >> >> > > > > > > > 
> > >> >> > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > >> >> > > > > > > > should we be concerned about the performance impact here?
> > >> >> > > > > > > > Any chance it can be tested?  
> > >> >> > > > > > > We can have a bunch of devices using the same airq structure, and the
> > >> >> > > > > > > sync cb creates a choke point, same as registering/unregistering.  
> > >> >> > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?  
> > >> >> > > > > I'm not sure I understand the question.
> > >> >> > > > > 
> > >> >> > > > > I do think we can have multiple CPUs that are executing some portion of
> > >> >> > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > >> >> > > > > 
> > >> >> > > > > On the other hand we could also end up serializing synchronize_cbs()
> > >> >> > > > > calls for different devices if they happen to use the same airq_info. But
> > >> >> > > > > this probably was not your question  
> > >> >> > > > 
> > >> >> > > > I am less concerned about  synchronize_cbs being slow and more about
> > >> >> > > > the slowdown in interrupt processing itself.
> > >> >> > > >   
> > >> >> > > > > > this patch serializes them on a spinlock.
> > >> >> > > > > >   
> > >> >> > > > > Those could then pile up on the newly introduced spinlock.  
> > >> 
> > >> How bad would that be in practice? IIUC, we hit on the spinlock when
> > >> - doing synchronize_cbs (should be rare)
> > >> - processing queue interrupts for devices using per-device indicators
> > >>   (which is the non-preferred path, which I would basically only expect
> > >>   when running on an ancient or non-standard hypervisor)  
> > >
> > > this one is my concern. I am worried serializing everything on a single lock
> > > will drastically regress performance here.  
> > 
> > Yeah, that case could get much worse. OTOH, how likely is it that any
> > setup that runs a recent kernel will actually end up with devices using
> > per-device indicators? Anything running under a QEMU released in the
> > last couple of years is unlikely to not use airqs, I think. Halil, do
> > you think that the classic indicator setup would be more common on any
> > non-QEMU hypervisors?
> > 
> 
> I really don't know. My opinion is that, two stages indicators are kind
> of recommended for anybody who cares about notifications performance.
> 
> > IOW, how much effort is it worth spending on optimizing this case? We
> > certainly should explore any simple solutions, but I don't think we need
> > to twist ourselves into pretzels to solve it.
> > 
> 
> Frankly, I would be fine with an rwlock based solution as proposed by
> Jason. My rationale is: we recommend two stage indicators, and the two
> stage indicators are already encumbered by an rwlock on the interrupt
> path.

Which lock is it? Can we take that lock to synchronize?

> Yes, the coalescence of adapter interrupts is architecturally
> different, and so it is with GISA (without GISA, I'm not even sure), so
> this rwlock end up being worse than the one for 2 stage. But my feeling
> is, that it should be fine. On the other hand, I don't feel comfortable
> with plain spinlock, and I am curious about a more advanced solution.
> But my guess is that rwlock + some testing for the legacy indicator case
> just to double check if there is a heavy regression despite of our
> expectations to see none should do the trick.
> 
> Regards,
> Halil

Makes sense.

> > >
> > >  
> > >> - configuration change interrupts (should be rare)
> > >> - during setup, reset, etc. (should not be a concern)  
> > 

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Michael S. Tsirkin]

On Thu, Apr 28, 2022 at 04:43:15AM +0200, Halil Pasic wrote:
> On Wed, 27 Apr 2022 11:27:03 +0200
> Cornelia Huck <cohuck@redhat.com> wrote:
> 
> > On Tue, Apr 26 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > 
> > > On Tue, Apr 26, 2022 at 05:47:17PM +0200, Cornelia Huck wrote:  
> > >> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >>   
> > >> > On Mon, Apr 25, 2022 at 11:53:24PM -0400, Michael S. Tsirkin wrote:  
> > >> >> On Tue, Apr 26, 2022 at 11:42:45AM +0800, Jason Wang wrote:  
> > >> >> > 
> > >> >> > 在 2022/4/26 11:38, Michael S. Tsirkin 写道:  
> > >> >> > > On Mon, Apr 25, 2022 at 11:35:41PM -0400, Michael S. Tsirkin wrote:  
> > >> >> > > > On Tue, Apr 26, 2022 at 04:29:11AM +0200, Halil Pasic wrote:  
> > >> >> > > > > On Mon, 25 Apr 2022 09:59:55 -0400
> > >> >> > > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >> >> > > > >   
> > >> >> > > > > > On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:  
> > >> >> > > > > > > On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:  
> > >> >> > > > > > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:  
> > >> >> > > > > > > > > This patch tries to implement the synchronize_cbs() for ccw. For the
> > >> >> > > > > > > > > vring_interrupt() that is called via virtio_airq_handler(), the
> > >> >> > > > > > > > > synchronization is simply done via the airq_info's lock. For the
> > >> >> > > > > > > > > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > >> >> > > > > > > > > device spinlock for irq is introduced ans used in the synchronization
> > >> >> > > > > > > > > method.
> > >> >> > > > > > > > > 
> > >> >> > > > > > > > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > >> >> > > > > > > > > Cc: Peter Zijlstra <peterz@infradead.org>
> > >> >> > > > > > > > > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > >> >> > > > > > > > > Cc: Marc Zyngier <maz@kernel.org>
> > >> >> > > > > > > > > Cc: Halil Pasic <pasic@linux.ibm.com>
> > >> >> > > > > > > > > Cc: Cornelia Huck <cohuck@redhat.com>
> > >> >> > > > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com>  
> > >> >> > > > > > > > 
> > >> >> > > > > > > > This is the only one that is giving me pause. Halil, Cornelia,
> > >> >> > > > > > > > should we be concerned about the performance impact here?
> > >> >> > > > > > > > Any chance it can be tested?  
> > >> >> > > > > > > We can have a bunch of devices using the same airq structure, and the
> > >> >> > > > > > > sync cb creates a choke point, same as registering/unregistering.  
> > >> >> > > > > > BTW can callbacks for multiple VQs run on multiple CPUs at the moment?  
> > >> >> > > > > I'm not sure I understand the question.
> > >> >> > > > > 
> > >> >> > > > > I do think we can have multiple CPUs that are executing some portion of
> > >> >> > > > > virtio_ccw_int_handler(). So I guess the answer is yes. Connie what do you think?
> > >> >> > > > > 
> > >> >> > > > > On the other hand we could also end up serializing synchronize_cbs()
> > >> >> > > > > calls for different devices if they happen to use the same airq_info. But
> > >> >> > > > > this probably was not your question  
> > >> >> > > > 
> > >> >> > > > I am less concerned about  synchronize_cbs being slow and more about
> > >> >> > > > the slowdown in interrupt processing itself.
> > >> >> > > >   
> > >> >> > > > > > this patch serializes them on a spinlock.
> > >> >> > > > > >   
> > >> >> > > > > Those could then pile up on the newly introduced spinlock.  
> > >> 
> > >> How bad would that be in practice? IIUC, we hit on the spinlock when
> > >> - doing synchronize_cbs (should be rare)
> > >> - processing queue interrupts for devices using per-device indicators
> > >>   (which is the non-preferred path, which I would basically only expect
> > >>   when running on an ancient or non-standard hypervisor)  
> > >
> > > this one is my concern. I am worried serializing everything on a single lock
> > > will drastically regress performance here.  
> > 
> > Yeah, that case could get much worse. OTOH, how likely is it that any
> > setup that runs a recent kernel will actually end up with devices using
> > per-device indicators? Anything running under a QEMU released in the
> > last couple of years is unlikely to not use airqs, I think. Halil, do
> > you think that the classic indicator setup would be more common on any
> > non-QEMU hypervisors?
> > 
> 
> I really don't know. My opinion is that, two stages indicators are kind
> of recommended for anybody who cares about notifications performance.
> 
> > IOW, how much effort is it worth spending on optimizing this case? We
> > certainly should explore any simple solutions, but I don't think we need
> > to twist ourselves into pretzels to solve it.
> > 
> 
> Frankly, I would be fine with an rwlock based solution as proposed by
> Jason. My rationale is: we recommend two stage indicators, and the two
> stage indicators are already encumbered by an rwlock on the interrupt
> path.

Which lock is it? Can we take that lock to synchronize?

> Yes, the coalescence of adapter interrupts is architecturally
> different, and so it is with GISA (without GISA, I'm not even sure), so
> this rwlock end up being worse than the one for 2 stage. But my feeling
> is, that it should be fine. On the other hand, I don't feel comfortable
> with plain spinlock, and I am curious about a more advanced solution.
> But my guess is that rwlock + some testing for the legacy indicator case
> just to double check if there is a heavy regression despite of our
> expectations to see none should do the trick.
> 
> Regards,
> Halil

Makes sense.

> > >
> > >  
> > >> - configuration change interrupts (should be rare)
> > >> - during setup, reset, etc. (should not be a concern)  
> > 

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Halil Pasic]

On Mon, 25 Apr 2022 10:54:24 +0200
Cornelia Huck <cohuck@redhat.com> wrote:

> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:  
> >> This patch tries to implement the synchronize_cbs() for ccw. For the
> >> vring_interrupt() that is called via virtio_airq_handler(), the
> >> synchronization is simply done via the airq_info's lock. For the
> >> vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> >> device spinlock for irq is introduced ans used in the synchronization
> >> method.
> >> 
> >> Cc: Thomas Gleixner <tglx@linutronix.de>
> >> Cc: Peter Zijlstra <peterz@infradead.org>
> >> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> >> Cc: Marc Zyngier <maz@kernel.org>
> >> Cc: Halil Pasic <pasic@linux.ibm.com>
> >> Cc: Cornelia Huck <cohuck@redhat.com>
> >> Signed-off-by: Jason Wang <jasowang@redhat.com>  
> >
> >
> > This is the only one that is giving me pause. Halil, Cornelia,
> > should we be concerned about the performance impact here?
> > Any chance it can be tested?  
> 
> We can have a bunch of devices using the same airq structure, and the
> sync cb creates a choke point, same as registering/unregistering. If
> invoking the sync cb is a rare operation (same as (un)registering), it
> should not affect interrupt processing for other devices too much, but
> it really should be rare.

With the notable difference that the critical section in sync_cb is
basically empty, so it should be less intrusive that register/unregister.

I would also argue, that since after the reset we (re-)discover our
virtqueues and (re-)register adapter interrupts, and thus before or as a
part of the reset we probably do an unregister to clean up the adapter
interrupts and de-allocate the bits in the info, this should not incur
any mayor overhead for the airq case, which is the preferred one.

Or am I missing something?

> 
> For testing, you would probably want to use a setup with many devices
> that share the same airq area (you can fit a lot of devices if they have
> few queues), generate traffic on the queues, and then do something that
> triggers the callback (adding/removing a new device in a loop?)
> 
> I currently don't have such a setup handy; Halil, would you be able to
> test that?

Neither do I. I would also have to start from scratch. I guess it would
be also sufficient to do a setup with two devices: a nic with many busy
queues, and another device that is responsible for generating the resets.

Regards,
Halil
> 
> >  
> >> ---
> >>  drivers/s390/virtio/virtio_ccw.c | 27 +++++++++++++++++++++++++++
> >>  1 file changed, 27 insertions(+)
> >> 
> >> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
> >> index d35e7a3f7067..c19f07a82d62 100644
> >> --- a/drivers/s390/virtio/virtio_ccw.c
> >> +++ b/drivers/s390/virtio/virtio_ccw.c
> >> @@ -62,6 +62,7 @@ struct virtio_ccw_device {
> >>  	unsigned int revision; /* Transport revision */
> >>  	wait_queue_head_t wait_q;
> >>  	spinlock_t lock;
> >> +	spinlock_t irq_lock;
> >>  	struct mutex io_lock; /* Serializes I/O requests */
> >>  	struct list_head virtqueues;
> >>  	bool is_thinint;
> >> @@ -984,6 +985,27 @@ static const char *virtio_ccw_bus_name(struct virtio_device *vdev)
> >>  	return dev_name(&vcdev->cdev->dev);
> >>  }
> >>  
> >> +static void virtio_ccw_synchronize_cbs(struct virtio_device *vdev)
> >> +{
> >> +	struct virtio_ccw_device *vcdev = to_vc_device(vdev);
> >> +	struct airq_info *info = vcdev->airq_info;
> >> +
> >> +	/*
> >> +	 * Synchronize with the vring_interrupt() called by
> >> +	 * virtio_ccw_int_handler().
> >> +	 */
> >> +	spin_lock(&vcdev->irq_lock);
> >> +	spin_unlock(&vcdev->irq_lock);
> >> +
> >> +	if (info) {
> >> +		/*
> >> +		 * Synchronize with the vring_interrupt() with airq indicator
> >> +		 */
> >> +		write_lock(&info->lock);
> >> +		write_unlock(&info->lock);
> >> +	}  
> 
> I think we can make this an either/or operation (devices will either use
> classic interrupts or adapter interrupts)?

Right, for virtqueue notifications. I second Connie's motion!

> 
> >> +}
> >> +
> >>  static const struct virtio_config_ops virtio_ccw_config_ops = {
> >>  	.get_features = virtio_ccw_get_features,
> >>  	.finalize_features = virtio_ccw_finalize_features,  
> 

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Halil Pasic]

On Mon, 25 Apr 2022 10:54:24 +0200
Cornelia Huck <cohuck@redhat.com> wrote:

> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote:  
> >> This patch tries to implement the synchronize_cbs() for ccw. For the
> >> vring_interrupt() that is called via virtio_airq_handler(), the
> >> synchronization is simply done via the airq_info's lock. For the
> >> vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> >> device spinlock for irq is introduced ans used in the synchronization
> >> method.
> >> 
> >> Cc: Thomas Gleixner <tglx@linutronix.de>
> >> Cc: Peter Zijlstra <peterz@infradead.org>
> >> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> >> Cc: Marc Zyngier <maz@kernel.org>
> >> Cc: Halil Pasic <pasic@linux.ibm.com>
> >> Cc: Cornelia Huck <cohuck@redhat.com>
> >> Signed-off-by: Jason Wang <jasowang@redhat.com>  
> >
> >
> > This is the only one that is giving me pause. Halil, Cornelia,
> > should we be concerned about the performance impact here?
> > Any chance it can be tested?  
> 
> We can have a bunch of devices using the same airq structure, and the
> sync cb creates a choke point, same as registering/unregistering. If
> invoking the sync cb is a rare operation (same as (un)registering), it
> should not affect interrupt processing for other devices too much, but
> it really should be rare.

With the notable difference that the critical section in sync_cb is
basically empty, so it should be less intrusive that register/unregister.

I would also argue, that since after the reset we (re-)discover our
virtqueues and (re-)register adapter interrupts, and thus before or as a
part of the reset we probably do an unregister to clean up the adapter
interrupts and de-allocate the bits in the info, this should not incur
any mayor overhead for the airq case, which is the preferred one.

Or am I missing something?

> 
> For testing, you would probably want to use a setup with many devices
> that share the same airq area (you can fit a lot of devices if they have
> few queues), generate traffic on the queues, and then do something that
> triggers the callback (adding/removing a new device in a loop?)
> 
> I currently don't have such a setup handy; Halil, would you be able to
> test that?

Neither do I. I would also have to start from scratch. I guess it would
be also sufficient to do a setup with two devices: a nic with many busy
queues, and another device that is responsible for generating the resets.

Regards,
Halil
> 
> >  
> >> ---
> >>  drivers/s390/virtio/virtio_ccw.c | 27 +++++++++++++++++++++++++++
> >>  1 file changed, 27 insertions(+)
> >> 
> >> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
> >> index d35e7a3f7067..c19f07a82d62 100644
> >> --- a/drivers/s390/virtio/virtio_ccw.c
> >> +++ b/drivers/s390/virtio/virtio_ccw.c
> >> @@ -62,6 +62,7 @@ struct virtio_ccw_device {
> >>  	unsigned int revision; /* Transport revision */
> >>  	wait_queue_head_t wait_q;
> >>  	spinlock_t lock;
> >> +	spinlock_t irq_lock;
> >>  	struct mutex io_lock; /* Serializes I/O requests */
> >>  	struct list_head virtqueues;
> >>  	bool is_thinint;
> >> @@ -984,6 +985,27 @@ static const char *virtio_ccw_bus_name(struct virtio_device *vdev)
> >>  	return dev_name(&vcdev->cdev->dev);
> >>  }
> >>  
> >> +static void virtio_ccw_synchronize_cbs(struct virtio_device *vdev)
> >> +{
> >> +	struct virtio_ccw_device *vcdev = to_vc_device(vdev);
> >> +	struct airq_info *info = vcdev->airq_info;
> >> +
> >> +	/*
> >> +	 * Synchronize with the vring_interrupt() called by
> >> +	 * virtio_ccw_int_handler().
> >> +	 */
> >> +	spin_lock(&vcdev->irq_lock);
> >> +	spin_unlock(&vcdev->irq_lock);
> >> +
> >> +	if (info) {
> >> +		/*
> >> +		 * Synchronize with the vring_interrupt() with airq indicator
> >> +		 */
> >> +		write_lock(&info->lock);
> >> +		write_unlock(&info->lock);
> >> +	}  
> 
> I think we can make this an either/or operation (devices will either use
> classic interrupts or adapter interrupts)?

Right, for virtqueue notifications. I second Connie's motion!

> 
> >> +}
> >> +
> >>  static const struct virtio_config_ops virtio_ccw_config_ops = {
> >>  	.get_features = virtio_ccw_get_features,
> >>  	.finalize_features = virtio_ccw_finalize_features,  
> 

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

在 2022/4/26 10:50, Halil Pasic 写道:
>>>> +static void virtio_ccw_synchronize_cbs(struct virtio_device *vdev)
>>>> +{
>>>> +	struct virtio_ccw_device *vcdev = to_vc_device(vdev);
>>>> +	struct airq_info *info = vcdev->airq_info;
>>>> +
>>>> +	/*
>>>> +	 * Synchronize with the vring_interrupt() called by
>>>> +	 * virtio_ccw_int_handler().
>>>> +	 */
>>>> +	spin_lock(&vcdev->irq_lock);
>>>> +	spin_unlock(&vcdev->irq_lock);
>>>> +
>>>> +	if (info) {
>>>> +		/*
>>>> +		 * Synchronize with the vring_interrupt() with airq indicator
>>>> +		 */
>>>> +		write_lock(&info->lock);
>>>> +		write_unlock(&info->lock);
>>>> +	}
>> I think we can make this an either/or operation (devices will either use
>> classic interrupts or adapter interrupts)?
> Right, for virtqueue notifications. I second Connie's motion!


Will do this.

Thanks


>

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

在 2022/4/26 10:50, Halil Pasic 写道:
>>>> +static void virtio_ccw_synchronize_cbs(struct virtio_device *vdev)
>>>> +{
>>>> +	struct virtio_ccw_device *vcdev = to_vc_device(vdev);
>>>> +	struct airq_info *info = vcdev->airq_info;
>>>> +
>>>> +	/*
>>>> +	 * Synchronize with the vring_interrupt() called by
>>>> +	 * virtio_ccw_int_handler().
>>>> +	 */
>>>> +	spin_lock(&vcdev->irq_lock);
>>>> +	spin_unlock(&vcdev->irq_lock);
>>>> +
>>>> +	if (info) {
>>>> +		/*
>>>> +		 * Synchronize with the vring_interrupt() with airq indicator
>>>> +		 */
>>>> +		write_lock(&info->lock);
>>>> +		write_unlock(&info->lock);
>>>> +	}
>> I think we can make this an either/or operation (devices will either use
>> classic interrupts or adapter interrupts)?
> Right, for virtqueue notifications. I second Connie's motion!


Will do this.

Thanks


>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()

[Jason Wang]

On Tue, Apr 26, 2022 at 1:39 PM Hillf Danton <hdanton@sina.com> wrote:
>
> On Mon, 25 Apr 2022 10:44:15 +0800 Jason Wang wrote:
> > This patch tries to implement the synchronize_cbs() for ccw. For the
> > vring_interrupt() that is called via virtio_airq_handler(), the
> > synchronization is simply done via the airq_info's lock. For the
> > vring_interrupt() that is called via virtio_ccw_int_handler(), a per
> > device spinlock for irq is introduced ans used in the synchronization
> > method.
> >
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Cc: "Paul E. McKenney" <paulmck@kernel.org>
> > Cc: Marc Zyngier <maz@kernel.org>
> > Cc: Halil Pasic <pasic@linux.ibm.com>
> > Cc: Cornelia Huck <cohuck@redhat.com>
> > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > ---
> >  drivers/s390/virtio/virtio_ccw.c | 27 +++++++++++++++++++++++++++
> >  1 file changed, 27 insertions(+)
> >
> > diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
> > index d35e7a3f7067..c19f07a82d62 100644
> > --- a/drivers/s390/virtio/virtio_ccw.c
> > +++ b/drivers/s390/virtio/virtio_ccw.c
> > @@ -62,6 +62,7 @@ struct virtio_ccw_device {
> >       unsigned int revision; /* Transport revision */
> >       wait_queue_head_t wait_q;
> >       spinlock_t lock;
> > +     spinlock_t irq_lock;
> >       struct mutex io_lock; /* Serializes I/O requests */
> >       struct list_head virtqueues;
> >       bool is_thinint;
> > @@ -984,6 +985,27 @@ static const char *virtio_ccw_bus_name(struct virtio_device *vdev)
> >       return dev_name(&vcdev->cdev->dev);
> >  }
> >
> > +static void virtio_ccw_synchronize_cbs(struct virtio_device *vdev)
> > +{
> > +     struct virtio_ccw_device *vcdev = to_vc_device(vdev);
> > +     struct airq_info *info = vcdev->airq_info;
> > +
> > +     /*
> > +      * Synchronize with the vring_interrupt() called by
> > +      * virtio_ccw_int_handler().
> > +      */
> > +     spin_lock(&vcdev->irq_lock);
> > +     spin_unlock(&vcdev->irq_lock);
>
> Given five lock acquirers, is the sync able to ensure the other four
> parties have complete their lock/unlock operations?

Just to make sure I understand the meaning of "four parties" here. It
looks to me the lock is only used in virtio_ccw_int_handler().

Thanks

>
> > +
> > +     if (info) {
> > +             /*
> > +              * Synchronize with the vring_interrupt() with airq indicator
> > +              */
> > +             write_lock(&info->lock);
> > +             write_unlock(&info->lock);
> > +     }
> > +}
> > +
> >  static const struct virtio_config_ops virtio_ccw_config_ops = {
> >       .get_features = virtio_ccw_get_features,
> >       .finalize_features = virtio_ccw_finalize_features,
> > @@ -995,6 +1017,7 @@ static const struct virtio_config_ops virtio_ccw_config_ops = {
> >       .find_vqs = virtio_ccw_find_vqs,
> >       .del_vqs = virtio_ccw_del_vqs,
> >       .bus_name = virtio_ccw_bus_name,
> > +     .synchronize_cbs = virtio_ccw_synchronize_cbs,
> >  };
> >
> >
> > @@ -1079,6 +1102,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
> >  {
> >       __u32 activity = intparm & VIRTIO_CCW_INTPARM_MASK;
> >       struct virtio_ccw_device *vcdev = dev_get_drvdata(&cdev->dev);
> > +     unsigned long flags;
> >       int i;
> >       struct virtqueue *vq;
> >
> > @@ -1106,6 +1130,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
> >                       vcdev->err = -EIO;
> >       }
> >       virtio_ccw_check_activity(vcdev, activity);
> > +     spin_lock_irqsave(&vcdev->irq_lock, flags);
> >       for_each_set_bit(i, indicators(vcdev),
> >                        sizeof(*indicators(vcdev)) * BITS_PER_BYTE) {
> >               /* The bit clear must happen before the vring kick. */
> > @@ -1114,6 +1139,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
> >               vq = virtio_ccw_vq_by_ind(vcdev, i);
> >               vring_interrupt(0, vq);
> >       }
> > +     spin_unlock_irqrestore(&vcdev->irq_lock, flags);
> >       if (test_bit(0, indicators2(vcdev))) {
> >               virtio_config_changed(&vcdev->vdev);
> >               clear_bit(0, indicators2(vcdev));
> > @@ -1284,6 +1310,7 @@ static int virtio_ccw_online(struct ccw_device *cdev)
> >       init_waitqueue_head(&vcdev->wait_q);
> >       INIT_LIST_HEAD(&vcdev->virtqueues);
> >       spin_lock_init(&vcdev->lock);
> > +     spin_lock_init(&vcdev->irq_lock);
> >       mutex_init(&vcdev->io_lock);
> >
> >       spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
> > --
> > 2.25.1
> >
> >
>

[PATCH V3 7/9] virtio: allow to unbreak virtqueue

[Jason Wang]

This patch allows the virtio_break_device() to accept a boolean value
then we can unbreak the virtqueue.

Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/char/virtio_console.c              | 2 +-
 drivers/crypto/virtio/virtio_crypto_core.c | 2 +-
 drivers/s390/virtio/virtio_ccw.c           | 4 ++--
 drivers/virtio/virtio_pci_common.c         | 2 +-
 drivers/virtio/virtio_ring.c               | 4 ++--
 include/linux/virtio.h                     | 2 +-
 6 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
index e3c430539a17..afede977f7b3 100644
--- a/drivers/char/virtio_console.c
+++ b/drivers/char/virtio_console.c
@@ -1958,7 +1958,7 @@ static void virtcons_remove(struct virtio_device *vdev)
 	spin_unlock_irq(&pdrvdata_lock);
 
 	/* Device is going away, exit any polling for buffers */
-	virtio_break_device(vdev);
+	virtio_break_device(vdev, true);
 	if (use_multiport(portdev))
 		flush_work(&portdev->control_work);
 	else
diff --git a/drivers/crypto/virtio/virtio_crypto_core.c b/drivers/crypto/virtio/virtio_crypto_core.c
index c6f482db0bc0..fd17f3f2e958 100644
--- a/drivers/crypto/virtio/virtio_crypto_core.c
+++ b/drivers/crypto/virtio/virtio_crypto_core.c
@@ -215,7 +215,7 @@ static int virtcrypto_update_status(struct virtio_crypto *vcrypto)
 		dev_warn(&vcrypto->vdev->dev,
 				"Unknown status bits: 0x%x\n", status);
 
-		virtio_break_device(vcrypto->vdev);
+		virtio_break_device(vcrypto->vdev, true);
 		return -EPERM;
 	}
 
diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
index c19f07a82d62..9a963f5af5b5 100644
--- a/drivers/s390/virtio/virtio_ccw.c
+++ b/drivers/s390/virtio/virtio_ccw.c
@@ -1211,7 +1211,7 @@ static void virtio_ccw_remove(struct ccw_device *cdev)
 
 	if (vcdev && cdev->online) {
 		if (vcdev->device_lost)
-			virtio_break_device(&vcdev->vdev);
+			virtio_break_device(&vcdev->vdev, true);
 		unregister_virtio_device(&vcdev->vdev);
 		spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
 		dev_set_drvdata(&cdev->dev, NULL);
@@ -1228,7 +1228,7 @@ static int virtio_ccw_offline(struct ccw_device *cdev)
 	if (!vcdev)
 		return 0;
 	if (vcdev->device_lost)
-		virtio_break_device(&vcdev->vdev);
+		virtio_break_device(&vcdev->vdev, true);
 	unregister_virtio_device(&vcdev->vdev);
 	spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
 	dev_set_drvdata(&cdev->dev, NULL);
diff --git a/drivers/virtio/virtio_pci_common.c b/drivers/virtio/virtio_pci_common.c
index d724f676608b..39a711ddff30 100644
--- a/drivers/virtio/virtio_pci_common.c
+++ b/drivers/virtio/virtio_pci_common.c
@@ -583,7 +583,7 @@ static void virtio_pci_remove(struct pci_dev *pci_dev)
 	 * layers can abort any ongoing operation.
 	 */
 	if (!pci_device_is_present(pci_dev))
-		virtio_break_device(&vp_dev->vdev);
+		virtio_break_device(&vp_dev->vdev, true);
 
 	pci_disable_sriov(pci_dev);
 
diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index cfb028ca238e..6da13495a70c 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -2382,7 +2382,7 @@ EXPORT_SYMBOL_GPL(virtqueue_is_broken);
  * This should prevent the device from being used, allowing drivers to
  * recover.  You may need to grab appropriate locks to flush.
  */
-void virtio_break_device(struct virtio_device *dev)
+void virtio_break_device(struct virtio_device *dev, bool broken)
 {
 	struct virtqueue *_vq;
 
@@ -2391,7 +2391,7 @@ void virtio_break_device(struct virtio_device *dev)
 		struct vring_virtqueue *vq = to_vvq(_vq);
 
 		/* Pairs with READ_ONCE() in virtqueue_is_broken(). */
-		WRITE_ONCE(vq->broken, true);
+		WRITE_ONCE(vq->broken, broken);
 	}
 	spin_unlock(&dev->vqs_list_lock);
 }
diff --git a/include/linux/virtio.h b/include/linux/virtio.h
index 5464f398912a..24bff3b314c8 100644
--- a/include/linux/virtio.h
+++ b/include/linux/virtio.h
@@ -130,7 +130,7 @@ int register_virtio_device(struct virtio_device *dev);
 void unregister_virtio_device(struct virtio_device *dev);
 bool is_virtio_device(struct device *dev);
 
-void virtio_break_device(struct virtio_device *dev);
+void virtio_break_device(struct virtio_device *dev, bool broken);
 
 void virtio_config_changed(struct virtio_device *dev);
 #ifdef CONFIG_PM_SLEEP
-- 
2.25.1

[PATCH V3 7/9] virtio: allow to unbreak virtqueue

[Jason Wang]

This patch allows the virtio_break_device() to accept a boolean value
then we can unbreak the virtqueue.

Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/char/virtio_console.c              | 2 +-
 drivers/crypto/virtio/virtio_crypto_core.c | 2 +-
 drivers/s390/virtio/virtio_ccw.c           | 4 ++--
 drivers/virtio/virtio_pci_common.c         | 2 +-
 drivers/virtio/virtio_ring.c               | 4 ++--
 include/linux/virtio.h                     | 2 +-
 6 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
index e3c430539a17..afede977f7b3 100644
--- a/drivers/char/virtio_console.c
+++ b/drivers/char/virtio_console.c
@@ -1958,7 +1958,7 @@ static void virtcons_remove(struct virtio_device *vdev)
 	spin_unlock_irq(&pdrvdata_lock);
 
 	/* Device is going away, exit any polling for buffers */
-	virtio_break_device(vdev);
+	virtio_break_device(vdev, true);
 	if (use_multiport(portdev))
 		flush_work(&portdev->control_work);
 	else
diff --git a/drivers/crypto/virtio/virtio_crypto_core.c b/drivers/crypto/virtio/virtio_crypto_core.c
index c6f482db0bc0..fd17f3f2e958 100644
--- a/drivers/crypto/virtio/virtio_crypto_core.c
+++ b/drivers/crypto/virtio/virtio_crypto_core.c
@@ -215,7 +215,7 @@ static int virtcrypto_update_status(struct virtio_crypto *vcrypto)
 		dev_warn(&vcrypto->vdev->dev,
 				"Unknown status bits: 0x%x\n", status);
 
-		virtio_break_device(vcrypto->vdev);
+		virtio_break_device(vcrypto->vdev, true);
 		return -EPERM;
 	}
 
diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
index c19f07a82d62..9a963f5af5b5 100644
--- a/drivers/s390/virtio/virtio_ccw.c
+++ b/drivers/s390/virtio/virtio_ccw.c
@@ -1211,7 +1211,7 @@ static void virtio_ccw_remove(struct ccw_device *cdev)
 
 	if (vcdev && cdev->online) {
 		if (vcdev->device_lost)
-			virtio_break_device(&vcdev->vdev);
+			virtio_break_device(&vcdev->vdev, true);
 		unregister_virtio_device(&vcdev->vdev);
 		spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
 		dev_set_drvdata(&cdev->dev, NULL);
@@ -1228,7 +1228,7 @@ static int virtio_ccw_offline(struct ccw_device *cdev)
 	if (!vcdev)
 		return 0;
 	if (vcdev->device_lost)
-		virtio_break_device(&vcdev->vdev);
+		virtio_break_device(&vcdev->vdev, true);
 	unregister_virtio_device(&vcdev->vdev);
 	spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
 	dev_set_drvdata(&cdev->dev, NULL);
diff --git a/drivers/virtio/virtio_pci_common.c b/drivers/virtio/virtio_pci_common.c
index d724f676608b..39a711ddff30 100644
--- a/drivers/virtio/virtio_pci_common.c
+++ b/drivers/virtio/virtio_pci_common.c
@@ -583,7 +583,7 @@ static void virtio_pci_remove(struct pci_dev *pci_dev)
 	 * layers can abort any ongoing operation.
 	 */
 	if (!pci_device_is_present(pci_dev))
-		virtio_break_device(&vp_dev->vdev);
+		virtio_break_device(&vp_dev->vdev, true);
 
 	pci_disable_sriov(pci_dev);
 
diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index cfb028ca238e..6da13495a70c 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -2382,7 +2382,7 @@ EXPORT_SYMBOL_GPL(virtqueue_is_broken);
  * This should prevent the device from being used, allowing drivers to
  * recover.  You may need to grab appropriate locks to flush.
  */
-void virtio_break_device(struct virtio_device *dev)
+void virtio_break_device(struct virtio_device *dev, bool broken)
 {
 	struct virtqueue *_vq;
 
@@ -2391,7 +2391,7 @@ void virtio_break_device(struct virtio_device *dev)
 		struct vring_virtqueue *vq = to_vvq(_vq);
 
 		/* Pairs with READ_ONCE() in virtqueue_is_broken(). */
-		WRITE_ONCE(vq->broken, true);
+		WRITE_ONCE(vq->broken, broken);
 	}
 	spin_unlock(&dev->vqs_list_lock);
 }
diff --git a/include/linux/virtio.h b/include/linux/virtio.h
index 5464f398912a..24bff3b314c8 100644
--- a/include/linux/virtio.h
+++ b/include/linux/virtio.h
@@ -130,7 +130,7 @@ int register_virtio_device(struct virtio_device *dev);
 void unregister_virtio_device(struct virtio_device *dev);
 bool is_virtio_device(struct device *dev);
 
-void virtio_break_device(struct virtio_device *dev);
+void virtio_break_device(struct virtio_device *dev, bool broken);
 
 void virtio_config_changed(struct virtio_device *dev);
 #ifdef CONFIG_PM_SLEEP
-- 
2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 7/9] virtio: allow to unbreak virtqueue

[Cornelia Huck]

On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:

> This patch allows the virtio_break_device() to accept a boolean value
> then we can unbreak the virtqueue.
>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  drivers/char/virtio_console.c              | 2 +-
>  drivers/crypto/virtio/virtio_crypto_core.c | 2 +-
>  drivers/s390/virtio/virtio_ccw.c           | 4 ++--
>  drivers/virtio/virtio_pci_common.c         | 2 +-
>  drivers/virtio/virtio_ring.c               | 4 ++--
>  include/linux/virtio.h                     | 2 +-
>  6 files changed, 8 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
> index e3c430539a17..afede977f7b3 100644
> --- a/drivers/char/virtio_console.c
> +++ b/drivers/char/virtio_console.c
> @@ -1958,7 +1958,7 @@ static void virtcons_remove(struct virtio_device *vdev)
>  	spin_unlock_irq(&pdrvdata_lock);
>  
>  	/* Device is going away, exit any polling for buffers */
> -	virtio_break_device(vdev);
> +	virtio_break_device(vdev, true);
>  	if (use_multiport(portdev))
>  		flush_work(&portdev->control_work);
>  	else
> diff --git a/drivers/crypto/virtio/virtio_crypto_core.c b/drivers/crypto/virtio/virtio_crypto_core.c
> index c6f482db0bc0..fd17f3f2e958 100644
> --- a/drivers/crypto/virtio/virtio_crypto_core.c
> +++ b/drivers/crypto/virtio/virtio_crypto_core.c
> @@ -215,7 +215,7 @@ static int virtcrypto_update_status(struct virtio_crypto *vcrypto)
>  		dev_warn(&vcrypto->vdev->dev,
>  				"Unknown status bits: 0x%x\n", status);
>  
> -		virtio_break_device(vcrypto->vdev);
> +		virtio_break_device(vcrypto->vdev, true);
>  		return -EPERM;
>  	}
>  
> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
> index c19f07a82d62..9a963f5af5b5 100644
> --- a/drivers/s390/virtio/virtio_ccw.c
> +++ b/drivers/s390/virtio/virtio_ccw.c
> @@ -1211,7 +1211,7 @@ static void virtio_ccw_remove(struct ccw_device *cdev)
>  
>  	if (vcdev && cdev->online) {
>  		if (vcdev->device_lost)
> -			virtio_break_device(&vcdev->vdev);
> +			virtio_break_device(&vcdev->vdev, true);
>  		unregister_virtio_device(&vcdev->vdev);
>  		spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
>  		dev_set_drvdata(&cdev->dev, NULL);
> @@ -1228,7 +1228,7 @@ static int virtio_ccw_offline(struct ccw_device *cdev)
>  	if (!vcdev)
>  		return 0;
>  	if (vcdev->device_lost)
> -		virtio_break_device(&vcdev->vdev);
> +		virtio_break_device(&vcdev->vdev, true);
>  	unregister_virtio_device(&vcdev->vdev);
>  	spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
>  	dev_set_drvdata(&cdev->dev, NULL);
> diff --git a/drivers/virtio/virtio_pci_common.c b/drivers/virtio/virtio_pci_common.c
> index d724f676608b..39a711ddff30 100644
> --- a/drivers/virtio/virtio_pci_common.c
> +++ b/drivers/virtio/virtio_pci_common.c
> @@ -583,7 +583,7 @@ static void virtio_pci_remove(struct pci_dev *pci_dev)
>  	 * layers can abort any ongoing operation.
>  	 */
>  	if (!pci_device_is_present(pci_dev))
> -		virtio_break_device(&vp_dev->vdev);
> +		virtio_break_device(&vp_dev->vdev, true);
>  
>  	pci_disable_sriov(pci_dev);
>  
> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
> index cfb028ca238e..6da13495a70c 100644
> --- a/drivers/virtio/virtio_ring.c
> +++ b/drivers/virtio/virtio_ring.c
> @@ -2382,7 +2382,7 @@ EXPORT_SYMBOL_GPL(virtqueue_is_broken);
>   * This should prevent the device from being used, allowing drivers to
>   * recover.  You may need to grab appropriate locks to flush.
>   */
> -void virtio_break_device(struct virtio_device *dev)
> +void virtio_break_device(struct virtio_device *dev, bool broken)

I think we need to be careful to say when it is safe to unset 'broken'.

The current callers set all queues to broken in case of surprise removal
(ccw, pci), removal (console), or the device behaving badly
(crypto). There's also code setting individual queues to broken. We do
not want to undo any of these, unless the device has gone through a
reset in the meanwhile. Maybe add:

"It is only safe to call this function to *remove* the broken flag for a
device that is (re)transitioning to becoming usable; calling it that way
during normal usage may have unpredictable consequences."

(Not sure how to word this; especially if we consider future usage of
queue reset.)

Re: [PATCH V3 7/9] virtio: allow to unbreak virtqueue

[Cornelia Huck]

On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:

> This patch allows the virtio_break_device() to accept a boolean value
> then we can unbreak the virtqueue.
>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  drivers/char/virtio_console.c              | 2 +-
>  drivers/crypto/virtio/virtio_crypto_core.c | 2 +-
>  drivers/s390/virtio/virtio_ccw.c           | 4 ++--
>  drivers/virtio/virtio_pci_common.c         | 2 +-
>  drivers/virtio/virtio_ring.c               | 4 ++--
>  include/linux/virtio.h                     | 2 +-
>  6 files changed, 8 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
> index e3c430539a17..afede977f7b3 100644
> --- a/drivers/char/virtio_console.c
> +++ b/drivers/char/virtio_console.c
> @@ -1958,7 +1958,7 @@ static void virtcons_remove(struct virtio_device *vdev)
>  	spin_unlock_irq(&pdrvdata_lock);
>  
>  	/* Device is going away, exit any polling for buffers */
> -	virtio_break_device(vdev);
> +	virtio_break_device(vdev, true);
>  	if (use_multiport(portdev))
>  		flush_work(&portdev->control_work);
>  	else
> diff --git a/drivers/crypto/virtio/virtio_crypto_core.c b/drivers/crypto/virtio/virtio_crypto_core.c
> index c6f482db0bc0..fd17f3f2e958 100644
> --- a/drivers/crypto/virtio/virtio_crypto_core.c
> +++ b/drivers/crypto/virtio/virtio_crypto_core.c
> @@ -215,7 +215,7 @@ static int virtcrypto_update_status(struct virtio_crypto *vcrypto)
>  		dev_warn(&vcrypto->vdev->dev,
>  				"Unknown status bits: 0x%x\n", status);
>  
> -		virtio_break_device(vcrypto->vdev);
> +		virtio_break_device(vcrypto->vdev, true);
>  		return -EPERM;
>  	}
>  
> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
> index c19f07a82d62..9a963f5af5b5 100644
> --- a/drivers/s390/virtio/virtio_ccw.c
> +++ b/drivers/s390/virtio/virtio_ccw.c
> @@ -1211,7 +1211,7 @@ static void virtio_ccw_remove(struct ccw_device *cdev)
>  
>  	if (vcdev && cdev->online) {
>  		if (vcdev->device_lost)
> -			virtio_break_device(&vcdev->vdev);
> +			virtio_break_device(&vcdev->vdev, true);
>  		unregister_virtio_device(&vcdev->vdev);
>  		spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
>  		dev_set_drvdata(&cdev->dev, NULL);
> @@ -1228,7 +1228,7 @@ static int virtio_ccw_offline(struct ccw_device *cdev)
>  	if (!vcdev)
>  		return 0;
>  	if (vcdev->device_lost)
> -		virtio_break_device(&vcdev->vdev);
> +		virtio_break_device(&vcdev->vdev, true);
>  	unregister_virtio_device(&vcdev->vdev);
>  	spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
>  	dev_set_drvdata(&cdev->dev, NULL);
> diff --git a/drivers/virtio/virtio_pci_common.c b/drivers/virtio/virtio_pci_common.c
> index d724f676608b..39a711ddff30 100644
> --- a/drivers/virtio/virtio_pci_common.c
> +++ b/drivers/virtio/virtio_pci_common.c
> @@ -583,7 +583,7 @@ static void virtio_pci_remove(struct pci_dev *pci_dev)
>  	 * layers can abort any ongoing operation.
>  	 */
>  	if (!pci_device_is_present(pci_dev))
> -		virtio_break_device(&vp_dev->vdev);
> +		virtio_break_device(&vp_dev->vdev, true);
>  
>  	pci_disable_sriov(pci_dev);
>  
> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
> index cfb028ca238e..6da13495a70c 100644
> --- a/drivers/virtio/virtio_ring.c
> +++ b/drivers/virtio/virtio_ring.c
> @@ -2382,7 +2382,7 @@ EXPORT_SYMBOL_GPL(virtqueue_is_broken);
>   * This should prevent the device from being used, allowing drivers to
>   * recover.  You may need to grab appropriate locks to flush.
>   */
> -void virtio_break_device(struct virtio_device *dev)
> +void virtio_break_device(struct virtio_device *dev, bool broken)

I think we need to be careful to say when it is safe to unset 'broken'.

The current callers set all queues to broken in case of surprise removal
(ccw, pci), removal (console), or the device behaving badly
(crypto). There's also code setting individual queues to broken. We do
not want to undo any of these, unless the device has gone through a
reset in the meanwhile. Maybe add:

"It is only safe to call this function to *remove* the broken flag for a
device that is (re)transitioning to becoming usable; calling it that way
during normal usage may have unpredictable consequences."

(Not sure how to word this; especially if we consider future usage of
queue reset.)

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 7/9] virtio: allow to unbreak virtqueue

[Michael S. Tsirkin]

On Mon, Apr 25, 2022 at 02:44:06PM +0200, Cornelia Huck wrote:
> On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:
> 
> > This patch allows the virtio_break_device() to accept a boolean value
> > then we can unbreak the virtqueue.
> >
> > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > ---
> >  drivers/char/virtio_console.c              | 2 +-
> >  drivers/crypto/virtio/virtio_crypto_core.c | 2 +-
> >  drivers/s390/virtio/virtio_ccw.c           | 4 ++--
> >  drivers/virtio/virtio_pci_common.c         | 2 +-
> >  drivers/virtio/virtio_ring.c               | 4 ++--
> >  include/linux/virtio.h                     | 2 +-
> >  6 files changed, 8 insertions(+), 8 deletions(-)
> >
> > diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
> > index e3c430539a17..afede977f7b3 100644
> > --- a/drivers/char/virtio_console.c
> > +++ b/drivers/char/virtio_console.c
> > @@ -1958,7 +1958,7 @@ static void virtcons_remove(struct virtio_device *vdev)
> >  	spin_unlock_irq(&pdrvdata_lock);
> >  
> >  	/* Device is going away, exit any polling for buffers */
> > -	virtio_break_device(vdev);
> > +	virtio_break_device(vdev, true);
> >  	if (use_multiport(portdev))
> >  		flush_work(&portdev->control_work);
> >  	else
> > diff --git a/drivers/crypto/virtio/virtio_crypto_core.c b/drivers/crypto/virtio/virtio_crypto_core.c
> > index c6f482db0bc0..fd17f3f2e958 100644
> > --- a/drivers/crypto/virtio/virtio_crypto_core.c
> > +++ b/drivers/crypto/virtio/virtio_crypto_core.c
> > @@ -215,7 +215,7 @@ static int virtcrypto_update_status(struct virtio_crypto *vcrypto)
> >  		dev_warn(&vcrypto->vdev->dev,
> >  				"Unknown status bits: 0x%x\n", status);
> >  
> > -		virtio_break_device(vcrypto->vdev);
> > +		virtio_break_device(vcrypto->vdev, true);
> >  		return -EPERM;
> >  	}
> >  
> > diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
> > index c19f07a82d62..9a963f5af5b5 100644
> > --- a/drivers/s390/virtio/virtio_ccw.c
> > +++ b/drivers/s390/virtio/virtio_ccw.c
> > @@ -1211,7 +1211,7 @@ static void virtio_ccw_remove(struct ccw_device *cdev)
> >  
> >  	if (vcdev && cdev->online) {
> >  		if (vcdev->device_lost)
> > -			virtio_break_device(&vcdev->vdev);
> > +			virtio_break_device(&vcdev->vdev, true);
> >  		unregister_virtio_device(&vcdev->vdev);
> >  		spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
> >  		dev_set_drvdata(&cdev->dev, NULL);
> > @@ -1228,7 +1228,7 @@ static int virtio_ccw_offline(struct ccw_device *cdev)
> >  	if (!vcdev)
> >  		return 0;
> >  	if (vcdev->device_lost)
> > -		virtio_break_device(&vcdev->vdev);
> > +		virtio_break_device(&vcdev->vdev, true);
> >  	unregister_virtio_device(&vcdev->vdev);
> >  	spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
> >  	dev_set_drvdata(&cdev->dev, NULL);
> > diff --git a/drivers/virtio/virtio_pci_common.c b/drivers/virtio/virtio_pci_common.c
> > index d724f676608b..39a711ddff30 100644
> > --- a/drivers/virtio/virtio_pci_common.c
> > +++ b/drivers/virtio/virtio_pci_common.c
> > @@ -583,7 +583,7 @@ static void virtio_pci_remove(struct pci_dev *pci_dev)
> >  	 * layers can abort any ongoing operation.
> >  	 */
> >  	if (!pci_device_is_present(pci_dev))
> > -		virtio_break_device(&vp_dev->vdev);
> > +		virtio_break_device(&vp_dev->vdev, true);
> >  
> >  	pci_disable_sriov(pci_dev);
> >  
> > diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
> > index cfb028ca238e..6da13495a70c 100644
> > --- a/drivers/virtio/virtio_ring.c
> > +++ b/drivers/virtio/virtio_ring.c
> > @@ -2382,7 +2382,7 @@ EXPORT_SYMBOL_GPL(virtqueue_is_broken);
> >   * This should prevent the device from being used, allowing drivers to
> >   * recover.  You may need to grab appropriate locks to flush.
> >   */
> > -void virtio_break_device(struct virtio_device *dev)
> > +void virtio_break_device(struct virtio_device *dev, bool broken)
> 
> I think we need to be careful to say when it is safe to unset 'broken'.
> 
> The current callers set all queues to broken in case of surprise removal
> (ccw, pci), removal (console), or the device behaving badly
> (crypto). There's also code setting individual queues to broken. We do
> not want to undo any of these, unless the device has gone through a
> reset in the meanwhile. Maybe add:
> 
> "It is only safe to call this function to *remove* the broken flag for a
> device that is (re)transitioning to becoming usable; calling it that way
> during normal usage may have unpredictable consequences."
> 
> (Not sure how to word this; especially if we consider future usage of
> queue reset.)


Right. I would prefer __virtio_unbreak_device or something similar
with a bit comment explaining it's only safe to call during probe.

-- 
MST

Re: [PATCH V3 7/9] virtio: allow to unbreak virtqueue

[Michael S. Tsirkin]

On Mon, Apr 25, 2022 at 02:44:06PM +0200, Cornelia Huck wrote:
> On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:
> 
> > This patch allows the virtio_break_device() to accept a boolean value
> > then we can unbreak the virtqueue.
> >
> > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > ---
> >  drivers/char/virtio_console.c              | 2 +-
> >  drivers/crypto/virtio/virtio_crypto_core.c | 2 +-
> >  drivers/s390/virtio/virtio_ccw.c           | 4 ++--
> >  drivers/virtio/virtio_pci_common.c         | 2 +-
> >  drivers/virtio/virtio_ring.c               | 4 ++--
> >  include/linux/virtio.h                     | 2 +-
> >  6 files changed, 8 insertions(+), 8 deletions(-)
> >
> > diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
> > index e3c430539a17..afede977f7b3 100644
> > --- a/drivers/char/virtio_console.c
> > +++ b/drivers/char/virtio_console.c
> > @@ -1958,7 +1958,7 @@ static void virtcons_remove(struct virtio_device *vdev)
> >  	spin_unlock_irq(&pdrvdata_lock);
> >  
> >  	/* Device is going away, exit any polling for buffers */
> > -	virtio_break_device(vdev);
> > +	virtio_break_device(vdev, true);
> >  	if (use_multiport(portdev))
> >  		flush_work(&portdev->control_work);
> >  	else
> > diff --git a/drivers/crypto/virtio/virtio_crypto_core.c b/drivers/crypto/virtio/virtio_crypto_core.c
> > index c6f482db0bc0..fd17f3f2e958 100644
> > --- a/drivers/crypto/virtio/virtio_crypto_core.c
> > +++ b/drivers/crypto/virtio/virtio_crypto_core.c
> > @@ -215,7 +215,7 @@ static int virtcrypto_update_status(struct virtio_crypto *vcrypto)
> >  		dev_warn(&vcrypto->vdev->dev,
> >  				"Unknown status bits: 0x%x\n", status);
> >  
> > -		virtio_break_device(vcrypto->vdev);
> > +		virtio_break_device(vcrypto->vdev, true);
> >  		return -EPERM;
> >  	}
> >  
> > diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
> > index c19f07a82d62..9a963f5af5b5 100644
> > --- a/drivers/s390/virtio/virtio_ccw.c
> > +++ b/drivers/s390/virtio/virtio_ccw.c
> > @@ -1211,7 +1211,7 @@ static void virtio_ccw_remove(struct ccw_device *cdev)
> >  
> >  	if (vcdev && cdev->online) {
> >  		if (vcdev->device_lost)
> > -			virtio_break_device(&vcdev->vdev);
> > +			virtio_break_device(&vcdev->vdev, true);
> >  		unregister_virtio_device(&vcdev->vdev);
> >  		spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
> >  		dev_set_drvdata(&cdev->dev, NULL);
> > @@ -1228,7 +1228,7 @@ static int virtio_ccw_offline(struct ccw_device *cdev)
> >  	if (!vcdev)
> >  		return 0;
> >  	if (vcdev->device_lost)
> > -		virtio_break_device(&vcdev->vdev);
> > +		virtio_break_device(&vcdev->vdev, true);
> >  	unregister_virtio_device(&vcdev->vdev);
> >  	spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
> >  	dev_set_drvdata(&cdev->dev, NULL);
> > diff --git a/drivers/virtio/virtio_pci_common.c b/drivers/virtio/virtio_pci_common.c
> > index d724f676608b..39a711ddff30 100644
> > --- a/drivers/virtio/virtio_pci_common.c
> > +++ b/drivers/virtio/virtio_pci_common.c
> > @@ -583,7 +583,7 @@ static void virtio_pci_remove(struct pci_dev *pci_dev)
> >  	 * layers can abort any ongoing operation.
> >  	 */
> >  	if (!pci_device_is_present(pci_dev))
> > -		virtio_break_device(&vp_dev->vdev);
> > +		virtio_break_device(&vp_dev->vdev, true);
> >  
> >  	pci_disable_sriov(pci_dev);
> >  
> > diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
> > index cfb028ca238e..6da13495a70c 100644
> > --- a/drivers/virtio/virtio_ring.c
> > +++ b/drivers/virtio/virtio_ring.c
> > @@ -2382,7 +2382,7 @@ EXPORT_SYMBOL_GPL(virtqueue_is_broken);
> >   * This should prevent the device from being used, allowing drivers to
> >   * recover.  You may need to grab appropriate locks to flush.
> >   */
> > -void virtio_break_device(struct virtio_device *dev)
> > +void virtio_break_device(struct virtio_device *dev, bool broken)
> 
> I think we need to be careful to say when it is safe to unset 'broken'.
> 
> The current callers set all queues to broken in case of surprise removal
> (ccw, pci), removal (console), or the device behaving badly
> (crypto). There's also code setting individual queues to broken. We do
> not want to undo any of these, unless the device has gone through a
> reset in the meanwhile. Maybe add:
> 
> "It is only safe to call this function to *remove* the broken flag for a
> device that is (re)transitioning to becoming usable; calling it that way
> during normal usage may have unpredictable consequences."
> 
> (Not sure how to word this; especially if we consider future usage of
> queue reset.)


Right. I would prefer __virtio_unbreak_device or something similar
with a bit comment explaining it's only safe to call during probe.

-- 
MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V3 7/9] virtio: allow to unbreak virtqueue

[Jason Wang]

在 2022/4/25 22:04, Michael S. Tsirkin 写道:
> On Mon, Apr 25, 2022 at 02:44:06PM +0200, Cornelia Huck wrote:
>> On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:
>>
>>> This patch allows the virtio_break_device() to accept a boolean value
>>> then we can unbreak the virtqueue.
>>>
>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>> ---
>>>   drivers/char/virtio_console.c              | 2 +-
>>>   drivers/crypto/virtio/virtio_crypto_core.c | 2 +-
>>>   drivers/s390/virtio/virtio_ccw.c           | 4 ++--
>>>   drivers/virtio/virtio_pci_common.c         | 2 +-
>>>   drivers/virtio/virtio_ring.c               | 4 ++--
>>>   include/linux/virtio.h                     | 2 +-
>>>   6 files changed, 8 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
>>> index e3c430539a17..afede977f7b3 100644
>>> --- a/drivers/char/virtio_console.c
>>> +++ b/drivers/char/virtio_console.c
>>> @@ -1958,7 +1958,7 @@ static void virtcons_remove(struct virtio_device *vdev)
>>>   	spin_unlock_irq(&pdrvdata_lock);
>>>   
>>>   	/* Device is going away, exit any polling for buffers */
>>> -	virtio_break_device(vdev);
>>> +	virtio_break_device(vdev, true);
>>>   	if (use_multiport(portdev))
>>>   		flush_work(&portdev->control_work);
>>>   	else
>>> diff --git a/drivers/crypto/virtio/virtio_crypto_core.c b/drivers/crypto/virtio/virtio_crypto_core.c
>>> index c6f482db0bc0..fd17f3f2e958 100644
>>> --- a/drivers/crypto/virtio/virtio_crypto_core.c
>>> +++ b/drivers/crypto/virtio/virtio_crypto_core.c
>>> @@ -215,7 +215,7 @@ static int virtcrypto_update_status(struct virtio_crypto *vcrypto)
>>>   		dev_warn(&vcrypto->vdev->dev,
>>>   				"Unknown status bits: 0x%x\n", status);
>>>   
>>> -		virtio_break_device(vcrypto->vdev);
>>> +		virtio_break_device(vcrypto->vdev, true);
>>>   		return -EPERM;
>>>   	}
>>>   
>>> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
>>> index c19f07a82d62..9a963f5af5b5 100644
>>> --- a/drivers/s390/virtio/virtio_ccw.c
>>> +++ b/drivers/s390/virtio/virtio_ccw.c
>>> @@ -1211,7 +1211,7 @@ static void virtio_ccw_remove(struct ccw_device *cdev)
>>>   
>>>   	if (vcdev && cdev->online) {
>>>   		if (vcdev->device_lost)
>>> -			virtio_break_device(&vcdev->vdev);
>>> +			virtio_break_device(&vcdev->vdev, true);
>>>   		unregister_virtio_device(&vcdev->vdev);
>>>   		spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
>>>   		dev_set_drvdata(&cdev->dev, NULL);
>>> @@ -1228,7 +1228,7 @@ static int virtio_ccw_offline(struct ccw_device *cdev)
>>>   	if (!vcdev)
>>>   		return 0;
>>>   	if (vcdev->device_lost)
>>> -		virtio_break_device(&vcdev->vdev);
>>> +		virtio_break_device(&vcdev->vdev, true);
>>>   	unregister_virtio_device(&vcdev->vdev);
>>>   	spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
>>>   	dev_set_drvdata(&cdev->dev, NULL);
>>> diff --git a/drivers/virtio/virtio_pci_common.c b/drivers/virtio/virtio_pci_common.c
>>> index d724f676608b..39a711ddff30 100644
>>> --- a/drivers/virtio/virtio_pci_common.c
>>> +++ b/drivers/virtio/virtio_pci_common.c
>>> @@ -583,7 +583,7 @@ static void virtio_pci_remove(struct pci_dev *pci_dev)
>>>   	 * layers can abort any ongoing operation.
>>>   	 */
>>>   	if (!pci_device_is_present(pci_dev))
>>> -		virtio_break_device(&vp_dev->vdev);
>>> +		virtio_break_device(&vp_dev->vdev, true);
>>>   
>>>   	pci_disable_sriov(pci_dev);
>>>   
>>> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
>>> index cfb028ca238e..6da13495a70c 100644
>>> --- a/drivers/virtio/virtio_ring.c
>>> +++ b/drivers/virtio/virtio_ring.c
>>> @@ -2382,7 +2382,7 @@ EXPORT_SYMBOL_GPL(virtqueue_is_broken);
>>>    * This should prevent the device from being used, allowing drivers to
>>>    * recover.  You may need to grab appropriate locks to flush.
>>>    */
>>> -void virtio_break_device(struct virtio_device *dev)
>>> +void virtio_break_device(struct virtio_device *dev, bool broken)
>> I think we need to be careful to say when it is safe to unset 'broken'.
>>
>> The current callers set all queues to broken in case of surprise removal
>> (ccw, pci), removal (console), or the device behaving badly
>> (crypto). There's also code setting individual queues to broken. We do
>> not want to undo any of these, unless the device has gone through a
>> reset in the meanwhile. Maybe add:
>>
>> "It is only safe to call this function to *remove* the broken flag for a
>> device that is (re)transitioning to becoming usable; calling it that way
>> during normal usage may have unpredictable consequences."
>>
>> (Not sure how to word this; especially if we consider future usage of
>> queue reset.)
>
> Right. I would prefer __virtio_unbreak_device or something similar


Ok, so I think then we don't want it to be exported.


> with a bit comment explaining it's only safe to call during probe.


Will do.

Thanks


>

Re: [PATCH V3 7/9] virtio: allow to unbreak virtqueue

[Jason Wang]

在 2022/4/25 22:04, Michael S. Tsirkin 写道:
> On Mon, Apr 25, 2022 at 02:44:06PM +0200, Cornelia Huck wrote:
>> On Mon, Apr 25 2022, Jason Wang <jasowang@redhat.com> wrote:
>>
>>> This patch allows the virtio_break_device() to accept a boolean value
>>> then we can unbreak the virtqueue.
>>>
>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>> ---
>>>   drivers/char/virtio_console.c              | 2 +-
>>>   drivers/crypto/virtio/virtio_crypto_core.c | 2 +-
>>>   drivers/s390/virtio/virtio_ccw.c           | 4 ++--
>>>   drivers/virtio/virtio_pci_common.c         | 2 +-
>>>   drivers/virtio/virtio_ring.c               | 4 ++--
>>>   include/linux/virtio.h                     | 2 +-
>>>   6 files changed, 8 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
>>> index e3c430539a17..afede977f7b3 100644
>>> --- a/drivers/char/virtio_console.c
>>> +++ b/drivers/char/virtio_console.c
>>> @@ -1958,7 +1958,7 @@ static void virtcons_remove(struct virtio_device *vdev)
>>>   	spin_unlock_irq(&pdrvdata_lock);
>>>   
>>>   	/* Device is going away, exit any polling for buffers */
>>> -	virtio_break_device(vdev);
>>> +	virtio_break_device(vdev, true);
>>>   	if (use_multiport(portdev))
>>>   		flush_work(&portdev->control_work);
>>>   	else
>>> diff --git a/drivers/crypto/virtio/virtio_crypto_core.c b/drivers/crypto/virtio/virtio_crypto_core.c
>>> index c6f482db0bc0..fd17f3f2e958 100644
>>> --- a/drivers/crypto/virtio/virtio_crypto_core.c
>>> +++ b/drivers/crypto/virtio/virtio_crypto_core.c
>>> @@ -215,7 +215,7 @@ static int virtcrypto_update_status(struct virtio_crypto *vcrypto)
>>>   		dev_warn(&vcrypto->vdev->dev,
>>>   				"Unknown status bits: 0x%x\n", status);
>>>   
>>> -		virtio_break_device(vcrypto->vdev);
>>> +		virtio_break_device(vcrypto->vdev, true);
>>>   		return -EPERM;
>>>   	}
>>>   
>>> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
>>> index c19f07a82d62..9a963f5af5b5 100644
>>> --- a/drivers/s390/virtio/virtio_ccw.c
>>> +++ b/drivers/s390/virtio/virtio_ccw.c
>>> @@ -1211,7 +1211,7 @@ static void virtio_ccw_remove(struct ccw_device *cdev)
>>>   
>>>   	if (vcdev && cdev->online) {
>>>   		if (vcdev->device_lost)
>>> -			virtio_break_device(&vcdev->vdev);
>>> +			virtio_break_device(&vcdev->vdev, true);
>>>   		unregister_virtio_device(&vcdev->vdev);
>>>   		spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
>>>   		dev_set_drvdata(&cdev->dev, NULL);
>>> @@ -1228,7 +1228,7 @@ static int virtio_ccw_offline(struct ccw_device *cdev)
>>>   	if (!vcdev)
>>>   		return 0;
>>>   	if (vcdev->device_lost)
>>> -		virtio_break_device(&vcdev->vdev);
>>> +		virtio_break_device(&vcdev->vdev, true);
>>>   	unregister_virtio_device(&vcdev->vdev);
>>>   	spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
>>>   	dev_set_drvdata(&cdev->dev, NULL);
>>> diff --git a/drivers/virtio/virtio_pci_common.c b/drivers/virtio/virtio_pci_common.c
>>> index d724f676608b..39a711ddff30 100644
>>> --- a/drivers/virtio/virtio_pci_common.c
>>> +++ b/drivers/virtio/virtio_pci_common.c
>>> @@ -583,7 +583,7 @@ static void virtio_pci_remove(struct pci_dev *pci_dev)
>>>   	 * layers can abort any ongoing operation.
>>>   	 */
>>>   	if (!pci_device_is_present(pci_dev))
>>> -		virtio_break_device(&vp_dev->vdev);
>>> +		virtio_break_device(&vp_dev->vdev, true);
>>>   
>>>   	pci_disable_sriov(pci_dev);
>>>   
>>> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
>>> index cfb028ca238e..6da13495a70c 100644
>>> --- a/drivers/virtio/virtio_ring.c
>>> +++ b/drivers/virtio/virtio_ring.c
>>> @@ -2382,7 +2382,7 @@ EXPORT_SYMBOL_GPL(virtqueue_is_broken);
>>>    * This should prevent the device from being used, allowing drivers to
>>>    * recover.  You may need to grab appropriate locks to flush.
>>>    */
>>> -void virtio_break_device(struct virtio_device *dev)
>>> +void virtio_break_device(struct virtio_device *dev, bool broken)
>> I think we need to be careful to say when it is safe to unset 'broken'.
>>
>> The current callers set all queues to broken in case of surprise removal
>> (ccw, pci), removal (console), or the device behaving badly
>> (crypto). There's also code setting individual queues to broken. We do
>> not want to undo any of these, unless the device has gone through a
>> reset in the meanwhile. Maybe add:
>>
>> "It is only safe to call this function to *remove* the broken flag for a
>> device that is (re)transitioning to becoming usable; calling it that way
>> during normal usage may have unpredictable consequences."
>>
>> (Not sure how to word this; especially if we consider future usage of
>> queue reset.)
>
> Right. I would prefer __virtio_unbreak_device or something similar


Ok, so I think then we don't want it to be exported.


> with a bit comment explaining it's only safe to call during probe.


Will do.

Thanks


>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH V3 8/9] virtio: harden vring IRQ

[Jason Wang]

This is a rework on the previous IRQ hardening that is done for
virtio-pci where several drawbacks were found and were reverted:

1) try to use IRQF_NO_AUTOEN which is not friendly to affinity managed IRQ
   that is used by some device such as virtio-blk
2) done only for PCI transport

The vq->broken is re-used in this patch for implementing the IRQ
hardening. The vq->broken is set to true during both initialization
and reset. And the vq->broken is set to false in
virtio_device_ready(). Then vring_interrupt can check and return when
vq->broken is true. And in this case, switch to return IRQ_NONE to let
the interrupt core aware of such invalid interrupt to prevent IRQ
storm.

The reason of using a per queue variable instead of a per device one
is that we may need it for per queue reset hardening in the future.

Note that the hardening is only done for vring interrupt since the
config interrupt hardening is already done in commit 22b7050a024d7
("virtio: defer config changed notifications"). But the method that is
used by config interrupt can't be reused by the vring interrupt
handler because it uses spinlock to do the synchronization which is
expensive.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/virtio/virtio.c       | 15 ++++++++++++---
 drivers/virtio/virtio_ring.c  | 11 +++++++----
 include/linux/virtio_config.h | 12 ++++++++++++
 3 files changed, 31 insertions(+), 7 deletions(-)

diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index 8dde44ea044a..f575df1f85d0 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -220,6 +220,15 @@ static int virtio_features_ok(struct virtio_device *dev)
  * */
 void virtio_reset_device(struct virtio_device *dev)
 {
+	/*
+	 * The below virtio_synchronize_cbs() guarantees that any
+	 * interrupt for this line arriving after
+	 * virtio_synchronize_vqs() has completed is guaranteed to see
+	 * driver_ready == false.
+	 */
+	virtio_break_device(dev, true);
+	virtio_synchronize_cbs(dev);
+
 	dev->config->reset(dev);
 }
 EXPORT_SYMBOL_GPL(virtio_reset_device);
@@ -428,6 +437,9 @@ int register_virtio_device(struct virtio_device *dev)
 	dev->config_enabled = false;
 	dev->config_change_pending = false;
 
+	INIT_LIST_HEAD(&dev->vqs);
+	spin_lock_init(&dev->vqs_list_lock);
+
 	/* We always start by resetting the device, in case a previous
 	 * driver messed it up.  This also tests that code path a little. */
 	virtio_reset_device(dev);
@@ -435,9 +447,6 @@ int register_virtio_device(struct virtio_device *dev)
 	/* Acknowledge that we've seen the device. */
 	virtio_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE);
 
-	INIT_LIST_HEAD(&dev->vqs);
-	spin_lock_init(&dev->vqs_list_lock);
-
 	/*
 	 * device_add() causes the bus infrastructure to look for a matching
 	 * driver.
diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index 6da13495a70c..12f5fe2ee8ef 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -1690,7 +1690,7 @@ static struct virtqueue *vring_create_virtqueue_packed(
 	vq->we_own_ring = true;
 	vq->notify = notify;
 	vq->weak_barriers = weak_barriers;
-	vq->broken = false;
+	vq->broken = true;
 	vq->last_used_idx = 0;
 	vq->event_triggered = false;
 	vq->num_added = 0;
@@ -2136,8 +2136,11 @@ irqreturn_t vring_interrupt(int irq, void *_vq)
 		return IRQ_NONE;
 	}
 
-	if (unlikely(vq->broken))
-		return IRQ_HANDLED;
+	if (unlikely(vq->broken)) {
+		dev_warn_once(&vq->vq.vdev->dev,
+			      "virtio vring IRQ raised before DRIVER_OK");
+		return IRQ_NONE;
+	}
 
 	/* Just a hint for performance: so it's ok that this can be racy! */
 	if (vq->event)
@@ -2179,7 +2182,7 @@ struct virtqueue *__vring_new_virtqueue(unsigned int index,
 	vq->we_own_ring = false;
 	vq->notify = notify;
 	vq->weak_barriers = weak_barriers;
-	vq->broken = false;
+	vq->broken = true;
 	vq->last_used_idx = 0;
 	vq->event_triggered = false;
 	vq->num_added = 0;
diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
index 14fe89ff99c7..3e2bc8aff51e 100644
--- a/include/linux/virtio_config.h
+++ b/include/linux/virtio_config.h
@@ -255,6 +255,18 @@ void virtio_device_ready(struct virtio_device *dev)
 	unsigned status = dev->config->get_status(dev);
 
 	BUG_ON(status & VIRTIO_CONFIG_S_DRIVER_OK);
+
+	/*
+	 * The virtio_synchronize_cbs() makes sure vring_interrupt()
+	 * will see the driver specific setup if it sees vq->broken
+	 * as false.
+	 */
+	virtio_synchronize_cbs(dev);
+	virtio_break_device(dev, false);
+	/*
+	 * The transport is expected ensure the visibility of
+	 * vq->broken before setting VIRTIO_CONFIG_S_DRIVER_OK.
+	 */
 	dev->config->set_status(dev, status | VIRTIO_CONFIG_S_DRIVER_OK);
 }
 
-- 
2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH V3 8/9] virtio: harden vring IRQ

[Jason Wang]

This is a rework on the previous IRQ hardening that is done for
virtio-pci where several drawbacks were found and were reverted:

1) try to use IRQF_NO_AUTOEN which is not friendly to affinity managed IRQ
   that is used by some device such as virtio-blk
2) done only for PCI transport

The vq->broken is re-used in this patch for implementing the IRQ
hardening. The vq->broken is set to true during both initialization
and reset. And the vq->broken is set to false in
virtio_device_ready(). Then vring_interrupt can check and return when
vq->broken is true. And in this case, switch to return IRQ_NONE to let
the interrupt core aware of such invalid interrupt to prevent IRQ
storm.

The reason of using a per queue variable instead of a per device one
is that we may need it for per queue reset hardening in the future.

Note that the hardening is only done for vring interrupt since the
config interrupt hardening is already done in commit 22b7050a024d7
("virtio: defer config changed notifications"). But the method that is
used by config interrupt can't be reused by the vring interrupt
handler because it uses spinlock to do the synchronization which is
expensive.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/virtio/virtio.c       | 15 ++++++++++++---
 drivers/virtio/virtio_ring.c  | 11 +++++++----
 include/linux/virtio_config.h | 12 ++++++++++++
 3 files changed, 31 insertions(+), 7 deletions(-)

diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index 8dde44ea044a..f575df1f85d0 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -220,6 +220,15 @@ static int virtio_features_ok(struct virtio_device *dev)
  * */
 void virtio_reset_device(struct virtio_device *dev)
 {
+	/*
+	 * The below virtio_synchronize_cbs() guarantees that any
+	 * interrupt for this line arriving after
+	 * virtio_synchronize_vqs() has completed is guaranteed to see
+	 * driver_ready == false.
+	 */
+	virtio_break_device(dev, true);
+	virtio_synchronize_cbs(dev);
+
 	dev->config->reset(dev);
 }
 EXPORT_SYMBOL_GPL(virtio_reset_device);
@@ -428,6 +437,9 @@ int register_virtio_device(struct virtio_device *dev)
 	dev->config_enabled = false;
 	dev->config_change_pending = false;
 
+	INIT_LIST_HEAD(&dev->vqs);
+	spin_lock_init(&dev->vqs_list_lock);
+
 	/* We always start by resetting the device, in case a previous
 	 * driver messed it up.  This also tests that code path a little. */
 	virtio_reset_device(dev);
@@ -435,9 +447,6 @@ int register_virtio_device(struct virtio_device *dev)
 	/* Acknowledge that we've seen the device. */
 	virtio_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE);
 
-	INIT_LIST_HEAD(&dev->vqs);
-	spin_lock_init(&dev->vqs_list_lock);
-
 	/*
 	 * device_add() causes the bus infrastructure to look for a matching
 	 * driver.
diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index 6da13495a70c..12f5fe2ee8ef 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -1690,7 +1690,7 @@ static struct virtqueue *vring_create_virtqueue_packed(
 	vq->we_own_ring = true;
 	vq->notify = notify;
 	vq->weak_barriers = weak_barriers;
-	vq->broken = false;
+	vq->broken = true;
 	vq->last_used_idx = 0;
 	vq->event_triggered = false;
 	vq->num_added = 0;
@@ -2136,8 +2136,11 @@ irqreturn_t vring_interrupt(int irq, void *_vq)
 		return IRQ_NONE;
 	}
 
-	if (unlikely(vq->broken))
-		return IRQ_HANDLED;
+	if (unlikely(vq->broken)) {
+		dev_warn_once(&vq->vq.vdev->dev,
+			      "virtio vring IRQ raised before DRIVER_OK");
+		return IRQ_NONE;
+	}
 
 	/* Just a hint for performance: so it's ok that this can be racy! */
 	if (vq->event)
@@ -2179,7 +2182,7 @@ struct virtqueue *__vring_new_virtqueue(unsigned int index,
 	vq->we_own_ring = false;
 	vq->notify = notify;
 	vq->weak_barriers = weak_barriers;
-	vq->broken = false;
+	vq->broken = true;
 	vq->last_used_idx = 0;
 	vq->event_triggered = false;
 	vq->num_added = 0;
diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
index 14fe89ff99c7..3e2bc8aff51e 100644
--- a/include/linux/virtio_config.h
+++ b/include/linux/virtio_config.h
@@ -255,6 +255,18 @@ void virtio_device_ready(struct virtio_device *dev)
 	unsigned status = dev->config->get_status(dev);
 
 	BUG_ON(status & VIRTIO_CONFIG_S_DRIVER_OK);
+
+	/*
+	 * The virtio_synchronize_cbs() makes sure vring_interrupt()
+	 * will see the driver specific setup if it sees vq->broken
+	 * as false.
+	 */
+	virtio_synchronize_cbs(dev);
+	virtio_break_device(dev, false);
+	/*
+	 * The transport is expected ensure the visibility of
+	 * vq->broken before setting VIRTIO_CONFIG_S_DRIVER_OK.
+	 */
 	dev->config->set_status(dev, status | VIRTIO_CONFIG_S_DRIVER_OK);
 }
 
-- 
2.25.1

[PATCH V3 9/9] virtio: use WARN_ON() to warning illegal status value

[Jason Wang]

We used to use BUG_ON() in virtio_device_ready() to detect illegal
status value, this seems sub-optimal since the value is under the
control of the device. Switch to use WARN_ON() instead.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 include/linux/virtio_config.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
index 3e2bc8aff51e..b0010b2ca7c2 100644
--- a/include/linux/virtio_config.h
+++ b/include/linux/virtio_config.h
@@ -254,7 +254,7 @@ void virtio_device_ready(struct virtio_device *dev)
 {
 	unsigned status = dev->config->get_status(dev);
 
-	BUG_ON(status & VIRTIO_CONFIG_S_DRIVER_OK);
+	WARN_ON(status & VIRTIO_CONFIG_S_DRIVER_OK);
 
 	/*
 	 * The virtio_synchronize_cbs() makes sure vring_interrupt()
-- 
2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH V3 9/9] virtio: use WARN_ON() to warning illegal status value

[Jason Wang]

We used to use BUG_ON() in virtio_device_ready() to detect illegal
status value, this seems sub-optimal since the value is under the
control of the device. Switch to use WARN_ON() instead.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Halil Pasic <pasic@linux.ibm.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 include/linux/virtio_config.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
index 3e2bc8aff51e..b0010b2ca7c2 100644
--- a/include/linux/virtio_config.h
+++ b/include/linux/virtio_config.h
@@ -254,7 +254,7 @@ void virtio_device_ready(struct virtio_device *dev)
 {
 	unsigned status = dev->config->get_status(dev);
 
-	BUG_ON(status & VIRTIO_CONFIG_S_DRIVER_OK);
+	WARN_ON(status & VIRTIO_CONFIG_S_DRIVER_OK);
 
 	/*
 	 * The virtio_synchronize_cbs() makes sure vring_interrupt()
-- 
2.25.1

Re: [PATCH V3 0/9] rework on the IRQ hardening of virtio

[Jason Wang]

On Mon, Apr 25, 2022 at 10:44 AM Jason Wang <jasowang@redhat.com> wrote:
>
> Hi All:
>

Forgot to add Xuan to the series.

Thanks

> This is a rework on the IRQ hardening for virtio which is done
> previously by the following commits are reverted:
>
> 9e35276a5344 ("virtio_pci: harden MSI-X interrupts")
> 080cd7c3ac87 ("virtio-pci: harden INTX interrupts")
>
> The reason is that it depends on the IRQF_NO_AUTOEN which may conflict
> with the assumption of the affinity managed IRQ that is used by some
> virtio drivers. And what's more, it is only done for virtio-pci but
> not other transports.
>
> In this rework, I try to implement a general virtio solution which
> borrows the idea of the INTX hardening by re-using per virtqueue
> boolean vq->broken and toggle it in virtio_device_ready() and
> virtio_reset_device(). Then we can simply reuse the existing checks in
> the vring_interrupt() and return early if the driver is not ready.
>
> Note that, I only did compile test on ccw and MMIO transport.
>
> Please review.
>
> Changes since v1:
>
> - Use transport specific irq synchronization method when possible
> - Drop the module parameter and enable the hardening unconditonally
> - Tweak the barrier/ordering facilities used in the code
> - Reanme irq_soft_enabled to driver_ready
> - Avoid unnecssary IRQ synchornization (e.g during boot)
>
> Changes since V2:
>
> - add ccw and MMIO support
> - rename synchronize_vqs() to synchronize_cbs()
> - switch to re-use vq->broken instead of introducing new device
>   attributes for the future virtqueue reset support
> - remove unnecssary READ_ONCE()/WRITE_ONCE()
> - a new patch to remove device triggerable BUG_ON()
> - more tweaks on the comments
>
> Jason Wang (8):
>   virtio: use virtio_reset_device() when possible
>   virtio: introduce config op to synchronize vring callbacks
>   virtio-pci: implement synchronize_cbs()
>   virtio-mmio: implement synchronize_cbs()
>   virtio-ccw: implement synchronize_cbs()
>   virtio: allow to unbreak virtqueue
>   virtio: harden vring IRQ
>   virtio: use WARN_ON() to warning illegal status value
>
> Stefano Garzarella (1):
>   virtio: use virtio_device_ready() in virtio_device_restore()
>
>  drivers/char/virtio_console.c              |  2 +-
>  drivers/crypto/virtio/virtio_crypto_core.c |  2 +-
>  drivers/s390/virtio/virtio_ccw.c           | 31 ++++++++++++++++--
>  drivers/virtio/virtio.c                    | 24 ++++++++++----
>  drivers/virtio/virtio_mmio.c               |  9 +++++
>  drivers/virtio/virtio_pci_common.c         |  2 +-
>  drivers/virtio/virtio_pci_common.h         |  2 ++
>  drivers/virtio/virtio_pci_legacy.c         |  1 +
>  drivers/virtio/virtio_pci_modern.c         |  2 ++
>  drivers/virtio/virtio_ring.c               | 15 +++++----
>  include/linux/virtio.h                     |  2 +-
>  include/linux/virtio_config.h              | 38 +++++++++++++++++++++-
>  12 files changed, 110 insertions(+), 20 deletions(-)
>
> --
> 2.25.1
>
>

Re: [PATCH V3 0/9] rework on the IRQ hardening of virtio

[Jason Wang]

On Mon, Apr 25, 2022 at 10:44 AM Jason Wang <jasowang@redhat.com> wrote:
>
> Hi All:
>

Forgot to add Xuan to the series.

Thanks

> This is a rework on the IRQ hardening for virtio which is done
> previously by the following commits are reverted:
>
> 9e35276a5344 ("virtio_pci: harden MSI-X interrupts")
> 080cd7c3ac87 ("virtio-pci: harden INTX interrupts")
>
> The reason is that it depends on the IRQF_NO_AUTOEN which may conflict
> with the assumption of the affinity managed IRQ that is used by some
> virtio drivers. And what's more, it is only done for virtio-pci but
> not other transports.
>
> In this rework, I try to implement a general virtio solution which
> borrows the idea of the INTX hardening by re-using per virtqueue
> boolean vq->broken and toggle it in virtio_device_ready() and
> virtio_reset_device(). Then we can simply reuse the existing checks in
> the vring_interrupt() and return early if the driver is not ready.
>
> Note that, I only did compile test on ccw and MMIO transport.
>
> Please review.
>
> Changes since v1:
>
> - Use transport specific irq synchronization method when possible
> - Drop the module parameter and enable the hardening unconditonally
> - Tweak the barrier/ordering facilities used in the code
> - Reanme irq_soft_enabled to driver_ready
> - Avoid unnecssary IRQ synchornization (e.g during boot)
>
> Changes since V2:
>
> - add ccw and MMIO support
> - rename synchronize_vqs() to synchronize_cbs()
> - switch to re-use vq->broken instead of introducing new device
>   attributes for the future virtqueue reset support
> - remove unnecssary READ_ONCE()/WRITE_ONCE()
> - a new patch to remove device triggerable BUG_ON()
> - more tweaks on the comments
>
> Jason Wang (8):
>   virtio: use virtio_reset_device() when possible
>   virtio: introduce config op to synchronize vring callbacks
>   virtio-pci: implement synchronize_cbs()
>   virtio-mmio: implement synchronize_cbs()
>   virtio-ccw: implement synchronize_cbs()
>   virtio: allow to unbreak virtqueue
>   virtio: harden vring IRQ
>   virtio: use WARN_ON() to warning illegal status value
>
> Stefano Garzarella (1):
>   virtio: use virtio_device_ready() in virtio_device_restore()
>
>  drivers/char/virtio_console.c              |  2 +-
>  drivers/crypto/virtio/virtio_crypto_core.c |  2 +-
>  drivers/s390/virtio/virtio_ccw.c           | 31 ++++++++++++++++--
>  drivers/virtio/virtio.c                    | 24 ++++++++++----
>  drivers/virtio/virtio_mmio.c               |  9 +++++
>  drivers/virtio/virtio_pci_common.c         |  2 +-
>  drivers/virtio/virtio_pci_common.h         |  2 ++
>  drivers/virtio/virtio_pci_legacy.c         |  1 +
>  drivers/virtio/virtio_pci_modern.c         |  2 ++
>  drivers/virtio/virtio_ring.c               | 15 +++++----
>  include/linux/virtio.h                     |  2 +-
>  include/linux/virtio_config.h              | 38 +++++++++++++++++++++-
>  12 files changed, 110 insertions(+), 20 deletions(-)
>
> --
> 2.25.1
>
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization