[Qemu-devel] [PATCH V3 00/14] Support more virtio queues

[Qemu-devel] [PATCH V3 00/14] Support more virtio queues

[Jason Wang]

We current limit the max virtio queues to 64. This is not sufficient
to support multiqueue deivces (e.g recent Linux support up to 256
tap queues). So this series try to let virtio to support more queues.

No much works need to be done except:

- Patch 1 add a check to validate the queues supported by backend
  against the virtio queue limitation
- Patch 2 fixes a bug in virtio_del_queue()
- Patch 3 introdues a bus specific queue limitation
- Patch 4 - Patch 8 let each transport to use bus specific queue
  limitation instead of VIRTIO_PCI_QUEUE_MAX
- Patch 9 introduces a map from vector to queues that use this vector
- Patch 10 introduces a helpr for get the queue index
- Patch 11 speedups the MSI-X masking and unmasking through the vector
  to queues mapping that introduced by Patch 9
- Patch 12 increase the maximum number of queues supported by
  virtio-pci from 64 to 513, and deal with the migration compatibility
  with the changes.
- Patch 13 tries to remove the hard coded MSI-X bar size (4096) and
  allow up to 2048 MSI-X entries.
- Patch 14 add a boolean property to let the virtio-net can calculate
  the MSI-X bar size based on the number of MSI-X vectors and keep
  migration compatibility with legacy version whose bar size is 4096.

With this patch, we can support up to 256 queues. Since x86 can only
allow about 240 interrupt vectors for MSI-X, current Linux driver can
only have about 80 queue pairs has their private MSI-X interrupt
vectors. With sharing IRQ with queue pairs (RFC posted in
https://lkml.org/lkml/2014/12/25/169), Linux driver can have up
to about 186 queue pairs has their private MSI-X interrupt vectors.

Stress/migration test on virtio-pci, compile test/make check on
s390x-softmmu.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <rth@twiddle.net>
Cc: Anthony Liguori <aliguori@amazon.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Cc: Alexander Graf <agraf@suse.de>
Cc: Keith Busch <keith.busch@intel.com>
Cc: Kevin Wolf <kwolf@redhat.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Anthony Liguori <aliguori@amazon.com>
Cc: Christian Borntraeger <borntraeger@de.ibm.com>
Cc: Cornelia Huck <cornelia.huck@de.ibm.com>
Cc: Amit Shah <amit.shah@redhat.com>

Please review

Thanks

Changes from V2:
- Move transport specific limitation to their implementation. (The
  left is VIRTIO_CCW_QUEUE_MAX since I fail to find a common header
  files other than virtio.h)
- Use virtio_get_queue_max() if possbile in virtio.c
- AdapterRoutes should use ccw limit
- Introduce a vector to queue mapping for virito devices and speedup
  the MSI-X masking and unmasking through this.

Changes from V1:
- add a validation against the bus limitation
- switch to use a bus specific queue limit instead of a global one,
  this will allow us to just increase the limit of one transport
  without disturbing others.
- only increase the queue limit of virtio-pci
- limit the maximum number of virtio queues to 64 for legacy machine
  types

Jason Wang (14):
  virtio-net: validate backend queue numbers against bus limitation
  virtio-net: fix the upper bound when trying to delete queues
  virito: introduce bus specific queue limit
  virtio-ccw: introduce ccw specific queue limit
  virtio-s390: switch to bus specific queue limit
  virtio-serial-bus: switch to bus specific queue limit
  virtio-mmio: switch to bus specific queue limit
  virtio-pci: switch to use bus specific queue limit
  virtio: introduce vector to virtqueues mapping
  virtio: introduce virtio_queue_get_index()
  virtio-pci: speedup MSI-X masking and unmasking
  virtio-pci: increase the maximum number of virtqueues to 513
  pci: remove hard-coded bar size in msix_init_exclusive_bar()
  virtio-pci: introduce auto_msix_bar_size property

 hw/block/nvme.c                |  2 +-
 hw/char/virtio-serial-bus.c    |  4 ++-
 hw/i386/pc_piix.c              | 10 ++++++
 hw/i386/pc_q35.c               |  9 +++++
 hw/misc/ivshmem.c              |  2 +-
 hw/net/virtio-net.c            |  9 ++++-
 hw/pci/msix.c                  | 18 ++++------
 hw/ppc/spapr.c                 |  5 +++
 hw/s390x/s390-virtio-bus.c     |  7 ++--
 hw/s390x/s390-virtio-ccw.c     |  7 ++--
 hw/s390x/virtio-ccw.c          | 12 ++++---
 hw/scsi/virtio-scsi.c          |  4 +--
 hw/virtio/virtio-mmio.c        |  7 ++--
 hw/virtio/virtio-pci.c         | 78 ++++++++++++++++++++++++++++--------------
 hw/virtio/virtio-pci.h         |  3 ++
 hw/virtio/virtio.c             | 74 ++++++++++++++++++++++++++++++---------
 include/hw/compat.h            |  8 +++++
 include/hw/pci/msix.h          |  2 +-
 include/hw/s390x/s390_flic.h   |  2 +-
 include/hw/virtio/virtio-bus.h |  2 ++
 include/hw/virtio/virtio.h     |  7 +++-
 21 files changed, 199 insertions(+), 73 deletions(-)

-- 
2.1.0

[Qemu-devel] [PATCH V3 01/14] virtio-net: validate backend queue numbers against bus limitation

[Jason Wang]

We don't validate the backend queue numbers against bus limitation,
this will easily crash qemu if it exceeds the limitation. Fixing this
by doing the validation and fail early.

Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/net/virtio-net.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 45da34a..b4ac2b3 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -1585,6 +1585,13 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp)
     virtio_init(vdev, "virtio-net", VIRTIO_ID_NET, n->config_size);
 
     n->max_queues = MAX(n->nic_conf.peers.queues, 1);
+    if (n->max_queues * 2 + 1 > VIRTIO_PCI_QUEUE_MAX) {
+        error_setg(errp, "Invalid number of queues (= %" PRIu32 "), "
+                   "must be a postive integer less than %d.",
+                   n->max_queues, (VIRTIO_PCI_QUEUE_MAX - 1) / 2);
+        virtio_cleanup(vdev);
+        return;
+    }
     n->vqs = g_malloc0(sizeof(VirtIONetQueue) * n->max_queues);
     n->vqs[0].rx_vq = virtio_add_queue(vdev, 256, virtio_net_handle_rx);
     n->curr_queues = 1;
-- 
2.1.0

[Qemu-devel] [PATCH V3 02/14] virtio-net: fix the upper bound when trying to delete queues

[Jason Wang]

Virtqueue were indexed from zero, so don't delete virtqueue whose
index is n->max_queues * 2 + 1.

Cc: Anthony Liguori <aliguori@amazon.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/net/virtio-net.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index b4ac2b3..c8d2cca 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -1306,7 +1306,7 @@ static void virtio_net_set_multiqueue(VirtIONet *n, int multiqueue)
 
     n->multiqueue = multiqueue;
 
-    for (i = 2; i <= n->max_queues * 2 + 1; i++) {
+    for (i = 2; i < n->max_queues * 2 + 1; i++) {
         virtio_del_queue(vdev, i);
     }
 
-- 
2.1.0

[Qemu-devel] [PATCH V3 03/14] virito: introduce bus specific queue limit

[Jason Wang]

This patch introduces a bus specific queue limitation. It will be
useful for increasing the limit for one of the bus without disturbing
other buses.

Cc: Anthony Liguori <aliguori@amazon.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Cc: Alexander Graf <agraf@suse.de>
Cc: Richard Henderson <rth@twiddle.net>
Cc: Cornelia Huck <cornelia.huck@de.ibm.com>
Cc: Christian Borntraeger <borntraeger@de.ibm.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/net/virtio-net.c            |  4 ++--
 hw/s390x/s390-virtio-bus.c     |  1 +
 hw/s390x/virtio-ccw.c          |  1 +
 hw/scsi/virtio-scsi.c          |  4 ++--
 hw/virtio/virtio-mmio.c        |  1 +
 hw/virtio/virtio-pci.c         |  1 +
 hw/virtio/virtio.c             | 39 ++++++++++++++++++++++++---------------
 include/hw/virtio/virtio-bus.h |  1 +
 include/hw/virtio/virtio.h     |  1 +
 9 files changed, 34 insertions(+), 19 deletions(-)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index c8d2cca..260345c 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -1585,10 +1585,10 @@ static void virtio_net_device_realize(DeviceState *dev, Error **errp)
     virtio_init(vdev, "virtio-net", VIRTIO_ID_NET, n->config_size);
 
     n->max_queues = MAX(n->nic_conf.peers.queues, 1);
-    if (n->max_queues * 2 + 1 > VIRTIO_PCI_QUEUE_MAX) {
+    if (n->max_queues * 2 + 1 > virtio_get_queue_max(vdev)) {
         error_setg(errp, "Invalid number of queues (= %" PRIu32 "), "
                    "must be a postive integer less than %d.",
-                   n->max_queues, (VIRTIO_PCI_QUEUE_MAX - 1) / 2);
+                   n->max_queues, (virtio_get_queue_max(vdev) - 1) / 2);
         virtio_cleanup(vdev);
         return;
     }
diff --git a/hw/s390x/s390-virtio-bus.c b/hw/s390x/s390-virtio-bus.c
index 39dc201..d48590a 100644
--- a/hw/s390x/s390-virtio-bus.c
+++ b/hw/s390x/s390-virtio-bus.c
@@ -707,6 +707,7 @@ static void virtio_s390_bus_class_init(ObjectClass *klass, void *data)
     bus_class->max_dev = 1;
     k->notify = virtio_s390_notify;
     k->get_features = virtio_s390_get_features;
+    k->queue_max = VIRTIO_PCI_QUEUE_MAX;
 }
 
 static const TypeInfo virtio_s390_bus_info = {
diff --git a/hw/s390x/virtio-ccw.c b/hw/s390x/virtio-ccw.c
index ea236c9..4874622 100644
--- a/hw/s390x/virtio-ccw.c
+++ b/hw/s390x/virtio-ccw.c
@@ -1695,6 +1695,7 @@ static void virtio_ccw_bus_class_init(ObjectClass *klass, void *data)
     k->load_queue = virtio_ccw_load_queue;
     k->save_config = virtio_ccw_save_config;
     k->load_config = virtio_ccw_load_config;
+    k->queue_max = VIRTIO_PCI_QUEUE_MAX;
 }
 
 static const TypeInfo virtio_ccw_bus_info = {
diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c
index 9e2c718..91d49f1 100644
--- a/hw/scsi/virtio-scsi.c
+++ b/hw/scsi/virtio-scsi.c
@@ -822,10 +822,10 @@ void virtio_scsi_common_realize(DeviceState *dev, Error **errp,
                 sizeof(VirtIOSCSIConfig));
 
     if (s->conf.num_queues == 0 ||
-            s->conf.num_queues > VIRTIO_PCI_QUEUE_MAX - 2) {
+            s->conf.num_queues > virtio_get_queue_max(vdev) - 2) {
         error_setg(errp, "Invalid number of queues (= %" PRIu32 "), "
                          "must be a positive integer less than %d.",
-                   s->conf.num_queues, VIRTIO_PCI_QUEUE_MAX - 2);
+                   s->conf.num_queues, virtio_get_queue_max(vdev) - 2);
         virtio_cleanup(vdev);
         return;
     }
diff --git a/hw/virtio/virtio-mmio.c b/hw/virtio/virtio-mmio.c
index 2450c13..ad03218 100644
--- a/hw/virtio/virtio-mmio.c
+++ b/hw/virtio/virtio-mmio.c
@@ -403,6 +403,7 @@ static void virtio_mmio_bus_class_init(ObjectClass *klass, void *data)
     k->device_plugged = virtio_mmio_device_plugged;
     k->has_variable_vring_alignment = true;
     bus_class->max_dev = 1;
+    k->queue_max = VIRTIO_PCI_QUEUE_MAX;
 }
 
 static const TypeInfo virtio_mmio_bus_info = {
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index dde1d73..7fa8141 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -1559,6 +1559,7 @@ static void virtio_pci_bus_class_init(ObjectClass *klass, void *data)
     k->vmstate_change = virtio_pci_vmstate_change;
     k->device_plugged = virtio_pci_device_plugged;
     k->device_unplugged = virtio_pci_device_unplugged;
+    k->queue_max = VIRTIO_PCI_QUEUE_MAX;
 }
 
 static const TypeInfo virtio_pci_bus_info = {
diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
index ffc22e8..ac815cf 100644
--- a/hw/virtio/virtio.c
+++ b/hw/virtio/virtio.c
@@ -541,6 +541,14 @@ void virtio_update_irq(VirtIODevice *vdev)
     virtio_notify_vector(vdev, VIRTIO_NO_VECTOR);
 }
 
+int virtio_get_queue_max(VirtIODevice *vdev)
+{
+    BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
+    VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
+
+    return k->queue_max;
+}
+
 void virtio_set_status(VirtIODevice *vdev, uint8_t val)
 {
     VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
@@ -599,7 +607,7 @@ void virtio_reset(void *opaque)
     vdev->config_vector = VIRTIO_NO_VECTOR;
     virtio_notify_vector(vdev, vdev->config_vector);
 
-    for(i = 0; i < VIRTIO_PCI_QUEUE_MAX; i++) {
+    for(i = 0; i < virtio_get_queue_max(vdev); i++) {
         vdev->vq[i].vring.desc = 0;
         vdev->vq[i].vring.avail = 0;
         vdev->vq[i].vring.used = 0;
@@ -738,7 +746,8 @@ int virtio_queue_get_num(VirtIODevice *vdev, int n)
 int virtio_queue_get_id(VirtQueue *vq)
 {
     VirtIODevice *vdev = vq->vdev;
-    assert(vq >= &vdev->vq[0] && vq < &vdev->vq[VIRTIO_PCI_QUEUE_MAX]);
+
+    assert(vq >= &vdev->vq[0] && vq < &vdev->vq[virtio_get_queue_max(vdev)]);
     return vq - &vdev->vq[0];
 }
 
@@ -777,27 +786,27 @@ void virtio_queue_notify(VirtIODevice *vdev, int n)
 
 uint16_t virtio_queue_vector(VirtIODevice *vdev, int n)
 {
-    return n < VIRTIO_PCI_QUEUE_MAX ? vdev->vq[n].vector :
+    return n < virtio_get_queue_max(vdev) ? vdev->vq[n].vector :
         VIRTIO_NO_VECTOR;
 }
 
 void virtio_queue_set_vector(VirtIODevice *vdev, int n, uint16_t vector)
 {
-    if (n < VIRTIO_PCI_QUEUE_MAX)
+    if (n < virtio_get_queue_max(vdev))
         vdev->vq[n].vector = vector;
 }
 
 VirtQueue *virtio_add_queue(VirtIODevice *vdev, int queue_size,
                             void (*handle_output)(VirtIODevice *, VirtQueue *))
 {
-    int i;
+    int i, queue_max = virtio_get_queue_max(vdev);
 
-    for (i = 0; i < VIRTIO_PCI_QUEUE_MAX; i++) {
+    for (i = 0; i < queue_max; i++) {
         if (vdev->vq[i].vring.num == 0)
             break;
     }
 
-    if (i == VIRTIO_PCI_QUEUE_MAX || queue_size > VIRTQUEUE_MAX_SIZE)
+    if (i == queue_max || queue_size > VIRTQUEUE_MAX_SIZE)
         abort();
 
     vdev->vq[i].vring.num = queue_size;
@@ -809,7 +818,7 @@ VirtQueue *virtio_add_queue(VirtIODevice *vdev, int queue_size,
 
 void virtio_del_queue(VirtIODevice *vdev, int n)
 {
-    if (n < 0 || n >= VIRTIO_PCI_QUEUE_MAX) {
+    if (n < 0 || n >= virtio_get_queue_max(vdev)) {
         abort();
     }
 
@@ -932,14 +941,14 @@ void virtio_save(VirtIODevice *vdev, QEMUFile *f)
     qemu_put_be32(f, vdev->config_len);
     qemu_put_buffer(f, vdev->config, vdev->config_len);
 
-    for (i = 0; i < VIRTIO_PCI_QUEUE_MAX; i++) {
+    for (i = 0; i < k->queue_max; i++) {
         if (vdev->vq[i].vring.num == 0)
             break;
     }
 
     qemu_put_be32(f, i);
 
-    for (i = 0; i < VIRTIO_PCI_QUEUE_MAX; i++) {
+    for (i = 0; i < k->queue_max; i++) {
         if (vdev->vq[i].vring.num == 0)
             break;
 
@@ -1004,7 +1013,7 @@ int virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
     qemu_get_8s(f, &vdev->status);
     qemu_get_8s(f, &vdev->isr);
     qemu_get_be16s(f, &vdev->queue_sel);
-    if (vdev->queue_sel >= VIRTIO_PCI_QUEUE_MAX) {
+    if (vdev->queue_sel >= k->queue_max) {
         return -1;
     }
     qemu_get_be32s(f, &features);
@@ -1031,7 +1040,7 @@ int virtio_load(VirtIODevice *vdev, QEMUFile *f, int version_id)
 
     num = qemu_get_be32(f);
 
-    if (num > VIRTIO_PCI_QUEUE_MAX) {
+    if (num > k->queue_max) {
         error_report("Invalid number of PCI queues: 0x%x", num);
         return -1;
     }
@@ -1141,15 +1150,15 @@ void virtio_instance_init_common(Object *proxy_obj, void *data,
 void virtio_init(VirtIODevice *vdev, const char *name,
                  uint16_t device_id, size_t config_size)
 {
-    int i;
+    int i, queue_max = virtio_get_queue_max(vdev);
     vdev->device_id = device_id;
     vdev->status = 0;
     vdev->isr = 0;
     vdev->queue_sel = 0;
     vdev->config_vector = VIRTIO_NO_VECTOR;
-    vdev->vq = g_malloc0(sizeof(VirtQueue) * VIRTIO_PCI_QUEUE_MAX);
+    vdev->vq = g_malloc0(sizeof(VirtQueue) * queue_max);
     vdev->vm_running = runstate_is_running();
-    for (i = 0; i < VIRTIO_PCI_QUEUE_MAX; i++) {
+    for (i = 0; i < queue_max; i++) {
         vdev->vq[i].vector = VIRTIO_NO_VECTOR;
         vdev->vq[i].vdev = vdev;
         vdev->vq[i].queue_index = i;
diff --git a/include/hw/virtio/virtio-bus.h b/include/hw/virtio/virtio-bus.h
index 0756545..979835c 100644
--- a/include/hw/virtio/virtio-bus.h
+++ b/include/hw/virtio/virtio-bus.h
@@ -68,6 +68,7 @@ typedef struct VirtioBusClass {
      * Note that changing this will break migration for this transport.
      */
     bool has_variable_vring_alignment;
+    uint16_t queue_max;
 } VirtioBusClass;
 
 struct VirtioBusState {
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index f24997d..9fe0d92 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -223,6 +223,7 @@ void virtio_queue_notify(VirtIODevice *vdev, int n);
 uint16_t virtio_queue_vector(VirtIODevice *vdev, int n);
 void virtio_queue_set_vector(VirtIODevice *vdev, int n, uint16_t vector);
 void virtio_set_status(VirtIODevice *vdev, uint8_t val);
+int virtio_get_queue_max(VirtIODevice *vdev);
 void virtio_reset(void *opaque);
 void virtio_update_irq(VirtIODevice *vdev);
 int virtio_set_features(VirtIODevice *vdev, uint32_t val);
-- 
2.1.0

[Qemu-devel] [PATCH V3 04/14] virtio-ccw: introduce ccw specific queue limit

[Jason Wang]

Instead of depending on marco, using a bus specific limit.

Cc: Alexander Graf <agraf@suse.de>
Cc: Cornelia Huck <cornelia.huck@de.ibm.com>
Cc: Christian Borntraeger <borntraeger@de.ibm.com>
Cc: Richard Henderson <rth@twiddle.net>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/s390x/s390-virtio-ccw.c   |  7 +++++--
 hw/s390x/virtio-ccw.c        | 13 +++++++------
 include/hw/s390x/s390_flic.h |  2 +-
 include/hw/virtio/virtio.h   |  1 +
 4 files changed, 14 insertions(+), 9 deletions(-)

diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
index 71bafe0..6aeb815 100644
--- a/hw/s390x/s390-virtio-ccw.c
+++ b/hw/s390x/s390-virtio-ccw.c
@@ -43,6 +43,7 @@ void io_subsystem_reset(void)
 
 static int virtio_ccw_hcall_notify(const uint64_t *args)
 {
+    VirtIODevice *vdev;
     uint64_t subch_id = args[0];
     uint64_t queue = args[1];
     SubchDev *sch;
@@ -55,10 +56,12 @@ static int virtio_ccw_hcall_notify(const uint64_t *args)
     if (!sch || !css_subch_visible(sch)) {
         return -EINVAL;
     }
-    if (queue >= VIRTIO_PCI_QUEUE_MAX) {
+
+    vdev = virtio_ccw_get_vdev(sch);
+    if (queue >= virtio_get_queue_max(vdev)) {
         return -EINVAL;
     }
-    virtio_queue_notify(virtio_ccw_get_vdev(sch), queue);
+    virtio_queue_notify(vdev, queue);
     return 0;
 
 }
diff --git a/hw/s390x/virtio-ccw.c b/hw/s390x/virtio-ccw.c
index 4874622..98a1a90 100644
--- a/hw/s390x/virtio-ccw.c
+++ b/hw/s390x/virtio-ccw.c
@@ -170,7 +170,7 @@ static void virtio_ccw_start_ioeventfd(VirtioCcwDevice *dev)
         return;
     }
     vdev = virtio_bus_get_device(&dev->bus);
-    for (n = 0; n < VIRTIO_PCI_QUEUE_MAX; n++) {
+    for (n = 0; n < virtio_get_queue_max(vdev); n++) {
         if (!virtio_queue_get_num(vdev, n)) {
             continue;
         }
@@ -205,7 +205,7 @@ static void virtio_ccw_stop_ioeventfd(VirtioCcwDevice *dev)
         return;
     }
     vdev = virtio_bus_get_device(&dev->bus);
-    for (n = 0; n < VIRTIO_PCI_QUEUE_MAX; n++) {
+    for (n = 0; n < virtio_get_queue_max(vdev); n++) {
         if (!virtio_queue_get_num(vdev, n)) {
             continue;
         }
@@ -265,8 +265,9 @@ static int virtio_ccw_set_vqs(SubchDev *sch, uint64_t addr, uint32_t align,
                               uint16_t index, uint16_t num)
 {
     VirtIODevice *vdev = virtio_ccw_get_vdev(sch);
+    int queue_max = virtio_get_queue_max(vdev);
 
-    if (index > VIRTIO_PCI_QUEUE_MAX) {
+    if (index > queue_max) {
         return -EINVAL;
     }
 
@@ -291,7 +292,7 @@ static int virtio_ccw_set_vqs(SubchDev *sch, uint64_t addr, uint32_t align,
         virtio_queue_set_vector(vdev, index, index);
     }
     /* tell notify handler in case of config change */
-    vdev->config_vector = VIRTIO_PCI_QUEUE_MAX;
+    vdev->config_vector = queue_max;
     return 0;
 }
 
@@ -1026,7 +1027,7 @@ static void virtio_ccw_notify(DeviceState *d, uint16_t vector)
         return;
     }
 
-    if (vector < VIRTIO_PCI_QUEUE_MAX) {
+    if (vector < virtio_get_queue_max(virtio_bus_get_device(&dev->bus))) {
         if (!dev->indicators) {
             return;
         }
@@ -1695,7 +1696,7 @@ static void virtio_ccw_bus_class_init(ObjectClass *klass, void *data)
     k->load_queue = virtio_ccw_load_queue;
     k->save_config = virtio_ccw_save_config;
     k->load_config = virtio_ccw_load_config;
-    k->queue_max = VIRTIO_PCI_QUEUE_MAX;
+    k->queue_max = VIRTIO_CCW_QUEUE_MAX;
 }
 
 static const TypeInfo virtio_ccw_bus_info = {
diff --git a/include/hw/s390x/s390_flic.h b/include/hw/s390x/s390_flic.h
index 489d73b..b92f650 100644
--- a/include/hw/s390x/s390_flic.h
+++ b/include/hw/s390x/s390_flic.h
@@ -20,7 +20,7 @@
 typedef struct AdapterRoutes {
     AdapterInfo adapter;
     int num_routes;
-    int gsi[VIRTIO_PCI_QUEUE_MAX];
+    int gsi[VIRTIO_CCW_QUEUE_MAX];
 } AdapterRoutes;
 
 #define TYPE_S390_FLIC_COMMON "s390-flic"
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index 9fe0d92..04ad532 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -93,6 +93,7 @@ typedef struct VirtQueueElement
 } VirtQueueElement;
 
 #define VIRTIO_PCI_QUEUE_MAX 64
+#define VIRTIO_CCW_QUEUE_MAX 64
 
 #define VIRTIO_NO_VECTOR 0xffff
 
-- 
2.1.0

[Qemu-devel] [PATCH V3 05/14] virtio-s390: switch to bus specific queue limit

[Jason Wang]

Instead of depending on marco, switch to use a bus specific queue
limit. Left is AdapterRouters->gsi[], this could be done in the future
if we want to increase s390's queue limit really.

Cc: Alexander Graf <agraf@suse.de>
Cc: Richard Henderson <rth@twiddle.net>
Cc: Christian Borntraeger <borntraeger@de.ibm.com>
Cc: Cornelia Huck <cornelia.huck@de.ibm.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/s390x/s390-virtio-bus.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/hw/s390x/s390-virtio-bus.c b/hw/s390x/s390-virtio-bus.c
index d48590a..dc147e0 100644
--- a/hw/s390x/s390-virtio-bus.c
+++ b/hw/s390x/s390-virtio-bus.c
@@ -45,6 +45,8 @@
     do { } while (0)
 #endif
 
+#define VIRTIO_S390_QUEUE_MAX 64
+
 static void virtio_s390_bus_new(VirtioBusState *bus, size_t bus_size,
                                 VirtIOS390Device *dev);
 
@@ -331,7 +333,7 @@ static ram_addr_t s390_virtio_device_num_vq(VirtIOS390Device *dev)
     VirtIODevice *vdev = dev->vdev;
     int num_vq;
 
-    for (num_vq = 0; num_vq < VIRTIO_PCI_QUEUE_MAX; num_vq++) {
+    for (num_vq = 0; num_vq < virtio_get_queue_max(vdev); num_vq++) {
         if (!virtio_queue_get_num(vdev, num_vq)) {
             break;
         }
@@ -438,7 +440,7 @@ VirtIOS390Device *s390_virtio_bus_find_vring(VirtIOS390Bus *bus,
     QTAILQ_FOREACH(kid, &bus->bus.children, sibling) {
         VirtIOS390Device *dev = (VirtIOS390Device *)kid->child;
 
-        for(i = 0; i < VIRTIO_PCI_QUEUE_MAX; i++) {
+        for(i = 0; i < virtio_get_queue_max(dev->vdev); i++) {
             if (!virtio_queue_get_addr(dev->vdev, i))
                 break;
             if (virtio_queue_get_addr(dev->vdev, i) == mem) {
@@ -707,7 +709,7 @@ static void virtio_s390_bus_class_init(ObjectClass *klass, void *data)
     bus_class->max_dev = 1;
     k->notify = virtio_s390_notify;
     k->get_features = virtio_s390_get_features;
-    k->queue_max = VIRTIO_PCI_QUEUE_MAX;
+    k->queue_max = VIRTIO_S390_QUEUE_MAX;
 }
 
 static const TypeInfo virtio_s390_bus_info = {
-- 
2.1.0

[Qemu-devel] [PATCH V3 06/14] virtio-serial-bus: switch to bus specific queue limit

[Jason Wang]

Cc: Amit Shah <amit.shah@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/char/virtio-serial-bus.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/hw/char/virtio-serial-bus.c b/hw/char/virtio-serial-bus.c
index 37a6f44..f280e95 100644
--- a/hw/char/virtio-serial-bus.c
+++ b/hw/char/virtio-serial-bus.c
@@ -26,6 +26,8 @@
 #include "hw/virtio/virtio-serial.h"
 #include "hw/virtio/virtio-access.h"
 
+#define VIRTIO_SERIAL_BUS_QUEUE_MAX 64
+
 struct VirtIOSerialDevices {
     QLIST_HEAD(, VirtIOSerial) devices;
 } vserdevices;
@@ -942,7 +944,7 @@ static void virtio_serial_device_realize(DeviceState *dev, Error **errp)
     }
 
     /* Each port takes 2 queues, and one pair is for the control queue */
-    max_supported_ports = VIRTIO_PCI_QUEUE_MAX / 2 - 1;
+    max_supported_ports = VIRTIO_SERIAL_BUS_QUEUE_MAX / 2 - 1;
 
     if (vser->serial.max_virtserial_ports > max_supported_ports) {
         error_setg(errp, "maximum ports supported: %u", max_supported_ports);
-- 
2.1.0

[Qemu-devel] [PATCH V3 07/14] virtio-mmio: switch to bus specific queue limit

[Jason Wang]

Cc: Anthony Liguori <aliguori@amazon.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/virtio/virtio-mmio.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/hw/virtio/virtio-mmio.c b/hw/virtio/virtio-mmio.c
index ad03218..7884fb7 100644
--- a/hw/virtio/virtio-mmio.c
+++ b/hw/virtio/virtio-mmio.c
@@ -34,6 +34,8 @@ do { printf("virtio_mmio: " fmt , ## __VA_ARGS__); } while (0)
 #define DPRINTF(fmt, ...) do {} while (0)
 #endif
 
+#define VIRTIO_MMIO_QUEUE_MAX 64
+
 /* QOM macros */
 /* virtio-mmio-bus */
 #define TYPE_VIRTIO_MMIO_BUS "virtio-mmio-bus"
@@ -237,7 +239,7 @@ static void virtio_mmio_write(void *opaque, hwaddr offset, uint64_t value,
                 proxy->guest_page_shift);
         break;
     case VIRTIO_MMIO_QUEUESEL:
-        if (value < VIRTIO_PCI_QUEUE_MAX) {
+	    if (value < virtio_get_queue_max(vdev)) {
             vdev->queue_sel = value;
         }
         break;
@@ -257,7 +259,7 @@ static void virtio_mmio_write(void *opaque, hwaddr offset, uint64_t value,
         }
         break;
     case VIRTIO_MMIO_QUEUENOTIFY:
-        if (value < VIRTIO_PCI_QUEUE_MAX) {
+        if (value < virtio_get_queue_max(vdev)) {
             virtio_queue_notify(vdev, value);
         }
         break;
@@ -403,7 +405,7 @@ static void virtio_mmio_bus_class_init(ObjectClass *klass, void *data)
     k->device_plugged = virtio_mmio_device_plugged;
     k->has_variable_vring_alignment = true;
     bus_class->max_dev = 1;
-    k->queue_max = VIRTIO_PCI_QUEUE_MAX;
+    k->queue_max = VIRTIO_MMIO_QUEUE_MAX;
 }
 
 static const TypeInfo virtio_mmio_bus_info = {
-- 
2.1.0

[Qemu-devel] [PATCH V3 08/14] virtio-pci: switch to use bus specific queue limit

[Jason Wang]

Instead of depending on a macro, switch to use a bus specific queue
limit.

Cc: Anthony Liguori <aliguori@amazon.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/virtio/virtio-pci.c     | 12 +++++++-----
 include/hw/virtio/virtio.h |  1 -
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index 7fa8141..a261515 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -86,6 +86,8 @@
  * 12 is historical, and due to x86 page size. */
 #define VIRTIO_PCI_QUEUE_ADDR_SHIFT    12
 
+#define VIRTIO_PCI_QUEUE_MAX 64
+
 static void virtio_pci_bus_new(VirtioBusState *bus, size_t bus_size,
                                VirtIOPCIProxy *dev);
 
@@ -215,7 +217,7 @@ static void virtio_pci_start_ioeventfd(VirtIOPCIProxy *proxy)
         return;
     }
 
-    for (n = 0; n < VIRTIO_PCI_QUEUE_MAX; n++) {
+    for (n = 0; n < virtio_get_queue_max(vdev); n++) {
         if (!virtio_queue_get_num(vdev, n)) {
             continue;
         }
@@ -251,7 +253,7 @@ static void virtio_pci_stop_ioeventfd(VirtIOPCIProxy *proxy)
         return;
     }
 
-    for (n = 0; n < VIRTIO_PCI_QUEUE_MAX; n++) {
+    for (n = 0; n < virtio_get_queue_max(vdev); n++) {
         if (!virtio_queue_get_num(vdev, n)) {
             continue;
         }
@@ -287,11 +289,11 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
             virtio_queue_set_addr(vdev, vdev->queue_sel, pa);
         break;
     case VIRTIO_PCI_QUEUE_SEL:
-        if (val < VIRTIO_PCI_QUEUE_MAX)
+        if (val < virtio_get_queue_max(vdev))
             vdev->queue_sel = val;
         break;
     case VIRTIO_PCI_QUEUE_NOTIFY:
-        if (val < VIRTIO_PCI_QUEUE_MAX) {
+        if (val < virtio_get_queue_max(vdev)) {
             virtio_queue_notify(vdev, val);
         }
         break;
@@ -792,7 +794,7 @@ static int virtio_pci_set_guest_notifiers(DeviceState *d, int nvqs, bool assign)
     bool with_irqfd = msix_enabled(&proxy->pci_dev) &&
         kvm_msi_via_irqfd_enabled();
 
-    nvqs = MIN(nvqs, VIRTIO_PCI_QUEUE_MAX);
+    nvqs = MIN(nvqs, virtio_get_queue_max(vdev));
 
     /* When deassigning, pass a consistent nvqs value
      * to avoid leaking notifiers.
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index 04ad532..9a81edf 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -92,7 +92,6 @@ typedef struct VirtQueueElement
     struct iovec out_sg[VIRTQUEUE_MAX_SIZE];
 } VirtQueueElement;
 
-#define VIRTIO_PCI_QUEUE_MAX 64
 #define VIRTIO_CCW_QUEUE_MAX 64
 
 #define VIRTIO_NO_VECTOR 0xffff
-- 
2.1.0

[Qemu-devel] [PATCH V3 09/14] virtio: introduce vector to virtqueues mapping

[Jason Wang]

Currently we will try to traverse all virtqueues to find a subset that
using a specific vector. This is sub optimal when we will support
hundreds or even thousands of virtqueues. So this patch introduces a
method which could be used by transport to get all virtqueues that
using a same vector. This is done through QLISTs and the number of
QLISTs was queried through a transport specific method. When guest
setting vectors, the virtqueue will be linked and helpers for traverse
the list was also introduced.

The first user will be virtio pci which will use this to speed up
MSI-X masking and unmasking handling.

Cc: Anthony Liguori <aliguori@amazon.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/virtio/virtio-pci.c         |  8 ++++++++
 hw/virtio/virtio.c             | 32 ++++++++++++++++++++++++++++++--
 include/hw/virtio/virtio-bus.h |  1 +
 include/hw/virtio/virtio.h     |  3 +++
 4 files changed, 42 insertions(+), 2 deletions(-)

diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index a261515..280bba2 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -957,6 +957,13 @@ static const TypeInfo virtio_9p_pci_info = {
  * virtio-pci: This is the PCIDevice which has a virtio-pci-bus.
  */
 
+static int virtio_pci_query_nvectors(DeviceState *d)
+{
+    VirtIOPCIProxy *proxy = VIRTIO_PCI(d);
+
+    return proxy->nvectors;
+}
+
 /* This is called by virtio-bus just after the device is plugged. */
 static void virtio_pci_device_plugged(DeviceState *d)
 {
@@ -1561,6 +1568,7 @@ static void virtio_pci_bus_class_init(ObjectClass *klass, void *data)
     k->vmstate_change = virtio_pci_vmstate_change;
     k->device_plugged = virtio_pci_device_plugged;
     k->device_unplugged = virtio_pci_device_unplugged;
+    k->query_nvectors = virtio_pci_query_nvectors;
     k->queue_max = VIRTIO_PCI_QUEUE_MAX;
 }
 
diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
index ac815cf..5e6b879 100644
--- a/hw/virtio/virtio.c
+++ b/hw/virtio/virtio.c
@@ -89,6 +89,7 @@ struct VirtQueue
     VirtIODevice *vdev;
     EventNotifier guest_notifier;
     EventNotifier host_notifier;
+    QLIST_ENTRY(VirtQueue) node;
 };
 
 /* virt queue functions */
@@ -613,7 +614,7 @@ void virtio_reset(void *opaque)
         vdev->vq[i].vring.used = 0;
         vdev->vq[i].last_avail_idx = 0;
         vdev->vq[i].pa = 0;
-        vdev->vq[i].vector = VIRTIO_NO_VECTOR;
+        virtio_queue_set_vector(vdev, i, VIRTIO_NO_VECTOR);
         vdev->vq[i].signalled_used = 0;
         vdev->vq[i].signalled_used_valid = false;
         vdev->vq[i].notification = true;
@@ -738,6 +739,16 @@ void virtio_queue_set_num(VirtIODevice *vdev, int n, int num)
     virtqueue_init(&vdev->vq[n]);
 }
 
+VirtQueue *virtio_vector_first_queue(VirtIODevice *vdev, uint16_t vector)
+{
+    return QLIST_FIRST(&vdev->vector_queues[vector]);
+}
+
+VirtQueue *virtio_vector_next_queue(VirtQueue *vq)
+{
+    return QLIST_NEXT(vq, node);
+}
+
 int virtio_queue_get_num(VirtIODevice *vdev, int n)
 {
     return vdev->vq[n].vring.num;
@@ -792,8 +803,17 @@ uint16_t virtio_queue_vector(VirtIODevice *vdev, int n)
 
 void virtio_queue_set_vector(VirtIODevice *vdev, int n, uint16_t vector)
 {
-    if (n < virtio_get_queue_max(vdev))
+    VirtQueue *vq = &vdev->vq[n];
+
+    if (n < virtio_get_queue_max(vdev)) {
+        if (vdev->vq[n].vector != VIRTIO_NO_VECTOR) {
+            QLIST_REMOVE(vq, node);
+        }
         vdev->vq[n].vector = vector;
+        if (vector != VIRTIO_NO_VECTOR) {
+            QLIST_INSERT_HEAD(&vdev->vector_queues[vector], vq, node);
+        }
+    }
 }
 
 VirtQueue *virtio_add_queue(VirtIODevice *vdev, int queue_size,
@@ -1113,6 +1133,7 @@ void virtio_cleanup(VirtIODevice *vdev)
     qemu_del_vm_change_state_handler(vdev->vmstate);
     g_free(vdev->config);
     g_free(vdev->vq);
+    g_free(vdev->vector_queues);
 }
 
 static void virtio_vmstate_change(void *opaque, int running, RunState state)
@@ -1150,7 +1171,14 @@ void virtio_instance_init_common(Object *proxy_obj, void *data,
 void virtio_init(VirtIODevice *vdev, const char *name,
                  uint16_t device_id, size_t config_size)
 {
+    BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
+    VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
     int i, queue_max = virtio_get_queue_max(vdev);
+    int nvectors = k->query_nvectors ?
+        k->query_nvectors(qbus->parent) : queue_max;
+
+    vdev->vector_queues = g_malloc0(sizeof(*vdev->vector_queues) * nvectors);
+
     vdev->device_id = device_id;
     vdev->status = 0;
     vdev->isr = 0;
diff --git a/include/hw/virtio/virtio-bus.h b/include/hw/virtio/virtio-bus.h
index 979835c..4487556 100644
--- a/include/hw/virtio/virtio-bus.h
+++ b/include/hw/virtio/virtio-bus.h
@@ -62,6 +62,7 @@ typedef struct VirtioBusClass {
      * This is called by virtio-bus just before the device is unplugged.
      */
     void (*device_unplugged)(DeviceState *d);
+    int (*query_nvectors)(DeviceState *d);
     /*
      * Does the transport have variable vring alignment?
      * (ie can it ever call virtio_queue_set_align()?)
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index 9a81edf..ceab8e8 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -128,6 +128,7 @@ struct VirtIODevice
     VMChangeStateEntry *vmstate;
     char *bus_name;
     uint8_t device_endian;
+    QLIST_HEAD(, VirtQueue) * vector_queues;
 };
 
 typedef struct VirtioDeviceClass {
@@ -263,6 +264,8 @@ void virtio_queue_set_host_notifier_fd_handler(VirtQueue *vq, bool assign,
                                                bool set_handler);
 void virtio_queue_notify_vq(VirtQueue *vq);
 void virtio_irq(VirtQueue *vq);
+VirtQueue *virtio_vector_first_queue(VirtIODevice *vdev, uint16_t vector);
+VirtQueue *virtio_vector_next_queue(VirtQueue *vq);
 
 static inline bool virtio_is_big_endian(VirtIODevice *vdev)
 {
-- 
2.1.0

[Qemu-devel] [PATCH V3 10/14] virtio: introduce virtio_queue_get_index()

[Jason Wang]

This patch introduces a helper that can get the queue index of a
VirtQueue. This is useful when traversing the list of VirtQueues.

Cc: Anthony Liguori <aliguori@amazon.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/virtio/virtio.c         | 5 +++++
 include/hw/virtio/virtio.h | 1 +
 2 files changed, 6 insertions(+)

diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
index 5e6b879..bd32518 100644
--- a/hw/virtio/virtio.c
+++ b/hw/virtio/virtio.c
@@ -725,6 +725,11 @@ hwaddr virtio_queue_get_addr(VirtIODevice *vdev, int n)
     return vdev->vq[n].pa;
 }
 
+int virtio_queue_get_index(VirtIODevice *vdev, VirtQueue *vq)
+{
+    return vq - vdev->vq;
+}
+
 void virtio_queue_set_num(VirtIODevice *vdev, int n, int num)
 {
     /* Don't allow guest to flip queue between existent and
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index ceab8e8..37aa9c4 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -217,6 +217,7 @@ void virtio_config_writew(VirtIODevice *vdev, uint32_t addr, uint32_t data);
 void virtio_config_writel(VirtIODevice *vdev, uint32_t addr, uint32_t data);
 void virtio_queue_set_addr(VirtIODevice *vdev, int n, hwaddr addr);
 hwaddr virtio_queue_get_addr(VirtIODevice *vdev, int n);
+int virtio_queue_get_index(VirtIODevice *vdev, VirtQueue *vq);
 void virtio_queue_set_num(VirtIODevice *vdev, int n, int num);
 int virtio_queue_get_num(VirtIODevice *vdev, int n);
 void virtio_queue_set_align(VirtIODevice *vdev, int n, int align);
-- 
2.1.0

[Qemu-devel] [PATCH V3 11/14] virtio-pci: speedup MSI-X masking and unmasking

[Jason Wang]

This patch tries to speed up the MSI-X masking and unmasking through
the mapping between vector and queues. With this patch it will there's
no need to go through all possible virtqueues, which may help to
reduce the time spent when doing MSI-X masking/unmasking a single
vector when more than hundreds or even thousands of virtqueues were
supported.

Tested with 80 queue pairs virito-net-pci by changing the smp affinity
in the background and doing netperf in the same time:

Before the patch:
5711.70 Gbits/sec
After the patch:
6830.98 Gbits/sec

About 19.6% improvements in throughput.

Cc: Anthony Liguori <aliguori@amazon.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/virtio/virtio-pci.c | 40 +++++++++++++++++++++-------------------
 1 file changed, 21 insertions(+), 19 deletions(-)

diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index 280bba2..327a3fc 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -676,28 +676,30 @@ static int virtio_pci_vector_unmask(PCIDevice *dev, unsigned vector,
 {
     VirtIOPCIProxy *proxy = container_of(dev, VirtIOPCIProxy, pci_dev);
     VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus);
-    int ret, queue_no;
+    VirtQueue *vq = virtio_vector_first_queue(vdev, vector);
+    int ret, index, unmasked = 0;
 
-    for (queue_no = 0; queue_no < proxy->nvqs_with_notifiers; queue_no++) {
-        if (!virtio_queue_get_num(vdev, queue_no)) {
+    while (vq) {
+        index = virtio_queue_get_index(vdev, vq);
+        if (!virtio_queue_get_num(vdev, index)) {
             break;
         }
-        if (virtio_queue_vector(vdev, queue_no) != vector) {
-            continue;
-        }
-        ret = virtio_pci_vq_vector_unmask(proxy, queue_no, vector, msg);
+        ret = virtio_pci_vq_vector_unmask(proxy, index, vector, msg);
         if (ret < 0) {
             goto undo;
         }
+        vq = virtio_vector_next_queue(vq);
+        ++unmasked;
     }
+
     return 0;
 
 undo:
-    while (--queue_no >= 0) {
-        if (virtio_queue_vector(vdev, queue_no) != vector) {
-            continue;
-        }
-        virtio_pci_vq_vector_mask(proxy, queue_no, vector);
+    vq = virtio_vector_first_queue(vdev, vector);
+    while (vq && --unmasked >= 0) {
+        index = virtio_queue_get_index(vdev, vq);
+        virtio_pci_vq_vector_mask(proxy, index, vector);
+        vq = virtio_vector_next_queue(vq);
     }
     return ret;
 }
@@ -706,16 +708,16 @@ static void virtio_pci_vector_mask(PCIDevice *dev, unsigned vector)
 {
     VirtIOPCIProxy *proxy = container_of(dev, VirtIOPCIProxy, pci_dev);
     VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus);
-    int queue_no;
+    VirtQueue *vq = virtio_vector_first_queue(vdev, vector);
+    int index;
 
-    for (queue_no = 0; queue_no < proxy->nvqs_with_notifiers; queue_no++) {
-        if (!virtio_queue_get_num(vdev, queue_no)) {
+    while (vq) {
+        index = virtio_queue_get_index(vdev, vq);
+        if (!virtio_queue_get_num(vdev, index)) {
             break;
         }
-        if (virtio_queue_vector(vdev, queue_no) != vector) {
-            continue;
-        }
-        virtio_pci_vq_vector_mask(proxy, queue_no, vector);
+        virtio_pci_vq_vector_mask(proxy, index, vector);
+        vq = virtio_vector_next_queue(vq);
     }
 }
 
-- 
2.1.0

[Qemu-devel] [PATCH V3 12/14] virtio-pci: increase the maximum number of virtqueues to 513

[Jason Wang]

This patch increases the maximum number of virtqueues for pci from 64
to 513. This will allow booting a virtio-net-pci device with 256 queue
pairs.

To keep migration compatibility, 64 was kept for legacy machine types.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <rth@twiddle.net>
Cc: Anthony Liguori <aliguori@amazon.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/i386/pc_piix.c      | 6 ++++++
 hw/i386/pc_q35.c       | 5 +++++
 hw/virtio/virtio-pci.c | 2 +-
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
index 38b42b0..821d44c 100644
--- a/hw/i386/pc_piix.c
+++ b/hw/i386/pc_piix.c
@@ -49,6 +49,7 @@
 #include "hw/acpi/acpi.h"
 #include "cpu.h"
 #include "qemu/error-report.h"
+#include "include/hw/virtio/virtio-bus.h"
 #ifdef CONFIG_XEN
 #  include <xen/hvm/hvm_info_table.h>
 #endif
@@ -310,6 +311,11 @@ static void pc_init_pci(MachineState *machine)
 
 static void pc_compat_2_2(MachineState *machine)
 {
+    ObjectClass *klass = object_class_by_name("virtio-pci-bus");
+    VirtioBusClass *k = VIRTIO_BUS_CLASS(klass);
+
+    k->queue_max = 64;
+
     x86_cpu_compat_set_features("kvm64", FEAT_1_EDX, 0, CPUID_VME);
     x86_cpu_compat_set_features("kvm32", FEAT_1_EDX, 0, CPUID_VME);
     x86_cpu_compat_set_features("Conroe", FEAT_1_EDX, 0, CPUID_VME);
diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index 63027ee..dc5ada4 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -45,6 +45,7 @@
 #include "hw/usb.h"
 #include "hw/cpu/icc_bus.h"
 #include "qemu/error-report.h"
+#include "include/hw/virtio/virtio-bus.h"
 
 /* ICH9 AHCI has 6 ports */
 #define MAX_SATA_PORTS     6
@@ -289,6 +290,10 @@ static void pc_q35_init(MachineState *machine)
 
 static void pc_compat_2_2(MachineState *machine)
 {
+    ObjectClass *klass = object_class_by_name("virtio-pci-bus");
+    VirtioBusClass *k = VIRTIO_BUS_CLASS(klass);
+
+    k->queue_max = 64;
     x86_cpu_compat_set_features("kvm64", FEAT_1_EDX, 0, CPUID_VME);
     x86_cpu_compat_set_features("kvm32", FEAT_1_EDX, 0, CPUID_VME);
     x86_cpu_compat_set_features("Conroe", FEAT_1_EDX, 0, CPUID_VME);
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index 327a3fc..c5d410b 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -86,7 +86,7 @@
  * 12 is historical, and due to x86 page size. */
 #define VIRTIO_PCI_QUEUE_ADDR_SHIFT    12
 
-#define VIRTIO_PCI_QUEUE_MAX 64
+#define VIRTIO_PCI_QUEUE_MAX 513
 
 static void virtio_pci_bus_new(VirtioBusState *bus, size_t bus_size,
                                VirtIOPCIProxy *dev);
-- 
2.1.0

[Qemu-devel] [PATCH V3 13/14] pci: remove hard-coded bar size in msix_init_exclusive_bar()

[Jason Wang]

This patch let msix_init_exclusive_bar() can accept bar_size parameter
other than a hard-coded limit 4096. Then caller can specify a bar_size
depends on msix entries and can use up to 2048 msix entries as PCI
spec allows. To keep migration compatibility, 4096 is used for all
callers and pba were start from half of bar size.

Cc: Keith Busch <keith.busch@intel.com>
Cc: Kevin Wolf <kwolf@redhat.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Cc: Anthony Liguori <aliguori@amazon.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/block/nvme.c        |  2 +-
 hw/misc/ivshmem.c      |  2 +-
 hw/pci/msix.c          | 18 +++++++-----------
 hw/virtio/virtio-pci.c |  2 +-
 include/hw/pci/msix.h  |  2 +-
 5 files changed, 11 insertions(+), 15 deletions(-)

diff --git a/hw/block/nvme.c b/hw/block/nvme.c
index ce079ae..0fa1396 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme.c
@@ -786,7 +786,7 @@ static int nvme_init(PCIDevice *pci_dev)
     pci_register_bar(&n->parent_obj, 0,
         PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64,
         &n->iomem);
-    msix_init_exclusive_bar(&n->parent_obj, n->num_queues, 4);
+    msix_init_exclusive_bar(&n->parent_obj, n->num_queues, 4, 4096);
 
     id->vid = cpu_to_le16(pci_get_word(pci_conf + PCI_VENDOR_ID));
     id->ssvid = cpu_to_le16(pci_get_word(pci_conf + PCI_SUBSYSTEM_VENDOR_ID));
diff --git a/hw/misc/ivshmem.c b/hw/misc/ivshmem.c
index 5d272c8..3e2a2d4 100644
--- a/hw/misc/ivshmem.c
+++ b/hw/misc/ivshmem.c
@@ -631,7 +631,7 @@ static uint64_t ivshmem_get_size(IVShmemState * s) {
 
 static void ivshmem_setup_msi(IVShmemState * s)
 {
-    if (msix_init_exclusive_bar(PCI_DEVICE(s), s->vectors, 1)) {
+    if (msix_init_exclusive_bar(PCI_DEVICE(s), s->vectors, 1, 4096)) {
         IVSHMEM_DPRINTF("msix initialization failed\n");
         exit(1);
     }
diff --git a/hw/pci/msix.c b/hw/pci/msix.c
index 24de260..9a1894f 100644
--- a/hw/pci/msix.c
+++ b/hw/pci/msix.c
@@ -291,33 +291,29 @@ int msix_init(struct PCIDevice *dev, unsigned short nentries,
 }
 
 int msix_init_exclusive_bar(PCIDevice *dev, unsigned short nentries,
-                            uint8_t bar_nr)
+                            uint8_t bar_nr, uint32_t bar_size)
 {
     int ret;
     char *name;
+    uint32_t bar_pba_offset = bar_size / 2;
 
     /*
      * Migration compatibility dictates that this remains a 4k
      * BAR with the vector table in the lower half and PBA in
      * the upper half.  Do not use these elsewhere!
      */
-#define MSIX_EXCLUSIVE_BAR_SIZE 4096
-#define MSIX_EXCLUSIVE_BAR_TABLE_OFFSET 0
-#define MSIX_EXCLUSIVE_BAR_PBA_OFFSET (MSIX_EXCLUSIVE_BAR_SIZE / 2)
-#define MSIX_EXCLUSIVE_CAP_OFFSET 0
-
-    if (nentries * PCI_MSIX_ENTRY_SIZE > MSIX_EXCLUSIVE_BAR_PBA_OFFSET) {
+    if (nentries * PCI_MSIX_ENTRY_SIZE > bar_pba_offset) {
         return -EINVAL;
     }
 
     name = g_strdup_printf("%s-msix", dev->name);
-    memory_region_init(&dev->msix_exclusive_bar, OBJECT(dev), name, MSIX_EXCLUSIVE_BAR_SIZE);
+    memory_region_init(&dev->msix_exclusive_bar, OBJECT(dev), name, bar_size);
     g_free(name);
 
     ret = msix_init(dev, nentries, &dev->msix_exclusive_bar, bar_nr,
-                    MSIX_EXCLUSIVE_BAR_TABLE_OFFSET, &dev->msix_exclusive_bar,
-                    bar_nr, MSIX_EXCLUSIVE_BAR_PBA_OFFSET,
-                    MSIX_EXCLUSIVE_CAP_OFFSET);
+                    0, &dev->msix_exclusive_bar,
+                    bar_nr, bar_pba_offset,
+                    0);
     if (ret) {
         return ret;
     }
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index c5d410b..0d26cb4 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -984,7 +984,7 @@ static void virtio_pci_device_plugged(DeviceState *d)
     config[PCI_INTERRUPT_PIN] = 1;
 
     if (proxy->nvectors &&
-        msix_init_exclusive_bar(&proxy->pci_dev, proxy->nvectors, 1)) {
+        msix_init_exclusive_bar(&proxy->pci_dev, proxy->nvectors, 1, 4096)) {
         error_report("unable to init msix vectors to %" PRIu32,
                      proxy->nvectors);
         proxy->nvectors = 0;
diff --git a/include/hw/pci/msix.h b/include/hw/pci/msix.h
index 954d82b..43edebc 100644
--- a/include/hw/pci/msix.h
+++ b/include/hw/pci/msix.h
@@ -11,7 +11,7 @@ int msix_init(PCIDevice *dev, unsigned short nentries,
               unsigned table_offset, MemoryRegion *pba_bar,
               uint8_t pba_bar_nr, unsigned pba_offset, uint8_t cap_pos);
 int msix_init_exclusive_bar(PCIDevice *dev, unsigned short nentries,
-                            uint8_t bar_nr);
+                            uint8_t bar_nr, uint32_t bar_size);
 
 void msix_write_config(PCIDevice *dev, uint32_t address, uint32_t val, int len);
 
-- 
2.1.0

[Qemu-devel] [PATCH V3 14/14] virtio-pci: introduce auto_msix_bar_size property

[Jason Wang]

Currently we don't support more than 128 MSI-X vectors for a pci
devices, trying to use vector=129 for a virtio-net-pci device may get:

qemu-system-x86_64: -device virtio-net-pci,netdev=hn0,vectors=129:
unable to init msix vectors to 129

This this because the MSI-X bar size were hard-coded as 4096. So this
patch introduces boolean auto_msix_bar_size property for
virito-pcidevices. Enable this will let the device calculate the msix
bar sizebased on the number of MSI-X entries instead of previous 4096
hard-coded limit.

This is a must to let virtio-net can up to 256 queues and each queue
were associated with a specific MSI-X entry.

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <rth@twiddle.net>
Cc: Anthony Liguori <aliguori@amazon.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Cc: Alexander Graf <agraf@suse.de>
Cc: qemu-ppc@nongnu.org
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 hw/i386/pc_piix.c      |  4 ++++
 hw/i386/pc_q35.c       |  4 ++++
 hw/ppc/spapr.c         |  5 +++++
 hw/virtio/virtio-pci.c | 17 +++++++++++++++--
 hw/virtio/virtio-pci.h |  3 +++
 include/hw/compat.h    |  8 ++++++++
 6 files changed, 39 insertions(+), 2 deletions(-)

diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
index 821d44c..07bf0b4 100644
--- a/hw/i386/pc_piix.c
+++ b/hw/i386/pc_piix.c
@@ -537,6 +537,10 @@ static QEMUMachine pc_i440fx_machine_v2_2 = {
     PC_I440FX_2_2_MACHINE_OPTIONS,
     .name = "pc-i440fx-2.2",
     .init = pc_init_pci_2_2,
+    .compat_props = (GlobalProperty[]) {
+        HW_COMPAT_2_2,
+        { /* end of list */ }
+    },
 };
 
 #define PC_I440FX_2_1_MACHINE_OPTIONS                           \
diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index dc5ada4..42799ab 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -433,6 +433,10 @@ static QEMUMachine pc_q35_machine_v2_2 = {
     PC_Q35_2_2_MACHINE_OPTIONS,
     .name = "pc-q35-2.2",
     .init = pc_q35_init_2_2,
+    .compat_props = (GlobalProperty[]) {
+        HW_COMPAT_2_2,
+        { /* end of list */ }
+    },
 };
 
 #define PC_Q35_2_1_MACHINE_OPTIONS                      \
diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index b560459..b50d600 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1777,11 +1777,16 @@ static const TypeInfo spapr_machine_2_1_info = {
 static void spapr_machine_2_2_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
+    static GlobalProperty compat_props[] = {
+        HW_COMPAT_2_2,
+        { /* end of list */ }
+    };
 
     mc->name = "pseries-2.2";
     mc->desc = "pSeries Logical Partition (PAPR compliant) v2.2";
     mc->alias = "pseries";
     mc->is_default = 1;
+    mc->compat_props = compat_props;
 }
 
 static const TypeInfo spapr_machine_2_2_info = {
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index 0d26cb4..26d4773 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -972,7 +972,7 @@ static void virtio_pci_device_plugged(DeviceState *d)
     VirtIOPCIProxy *proxy = VIRTIO_PCI(d);
     VirtioBusState *bus = &proxy->bus;
     uint8_t *config;
-    uint32_t size;
+    uint32_t size, bar_size;
 
     config = proxy->pci_dev.config;
     if (proxy->class_code) {
@@ -983,8 +983,19 @@ static void virtio_pci_device_plugged(DeviceState *d)
     pci_set_word(config + PCI_SUBSYSTEM_ID, virtio_bus_get_vdev_id(bus));
     config[PCI_INTERRUPT_PIN] = 1;
 
+    if (proxy->flags & VIRTIO_PCI_FLAG_AUTO_MSIX_SIZE) {
+        bar_size = proxy->nvectors * PCI_MSIX_ENTRY_SIZE * 2;
+        if (bar_size & (bar_size - 1)) {
+            bar_size = 1 << qemu_fls(bar_size);
+        }
+    } else {
+        /* For migration compatibility */
+        bar_size = 4096;
+    }
+
     if (proxy->nvectors &&
-        msix_init_exclusive_bar(&proxy->pci_dev, proxy->nvectors, 1, 4096)) {
+        msix_init_exclusive_bar(&proxy->pci_dev, proxy->nvectors, 1,
+                                bar_size)) {
         error_report("unable to init msix vectors to %" PRIu32,
                      proxy->nvectors);
         proxy->nvectors = 0;
@@ -1429,6 +1440,8 @@ static const TypeInfo virtio_serial_pci_info = {
 static Property virtio_net_properties[] = {
     DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags,
                     VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, false),
+    DEFINE_PROP_BIT("auto_msix_bar_size", VirtIOPCIProxy, flags,
+                    VIRTIO_PCI_FLAG_AUTO_MSIX_SIZE_BIT, true),
     DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, 3),
     DEFINE_VIRTIO_NET_FEATURES(VirtIOPCIProxy, host_features),
     DEFINE_PROP_END_OF_LIST(),
diff --git a/hw/virtio/virtio-pci.h b/hw/virtio/virtio-pci.h
index 8873b6d..96dea82 100644
--- a/hw/virtio/virtio-pci.h
+++ b/hw/virtio/virtio-pci.h
@@ -62,6 +62,9 @@ typedef struct VirtioBusClass VirtioPCIBusClass;
  * vcpu thread using ioeventfd for some devices. */
 #define VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT 1
 #define VIRTIO_PCI_FLAG_USE_IOEVENTFD   (1 << VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT)
+#define VIRTIO_PCI_FLAG_AUTO_MSIX_SIZE_BIT 2
+#define VIRTIO_PCI_FLAG_AUTO_MSIX_SIZE \
+    (1 << VIRTIO_PCI_FLAG_AUTO_MSIX_SIZE_BIT)
 
 typedef struct {
     MSIMessage msg;
diff --git a/include/hw/compat.h b/include/hw/compat.h
index 313682a..3e5c5ae 100644
--- a/include/hw/compat.h
+++ b/include/hw/compat.h
@@ -1,7 +1,15 @@
 #ifndef HW_COMPAT_H
 #define HW_COMPAT_H
 
+#define HW_COMPAT_2_2 \
+        {\
+            .driver   = "virtio-net-pci",\
+            .property = "auto_msix_bar_size",\
+            .value    = "off",\
+        }
+
 #define HW_COMPAT_2_1 \
+        HW_COMPAT_2_2, \
         {\
             .driver   = "intel-hda",\
             .property = "old_msi_addr",\
-- 
2.1.0

Re: [Qemu-devel] [PATCH V3 10/14] virtio: introduce virtio_queue_get_index()

[Fam Zheng]

On Thu, 03/05 13:48, Jason Wang wrote:
> This patch introduces a helper that can get the queue index of a
> VirtQueue. This is useful when traversing the list of VirtQueues.
> 
> Cc: Anthony Liguori <aliguori@amazon.com>
> Cc: Michael S. Tsirkin <mst@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  hw/virtio/virtio.c         | 5 +++++
>  include/hw/virtio/virtio.h | 1 +
>  2 files changed, 6 insertions(+)
> 
> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
> index 5e6b879..bd32518 100644
> --- a/hw/virtio/virtio.c
> +++ b/hw/virtio/virtio.c
> @@ -725,6 +725,11 @@ hwaddr virtio_queue_get_addr(VirtIODevice *vdev, int n)
>      return vdev->vq[n].pa;
>  }
>  
> +int virtio_queue_get_index(VirtIODevice *vdev, VirtQueue *vq)
> +{
> +    return vq - vdev->vq;

Probably assert the return value is within [0, virtio_get_queue_max(vdev))?

Fam

> +}
> +
>  void virtio_queue_set_num(VirtIODevice *vdev, int n, int num)
>  {
>      /* Don't allow guest to flip queue between existent and
> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
> index ceab8e8..37aa9c4 100644
> --- a/include/hw/virtio/virtio.h
> +++ b/include/hw/virtio/virtio.h
> @@ -217,6 +217,7 @@ void virtio_config_writew(VirtIODevice *vdev, uint32_t addr, uint32_t data);
>  void virtio_config_writel(VirtIODevice *vdev, uint32_t addr, uint32_t data);
>  void virtio_queue_set_addr(VirtIODevice *vdev, int n, hwaddr addr);
>  hwaddr virtio_queue_get_addr(VirtIODevice *vdev, int n);
> +int virtio_queue_get_index(VirtIODevice *vdev, VirtQueue *vq);
>  void virtio_queue_set_num(VirtIODevice *vdev, int n, int num);
>  int virtio_queue_get_num(VirtIODevice *vdev, int n);
>  void virtio_queue_set_align(VirtIODevice *vdev, int n, int align);
> -- 
> 2.1.0
> 
> 

Re: [Qemu-devel] [PATCH V3 10/14] virtio: introduce virtio_queue_get_index()

[Jason Wang]

On Thu, Mar 5, 2015 at 2:12 PM, Fam Zheng <famz@redhat.com> wrote:
> On Thu, 03/05 13:48, Jason Wang wrote:
>>  This patch introduces a helper that can get the queue index of a
>>  VirtQueue. This is useful when traversing the list of VirtQueues.
>>  
>>  Cc: Anthony Liguori <aliguori@amazon.com>
>>  Cc: Michael S. Tsirkin <mst@redhat.com>
>>  Signed-off-by: Jason Wang <jasowang@redhat.com>
>>  ---
>>   hw/virtio/virtio.c         | 5 +++++
>>   include/hw/virtio/virtio.h | 1 +
>>   2 files changed, 6 insertions(+)
>>  
>>  diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
>>  index 5e6b879..bd32518 100644
>>  --- a/hw/virtio/virtio.c
>>  +++ b/hw/virtio/virtio.c
>>  @@ -725,6 +725,11 @@ hwaddr virtio_queue_get_addr(VirtIODevice 
>> *vdev, int n)
>>       return vdev->vq[n].pa;
>>   }
>>   
>>  +int virtio_queue_get_index(VirtIODevice *vdev, VirtQueue *vq)
>>  +{
>>  +    return vq - vdev->vq;
> 
> Probably assert the return value is within [0, 
> virtio_get_queue_max(vdev))?
> 
> Fam

Not sure this is necessary. If we add one such here, we probably needs 
to add more elsewhere.

Re: [Qemu-devel] [PATCH V3 04/14] virtio-ccw: introduce ccw specific queue limit

[Cornelia Huck]

On Thu,  5 Mar 2015 13:48:41 +0800
Jason Wang <jasowang@redhat.com> wrote:

> Instead of depending on marco, using a bus specific limit.
> 
> Cc: Alexander Graf <agraf@suse.de>
> Cc: Cornelia Huck <cornelia.huck@de.ibm.com>
> Cc: Christian Borntraeger <borntraeger@de.ibm.com>
> Cc: Richard Henderson <rth@twiddle.net>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  hw/s390x/s390-virtio-ccw.c   |  7 +++++--
>  hw/s390x/virtio-ccw.c        | 13 +++++++------
>  include/hw/s390x/s390_flic.h |  2 +-
>  include/hw/virtio/virtio.h   |  1 +
>  4 files changed, 14 insertions(+), 9 deletions(-)
> 

> diff --git a/hw/s390x/virtio-ccw.c b/hw/s390x/virtio-ccw.c
> index 4874622..98a1a90 100644
> --- a/hw/s390x/virtio-ccw.c
> +++ b/hw/s390x/virtio-ccw.c

> @@ -1026,7 +1027,7 @@ static void virtio_ccw_notify(DeviceState *d, uint16_t vector)
Unfortunately just snipped off in this diff, but the code says

if (vector >= 128) {
>          return;
>      }

This is originating from (64 bits for queues + 64 bits for secondary
indicators) - only the first bit of secondary indicators is currently
used (for configuration changes). I'd suggest to change that to
queue_max + 64 and add a comment /* queue indicators + secondary
indicators */ or so.

> 
> -    if (vector < VIRTIO_PCI_QUEUE_MAX) {
> +    if (vector < virtio_get_queue_max(virtio_bus_get_device(&dev->bus))) {
>          if (!dev->indicators) {
>              return;
>          }

Re: [Qemu-devel] [PATCH V3 05/14] virtio-s390: switch to bus specific queue limit

[Cornelia Huck]

On Thu,  5 Mar 2015 13:48:42 +0800
Jason Wang <jasowang@redhat.com> wrote:

> Instead of depending on marco, switch to use a bus specific queue
> limit. Left is AdapterRouters->gsi[], this could be done in the future
> if we want to increase s390's queue limit really.

Stale comment regarding AdapterRoutes? You changed that in the ccw
patch.

> 
> Cc: Alexander Graf <agraf@suse.de>
> Cc: Richard Henderson <rth@twiddle.net>
> Cc: Christian Borntraeger <borntraeger@de.ibm.com>
> Cc: Cornelia Huck <cornelia.huck@de.ibm.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  hw/s390x/s390-virtio-bus.c | 8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)
> 

Re: [Qemu-devel] [PATCH V3 06/14] virtio-serial-bus: switch to bus specific queue limit

[Cornelia Huck]

On Thu,  5 Mar 2015 13:48:43 +0800
Jason Wang <jasowang@redhat.com> wrote:

> Cc: Amit Shah <amit.shah@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  hw/char/virtio-serial-bus.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/hw/char/virtio-serial-bus.c b/hw/char/virtio-serial-bus.c
> index 37a6f44..f280e95 100644
> --- a/hw/char/virtio-serial-bus.c
> +++ b/hw/char/virtio-serial-bus.c
> @@ -26,6 +26,8 @@
>  #include "hw/virtio/virtio-serial.h"
>  #include "hw/virtio/virtio-access.h"
> 
> +#define VIRTIO_SERIAL_BUS_QUEUE_MAX 64
> +
>  struct VirtIOSerialDevices {
>      QLIST_HEAD(, VirtIOSerial) devices;
>  } vserdevices;
> @@ -942,7 +944,7 @@ static void virtio_serial_device_realize(DeviceState *dev, Error **errp)
>      }
> 
>      /* Each port takes 2 queues, and one pair is for the control queue */
> -    max_supported_ports = VIRTIO_PCI_QUEUE_MAX / 2 - 1;
> +    max_supported_ports = VIRTIO_SERIAL_BUS_QUEUE_MAX / 2 - 1;

Shouldn't this be determined via the VirtIODevice instead? Or be the
maximum of those two values?

> 
>      if (vser->serial.max_virtserial_ports > max_supported_ports) {
>          error_setg(errp, "maximum ports supported: %u", max_supported_ports);

Re: [Qemu-devel] [PATCH V3 08/14] virtio-pci: switch to use bus specific queue limit

[Cornelia Huck]

On Thu,  5 Mar 2015 13:48:45 +0800
Jason Wang <jasowang@redhat.com> wrote:

> Instead of depending on a macro, switch to use a bus specific queue
> limit.
> 
> Cc: Anthony Liguori <aliguori@amazon.com>
> Cc: Michael S. Tsirkin <mst@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  hw/virtio/virtio-pci.c     | 12 +++++++-----
>  include/hw/virtio/virtio.h |  1 -
>  2 files changed, 7 insertions(+), 6 deletions(-)
> 

> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
> index 04ad532..9a81edf 100644
> --- a/include/hw/virtio/virtio.h
> +++ b/include/hw/virtio/virtio.h
> @@ -92,7 +92,6 @@ typedef struct VirtQueueElement
>      struct iovec out_sg[VIRTQUEUE_MAX_SIZE];
>  } VirtQueueElement;
> 
> -#define VIRTIO_PCI_QUEUE_MAX 64
>  #define VIRTIO_CCW_QUEUE_MAX 64

This hunk makes me think that the ccw limit shouldn't have been added
here :)

> 
>  #define VIRTIO_NO_VECTOR 0xffff

Re: [Qemu-devel] [PATCH V3 09/14] virtio: introduce vector to virtqueues mapping

[Cornelia Huck]

On Thu,  5 Mar 2015 13:48:46 +0800
Jason Wang <jasowang@redhat.com> wrote:

> Currently we will try to traverse all virtqueues to find a subset that
> using a specific vector. This is sub optimal when we will support
> hundreds or even thousands of virtqueues. So this patch introduces a
> method which could be used by transport to get all virtqueues that
> using a same vector. This is done through QLISTs and the number of
> QLISTs was queried through a transport specific method. When guest
> setting vectors, the virtqueue will be linked and helpers for traverse
> the list was also introduced.
> 
> The first user will be virtio pci which will use this to speed up
> MSI-X masking and unmasking handling.

Will there be any users beyond virtio-pci, though? For virtio-ccw, at
least, "vectors" are an identity mapping of the queue index. I'm not
sure if introducing (memory) overhead for everyone is worth it.

> 
> Cc: Anthony Liguori <aliguori@amazon.com>
> Cc: Michael S. Tsirkin <mst@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> ---
>  hw/virtio/virtio-pci.c         |  8 ++++++++
>  hw/virtio/virtio.c             | 32 ++++++++++++++++++++++++++++++--
>  include/hw/virtio/virtio-bus.h |  1 +
>  include/hw/virtio/virtio.h     |  3 +++
>  4 files changed, 42 insertions(+), 2 deletions(-)
> 

>  void virtio_queue_set_vector(VirtIODevice *vdev, int n, uint16_t vector)
>  {
> -    if (n < virtio_get_queue_max(vdev))
> +    VirtQueue *vq = &vdev->vq[n];
> +
> +    if (n < virtio_get_queue_max(vdev)) {
> +        if (vdev->vq[n].vector != VIRTIO_NO_VECTOR) {
> +            QLIST_REMOVE(vq, node);
> +        }
>          vdev->vq[n].vector = vector;
> +        if (vector != VIRTIO_NO_VECTOR) {
> +            QLIST_INSERT_HEAD(&vdev->vector_queues[vector], vq, node);
> +        }
> +    }
>  }

Is there any way to remove an entry? E.g., if the guest unassociates
virtqueues. (I just noticed I probably need to use VIRTIO_NO_VECTOR
instead of 0 for that case in ccw.)

Or maybe I'm completely misunderstanding what vectors are doing on
pci :)

Re: [Qemu-devel] [PATCH V3 04/14] virtio-ccw: introduce ccw specific queue limit

[Jason Wang]

On Fri, Mar 6, 2015 at 8:13 PM, Cornelia Huck 
<cornelia.huck@de.ibm.com> wrote:
> On Thu,  5 Mar 2015 13:48:41 +0800
> Jason Wang <jasowang@redhat.com> wrote:
> 
>>  Instead of depending on marco, using a bus specific limit.
>>  
>>  Cc: Alexander Graf <agraf@suse.de>
>>  Cc: Cornelia Huck <cornelia.huck@de.ibm.com>
>>  Cc: Christian Borntraeger <borntraeger@de.ibm.com>
>>  Cc: Richard Henderson <rth@twiddle.net>
>>  Signed-off-by: Jason Wang <jasowang@redhat.com>
>>  ---
>>   hw/s390x/s390-virtio-ccw.c   |  7 +++++--
>>   hw/s390x/virtio-ccw.c        | 13 +++++++------
>>   include/hw/s390x/s390_flic.h |  2 +-
>>   include/hw/virtio/virtio.h   |  1 +
>>   4 files changed, 14 insertions(+), 9 deletions(-)
>>  
> 
>>  diff --git a/hw/s390x/virtio-ccw.c b/hw/s390x/virtio-ccw.c
>>  index 4874622..98a1a90 100644
>>  --- a/hw/s390x/virtio-ccw.c
>>  +++ b/hw/s390x/virtio-ccw.c
> 
>>  @@ -1026,7 +1027,7 @@ static void virtio_ccw_notify(DeviceState *d, 
>> uint16_t vector)
> Unfortunately just snipped off in this diff, but the code says
> 
> if (vector >= 128) {
>>           return;
>>       }
> 
> This is originating from (64 bits for queues + 64 bits for secondary
> indicators) - only the first bit of secondary indicators is currently
> used (for configuration changes). I'd suggest to change that to
> queue_max + 64 and add a comment /* queue indicators + secondary
> indicators */ or so.

Will do this.

Thanks

Re: [Qemu-devel] [PATCH V3 05/14] virtio-s390: switch to bus specific queue limit

[Jason Wang]

On Fri, Mar 6, 2015 at 8:18 PM, Cornelia Huck 
<cornelia.huck@de.ibm.com> wrote:
> On Thu,  5 Mar 2015 13:48:42 +0800
> Jason Wang <jasowang@redhat.com> wrote:
> 
>>  Instead of depending on marco, switch to use a bus specific queue
>>  limit. Left is AdapterRouters->gsi[], this could be done in the 
>> future
>>  if we want to increase s390's queue limit really.
> 
> Stale comment regarding AdapterRoutes? You changed that in the ccw
> patch.

Yes, it is. Will move this to ccw patch.

Re: [Qemu-devel] [PATCH V3 06/14] virtio-serial-bus: switch to bus specific queue limit

[Jason Wang]

On Fri, Mar 6, 2015 at 8:27 PM, Cornelia Huck 
<cornelia.huck@de.ibm.com> wrote:
> On Thu,  5 Mar 2015 13:48:43 +0800
> Jason Wang <jasowang@redhat.com> wrote:
> 
>>  Cc: Amit Shah <amit.shah@redhat.com>
>>  Signed-off-by: Jason Wang <jasowang@redhat.com>
>>  ---
>>   hw/char/virtio-serial-bus.c | 4 +++-
>>   1 file changed, 3 insertions(+), 1 deletion(-)
>>  
>>  diff --git a/hw/char/virtio-serial-bus.c 
>> b/hw/char/virtio-serial-bus.c
>>  index 37a6f44..f280e95 100644
>>  --- a/hw/char/virtio-serial-bus.c
>>  +++ b/hw/char/virtio-serial-bus.c
>>  @@ -26,6 +26,8 @@
>>   #include "hw/virtio/virtio-serial.h"
>>   #include "hw/virtio/virtio-access.h"
>>  
>>  +#define VIRTIO_SERIAL_BUS_QUEUE_MAX 64
>>  +
>>   struct VirtIOSerialDevices {
>>       QLIST_HEAD(, VirtIOSerial) devices;
>>   } vserdevices;
>>  @@ -942,7 +944,7 @@ static void 
>> virtio_serial_device_realize(DeviceState *dev, Error **errp)
>>       }
>>  
>>       /* Each port takes 2 queues, and one pair is for the control 
>> queue */
>>  -    max_supported_ports = VIRTIO_PCI_QUEUE_MAX / 2 - 1;
>>  +    max_supported_ports = VIRTIO_SERIAL_BUS_QUEUE_MAX / 2 - 1;
> 
> Shouldn't this be determined via the VirtIODevice instead? Or be the
> maximum of those two values?

Rethink about this. I think it's ok to use the transport limit through 
VirtIODevice here. Then there's no need for virtio serial to handle 
migration compatibility.

Re: [Qemu-devel] [PATCH V3 08/14] virtio-pci: switch to use bus specific queue limit

[Jason Wang]

On Fri, Mar 6, 2015 at 8:28 PM, Cornelia Huck 
<cornelia.huck@de.ibm.com> wrote:
> On Thu,  5 Mar 2015 13:48:45 +0800
> Jason Wang <jasowang@redhat.com> wrote:
> 
>>  Instead of depending on a macro, switch to use a bus specific queue
>>  limit.
>>  
>>  Cc: Anthony Liguori <aliguori@amazon.com>
>>  Cc: Michael S. Tsirkin <mst@redhat.com>
>>  Signed-off-by: Jason Wang <jasowang@redhat.com>
>>  ---
>>   hw/virtio/virtio-pci.c     | 12 +++++++-----
>>   include/hw/virtio/virtio.h |  1 -
>>   2 files changed, 7 insertions(+), 6 deletions(-)
>>  
> 
>>  diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
>>  index 04ad532..9a81edf 100644
>>  --- a/include/hw/virtio/virtio.h
>>  +++ b/include/hw/virtio/virtio.hs
>>  @@ -92,7 +92,6 @@ typedef struct VirtQueueElement
>>       struct iovec out_sg[VIRTQUEUE_MAX_SIZE];
>>   } VirtQueueElement;
>>  
>>  -#define VIRTIO_PCI_QUEUE_MAX 64
>>   #define VIRTIO_CCW_QUEUE_MAX 64
> 
> This hunk makes me think that the ccw limit shouldn't have been added
> here :)

Yes, fail to find a common header at first try. But looks like I can 
move this in include/hw/s390x/s390_flic.h.

Re: [Qemu-devel] [PATCH V3 09/14] virtio: introduce vector to virtqueues mapping

[Jason Wang]

On Fri, Mar 6, 2015 at 8:55 PM, Cornelia Huck 
<cornelia.huck@de.ibm.com> wrote:
> On Thu,  5 Mar 2015 13:48:46 +0800
> Jason Wang <jasowang@redhat.com> wrote:
> 
>>  Currently we will try to traverse all virtqueues to find a subset 
>> that
>>  using a specific vector. This is sub optimal when we will support
>>  hundreds or even thousands of virtqueues. So this patch introduces a
>>  method which could be used by transport to get all virtqueues that
>>  using a same vector. This is done through QLISTs and the number of
>>  QLISTs was queried through a transport specific method. When guest
>>  setting vectors, the virtqueue will be linked and helpers for 
>> traverse
>>  the list was also introduced.
>>  
>>  The first user will be virtio pci which will use this to speed up
>>  MSI-X masking and unmasking handling.
> 
> Will there be any users beyond virtio-pci, though?

Currently not. But if the vector could be shared in another transport, 
this is still useful.
>  For virtio-ccw, at
> least, "vectors" are an identity mapping of the queue index. I'm not
> sure if introducing (memory) overhead for everyone is worth it.

I see. Another reason I make it generic is because I believe we try 
hard to not expose the internals of VirtQueue to transport specific 
code. 
> 
> 
>>  
>>  Cc: Anthony Liguori <aliguori@amazon.com>
>>  Cc: Michael S. Tsirkin <mst@redhat.com>
>>  Signed-off-by: Jason Wang <jasowang@redhat.com>
>>  ---
>>   hw/virtio/virtio-pci.c         |  8 ++++++++
>>   hw/virtio/virtio.c             | 32 
>> ++++++++++++++++++++++++++++++--
>>   include/hw/virtio/virtio-bus.h |  1 +
>>   include/hw/virtio/virtio.h     |  3 +++
>>   4 files changed, 42 insertions(+), 2 deletions(-)
>>  
> 
>>   void virtio_queue_set_vector(VirtIODevice *vdev, int n, uint16_t 
>> vector)
>>   {
>>  -    if (n < virtio_get_queue_max(vdev))
>>  +    VirtQueue *vq = &vdev->vq[n];
>>  +
>>  +    if (n < virtio_get_queue_max(vdev)) {
>>  +        if (vdev->vq[n].vector != VIRTIO_NO_VECTOR) {
>>  +            QLIST_REMOVE(vq, node);
>>  +        }
>>           vdev->vq[n].vector = vector;
>>  +        if (vector != VIRTIO_NO_VECTOR) {
>>  +            QLIST_INSERT_HEAD(&vdev->vector_queues[vector], vq, 
>> node);
>>  +        }
>>  +    }
>>   }
> 
> Is there any way to remove an entry?

Yes, e.g setting the vector to VIRTIO_NO_VECTOR (e.g during reset or 
driver exit).
>  E.g., if the guest unassociates
> virtqueues. (I just noticed I probably need to use VIRTIO_NO_VECTOR
> instead of 0 for that case in ccw.)

Yes, I see 0 is used in virtio_ccw_set_vqs() when addr is zero. If 
there's no special consideration here, need to use VIRITO_NO_VECTOR.
> 
> 
> Or maybe I'm completely misunderstanding what vectors are doing on
> pci :)

Re: [Qemu-devel] [PATCH V3 08/14] virtio-pci: switch to use bus specific queue limit

[Cornelia Huck]

On Mon, 09 Mar 2015 15:32:51 +0800
Jason Wang <jasowang@redhat.com> wrote:

> 
> 
> On Fri, Mar 6, 2015 at 8:28 PM, Cornelia Huck 
> <cornelia.huck@de.ibm.com> wrote:
> > On Thu,  5 Mar 2015 13:48:45 +0800
> > Jason Wang <jasowang@redhat.com> wrote:
> > 
> >>  Instead of depending on a macro, switch to use a bus specific queue
> >>  limit.
> >>  
> >>  Cc: Anthony Liguori <aliguori@amazon.com>
> >>  Cc: Michael S. Tsirkin <mst@redhat.com>
> >>  Signed-off-by: Jason Wang <jasowang@redhat.com>
> >>  ---
> >>   hw/virtio/virtio-pci.c     | 12 +++++++-----
> >>   include/hw/virtio/virtio.h |  1 -
> >>   2 files changed, 7 insertions(+), 6 deletions(-)
> >>  
> > 
> >>  diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
> >>  index 04ad532..9a81edf 100644
> >>  --- a/include/hw/virtio/virtio.h
> >>  +++ b/include/hw/virtio/virtio.hs
> >>  @@ -92,7 +92,6 @@ typedef struct VirtQueueElement
> >>       struct iovec out_sg[VIRTQUEUE_MAX_SIZE];
> >>   } VirtQueueElement;
> >>  
> >>  -#define VIRTIO_PCI_QUEUE_MAX 64
> >>   #define VIRTIO_CCW_QUEUE_MAX 64
> > 
> > This hunk makes me think that the ccw limit shouldn't have been added
> > here :)
> 
> Yes, fail to find a common header at first try. But looks like I can 
> move this in include/hw/s390x/s390_flic.h.
> 

Hm, I'm not 100% sure about that. Maybe if we introduce a #gsi define
there and have virtio-ccw inherit that as queue limit? Having a virtio
limit in the flic header feels a bit odd.

Re: [Qemu-devel] [PATCH V3 08/14] virtio-pci: switch to use bus specific queue limit

[Jason Wang]

On Mon, Mar 9, 2015 at 4:04 PM, Cornelia Huck 
<cornelia.huck@de.ibm.com> wrote:
> On Mon, 09 Mar 2015 15:32:51 +0800
> Jason Wang <jasowang@redhat.com> wrote:
> 
>>  
>>  
>>  On Fri, Mar 6, 2015 at 8:28 PM, Cornelia Huck 
>>  <cornelia.huck@de.ibm.com> wrote:
>>  > On Thu,  5 Mar 2015 13:48:45 +0800
>>  > Jason Wang <jasowang@redhat.com> wrote:
>>  > 
>>  >>  Instead of depending on a macro, switch to use a bus specific 
>> queue
>>  >>  limit.
>>  >>  
>>  >>  Cc: Anthony Liguori <aliguori@amazon.com>
>>  >>  Cc: Michael S. Tsirkin <mst@redhat.com>
>>  >>  Signed-off-by: Jason Wang <jasowang@redhat.com>
>>  >>  ---
>>  >>   hw/virtio/virtio-pci.c     | 12 +++++++-----
>>  >>   include/hw/virtio/virtio.h |  1 -
>>  >>   2 files changed, 7 insertions(+), 6 deletions(-)
>>  >>  
>>  > 
>>  >>  diff --git a/include/hw/virtio/virtio.h 
>> b/include/hw/virtio/virtio.h
>>  >>  index 04ad532..9a81edf 100644
>>  >>  --- a/include/hw/virtio/virtio.h
>>  >>  +++ b/include/hw/virtio/virtio.hs
>>  >>  @@ -92,7 +92,6 @@ typedef struct VirtQueueElement
>>  >>       struct iovec out_sg[VIRTQUEUE_MAX_SIZE];
>>  >>   } VirtQueueElement;
>>  >>  
>>  >>  -#define VIRTIO_PCI_QUEUE_MAX 64
>>  >>   #define VIRTIO_CCW_QUEUE_MAX 64
>>  > 
>>  > This hunk makes me think that the ccw limit shouldn't have been 
>> added
>>  > here :)
>>  
>>  Yes, fail to find a common header at first try. But looks like I 
>> can 
>>  move this in include/hw/s390x/s390_flic.h.
>>  
> 
> Hm, I'm not 100% sure about that. Maybe if we introduce a #gsi define
> there and have virtio-ccw inherit that as queue limit? Having a virtio
> limit in the flic header feels a bit odd.
> 

Ok, this sounds better. Will do this.

Thanks

Re: [Qemu-devel] [PATCH V3 00/14] Support more virtio queues

[Michael S. Tsirkin]

On Thu, Mar 05, 2015 at 01:48:37PM +0800, Jason Wang wrote:
> We current limit the max virtio queues to 64. This is not sufficient
> to support multiqueue deivces (e.g recent Linux support up to 256
> tap queues). So this series try to let virtio to support more queues.

For some reason I was Cc'd only on some patches.
If you want me to apply series, pls Cc on all of them.

> No much works need to be done except:
> 
> - Patch 1 add a check to validate the queues supported by backend
>   against the virtio queue limitation
> - Patch 2 fixes a bug in virtio_del_queue()
> - Patch 3 introdues a bus specific queue limitation
> - Patch 4 - Patch 8 let each transport to use bus specific queue
>   limitation instead of VIRTIO_PCI_QUEUE_MAX
> - Patch 9 introduces a map from vector to queues that use this vector
> - Patch 10 introduces a helpr for get the queue index
> - Patch 11 speedups the MSI-X masking and unmasking through the vector
>   to queues mapping that introduced by Patch 9
> - Patch 12 increase the maximum number of queues supported by
>   virtio-pci from 64 to 513, and deal with the migration compatibility
>   with the changes.
> - Patch 13 tries to remove the hard coded MSI-X bar size (4096) and
>   allow up to 2048 MSI-X entries.
> - Patch 14 add a boolean property to let the virtio-net can calculate
>   the MSI-X bar size based on the number of MSI-X vectors and keep
>   migration compatibility with legacy version whose bar size is 4096.
> 
> With this patch, we can support up to 256 queues. Since x86 can only
> allow about 240 interrupt vectors for MSI-X, current Linux driver can
> only have about 80 queue pairs has their private MSI-X interrupt
> vectors. With sharing IRQ with queue pairs (RFC posted in
> https://lkml.org/lkml/2014/12/25/169), Linux driver can have up
> to about 186 queue pairs has their private MSI-X interrupt vectors.
> 
> Stress/migration test on virtio-pci, compile test/make check on
> s390x-softmmu.
> 
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Richard Henderson <rth@twiddle.net>
> Cc: Anthony Liguori <aliguori@amazon.com>
> Cc: Michael S. Tsirkin <mst@redhat.com>
> Cc: Alexander Graf <agraf@suse.de>
> Cc: Keith Busch <keith.busch@intel.com>
> Cc: Kevin Wolf <kwolf@redhat.com>
> Cc: Stefan Hajnoczi <stefanha@redhat.com>
> Cc: Anthony Liguori <aliguori@amazon.com>
> Cc: Christian Borntraeger <borntraeger@de.ibm.com>
> Cc: Cornelia Huck <cornelia.huck@de.ibm.com>
> Cc: Amit Shah <amit.shah@redhat.com>
> 
> Please review
> 
> Thanks
> 
> Changes from V2:
> - Move transport specific limitation to their implementation. (The
>   left is VIRTIO_CCW_QUEUE_MAX since I fail to find a common header
>   files other than virtio.h)
> - Use virtio_get_queue_max() if possbile in virtio.c
> - AdapterRoutes should use ccw limit
> - Introduce a vector to queue mapping for virito devices and speedup
>   the MSI-X masking and unmasking through this.
> 
> Changes from V1:
> - add a validation against the bus limitation
> - switch to use a bus specific queue limit instead of a global one,
>   this will allow us to just increase the limit of one transport
>   without disturbing others.
> - only increase the queue limit of virtio-pci
> - limit the maximum number of virtio queues to 64 for legacy machine
>   types
> 
> Jason Wang (14):
>   virtio-net: validate backend queue numbers against bus limitation
>   virtio-net: fix the upper bound when trying to delete queues
>   virito: introduce bus specific queue limit
>   virtio-ccw: introduce ccw specific queue limit
>   virtio-s390: switch to bus specific queue limit
>   virtio-serial-bus: switch to bus specific queue limit
>   virtio-mmio: switch to bus specific queue limit
>   virtio-pci: switch to use bus specific queue limit
>   virtio: introduce vector to virtqueues mapping
>   virtio: introduce virtio_queue_get_index()
>   virtio-pci: speedup MSI-X masking and unmasking
>   virtio-pci: increase the maximum number of virtqueues to 513
>   pci: remove hard-coded bar size in msix_init_exclusive_bar()
>   virtio-pci: introduce auto_msix_bar_size property
> 
>  hw/block/nvme.c                |  2 +-
>  hw/char/virtio-serial-bus.c    |  4 ++-
>  hw/i386/pc_piix.c              | 10 ++++++
>  hw/i386/pc_q35.c               |  9 +++++
>  hw/misc/ivshmem.c              |  2 +-
>  hw/net/virtio-net.c            |  9 ++++-
>  hw/pci/msix.c                  | 18 ++++------
>  hw/ppc/spapr.c                 |  5 +++
>  hw/s390x/s390-virtio-bus.c     |  7 ++--
>  hw/s390x/s390-virtio-ccw.c     |  7 ++--
>  hw/s390x/virtio-ccw.c          | 12 ++++---
>  hw/scsi/virtio-scsi.c          |  4 +--
>  hw/virtio/virtio-mmio.c        |  7 ++--
>  hw/virtio/virtio-pci.c         | 78 ++++++++++++++++++++++++++++--------------
>  hw/virtio/virtio-pci.h         |  3 ++
>  hw/virtio/virtio.c             | 74 ++++++++++++++++++++++++++++++---------
>  include/hw/compat.h            |  8 +++++
>  include/hw/pci/msix.h          |  2 +-
>  include/hw/s390x/s390_flic.h   |  2 +-
>  include/hw/virtio/virtio-bus.h |  2 ++
>  include/hw/virtio/virtio.h     |  7 +++-
>  21 files changed, 199 insertions(+), 73 deletions(-)
> 
> -- 
> 2.1.0

Re: [Qemu-devel] [PATCH V3 00/14] Support more virtio queues

[Jason Wang]

On Thu, Mar 12, 2015 at 5:28 PM, Michael S. Tsirkin <mst@redhat.com> 
wrote:
> On Thu, Mar 05, 2015 at 01:48:37PM +0800, Jason Wang wrote:
>>  We current limit the max virtio queues to 64. This is not sufficient
>>  to support multiqueue deivces (e.g recent Linux support up to 256
>>  tap queues). So this series try to let virtio to support more 
>> queues.
> 
> For some reason I was Cc'd only on some patches.
> If you want me to apply series, pls Cc on all of them.

Ok, will cc you for all patches in V4.

And looks like 2.3 is a little bit risky. Will target this to 2.4

Thanks