[Qemu-devel] [RESEND PATCH v16 0/6] KVM platform device passthrough

[Qemu-devel] [RESEND PATCH v16 0/6] KVM platform device passthrough

[Eric Auger]

Now that the platform-bus is instantiated in virt, the ARM sysbus-fdt
infrastructure, the VFIO platform device and its first Calxeda xgmac
specialization also are on the master, let's complete the work by

- enabling the instantiation of the VFIO Calxeda xgmac device
- enabling irqfd usage, which is vital for performance.

So the resend only relates to patches 4-9 of the v16 series.

http://git.linaro.org/people/eric.auger/qemu.git
(branch vfio_integ_v16_partial_resend)

Tested by Vikram Sethi on a SATA device featuring 2 MMIO regions.
Code also exercised successfully by VOSYS for AMBA derivative series.

Thanks

Eric

v15 -> v16:
- no code change
- add Vikram's T-b
- rebase on 0915aed
- remove linux header update patch file

v14 -> v15:
- add Peter R-b on sysbus: add irq_routing_notifier
- correct g_malloc0_n usage in skeleton
- correct return values of reset related functions
- include Cornelia's patch for header update

v13 -> v14:
- remove v13 9, 10, 11 patch files and replace them by a single patch file
  "sysbus: add irq_routing_notifier".
- in skeleton, fix ENAMETOOLONG sign
- remove VFIOINTp virtualID in "add irq assignment" patch file
- removed trace_vfio_platform_start_eventfd

v12 -> v13:
- header update but same update was already sent by Cornelia
- Rework VFIO signaling & irqfd setup: restored 2-step setup featuring
  eventfd setup on realize and then irqfd setup on irq binding.
- irqfd setup now uses kvm_irqchip_add_irqfd_notifier and
  sysbus irq_set_hook override. This leads to the introduction of 6 patch
  files enabling those 2 features. Paolo advised to introduce
  kvm_irqchip_add_irqfd_notifier series in the VFIO one. I did the
  same for irq_set_hook series but if it is better I can submit it aside.
- above changes made possible to remove
  x hw/vfio/platform: add capability to start IRQ propagation"
  x hw/arm/virt: start VFIO IRQ propagation
- in sysbus-fdt.c, use platform_bus_get_mmio_addr instead of deprecated
  mmio[0] property. Thanks to Bharat who pointed this issue out. also
  cpu_to_be32 was used for size and base (Vikram input) .
- in skeleton misc corrections following Alex 

v11->v12:
- add x-mmap property definition, without which the default value of
  vbasedev.allow_mmap is false, hence preventing the reg space from
  being mmapped.

v10->v11:
- rebase onto v2.3.0-rc0 (mainly related to PCIe support in virt)
- add dma-coherent property for calxeda midway (fix revealed by removal
  of kernel-side "vfio: type1: support for ARM SMMUS with VFIO_IOMMU_TYPE1")
- virt modifications to start VFIO IRQ forwarding are now in a separate
  patch
- rearrange linux header exports (those are still partial exports
  waiting for definitive 4.1-rc0)
- take into account Alex Bennécomments:
  - use g_malloc0_n instead of g_malloc0
  - use block declarations when possible
  - rework readlink returned value treatment
  - use g_strlcat in place strncat
  - re-arrange mutex locking for multiple IRQ support (user-side handled
    eventfds)
- use g_snprintf instead of snprintf
- change the order of functions to avoid pre-declaration in platform.c
- add flags in VFIOINTp struct to detect whether the IRQ is automasked
- some comment rewriting

v9->v10:
- rebase on "vfio: cleanup vfio_get_device error path, remove
  vfio_populate_device": vfio_populate_device no more called in
  vfio_get_device but in vfio_base_device_init
- update VFIO header according to vfio platform driver v13 (no AMBA)

v8->v9:
- rebase on 2.2.0 and machvirt dynamic sysbus instantiation v10
- v8 1-11 were pulled
- patch files related to forwarding are moved in a seperate series since
  it depends on kernel series still in RFC.
- introduction of basic VFIO platform device split into 3 patch files to
  ease the review (hope it will help).
- add an author in platform.c
- add deallocation in vfio_populate_device error case
- add patch file doing the VFIO header sync
- use VFIO_DEVICE_FLAGS_PLATFORM in vfio_populate_device
- rename calxeda_xgmac.c into calxeda-xgmac.c
- sysbus-fdt: add_calxeda_midway_xgmac_fdt_node g_free in case of errors
- reword of linux-headers patch files

v7->v8:
- rebase on v2.2.0-rc3 and integrate
  "Add skip_dump flag to ignore memory region during dump"
- KVM header evolution with subindex addition in kvm_arch_forwarded_irq
- split [PATCH v7 03/16] hw/vfio/pci: introduce VFIODevice into 4 patches
- vfio_compute_needs_reset does not return bool anymore
- add some comments about exposed MMIO region and IRQ in calxeda xgmac
  device
- vfio_[un]mask_irqindex renamed into vfio_[un]mask_single_irqindex
- rework IRQ startup: former machine init done notifier is replaced by a
  reset notifier. machine file passes the interrupt controller
  DeviceState handle (not the platform bus first irq parameter).
- sysbus-fdt:
  - move the add_fdt_node_functions array declaration between the device
    specific code and the generic code to avoid forward declarations of
    decice specific functions
  - rename add_basic_vfio_fdt_node into add_calxeda_midway_xgmac_fdt_node
    emphasizing the fact it is xgmac specific

v6->v7:
- fake injection test modality removed
- VFIO_DEVICE_TYPE_PLATFORM only introduced with VFIO platform
- new helper functions to start VFIO IRQ on machine init done notifier
  (introduced in hw/vfio/platform: add vfio-platform support and notifier
  registration invoked in hw/arm/virt: add support for VFIO devices).
  vfio_start_irq_injection is replaced by vfio_register_irq_starter.

v5->v6:
- rebase on 2.1rc5 PCI code
- forwarded IRQ first integraton
- vfio_device property renamed into host property
- split IRQ setup in different functions that match the 3 supported
  injection techniques (user handled eventfd, irqfd, forwarded IRQ):
  removes dynamic switch between injection methods
- introduce fake interrupts as a test modality:
  x makes possible to test multiple IRQ user-side handling.
  x this is a test feature only: enable to trigger a fd as if the
    real physical IRQ hit. No virtual IRQ is injected into the guest
    but handling is simulated so that the state machine can be tested
- user handled eventfd:
  x add mutex to protect IRQ state & list manipulation,
  x correct misleading comment in vfio_intp_interrupt.
  x Fix bugs using fake interrupt modality
- irqfd no more advertised in this patchset (handled in [3])
- VFIOPlatformDeviceClass becomes abstract and Calxeda xgmac device
  and class is re-introduced (as per v4)
- all DPRINTF removed in platform and replaced by trace-points
- corrects compilation with configure --disable-kvm
- simplifies the split for vfio_get_device and introduce a unique
  specialized function named vfio_populate_device
- group_list renamed into vfio_group_list
- hw/arm/dyn_sysbus_devtree.c currently only support vfio-calxeda-xgmac
  instantiation. Needs to be specialized for other VFIO devices
- fix 2 bugs in dyn_sysbus_devtree(reg_attr index and compat)

v4->v5:
- rebase on v2.1.0 PCI code
- take into account Alex Williamson comments on PCI code rework
  - trace updates in vfio_region_write/read
  - remove fd from VFIORegion
  - get/put ckeanup
- bug fix: bar region's vbasedev field duly initialization
- misc cleanups in platform device
- device tree node generation removed from device and handled in
  hw/arm/dyn_sysbus_devtree.c
- remove "hw/vfio: add an example calxeda_xgmac": with removal of
  device tree node generation we do not have so many things to
  implement in that derived device yet. May be re-introduced later
  on if needed typically for reset/migration.
- no GSI routing table anymore

v3->v4 changes (Eric Auger, Alvise Rigo)
- rebase on last VFIO PCI code (v2.1.0-rc0)
- full git history rework to ease PCI code change review
- mv include files in hw/vfio
- DPRINTF reformatting temporarily moved out
- support of VFIO virq (removal of resamplefd handler on user-side)
- integration with sysbus dynamic instantiation framwork
- removal of unrealize and cleanup routines until it is better
  understood what is really needed
- Support of VFIO for Amba devices should be handled in an inherited
  device to specialize the device tree generation (clock handle currently
  missing in framework however)
- "Always use eventfd as notifying mechanism" temporarily moved out
- static instantiation is not mainstream (although it remains possible)
  note if static instantiation is used, irqfd must be setup in machine file
  when virtual IRQ is known
- create the GSI routing table on qemu side

v2->v3 changes (Alvise Rigo, Eric Auger):
- Following Alex W recommandations, further efforts to factorize the
  code between PCI:introduction of VFIODevice and VFIORegion
  as base classes
- unique reset handler for platform and PCI
- cleanup following Kim's comments
- multiple IRQ support mechanics should be in place although not
  tested
- Better handling of MMIO multiple regions
- New features and fixes by Alvise (multiple compat string, exec
  flag, force eventfd usage, amba device tree support)
- irqfd support

v1->v2 changes (Kim Phillips, Eric Auger):
- IRQ initial support (legacy mode where eventfds are handled on
  user side)
- hacked dynamic instantiation

v1 (Kim Phillips):
- initial split between PCI and platform
- MMIO support only
- static instantiation

Best Regards

Eric




Eric Auger (6):
  hw/arm/sysbus-fdt: enable vfio-calxeda-xgmac dynamic instantiation
  kvm: rename kvm_irqchip_[add,remove]_irqfd_notifier with gsi suffix
  kvm-all.c: add qemu_irq/gsi hash table and utility routines
  intc: arm_gic_kvm: set the qemu_irq/gsi mapping
  sysbus: add irq_routing_notifier
  hw/vfio/platform: add irqfd support

 hw/arm/sysbus-fdt.c             |  72 +++++++++++++++++++++++++++
 hw/core/sysbus.c                |   6 +++
 hw/intc/arm_gic_kvm.c           |   6 +++
 hw/s390x/virtio-ccw.c           |   8 +--
 hw/vfio/pci.c                   |   6 +--
 hw/vfio/platform.c              | 107 ++++++++++++++++++++++++++++++++++++++++
 hw/virtio/virtio-pci.c          |   4 +-
 include/hw/sysbus.h             |   1 +
 include/hw/vfio/vfio-platform.h |   2 +
 include/sysemu/kvm.h            |  11 ++++-
 kvm-all.c                       |  44 +++++++++++++++--
 kvm-stub.c                      |   7 +--
 trace-events                    |   1 +
 13 files changed, 258 insertions(+), 17 deletions(-)

-- 
1.8.3.2

[Qemu-devel] [RESEND PATCH v16 1/6] hw/arm/sysbus-fdt: enable vfio-calxeda-xgmac dynamic instantiation

[Eric Auger]

This patch allows the instantiation of the vfio-calxeda-xgmac device
from the QEMU command line (-device vfio-calxeda-xgmac,host="<device>").

A specialized device tree node is created for the guest, containing
compat, dma-coherent, reg and interrupts properties.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---
v12 -> v13:
- use platform_bus_get_mmio_addr instead of deprecated mmio[0] property.
  Thanks to Bharat who pointed this issue out.
- use cpu_to_be32 to mmio_base & size (Vikram report)

v10 -> v11:
- add dma-coherent property to calxeda midway xgmac node (fix)
- use qemu_fdt_setprop to add reg property instead of
  qemu_fdt_setprop_sized_cells_from_array
- commit message rewording

v8 -> v9:
- properly free resources in case of errors in
  add_calxeda_midway_xgmac_fdt_node

v7 -> v8:
- move the add_fdt_node_functions array declaration between the device
  specific code and the generic code to avoid forward declarations of
  decice specific functions
- rename add_basic_vfio_fdt_node into
  add_calxeda_midway_xgmac_fdt_node

v6 -> v7:
- compat string re-formatting removed since compat string is not exposed
  anymore as a user option
- VFIO IRQ kick-off removed from sysbus-fdt and moved to VFIO platform
  device
---
 hw/arm/sysbus-fdt.c | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)

diff --git a/hw/arm/sysbus-fdt.c b/hw/arm/sysbus-fdt.c
index 3038b94..3d67acf 100644
--- a/hw/arm/sysbus-fdt.c
+++ b/hw/arm/sysbus-fdt.c
@@ -26,6 +26,8 @@
 #include "sysemu/device_tree.h"
 #include "hw/platform-bus.h"
 #include "sysemu/sysemu.h"
+#include "hw/vfio/vfio-platform.h"
+#include "hw/vfio/vfio-calxeda-xgmac.h"
 
 /*
  * internal struct that contains the information to create dynamic
@@ -53,11 +55,81 @@ typedef struct NodeCreationPair {
     int (*add_fdt_node_fn)(SysBusDevice *sbdev, void *opaque);
 } NodeCreationPair;
 
+/* Device Specific Code */
+
+/**
+ * add_calxeda_midway_xgmac_fdt_node
+ *
+ * Generates a simple node with following properties:
+ * compatible string, regs, interrupts, dma-coherent
+ */
+static int add_calxeda_midway_xgmac_fdt_node(SysBusDevice *sbdev, void *opaque)
+{
+    PlatformBusFDTData *data = opaque;
+    PlatformBusDevice *pbus = data->pbus;
+    void *fdt = data->fdt;
+    const char *parent_node = data->pbus_node_name;
+    int compat_str_len, i, ret = -1;
+    char *nodename;
+    uint32_t *irq_attr, *reg_attr;
+    uint64_t mmio_base, irq_number;
+    VFIOPlatformDevice *vdev = VFIO_PLATFORM_DEVICE(sbdev);
+    VFIODevice *vbasedev = &vdev->vbasedev;
+
+    mmio_base = platform_bus_get_mmio_addr(pbus, sbdev, 0);
+    nodename = g_strdup_printf("%s/%s@%" PRIx64, parent_node,
+                               vbasedev->name, mmio_base);
+    qemu_fdt_add_subnode(fdt, nodename);
+
+    compat_str_len = strlen(vdev->compat) + 1;
+    qemu_fdt_setprop(fdt, nodename, "compatible",
+                          vdev->compat, compat_str_len);
+
+    qemu_fdt_setprop(fdt, nodename, "dma-coherent", "", 0);
+
+    reg_attr = g_new(uint32_t, vbasedev->num_regions*2);
+    for (i = 0; i < vbasedev->num_regions; i++) {
+        mmio_base = platform_bus_get_mmio_addr(pbus, sbdev, i);
+        reg_attr[2*i] = cpu_to_be32(mmio_base);
+        reg_attr[2*i+1] = cpu_to_be32(
+                                memory_region_size(&vdev->regions[i]->mem));
+    }
+    ret = qemu_fdt_setprop(fdt, nodename, "reg", reg_attr,
+                           vbasedev->num_regions*2*sizeof(uint32_t));
+    if (ret) {
+        error_report("could not set reg property of node %s", nodename);
+        goto fail_reg;
+    }
+
+    irq_attr = g_new(uint32_t, vbasedev->num_irqs*3);
+    for (i = 0; i < vbasedev->num_irqs; i++) {
+        irq_number = platform_bus_get_irqn(pbus, sbdev , i)
+                         + data->irq_start;
+        irq_attr[3*i] = cpu_to_be32(0);
+        irq_attr[3*i+1] = cpu_to_be32(irq_number);
+        irq_attr[3*i+2] = cpu_to_be32(0x4);
+    }
+   ret = qemu_fdt_setprop(fdt, nodename, "interrupts",
+                     irq_attr, vbasedev->num_irqs*3*sizeof(uint32_t));
+    if (ret) {
+        error_report("could not set interrupts property of node %s",
+                     nodename);
+    }
+    g_free(irq_attr);
+fail_reg:
+    g_free(reg_attr);
+    g_free(nodename);
+    return ret;
+}
+
 /* list of supported dynamic sysbus devices */
 static const NodeCreationPair add_fdt_node_functions[] = {
+    {TYPE_VFIO_CALXEDA_XGMAC, add_calxeda_midway_xgmac_fdt_node},
     {"", NULL}, /* last element */
 };
 
+/* Generic Code */
+
 /**
  * add_fdt_node - add the device tree node of a dynamic sysbus device
  *
-- 
1.8.3.2

[Qemu-devel] [RESEND PATCH v16 2/6] kvm: rename kvm_irqchip_[add, remove]_irqfd_notifier with gsi suffix

[Eric Auger]

Anticipating for the introduction of new add/remove functions taking
a qemu_irq parameter, let's rename existing ones with a gsi suffix.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
Tested-by: Vikram Sethi <vikrams@codeaurora.org>

---

v15 -> v16:
- added Vikram's T-b
- resolve rebase conflict in kvm.h
---
 hw/s390x/virtio-ccw.c  | 8 ++++----
 hw/vfio/pci.c          | 6 +++---
 hw/virtio/virtio-pci.c | 4 ++--
 include/sysemu/kvm.h   | 7 ++++---
 kvm-all.c              | 7 ++++---
 kvm-stub.c             | 7 ++++---
 6 files changed, 21 insertions(+), 18 deletions(-)

diff --git a/hw/s390x/virtio-ccw.c b/hw/s390x/virtio-ccw.c
index b7a88d6..4a2b0f1 100644
--- a/hw/s390x/virtio-ccw.c
+++ b/hw/s390x/virtio-ccw.c
@@ -1186,8 +1186,8 @@ static int virtio_ccw_add_irqfd(VirtioCcwDevice *dev, int n)
     VirtQueue *vq = virtio_get_queue(vdev, n);
     EventNotifier *notifier = virtio_queue_get_guest_notifier(vq);
 
-    return kvm_irqchip_add_irqfd_notifier(kvm_state, notifier, NULL,
-                                          dev->routes.gsi[n]);
+    return kvm_irqchip_add_irqfd_notifier_gsi(kvm_state, notifier, NULL,
+                                              dev->routes.gsi[n]);
 }
 
 static void virtio_ccw_remove_irqfd(VirtioCcwDevice *dev, int n)
@@ -1197,8 +1197,8 @@ static void virtio_ccw_remove_irqfd(VirtioCcwDevice *dev, int n)
     EventNotifier *notifier = virtio_queue_get_guest_notifier(vq);
     int ret;
 
-    ret = kvm_irqchip_remove_irqfd_notifier(kvm_state, notifier,
-                                            dev->routes.gsi[n]);
+    ret = kvm_irqchip_remove_irqfd_notifier_gsi(kvm_state, notifier,
+                                                dev->routes.gsi[n]);
     assert(ret == 0);
 }
 
diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index e0e339a..ac85624 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -597,7 +597,7 @@ static void vfio_add_kvm_msi_virq(VFIOMSIVector *vector, MSIMessage *msg,
         return;
     }
 
-    if (kvm_irqchip_add_irqfd_notifier(kvm_state, &vector->kvm_interrupt,
+    if (kvm_irqchip_add_irqfd_notifier_gsi(kvm_state, &vector->kvm_interrupt,
                                        NULL, virq) < 0) {
         kvm_irqchip_release_virq(kvm_state, virq);
         event_notifier_cleanup(&vector->kvm_interrupt);
@@ -609,8 +609,8 @@ static void vfio_add_kvm_msi_virq(VFIOMSIVector *vector, MSIMessage *msg,
 
 static void vfio_remove_kvm_msi_virq(VFIOMSIVector *vector)
 {
-    kvm_irqchip_remove_irqfd_notifier(kvm_state, &vector->kvm_interrupt,
-                                      vector->virq);
+    kvm_irqchip_remove_irqfd_notifier_gsi(kvm_state, &vector->kvm_interrupt,
+                                          vector->virq);
     kvm_irqchip_release_virq(kvm_state, vector->virq);
     vector->virq = -1;
     event_notifier_cleanup(&vector->kvm_interrupt);
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index d7cf34c..1d4435a 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -495,7 +495,7 @@ static int kvm_virtio_pci_irqfd_use(VirtIOPCIProxy *proxy,
     VirtQueue *vq = virtio_get_queue(vdev, queue_no);
     EventNotifier *n = virtio_queue_get_guest_notifier(vq);
     int ret;
-    ret = kvm_irqchip_add_irqfd_notifier(kvm_state, n, NULL, irqfd->virq);
+    ret = kvm_irqchip_add_irqfd_notifier_gsi(kvm_state, n, NULL, irqfd->virq);
     return ret;
 }
 
@@ -509,7 +509,7 @@ static void kvm_virtio_pci_irqfd_release(VirtIOPCIProxy *proxy,
     VirtIOIRQFD *irqfd = &proxy->vector_irqfd[vector];
     int ret;
 
-    ret = kvm_irqchip_remove_irqfd_notifier(kvm_state, n, irqfd->virq);
+    ret = kvm_irqchip_remove_irqfd_notifier_gsi(kvm_state, n, irqfd->virq);
     assert(ret == 0);
 }
 
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index f459fbd..acb3025 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -416,9 +416,10 @@ void kvm_irqchip_release_virq(KVMState *s, int virq);
 
 int kvm_irqchip_add_adapter_route(KVMState *s, AdapterInfo *adapter);
 
-int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n,
-                                   EventNotifier *rn, int virq);
-int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n, int virq);
+int kvm_irqchip_add_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
+                                       EventNotifier *rn, int virq);
+int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
+                                          int virq);
 void kvm_pc_gsi_handler(void *opaque, int n, int level);
 void kvm_pc_setup_irq_routing(bool pci_enabled);
 void kvm_init_irq_routing(KVMState *s);
diff --git a/kvm-all.c b/kvm-all.c
index 53e01d4..730b818 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -1325,14 +1325,15 @@ int kvm_irqchip_update_msi_route(KVMState *s, int virq, MSIMessage msg)
 }
 #endif /* !KVM_CAP_IRQ_ROUTING */
 
-int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n,
-                                   EventNotifier *rn, int virq)
+int kvm_irqchip_add_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
+                                       EventNotifier *rn, int virq)
 {
     return kvm_irqchip_assign_irqfd(s, event_notifier_get_fd(n),
            rn ? event_notifier_get_fd(rn) : -1, virq, true);
 }
 
-int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n, int virq)
+int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
+                                          int virq)
 {
     return kvm_irqchip_assign_irqfd(s, event_notifier_get_fd(n), -1, virq,
            false);
diff --git a/kvm-stub.c b/kvm-stub.c
index 7ba90c5..977de21 100644
--- a/kvm-stub.c
+++ b/kvm-stub.c
@@ -137,13 +137,14 @@ int kvm_irqchip_add_adapter_route(KVMState *s, AdapterInfo *adapter)
     return -ENOSYS;
 }
 
-int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n,
-                                   EventNotifier *rn, int virq)
+int kvm_irqchip_add_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
+                                       EventNotifier *rn, int virq)
 {
     return -ENOSYS;
 }
 
-int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n, int virq)
+int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
+                                          int virq)
 {
     return -ENOSYS;
 }
-- 
1.8.3.2

[Qemu-devel] [RESEND PATCH v16 3/6] kvm-all.c: add qemu_irq/gsi hash table and utility routines

[Eric Auger]

VFIO platform device needs to setup irqfd but it does not know the
gsi corresponding to the device qemu_irq. This series proposes to
store a hash table in kvm_state using the qemu_irq as key and the gsi
as a value.

kvm_irqchip_set_qemuirq_gsi allows to insert such a pair. The interrupt
controller is supposed to use it.

kvm_irqchip_[add, remove]_irqfd_notifier allows to setup/tear down
irqfd directly from the qemu_irq.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
Tested-by: Vikram Sethi <vikrams@codeaurora.org>

---

v15 -> v16:
- Added Vikram's T-b

v13 -> v14:
- correct checkpatch warning

v2 -> v3 (integration into VFIO series v13):
- rename kvm_irqchip_[add, remove]_qemuirq_irqfd_notifier into
  kvm_irqchip_[add, remove]_irqfd_notifier. Possible since legacy
  functions were also renamed with _gsi suffix.

V1 -> v2:
- qemu_irq get_gsi callback replaced by hash table stored in kvm
---
 include/sysemu/kvm.h |  6 ++++++
 kvm-all.c            | 37 +++++++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+)

diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index acb3025..ba612fc 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -19,6 +19,7 @@
 #include "qemu/queue.h"
 #include "qom/cpu.h"
 #include "exec/memattrs.h"
+#include "hw/irq.h"
 
 #ifdef CONFIG_KVM
 #include <linux/kvm.h>
@@ -420,6 +421,11 @@ int kvm_irqchip_add_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
                                        EventNotifier *rn, int virq);
 int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
                                           int virq);
+int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n,
+                                   EventNotifier *rn, qemu_irq irq);
+int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n,
+                                      qemu_irq irq);
+void kvm_irqchip_set_qemuirq_gsi(KVMState *s, qemu_irq irq, int gsi);
 void kvm_pc_gsi_handler(void *opaque, int n, int level);
 void kvm_pc_setup_irq_routing(bool pci_enabled);
 void kvm_init_irq_routing(KVMState *s);
diff --git a/kvm-all.c b/kvm-all.c
index 730b818..939bfc4 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -37,6 +37,7 @@
 #include "exec/address-spaces.h"
 #include "qemu/event_notifier.h"
 #include "trace.h"
+#include "hw/irq.h"
 
 #include "hw/boards.h"
 
@@ -98,6 +99,7 @@ struct KVMState
      * unsigned, and treating them as signed here can break things */
     unsigned irq_set_ioctl;
     unsigned int sigmask_len;
+    GHashTable *gsimap;
 #ifdef KVM_CAP_IRQ_ROUTING
     struct kvm_irq_routing *irq_routes;
     int nr_allocated_irq_routes;
@@ -1339,6 +1341,39 @@ int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
            false);
 }
 
+int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n,
+                                   EventNotifier *rn, qemu_irq irq)
+{
+    gpointer key, gsi;
+    gboolean found = g_hash_table_lookup_extended(s->gsimap, irq, &key, &gsi);
+
+    if (!found) {
+        return -ENXIO;
+    } else {
+        return kvm_irqchip_add_irqfd_notifier_gsi(s, n, rn,
+                                                  GPOINTER_TO_UINT(gsi));
+    }
+}
+
+int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n,
+                                      qemu_irq irq)
+{
+    gpointer key, gsi;
+    gboolean found = g_hash_table_lookup_extended(s->gsimap, irq, &key, &gsi);
+
+    if (!found) {
+        return -ENXIO;
+    } else {
+        return kvm_irqchip_remove_irqfd_notifier_gsi(s, n,
+                                                     GPOINTER_TO_INT(gsi));
+    }
+}
+
+void kvm_irqchip_set_qemuirq_gsi(KVMState *s, qemu_irq irq, int gsi)
+{
+    g_hash_table_insert(s->gsimap, irq, GINT_TO_POINTER(gsi));
+}
+
 static int kvm_irqchip_create(MachineState *machine, KVMState *s)
 {
     int ret;
@@ -1371,6 +1406,8 @@ static int kvm_irqchip_create(MachineState *machine, KVMState *s)
 
     kvm_init_irq_routing(s);
 
+    s->gsimap = g_hash_table_new(g_direct_hash, g_direct_equal);
+
     return 0;
 }
 
-- 
1.8.3.2

[Qemu-devel] [RESEND PATCH v16 4/6] intc: arm_gic_kvm: set the qemu_irq/gsi mapping

[Eric Auger]

The arm_gic_kvm now calls kvm_irqchip_set_qemuirq_gsi to build
the hash table storing qemu_irq/gsi mappings. From that point on
irqfd can be setup directly from the qemu_irq using
kvm_irqchip_add_irqfd_notifier.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
Tested-by: Vikram Sethi <vikrams@codeaurora.org>

---
v15 -> v16:
- added Vikram's T-b
- Resolve rebase conflict

v2 -> v3:
- kvm_irqchip_add_qemuirq_irqfd_notifier renamed into
  kvm_irqchip_add_irqfd_notifier
---
 hw/intc/arm_gic_kvm.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/hw/intc/arm_gic_kvm.c b/hw/intc/arm_gic_kvm.c
index 2cb7d25..f56bff1 100644
--- a/hw/intc/arm_gic_kvm.c
+++ b/hw/intc/arm_gic_kvm.c
@@ -570,6 +570,12 @@ static void kvm_arm_gic_realize(DeviceState *dev, Error **errp)
      */
     i += (GIC_INTERNAL * s->num_cpu);
     qdev_init_gpio_in(dev, kvm_arm_gic_set_irq, i);
+
+    for (i = 0; i < s->num_irq - GIC_INTERNAL; i++) {
+        qemu_irq irq = qdev_get_gpio_in(dev, i);
+        kvm_irqchip_set_qemuirq_gsi(kvm_state, irq, i);
+    }
+
     /* We never use our outbound IRQ/FIQ lines but provide them so that
      * we maintain the same interface as the non-KVM GIC.
      */
-- 
1.8.3.2

[Qemu-devel] [RESEND PATCH v16 5/6] sysbus: add irq_routing_notifier

[Eric Auger]

Add a new connect_irq_notifier notifier in the SysBusDeviceClass. This
notifier, if populated, is called after sysbus_connect_irq.

This mechanism is used to setup VFIO signaling once VFIO platform
devices get attached to their platform bus, on a machine init done
notifier.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
Reviewed-by: Peter Crosthwaite <peter.crosthwaite@xilinx.com>
Tested-by: Vikram Sethi <vikrams@codeaurora.org>

---
v15 -> v16:
- add Vikram's T-b

v14 -> v15:
- add Peter R-b

v2 -> v3 (integrated into this series v14):
- rename irq_routing_notifier into connect_irq_notifier

v1 -> v2:
- duly put the notifier in the class and not in the device
---
 hw/core/sysbus.c    | 6 ++++++
 include/hw/sysbus.h | 1 +
 2 files changed, 7 insertions(+)

diff --git a/hw/core/sysbus.c b/hw/core/sysbus.c
index b53c351..2d22aec 100644
--- a/hw/core/sysbus.c
+++ b/hw/core/sysbus.c
@@ -109,7 +109,13 @@ qemu_irq sysbus_get_connected_irq(SysBusDevice *dev, int n)
 
 void sysbus_connect_irq(SysBusDevice *dev, int n, qemu_irq irq)
 {
+    SysBusDeviceClass *sbd = SYS_BUS_DEVICE_GET_CLASS(dev);
+
     qdev_connect_gpio_out_named(DEVICE(dev), SYSBUS_DEVICE_GPIO_IRQ, n, irq);
+
+    if (sbd->connect_irq_notifier) {
+        sbd->connect_irq_notifier(dev, irq);
+    }
 }
 
 /* Check whether an MMIO region exists */
diff --git a/include/hw/sysbus.h b/include/hw/sysbus.h
index d1f3f00..e80b26d 100644
--- a/include/hw/sysbus.h
+++ b/include/hw/sysbus.h
@@ -41,6 +41,7 @@ typedef struct SysBusDeviceClass {
     /*< public >*/
 
     int (*init)(SysBusDevice *dev);
+    void (*connect_irq_notifier)(SysBusDevice *dev, qemu_irq irq);
 } SysBusDeviceClass;
 
 struct SysBusDevice {
-- 
1.8.3.2

[Qemu-devel] [RESEND PATCH v16 6/6] hw/vfio/platform: add irqfd support

[Eric Auger]

This patch aims at optimizing IRQ handling using irqfd framework.

Instead of handling the eventfds on user-side they are handled on
kernel side using
- the KVM irqfd framework,
- the VFIO driver virqfd framework.

the virtual IRQ completion is trapped at interrupt controller
This removes the need for fast/slow path swap.

Overall this brings significant performance improvements.

Signed-off-by: Alvise Rigo <a.rigo@virtualopensystems.com>
Signed-off-by: Eric Auger <eric.auger@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Vikram Sethi <vikrams@codeaurora.org>

---
v15 -> v16:
- add Vikram's T-b

v13 -> v14:
- use connect_irq_notifier
- remove trace_vfio_platform_start_eventfd

v12 -> v13:
- setup the new mechanism for starting irqfd, based on
  LinkPropertySetter override
- use kvm_irqchip_[add,remove]_irqfd_notifier new functions: no need
  to bother about gsi (hence virtualID could be removed with small
  change in trace-events)

v10 -> v11:
- Add Alex' Reviewed-by
- introduce kvm_accel in this patch and initialize it

v5 -> v6
- rely on kvm_irqfds_enabled() and kvm_resamplefds_enabled()
- guard KVM code with #ifdef CONFIG_KVM

v3 -> v4:
[Alvise Rigo]
Use of VFIO Platform driver v6 unmask/virqfd feature and removal
of resamplefd handler. Physical IRQ unmasking is now done in
VFIO driver.

v3:
[Eric Auger]
initial support with resamplefd handled on QEMU side since the
unmask was not supported on VFIO platform driver v5.
---
 hw/vfio/platform.c              | 107 ++++++++++++++++++++++++++++++++++++++++
 include/hw/vfio/vfio-platform.h |   2 +
 trace-events                    |   1 +
 3 files changed, 110 insertions(+)

diff --git a/hw/vfio/platform.c b/hw/vfio/platform.c
index 9382bb7..1d44085 100644
--- a/hw/vfio/platform.c
+++ b/hw/vfio/platform.c
@@ -26,6 +26,7 @@
 #include "hw/sysbus.h"
 #include "trace.h"
 #include "hw/platform-bus.h"
+#include "sysemu/kvm.h"
 
 /*
  * Functions used whatever the injection method
@@ -51,6 +52,7 @@ static VFIOINTp *vfio_init_intp(VFIODevice *vbasedev,
     intp->pin = info.index;
     intp->flags = info.flags;
     intp->state = VFIO_IRQ_INACTIVE;
+    intp->kvm_accel = false;
 
     sysbus_init_irq(sbdev, &intp->qemuirq);
 
@@ -61,6 +63,13 @@ static VFIOINTp *vfio_init_intp(VFIODevice *vbasedev,
         error_report("vfio: Error: trigger event_notifier_init failed ");
         return NULL;
     }
+    /* Get an eventfd for resample/unmask */
+    ret = event_notifier_init(&intp->unmask, 0);
+    if (ret) {
+        g_free(intp);
+        error_report("vfio: Error: resample event_notifier_init failed eoi");
+        return NULL;
+    }
 
     QLIST_INSERT_HEAD(&vdev->intp_list, intp, next);
     return intp;
@@ -315,6 +324,95 @@ static int vfio_start_eventfd_injection(VFIOINTp *intp)
     return ret;
 }
 
+/*
+ * Functions used for irqfd
+ */
+
+#ifdef CONFIG_KVM
+
+/**
+ * vfio_set_resample_eventfd - sets the resamplefd for an IRQ
+ * @intp: the IRQ struct handle
+ * programs the VFIO driver to unmask this IRQ when the
+ * intp->unmask eventfd is triggered
+ */
+static int vfio_set_resample_eventfd(VFIOINTp *intp)
+{
+    VFIODevice *vbasedev = &intp->vdev->vbasedev;
+    struct vfio_irq_set *irq_set;
+    int argsz, ret;
+    int32_t *pfd;
+
+    argsz = sizeof(*irq_set) + sizeof(*pfd);
+    irq_set = g_malloc0(argsz);
+    irq_set->argsz = argsz;
+    irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_UNMASK;
+    irq_set->index = intp->pin;
+    irq_set->start = 0;
+    irq_set->count = 1;
+    pfd = (int32_t *)&irq_set->data;
+    *pfd = event_notifier_get_fd(&intp->unmask);
+    qemu_set_fd_handler(*pfd, NULL, NULL, NULL);
+    ret = ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set);
+    g_free(irq_set);
+    if (ret < 0) {
+        error_report("vfio: Failed to set resample eventfd: %m");
+    }
+    return ret;
+}
+
+static void vfio_start_irqfd_injection(SysBusDevice *sbdev, qemu_irq irq)
+{
+    VFIOPlatformDevice *vdev = VFIO_PLATFORM_DEVICE(sbdev);
+    VFIOINTp *intp;
+    bool found = false;
+
+    QLIST_FOREACH(intp, &vdev->intp_list, next) {
+        if (intp->qemuirq == irq) {
+            found  = true;
+            break;
+        }
+    }
+    assert(found);
+
+    /* Get to a known interrupt state */
+    qemu_set_fd_handler(event_notifier_get_fd(&intp->interrupt),
+                        NULL, NULL, vdev);
+
+    vfio_mask_single_irqindex(&vdev->vbasedev, intp->pin);
+    qemu_set_irq(intp->qemuirq, 0);
+
+    if (kvm_irqchip_add_irqfd_notifier(kvm_state, &intp->interrupt,
+                                   &intp->unmask, irq) < 0) {
+        goto fail_irqfd;
+    }
+
+    if (vfio_set_trigger_eventfd(intp, NULL) < 0) {
+        goto fail_vfio;
+    }
+    if (vfio_set_resample_eventfd(intp) < 0) {
+        goto fail_vfio;
+    }
+
+    /* Let'em rip */
+    vfio_unmask_single_irqindex(&vdev->vbasedev, intp->pin);
+
+    intp->kvm_accel = true;
+
+    trace_vfio_platform_start_irqfd_injection(intp->pin,
+                                     event_notifier_get_fd(&intp->interrupt),
+                                     event_notifier_get_fd(&intp->unmask));
+    return;
+fail_vfio:
+    kvm_irqchip_remove_irqfd_notifier(kvm_state, &intp->interrupt, irq);
+fail_irqfd:
+    vfio_start_eventfd_injection(intp);
+    vfio_unmask_single_irqindex(&vdev->vbasedev, intp->pin);
+    return;
+}
+
+#endif /* CONFIG_KVM */
+
 /* VFIO skeleton */
 
 static void vfio_platform_compute_needs_reset(VFIODevice *vbasedev)
@@ -548,6 +646,7 @@ static void vfio_platform_realize(DeviceState *dev, Error **errp)
 {
     VFIOPlatformDevice *vdev = VFIO_PLATFORM_DEVICE(dev);
     SysBusDevice *sbdev = SYS_BUS_DEVICE(dev);
+    SysBusDeviceClass *sbc = SYS_BUS_DEVICE_GET_CLASS(dev);
     VFIODevice *vbasedev = &vdev->vbasedev;
     VFIOINTp *intp;
     int i, ret;
@@ -555,6 +654,13 @@ static void vfio_platform_realize(DeviceState *dev, Error **errp)
     vbasedev->type = VFIO_DEVICE_TYPE_PLATFORM;
     vbasedev->ops = &vfio_platform_ops;
 
+#ifdef CONFIG_KVM
+    if (kvm_irqfds_enabled() && kvm_resamplefds_enabled() &&
+        vdev->irqfd_allowed) {
+        sbc->connect_irq_notifier = vfio_start_irqfd_injection;
+    }
+#endif
+
     trace_vfio_platform_realize(vbasedev->name, vdev->compat);
 
     ret = vfio_base_device_init(vbasedev);
@@ -584,6 +690,7 @@ static Property vfio_platform_dev_properties[] = {
     DEFINE_PROP_BOOL("x-mmap", VFIOPlatformDevice, vbasedev.allow_mmap, true),
     DEFINE_PROP_UINT32("mmap-timeout-ms", VFIOPlatformDevice,
                        mmap_timeout, 1100),
+    DEFINE_PROP_BOOL("x-irqfd", VFIOPlatformDevice, irqfd_allowed, true),
     DEFINE_PROP_END_OF_LIST(),
 };
 
diff --git a/include/hw/vfio/vfio-platform.h b/include/hw/vfio/vfio-platform.h
index 26b2ad6..c5cf1d7 100644
--- a/include/hw/vfio/vfio-platform.h
+++ b/include/hw/vfio/vfio-platform.h
@@ -41,6 +41,7 @@ typedef struct VFIOINTp {
     int state; /* inactive, pending, active */
     uint8_t pin; /* index */
     uint32_t flags; /* IRQ info flags */
+    bool kvm_accel; /* set when QEMU bypass through KVM enabled */
 } VFIOINTp;
 
 /* function type for user side eventfd handler */
@@ -57,6 +58,7 @@ typedef struct VFIOPlatformDevice {
     uint32_t mmap_timeout; /* delay to re-enable mmaps after interrupt */
     QEMUTimer *mmap_timer; /* allows fast-path resume after IRQ hit */
     QemuMutex intp_mutex; /* protect the intp_list IRQ state */
+    bool irqfd_allowed; /* debug option to force irqfd on/off */
 } VFIOPlatformDevice;
 
 typedef struct VFIOPlatformDeviceClass {
diff --git a/trace-events b/trace-events
index 6060d36..4390381 100644
--- a/trace-events
+++ b/trace-events
@@ -1594,6 +1594,7 @@ vfio_platform_intp_interrupt(int pin, int fd) "Inject IRQ #%d (fd = %d)"
 vfio_platform_intp_inject_pending_lockheld(int pin, int fd) "Inject pending IRQ #%d (fd = %d)"
 vfio_platform_populate_interrupts(int pin, int count, int flags) "- IRQ index %d: count %d, flags=0x%x"
 vfio_intp_interrupt_set_pending(int index) "irq %d is set PENDING"
+vfio_platform_start_irqfd_injection(int index, int fd, int resamplefd) "IRQ index=%d, fd = %d, resamplefd = %d"
 
 #hw/acpi/memory_hotplug.c
 mhp_acpi_invalid_slot_selected(uint32_t slot) "0x%"PRIx32
-- 
1.8.3.2

Re: [Qemu-devel] [RESEND PATCH v16 1/6] hw/arm/sysbus-fdt: enable vfio-calxeda-xgmac dynamic instantiation

[Peter Maydell]

On 15 June 2015 at 17:33, Eric Auger <eric.auger@linaro.org> wrote:
> This patch allows the instantiation of the vfio-calxeda-xgmac device
> from the QEMU command line (-device vfio-calxeda-xgmac,host="<device>").
>
> A specialized device tree node is created for the guest, containing
> compat, dma-coherent, reg and interrupts properties.
>
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>

Minor nits:

> +/* Device Specific Code */
> +
> +/**
> + * add_calxeda_midway_xgmac_fdt_node
> + *
> + * Generates a simple node with following properties:
> + * compatible string, regs, interrupts, dma-coherent
> + */
> +static int add_calxeda_midway_xgmac_fdt_node(SysBusDevice *sbdev, void *opaque)
> +{
> +    PlatformBusFDTData *data = opaque;
> +    PlatformBusDevice *pbus = data->pbus;
> +    void *fdt = data->fdt;
> +    const char *parent_node = data->pbus_node_name;
> +    int compat_str_len, i, ret = -1;
> +    char *nodename;
> +    uint32_t *irq_attr, *reg_attr;
> +    uint64_t mmio_base, irq_number;
> +    VFIOPlatformDevice *vdev = VFIO_PLATFORM_DEVICE(sbdev);
> +    VFIODevice *vbasedev = &vdev->vbasedev;
> +
> +    mmio_base = platform_bus_get_mmio_addr(pbus, sbdev, 0);
> +    nodename = g_strdup_printf("%s/%s@%" PRIx64, parent_node,
> +                               vbasedev->name, mmio_base);
> +    qemu_fdt_add_subnode(fdt, nodename);
> +
> +    compat_str_len = strlen(vdev->compat) + 1;
> +    qemu_fdt_setprop(fdt, nodename, "compatible",
> +                          vdev->compat, compat_str_len);
> +
> +    qemu_fdt_setprop(fdt, nodename, "dma-coherent", "", 0);
> +
> +    reg_attr = g_new(uint32_t, vbasedev->num_regions*2);

You should have spaces around the '*' operator here (and in a
couple of other places below).

> +    for (i = 0; i < vbasedev->num_regions; i++) {
> +        mmio_base = platform_bus_get_mmio_addr(pbus, sbdev, i);
> +        reg_attr[2*i] = cpu_to_be32(mmio_base);
> +        reg_attr[2*i+1] = cpu_to_be32(
> +                                memory_region_size(&vdev->regions[i]->mem));
> +    }
> +    ret = qemu_fdt_setprop(fdt, nodename, "reg", reg_attr,
> +                           vbasedev->num_regions*2*sizeof(uint32_t));
> +    if (ret) {
> +        error_report("could not set reg property of node %s", nodename);
> +        goto fail_reg;
> +    }
> +
> +    irq_attr = g_new(uint32_t, vbasedev->num_irqs*3);
> +    for (i = 0; i < vbasedev->num_irqs; i++) {
> +        irq_number = platform_bus_get_irqn(pbus, sbdev , i)
> +                         + data->irq_start;
> +        irq_attr[3*i] = cpu_to_be32(0);
> +        irq_attr[3*i+1] = cpu_to_be32(irq_number);
> +        irq_attr[3*i+2] = cpu_to_be32(0x4);

In virt.c the magic numbers in cells 0 and 2 have #defines
(GIC_FDT_IRQ_TYPE_SPI and GIC_FDT_IRQ_FLAGS_LEVEL_HI).

(1) maybe we should pull those out into a header so you can use them here
(2) is this device really using a level-triggered interrupt?

> +    }
> +   ret = qemu_fdt_setprop(fdt, nodename, "interrupts",
> +                     irq_attr, vbasedev->num_irqs*3*sizeof(uint32_t));
> +    if (ret) {
> +        error_report("could not set interrupts property of node %s",
> +                     nodename);
> +    }
> +    g_free(irq_attr);
> +fail_reg:
> +    g_free(reg_attr);
> +    g_free(nodename);
> +    return ret;
> +}
> +
>  /* list of supported dynamic sysbus devices */
>  static const NodeCreationPair add_fdt_node_functions[] = {
> +    {TYPE_VFIO_CALXEDA_XGMAC, add_calxeda_midway_xgmac_fdt_node},
>      {"", NULL}, /* last element */
>  };

This file could get big if we have a lot of platform passthrough
devices, but let's hope we don't...

Otherwise:
Acked-by: Peter Maydell <peter.maydell@linaro.org>

-- PMM

Re: [Qemu-devel] [RESEND PATCH v16 1/6] hw/arm/sysbus-fdt: enable vfio-calxeda-xgmac dynamic instantiation

[Eric Auger]

On 06/16/2015 10:29 AM, Peter Maydell wrote:
> On 15 June 2015 at 17:33, Eric Auger <eric.auger@linaro.org> wrote:
>> This patch allows the instantiation of the vfio-calxeda-xgmac device
>> from the QEMU command line (-device vfio-calxeda-xgmac,host="<device>").
>>
>> A specialized device tree node is created for the guest, containing
>> compat, dma-coherent, reg and interrupts properties.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>>
> 
> Minor nits:
> 
>> +/* Device Specific Code */
>> +
>> +/**
>> + * add_calxeda_midway_xgmac_fdt_node
>> + *
>> + * Generates a simple node with following properties:
>> + * compatible string, regs, interrupts, dma-coherent
>> + */
>> +static int add_calxeda_midway_xgmac_fdt_node(SysBusDevice *sbdev, void *opaque)
>> +{
>> +    PlatformBusFDTData *data = opaque;
>> +    PlatformBusDevice *pbus = data->pbus;
>> +    void *fdt = data->fdt;
>> +    const char *parent_node = data->pbus_node_name;
>> +    int compat_str_len, i, ret = -1;
>> +    char *nodename;
>> +    uint32_t *irq_attr, *reg_attr;
>> +    uint64_t mmio_base, irq_number;
>> +    VFIOPlatformDevice *vdev = VFIO_PLATFORM_DEVICE(sbdev);
>> +    VFIODevice *vbasedev = &vdev->vbasedev;
>> +
>> +    mmio_base = platform_bus_get_mmio_addr(pbus, sbdev, 0);
>> +    nodename = g_strdup_printf("%s/%s@%" PRIx64, parent_node,
>> +                               vbasedev->name, mmio_base);
>> +    qemu_fdt_add_subnode(fdt, nodename);
>> +
>> +    compat_str_len = strlen(vdev->compat) + 1;
>> +    qemu_fdt_setprop(fdt, nodename, "compatible",
>> +                          vdev->compat, compat_str_len);
>> +
>> +    qemu_fdt_setprop(fdt, nodename, "dma-coherent", "", 0);
>> +
>> +    reg_attr = g_new(uint32_t, vbasedev->num_regions*2);
> 
> You should have spaces around the '*' operator here (and in a
> couple of other places below).
OK
> 
>> +    for (i = 0; i < vbasedev->num_regions; i++) {
>> +        mmio_base = platform_bus_get_mmio_addr(pbus, sbdev, i);
>> +        reg_attr[2*i] = cpu_to_be32(mmio_base);
>> +        reg_attr[2*i+1] = cpu_to_be32(
>> +                                memory_region_size(&vdev->regions[i]->mem));
>> +    }
>> +    ret = qemu_fdt_setprop(fdt, nodename, "reg", reg_attr,
>> +                           vbasedev->num_regions*2*sizeof(uint32_t));
>> +    if (ret) {
>> +        error_report("could not set reg property of node %s", nodename);
>> +        goto fail_reg;
>> +    }
>> +
>> +    irq_attr = g_new(uint32_t, vbasedev->num_irqs*3);
>> +    for (i = 0; i < vbasedev->num_irqs; i++) {
>> +        irq_number = platform_bus_get_irqn(pbus, sbdev , i)
>> +                         + data->irq_start;
>> +        irq_attr[3*i] = cpu_to_be32(0);
>> +        irq_attr[3*i+1] = cpu_to_be32(irq_number);
>> +        irq_attr[3*i+2] = cpu_to_be32(0x4);
> 
> In virt.c the magic numbers in cells 0 and 2 have #defines
> (GIC_FDT_IRQ_TYPE_SPI and GIC_FDT_IRQ_FLAGS_LEVEL_HI).
> 
> (1) maybe we should pull those out into a header so you can use them here
OK maybe I can put them in newly created include/hw/arm/virt.h
> (2) is this device really using a level-triggered interrupt?
yes it does
> 
>> +    }
>> +   ret = qemu_fdt_setprop(fdt, nodename, "interrupts",
>> +                     irq_attr, vbasedev->num_irqs*3*sizeof(uint32_t));
>> +    if (ret) {
>> +        error_report("could not set interrupts property of node %s",
>> +                     nodename);
>> +    }
>> +    g_free(irq_attr);
>> +fail_reg:
>> +    g_free(reg_attr);
>> +    g_free(nodename);
>> +    return ret;
>> +}
>> +
>>  /* list of supported dynamic sysbus devices */
>>  static const NodeCreationPair add_fdt_node_functions[] = {
>> +    {TYPE_VFIO_CALXEDA_XGMAC, add_calxeda_midway_xgmac_fdt_node},
>>      {"", NULL}, /* last element */
>>  };
> 
> This file could get big if we have a lot of platform passthrough
> devices, but let's hope we don't...
Yes let's see how this feature gets used ;-)
> 
> Otherwise:
> Acked-by: Peter Maydell <peter.maydell@linaro.org>
If you allow me I will resend the patch file separately with the above
modifications.

Thanks!

Best Regards

Eric
> 
> -- PMM
> 

Re: [Qemu-devel] [RESEND PATCH v16 1/6] hw/arm/sysbus-fdt: enable vfio-calxeda-xgmac dynamic instantiation

[Peter Maydell]

On 16 June 2015 at 09:44, Eric Auger <eric.auger@linaro.org> wrote:
> On 06/16/2015 10:29 AM, Peter Maydell wrote:
>> In virt.c the magic numbers in cells 0 and 2 have #defines
>> (GIC_FDT_IRQ_TYPE_SPI and GIC_FDT_IRQ_FLAGS_LEVEL_HI).
>>
>> (1) maybe we should pull those out into a header so you can use them here
> OK maybe I can put them in newly created include/hw/arm/virt.h

They're not really virt-board specific though, a new header
would probably be better; maybe include/hw/arm/fdt.h ?

-- PMM

Re: [Qemu-devel] [RESEND PATCH v16 0/6] KVM platform device passthrough

[Eric Auger]

ping!

Peter acked and pulled the first patch file. This now makes possible to
instantiate the calxeda xgmac VFIO platform device.

Remain the irqfd setup helpers [2-4], the irq routing notifier [5] and
eventually the VFIO code using those pieces [6].

Those patches were already discussed on the ML:
- irqfd setup:
https://lists.gnu.org/archive/html/qemu-devel/2015-04/msg02837.html
- irq routing notifier:
https://lists.nongnu.org/archive/html/qemu-devel/2015-04/msg03019.html.

Irqfd usage is a must have to provide an efficient (platform)
passthrough and it would be a pity to introduce the functionality in 2.4
without the perf I think.

Thank you in advance

Best Regards

Eric



On 06/15/2015 06:33 PM, Eric Auger wrote:
> Now that the platform-bus is instantiated in virt, the ARM sysbus-fdt
> infrastructure, the VFIO platform device and its first Calxeda xgmac
> specialization also are on the master, let's complete the work by
> 
> - enabling the instantiation of the VFIO Calxeda xgmac device
> - enabling irqfd usage, which is vital for performance.
> 
> So the resend only relates to patches 4-9 of the v16 series.
> 
> http://git.linaro.org/people/eric.auger/qemu.git
> (branch vfio_integ_v16_partial_resend)
> 
> Tested by Vikram Sethi on a SATA device featuring 2 MMIO regions.
> Code also exercised successfully by VOSYS for AMBA derivative series.
> 
> Thanks
> 
> Eric
> 
> v15 -> v16:
> - no code change
> - add Vikram's T-b
> - rebase on 0915aed
> - remove linux header update patch file
> 
> v14 -> v15:
> - add Peter R-b on sysbus: add irq_routing_notifier
> - correct g_malloc0_n usage in skeleton
> - correct return values of reset related functions
> - include Cornelia's patch for header update
> 
> v13 -> v14:
> - remove v13 9, 10, 11 patch files and replace them by a single patch file
>   "sysbus: add irq_routing_notifier".
> - in skeleton, fix ENAMETOOLONG sign
> - remove VFIOINTp virtualID in "add irq assignment" patch file
> - removed trace_vfio_platform_start_eventfd
> 
> v12 -> v13:
> - header update but same update was already sent by Cornelia
> - Rework VFIO signaling & irqfd setup: restored 2-step setup featuring
>   eventfd setup on realize and then irqfd setup on irq binding.
> - irqfd setup now uses kvm_irqchip_add_irqfd_notifier and
>   sysbus irq_set_hook override. This leads to the introduction of 6 patch
>   files enabling those 2 features. Paolo advised to introduce
>   kvm_irqchip_add_irqfd_notifier series in the VFIO one. I did the
>   same for irq_set_hook series but if it is better I can submit it aside.
> - above changes made possible to remove
>   x hw/vfio/platform: add capability to start IRQ propagation"
>   x hw/arm/virt: start VFIO IRQ propagation
> - in sysbus-fdt.c, use platform_bus_get_mmio_addr instead of deprecated
>   mmio[0] property. Thanks to Bharat who pointed this issue out. also
>   cpu_to_be32 was used for size and base (Vikram input) .
> - in skeleton misc corrections following Alex 
> 
> v11->v12:
> - add x-mmap property definition, without which the default value of
>   vbasedev.allow_mmap is false, hence preventing the reg space from
>   being mmapped.
> 
> v10->v11:
> - rebase onto v2.3.0-rc0 (mainly related to PCIe support in virt)
> - add dma-coherent property for calxeda midway (fix revealed by removal
>   of kernel-side "vfio: type1: support for ARM SMMUS with VFIO_IOMMU_TYPE1")
> - virt modifications to start VFIO IRQ forwarding are now in a separate
>   patch
> - rearrange linux header exports (those are still partial exports
>   waiting for definitive 4.1-rc0)
> - take into account Alex Bennécomments:
>   - use g_malloc0_n instead of g_malloc0
>   - use block declarations when possible
>   - rework readlink returned value treatment
>   - use g_strlcat in place strncat
>   - re-arrange mutex locking for multiple IRQ support (user-side handled
>     eventfds)
> - use g_snprintf instead of snprintf
> - change the order of functions to avoid pre-declaration in platform.c
> - add flags in VFIOINTp struct to detect whether the IRQ is automasked
> - some comment rewriting
> 
> v9->v10:
> - rebase on "vfio: cleanup vfio_get_device error path, remove
>   vfio_populate_device": vfio_populate_device no more called in
>   vfio_get_device but in vfio_base_device_init
> - update VFIO header according to vfio platform driver v13 (no AMBA)
> 
> v8->v9:
> - rebase on 2.2.0 and machvirt dynamic sysbus instantiation v10
> - v8 1-11 were pulled
> - patch files related to forwarding are moved in a seperate series since
>   it depends on kernel series still in RFC.
> - introduction of basic VFIO platform device split into 3 patch files to
>   ease the review (hope it will help).
> - add an author in platform.c
> - add deallocation in vfio_populate_device error case
> - add patch file doing the VFIO header sync
> - use VFIO_DEVICE_FLAGS_PLATFORM in vfio_populate_device
> - rename calxeda_xgmac.c into calxeda-xgmac.c
> - sysbus-fdt: add_calxeda_midway_xgmac_fdt_node g_free in case of errors
> - reword of linux-headers patch files
> 
> v7->v8:
> - rebase on v2.2.0-rc3 and integrate
>   "Add skip_dump flag to ignore memory region during dump"
> - KVM header evolution with subindex addition in kvm_arch_forwarded_irq
> - split [PATCH v7 03/16] hw/vfio/pci: introduce VFIODevice into 4 patches
> - vfio_compute_needs_reset does not return bool anymore
> - add some comments about exposed MMIO region and IRQ in calxeda xgmac
>   device
> - vfio_[un]mask_irqindex renamed into vfio_[un]mask_single_irqindex
> - rework IRQ startup: former machine init done notifier is replaced by a
>   reset notifier. machine file passes the interrupt controller
>   DeviceState handle (not the platform bus first irq parameter).
> - sysbus-fdt:
>   - move the add_fdt_node_functions array declaration between the device
>     specific code and the generic code to avoid forward declarations of
>     decice specific functions
>   - rename add_basic_vfio_fdt_node into add_calxeda_midway_xgmac_fdt_node
>     emphasizing the fact it is xgmac specific
> 
> v6->v7:
> - fake injection test modality removed
> - VFIO_DEVICE_TYPE_PLATFORM only introduced with VFIO platform
> - new helper functions to start VFIO IRQ on machine init done notifier
>   (introduced in hw/vfio/platform: add vfio-platform support and notifier
>   registration invoked in hw/arm/virt: add support for VFIO devices).
>   vfio_start_irq_injection is replaced by vfio_register_irq_starter.
> 
> v5->v6:
> - rebase on 2.1rc5 PCI code
> - forwarded IRQ first integraton
> - vfio_device property renamed into host property
> - split IRQ setup in different functions that match the 3 supported
>   injection techniques (user handled eventfd, irqfd, forwarded IRQ):
>   removes dynamic switch between injection methods
> - introduce fake interrupts as a test modality:
>   x makes possible to test multiple IRQ user-side handling.
>   x this is a test feature only: enable to trigger a fd as if the
>     real physical IRQ hit. No virtual IRQ is injected into the guest
>     but handling is simulated so that the state machine can be tested
> - user handled eventfd:
>   x add mutex to protect IRQ state & list manipulation,
>   x correct misleading comment in vfio_intp_interrupt.
>   x Fix bugs using fake interrupt modality
> - irqfd no more advertised in this patchset (handled in [3])
> - VFIOPlatformDeviceClass becomes abstract and Calxeda xgmac device
>   and class is re-introduced (as per v4)
> - all DPRINTF removed in platform and replaced by trace-points
> - corrects compilation with configure --disable-kvm
> - simplifies the split for vfio_get_device and introduce a unique
>   specialized function named vfio_populate_device
> - group_list renamed into vfio_group_list
> - hw/arm/dyn_sysbus_devtree.c currently only support vfio-calxeda-xgmac
>   instantiation. Needs to be specialized for other VFIO devices
> - fix 2 bugs in dyn_sysbus_devtree(reg_attr index and compat)
> 
> v4->v5:
> - rebase on v2.1.0 PCI code
> - take into account Alex Williamson comments on PCI code rework
>   - trace updates in vfio_region_write/read
>   - remove fd from VFIORegion
>   - get/put ckeanup
> - bug fix: bar region's vbasedev field duly initialization
> - misc cleanups in platform device
> - device tree node generation removed from device and handled in
>   hw/arm/dyn_sysbus_devtree.c
> - remove "hw/vfio: add an example calxeda_xgmac": with removal of
>   device tree node generation we do not have so many things to
>   implement in that derived device yet. May be re-introduced later
>   on if needed typically for reset/migration.
> - no GSI routing table anymore
> 
> v3->v4 changes (Eric Auger, Alvise Rigo)
> - rebase on last VFIO PCI code (v2.1.0-rc0)
> - full git history rework to ease PCI code change review
> - mv include files in hw/vfio
> - DPRINTF reformatting temporarily moved out
> - support of VFIO virq (removal of resamplefd handler on user-side)
> - integration with sysbus dynamic instantiation framwork
> - removal of unrealize and cleanup routines until it is better
>   understood what is really needed
> - Support of VFIO for Amba devices should be handled in an inherited
>   device to specialize the device tree generation (clock handle currently
>   missing in framework however)
> - "Always use eventfd as notifying mechanism" temporarily moved out
> - static instantiation is not mainstream (although it remains possible)
>   note if static instantiation is used, irqfd must be setup in machine file
>   when virtual IRQ is known
> - create the GSI routing table on qemu side
> 
> v2->v3 changes (Alvise Rigo, Eric Auger):
> - Following Alex W recommandations, further efforts to factorize the
>   code between PCI:introduction of VFIODevice and VFIORegion
>   as base classes
> - unique reset handler for platform and PCI
> - cleanup following Kim's comments
> - multiple IRQ support mechanics should be in place although not
>   tested
> - Better handling of MMIO multiple regions
> - New features and fixes by Alvise (multiple compat string, exec
>   flag, force eventfd usage, amba device tree support)
> - irqfd support
> 
> v1->v2 changes (Kim Phillips, Eric Auger):
> - IRQ initial support (legacy mode where eventfds are handled on
>   user side)
> - hacked dynamic instantiation
> 
> v1 (Kim Phillips):
> - initial split between PCI and platform
> - MMIO support only
> - static instantiation
> 
> Best Regards
> 
> Eric
> 
> 
> 
> 
> Eric Auger (6):
>   hw/arm/sysbus-fdt: enable vfio-calxeda-xgmac dynamic instantiation
>   kvm: rename kvm_irqchip_[add,remove]_irqfd_notifier with gsi suffix
>   kvm-all.c: add qemu_irq/gsi hash table and utility routines
>   intc: arm_gic_kvm: set the qemu_irq/gsi mapping
>   sysbus: add irq_routing_notifier
>   hw/vfio/platform: add irqfd support
> 
>  hw/arm/sysbus-fdt.c             |  72 +++++++++++++++++++++++++++
>  hw/core/sysbus.c                |   6 +++
>  hw/intc/arm_gic_kvm.c           |   6 +++
>  hw/s390x/virtio-ccw.c           |   8 +--
>  hw/vfio/pci.c                   |   6 +--
>  hw/vfio/platform.c              | 107 ++++++++++++++++++++++++++++++++++++++++
>  hw/virtio/virtio-pci.c          |   4 +-
>  include/hw/sysbus.h             |   1 +
>  include/hw/vfio/vfio-platform.h |   2 +
>  include/sysemu/kvm.h            |  11 ++++-
>  kvm-all.c                       |  44 +++++++++++++++--
>  kvm-stub.c                      |   7 +--
>  trace-events                    |   1 +
>  13 files changed, 258 insertions(+), 17 deletions(-)
> 

Re: [Qemu-devel] [RESEND PATCH v16 2/6] kvm: rename kvm_irqchip_[add, remove]_irqfd_notifier with gsi suffix

[Paolo Bonzini]

On 15/06/2015 18:33, Eric Auger wrote:
> Anticipating for the introduction of new add/remove functions taking
> a qemu_irq parameter, let's rename existing ones with a gsi suffix.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> Tested-by: Vikram Sethi <vikrams@codeaurora.org>

Acked-by: Paolo Bonzini <pbonzini@redhat.com>

> ---
> 
> v15 -> v16:
> - added Vikram's T-b
> - resolve rebase conflict in kvm.h
> ---
>  hw/s390x/virtio-ccw.c  | 8 ++++----
>  hw/vfio/pci.c          | 6 +++---
>  hw/virtio/virtio-pci.c | 4 ++--
>  include/sysemu/kvm.h   | 7 ++++---
>  kvm-all.c              | 7 ++++---
>  kvm-stub.c             | 7 ++++---
>  6 files changed, 21 insertions(+), 18 deletions(-)
> 
> diff --git a/hw/s390x/virtio-ccw.c b/hw/s390x/virtio-ccw.c
> index b7a88d6..4a2b0f1 100644
> --- a/hw/s390x/virtio-ccw.c
> +++ b/hw/s390x/virtio-ccw.c
> @@ -1186,8 +1186,8 @@ static int virtio_ccw_add_irqfd(VirtioCcwDevice *dev, int n)
>      VirtQueue *vq = virtio_get_queue(vdev, n);
>      EventNotifier *notifier = virtio_queue_get_guest_notifier(vq);
>  
> -    return kvm_irqchip_add_irqfd_notifier(kvm_state, notifier, NULL,
> -                                          dev->routes.gsi[n]);
> +    return kvm_irqchip_add_irqfd_notifier_gsi(kvm_state, notifier, NULL,
> +                                              dev->routes.gsi[n]);
>  }
>  
>  static void virtio_ccw_remove_irqfd(VirtioCcwDevice *dev, int n)
> @@ -1197,8 +1197,8 @@ static void virtio_ccw_remove_irqfd(VirtioCcwDevice *dev, int n)
>      EventNotifier *notifier = virtio_queue_get_guest_notifier(vq);
>      int ret;
>  
> -    ret = kvm_irqchip_remove_irqfd_notifier(kvm_state, notifier,
> -                                            dev->routes.gsi[n]);
> +    ret = kvm_irqchip_remove_irqfd_notifier_gsi(kvm_state, notifier,
> +                                                dev->routes.gsi[n]);
>      assert(ret == 0);
>  }
>  
> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
> index e0e339a..ac85624 100644
> --- a/hw/vfio/pci.c
> +++ b/hw/vfio/pci.c
> @@ -597,7 +597,7 @@ static void vfio_add_kvm_msi_virq(VFIOMSIVector *vector, MSIMessage *msg,
>          return;
>      }
>  
> -    if (kvm_irqchip_add_irqfd_notifier(kvm_state, &vector->kvm_interrupt,
> +    if (kvm_irqchip_add_irqfd_notifier_gsi(kvm_state, &vector->kvm_interrupt,
>                                         NULL, virq) < 0) {
>          kvm_irqchip_release_virq(kvm_state, virq);
>          event_notifier_cleanup(&vector->kvm_interrupt);
> @@ -609,8 +609,8 @@ static void vfio_add_kvm_msi_virq(VFIOMSIVector *vector, MSIMessage *msg,
>  
>  static void vfio_remove_kvm_msi_virq(VFIOMSIVector *vector)
>  {
> -    kvm_irqchip_remove_irqfd_notifier(kvm_state, &vector->kvm_interrupt,
> -                                      vector->virq);
> +    kvm_irqchip_remove_irqfd_notifier_gsi(kvm_state, &vector->kvm_interrupt,
> +                                          vector->virq);
>      kvm_irqchip_release_virq(kvm_state, vector->virq);
>      vector->virq = -1;
>      event_notifier_cleanup(&vector->kvm_interrupt);
> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> index d7cf34c..1d4435a 100644
> --- a/hw/virtio/virtio-pci.c
> +++ b/hw/virtio/virtio-pci.c
> @@ -495,7 +495,7 @@ static int kvm_virtio_pci_irqfd_use(VirtIOPCIProxy *proxy,
>      VirtQueue *vq = virtio_get_queue(vdev, queue_no);
>      EventNotifier *n = virtio_queue_get_guest_notifier(vq);
>      int ret;
> -    ret = kvm_irqchip_add_irqfd_notifier(kvm_state, n, NULL, irqfd->virq);
> +    ret = kvm_irqchip_add_irqfd_notifier_gsi(kvm_state, n, NULL, irqfd->virq);
>      return ret;
>  }
>  
> @@ -509,7 +509,7 @@ static void kvm_virtio_pci_irqfd_release(VirtIOPCIProxy *proxy,
>      VirtIOIRQFD *irqfd = &proxy->vector_irqfd[vector];
>      int ret;
>  
> -    ret = kvm_irqchip_remove_irqfd_notifier(kvm_state, n, irqfd->virq);
> +    ret = kvm_irqchip_remove_irqfd_notifier_gsi(kvm_state, n, irqfd->virq);
>      assert(ret == 0);
>  }
>  
> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> index f459fbd..acb3025 100644
> --- a/include/sysemu/kvm.h
> +++ b/include/sysemu/kvm.h
> @@ -416,9 +416,10 @@ void kvm_irqchip_release_virq(KVMState *s, int virq);
>  
>  int kvm_irqchip_add_adapter_route(KVMState *s, AdapterInfo *adapter);
>  
> -int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n,
> -                                   EventNotifier *rn, int virq);
> -int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n, int virq);
> +int kvm_irqchip_add_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
> +                                       EventNotifier *rn, int virq);
> +int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
> +                                          int virq);
>  void kvm_pc_gsi_handler(void *opaque, int n, int level);
>  void kvm_pc_setup_irq_routing(bool pci_enabled);
>  void kvm_init_irq_routing(KVMState *s);
> diff --git a/kvm-all.c b/kvm-all.c
> index 53e01d4..730b818 100644
> --- a/kvm-all.c
> +++ b/kvm-all.c
> @@ -1325,14 +1325,15 @@ int kvm_irqchip_update_msi_route(KVMState *s, int virq, MSIMessage msg)
>  }
>  #endif /* !KVM_CAP_IRQ_ROUTING */
>  
> -int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n,
> -                                   EventNotifier *rn, int virq)
> +int kvm_irqchip_add_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
> +                                       EventNotifier *rn, int virq)
>  {
>      return kvm_irqchip_assign_irqfd(s, event_notifier_get_fd(n),
>             rn ? event_notifier_get_fd(rn) : -1, virq, true);
>  }
>  
> -int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n, int virq)
> +int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
> +                                          int virq)
>  {
>      return kvm_irqchip_assign_irqfd(s, event_notifier_get_fd(n), -1, virq,
>             false);
> diff --git a/kvm-stub.c b/kvm-stub.c
> index 7ba90c5..977de21 100644
> --- a/kvm-stub.c
> +++ b/kvm-stub.c
> @@ -137,13 +137,14 @@ int kvm_irqchip_add_adapter_route(KVMState *s, AdapterInfo *adapter)
>      return -ENOSYS;
>  }
>  
> -int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n,
> -                                   EventNotifier *rn, int virq)
> +int kvm_irqchip_add_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
> +                                       EventNotifier *rn, int virq)
>  {
>      return -ENOSYS;
>  }
>  
> -int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n, int virq)
> +int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
> +                                          int virq)
>  {
>      return -ENOSYS;
>  }
> 

Re: [Qemu-devel] [RESEND PATCH v16 3/6] kvm-all.c: add qemu_irq/gsi hash table and utility routines

[Paolo Bonzini]

On 15/06/2015 18:33, Eric Auger wrote:
> VFIO platform device needs to setup irqfd but it does not know the
> gsi corresponding to the device qemu_irq. This series proposes to
> store a hash table in kvm_state using the qemu_irq as key and the gsi
> as a value.
> 
> kvm_irqchip_set_qemuirq_gsi allows to insert such a pair. The interrupt
> controller is supposed to use it.
> 
> kvm_irqchip_[add, remove]_irqfd_notifier allows to setup/tear down
> irqfd directly from the qemu_irq.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> Tested-by: Vikram Sethi <vikrams@codeaurora.org>

Acked-by: Paolo Bonzini <pbonzini@redhat.com>

> ---
> 
> v15 -> v16:
> - Added Vikram's T-b
> 
> v13 -> v14:
> - correct checkpatch warning
> 
> v2 -> v3 (integration into VFIO series v13):
> - rename kvm_irqchip_[add, remove]_qemuirq_irqfd_notifier into
>   kvm_irqchip_[add, remove]_irqfd_notifier. Possible since legacy
>   functions were also renamed with _gsi suffix.
> 
> V1 -> v2:
> - qemu_irq get_gsi callback replaced by hash table stored in kvm
> ---
>  include/sysemu/kvm.h |  6 ++++++
>  kvm-all.c            | 37 +++++++++++++++++++++++++++++++++++++
>  2 files changed, 43 insertions(+)
> 
> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> index acb3025..ba612fc 100644
> --- a/include/sysemu/kvm.h
> +++ b/include/sysemu/kvm.h
> @@ -19,6 +19,7 @@
>  #include "qemu/queue.h"
>  #include "qom/cpu.h"
>  #include "exec/memattrs.h"
> +#include "hw/irq.h"
>  
>  #ifdef CONFIG_KVM
>  #include <linux/kvm.h>
> @@ -420,6 +421,11 @@ int kvm_irqchip_add_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
>                                         EventNotifier *rn, int virq);
>  int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
>                                            int virq);
> +int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n,
> +                                   EventNotifier *rn, qemu_irq irq);
> +int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n,
> +                                      qemu_irq irq);
> +void kvm_irqchip_set_qemuirq_gsi(KVMState *s, qemu_irq irq, int gsi);
>  void kvm_pc_gsi_handler(void *opaque, int n, int level);
>  void kvm_pc_setup_irq_routing(bool pci_enabled);
>  void kvm_init_irq_routing(KVMState *s);
> diff --git a/kvm-all.c b/kvm-all.c
> index 730b818..939bfc4 100644
> --- a/kvm-all.c
> +++ b/kvm-all.c
> @@ -37,6 +37,7 @@
>  #include "exec/address-spaces.h"
>  #include "qemu/event_notifier.h"
>  #include "trace.h"
> +#include "hw/irq.h"
>  
>  #include "hw/boards.h"
>  
> @@ -98,6 +99,7 @@ struct KVMState
>       * unsigned, and treating them as signed here can break things */
>      unsigned irq_set_ioctl;
>      unsigned int sigmask_len;
> +    GHashTable *gsimap;
>  #ifdef KVM_CAP_IRQ_ROUTING
>      struct kvm_irq_routing *irq_routes;
>      int nr_allocated_irq_routes;
> @@ -1339,6 +1341,39 @@ int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
>             false);
>  }
>  
> +int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n,
> +                                   EventNotifier *rn, qemu_irq irq)
> +{
> +    gpointer key, gsi;
> +    gboolean found = g_hash_table_lookup_extended(s->gsimap, irq, &key, &gsi);
> +
> +    if (!found) {
> +        return -ENXIO;
> +    } else {
> +        return kvm_irqchip_add_irqfd_notifier_gsi(s, n, rn,
> +                                                  GPOINTER_TO_UINT(gsi));
> +    }
> +}
> +
> +int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n,
> +                                      qemu_irq irq)
> +{
> +    gpointer key, gsi;
> +    gboolean found = g_hash_table_lookup_extended(s->gsimap, irq, &key, &gsi);
> +
> +    if (!found) {
> +        return -ENXIO;
> +    } else {
> +        return kvm_irqchip_remove_irqfd_notifier_gsi(s, n,
> +                                                     GPOINTER_TO_INT(gsi));
> +    }
> +}
> +
> +void kvm_irqchip_set_qemuirq_gsi(KVMState *s, qemu_irq irq, int gsi)
> +{
> +    g_hash_table_insert(s->gsimap, irq, GINT_TO_POINTER(gsi));
> +}
> +
>  static int kvm_irqchip_create(MachineState *machine, KVMState *s)
>  {
>      int ret;
> @@ -1371,6 +1406,8 @@ static int kvm_irqchip_create(MachineState *machine, KVMState *s)
>  
>      kvm_init_irq_routing(s);
>  
> +    s->gsimap = g_hash_table_new(g_direct_hash, g_direct_equal);
> +
>      return 0;
>  }
>  
> 

Re: [Qemu-devel] [RESEND PATCH v16 4/6] intc: arm_gic_kvm: set the qemu_irq/gsi mapping

[Paolo Bonzini]

On 15/06/2015 18:33, Eric Auger wrote:
> The arm_gic_kvm now calls kvm_irqchip_set_qemuirq_gsi to build
> the hash table storing qemu_irq/gsi mappings. From that point on
> irqfd can be setup directly from the qemu_irq using
> kvm_irqchip_add_irqfd_notifier.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> Tested-by: Vikram Sethi <vikrams@codeaurora.org>

Acked-by: Paolo Bonzini <pbonzini@redhat.com>

> ---
> v15 -> v16:
> - added Vikram's T-b
> - Resolve rebase conflict
> 
> v2 -> v3:
> - kvm_irqchip_add_qemuirq_irqfd_notifier renamed into
>   kvm_irqchip_add_irqfd_notifier
> ---
>  hw/intc/arm_gic_kvm.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/hw/intc/arm_gic_kvm.c b/hw/intc/arm_gic_kvm.c
> index 2cb7d25..f56bff1 100644
> --- a/hw/intc/arm_gic_kvm.c
> +++ b/hw/intc/arm_gic_kvm.c
> @@ -570,6 +570,12 @@ static void kvm_arm_gic_realize(DeviceState *dev, Error **errp)
>       */
>      i += (GIC_INTERNAL * s->num_cpu);
>      qdev_init_gpio_in(dev, kvm_arm_gic_set_irq, i);
> +
> +    for (i = 0; i < s->num_irq - GIC_INTERNAL; i++) {
> +        qemu_irq irq = qdev_get_gpio_in(dev, i);
> +        kvm_irqchip_set_qemuirq_gsi(kvm_state, irq, i);
> +    }
> +
>      /* We never use our outbound IRQ/FIQ lines but provide them so that
>       * we maintain the same interface as the non-KVM GIC.
>       */
> 

Re: [Qemu-devel] [RESEND PATCH v16 0/6] KVM platform device passthrough

[Paolo Bonzini]

On 15/06/2015 18:33, Eric Auger wrote:
> Now that the platform-bus is instantiated in virt, the ARM sysbus-fdt
> infrastructure, the VFIO platform device and its first Calxeda xgmac
> specialization also are on the master, let's complete the work by
> 
> - enabling the instantiation of the VFIO Calxeda xgmac device
> - enabling irqfd usage, which is vital for performance.
> 
> So the resend only relates to patches 4-9 of the v16 series.
> 
> http://git.linaro.org/people/eric.auger/qemu.git
> (branch vfio_integ_v16_partial_resend)
> 
> Tested by Vikram Sethi on a SATA device featuring 2 MMIO regions.
> Code also exercised successfully by VOSYS for AMBA derivative series.

I think I've now acked everything that's needed from me?

Paolo

> Thanks
> 
> Eric
> 
> v15 -> v16:
> - no code change
> - add Vikram's T-b
> - rebase on 0915aed
> - remove linux header update patch file
> 
> v14 -> v15:
> - add Peter R-b on sysbus: add irq_routing_notifier
> - correct g_malloc0_n usage in skeleton
> - correct return values of reset related functions
> - include Cornelia's patch for header update
> 
> v13 -> v14:
> - remove v13 9, 10, 11 patch files and replace them by a single patch file
>   "sysbus: add irq_routing_notifier".
> - in skeleton, fix ENAMETOOLONG sign
> - remove VFIOINTp virtualID in "add irq assignment" patch file
> - removed trace_vfio_platform_start_eventfd
> 
> v12 -> v13:
> - header update but same update was already sent by Cornelia
> - Rework VFIO signaling & irqfd setup: restored 2-step setup featuring
>   eventfd setup on realize and then irqfd setup on irq binding.
> - irqfd setup now uses kvm_irqchip_add_irqfd_notifier and
>   sysbus irq_set_hook override. This leads to the introduction of 6 patch
>   files enabling those 2 features. Paolo advised to introduce
>   kvm_irqchip_add_irqfd_notifier series in the VFIO one. I did the
>   same for irq_set_hook series but if it is better I can submit it aside.
> - above changes made possible to remove
>   x hw/vfio/platform: add capability to start IRQ propagation"
>   x hw/arm/virt: start VFIO IRQ propagation
> - in sysbus-fdt.c, use platform_bus_get_mmio_addr instead of deprecated
>   mmio[0] property. Thanks to Bharat who pointed this issue out. also
>   cpu_to_be32 was used for size and base (Vikram input) .
> - in skeleton misc corrections following Alex 
> 
> v11->v12:
> - add x-mmap property definition, without which the default value of
>   vbasedev.allow_mmap is false, hence preventing the reg space from
>   being mmapped.
> 
> v10->v11:
> - rebase onto v2.3.0-rc0 (mainly related to PCIe support in virt)
> - add dma-coherent property for calxeda midway (fix revealed by removal
>   of kernel-side "vfio: type1: support for ARM SMMUS with VFIO_IOMMU_TYPE1")
> - virt modifications to start VFIO IRQ forwarding are now in a separate
>   patch
> - rearrange linux header exports (those are still partial exports
>   waiting for definitive 4.1-rc0)
> - take into account Alex Bennécomments:
>   - use g_malloc0_n instead of g_malloc0
>   - use block declarations when possible
>   - rework readlink returned value treatment
>   - use g_strlcat in place strncat
>   - re-arrange mutex locking for multiple IRQ support (user-side handled
>     eventfds)
> - use g_snprintf instead of snprintf
> - change the order of functions to avoid pre-declaration in platform.c
> - add flags in VFIOINTp struct to detect whether the IRQ is automasked
> - some comment rewriting
> 
> v9->v10:
> - rebase on "vfio: cleanup vfio_get_device error path, remove
>   vfio_populate_device": vfio_populate_device no more called in
>   vfio_get_device but in vfio_base_device_init
> - update VFIO header according to vfio platform driver v13 (no AMBA)
> 
> v8->v9:
> - rebase on 2.2.0 and machvirt dynamic sysbus instantiation v10
> - v8 1-11 were pulled
> - patch files related to forwarding are moved in a seperate series since
>   it depends on kernel series still in RFC.
> - introduction of basic VFIO platform device split into 3 patch files to
>   ease the review (hope it will help).
> - add an author in platform.c
> - add deallocation in vfio_populate_device error case
> - add patch file doing the VFIO header sync
> - use VFIO_DEVICE_FLAGS_PLATFORM in vfio_populate_device
> - rename calxeda_xgmac.c into calxeda-xgmac.c
> - sysbus-fdt: add_calxeda_midway_xgmac_fdt_node g_free in case of errors
> - reword of linux-headers patch files
> 
> v7->v8:
> - rebase on v2.2.0-rc3 and integrate
>   "Add skip_dump flag to ignore memory region during dump"
> - KVM header evolution with subindex addition in kvm_arch_forwarded_irq
> - split [PATCH v7 03/16] hw/vfio/pci: introduce VFIODevice into 4 patches
> - vfio_compute_needs_reset does not return bool anymore
> - add some comments about exposed MMIO region and IRQ in calxeda xgmac
>   device
> - vfio_[un]mask_irqindex renamed into vfio_[un]mask_single_irqindex
> - rework IRQ startup: former machine init done notifier is replaced by a
>   reset notifier. machine file passes the interrupt controller
>   DeviceState handle (not the platform bus first irq parameter).
> - sysbus-fdt:
>   - move the add_fdt_node_functions array declaration between the device
>     specific code and the generic code to avoid forward declarations of
>     decice specific functions
>   - rename add_basic_vfio_fdt_node into add_calxeda_midway_xgmac_fdt_node
>     emphasizing the fact it is xgmac specific
> 
> v6->v7:
> - fake injection test modality removed
> - VFIO_DEVICE_TYPE_PLATFORM only introduced with VFIO platform
> - new helper functions to start VFIO IRQ on machine init done notifier
>   (introduced in hw/vfio/platform: add vfio-platform support and notifier
>   registration invoked in hw/arm/virt: add support for VFIO devices).
>   vfio_start_irq_injection is replaced by vfio_register_irq_starter.
> 
> v5->v6:
> - rebase on 2.1rc5 PCI code
> - forwarded IRQ first integraton
> - vfio_device property renamed into host property
> - split IRQ setup in different functions that match the 3 supported
>   injection techniques (user handled eventfd, irqfd, forwarded IRQ):
>   removes dynamic switch between injection methods
> - introduce fake interrupts as a test modality:
>   x makes possible to test multiple IRQ user-side handling.
>   x this is a test feature only: enable to trigger a fd as if the
>     real physical IRQ hit. No virtual IRQ is injected into the guest
>     but handling is simulated so that the state machine can be tested
> - user handled eventfd:
>   x add mutex to protect IRQ state & list manipulation,
>   x correct misleading comment in vfio_intp_interrupt.
>   x Fix bugs using fake interrupt modality
> - irqfd no more advertised in this patchset (handled in [3])
> - VFIOPlatformDeviceClass becomes abstract and Calxeda xgmac device
>   and class is re-introduced (as per v4)
> - all DPRINTF removed in platform and replaced by trace-points
> - corrects compilation with configure --disable-kvm
> - simplifies the split for vfio_get_device and introduce a unique
>   specialized function named vfio_populate_device
> - group_list renamed into vfio_group_list
> - hw/arm/dyn_sysbus_devtree.c currently only support vfio-calxeda-xgmac
>   instantiation. Needs to be specialized for other VFIO devices
> - fix 2 bugs in dyn_sysbus_devtree(reg_attr index and compat)
> 
> v4->v5:
> - rebase on v2.1.0 PCI code
> - take into account Alex Williamson comments on PCI code rework
>   - trace updates in vfio_region_write/read
>   - remove fd from VFIORegion
>   - get/put ckeanup
> - bug fix: bar region's vbasedev field duly initialization
> - misc cleanups in platform device
> - device tree node generation removed from device and handled in
>   hw/arm/dyn_sysbus_devtree.c
> - remove "hw/vfio: add an example calxeda_xgmac": with removal of
>   device tree node generation we do not have so many things to
>   implement in that derived device yet. May be re-introduced later
>   on if needed typically for reset/migration.
> - no GSI routing table anymore
> 
> v3->v4 changes (Eric Auger, Alvise Rigo)
> - rebase on last VFIO PCI code (v2.1.0-rc0)
> - full git history rework to ease PCI code change review
> - mv include files in hw/vfio
> - DPRINTF reformatting temporarily moved out
> - support of VFIO virq (removal of resamplefd handler on user-side)
> - integration with sysbus dynamic instantiation framwork
> - removal of unrealize and cleanup routines until it is better
>   understood what is really needed
> - Support of VFIO for Amba devices should be handled in an inherited
>   device to specialize the device tree generation (clock handle currently
>   missing in framework however)
> - "Always use eventfd as notifying mechanism" temporarily moved out
> - static instantiation is not mainstream (although it remains possible)
>   note if static instantiation is used, irqfd must be setup in machine file
>   when virtual IRQ is known
> - create the GSI routing table on qemu side
> 
> v2->v3 changes (Alvise Rigo, Eric Auger):
> - Following Alex W recommandations, further efforts to factorize the
>   code between PCI:introduction of VFIODevice and VFIORegion
>   as base classes
> - unique reset handler for platform and PCI
> - cleanup following Kim's comments
> - multiple IRQ support mechanics should be in place although not
>   tested
> - Better handling of MMIO multiple regions
> - New features and fixes by Alvise (multiple compat string, exec
>   flag, force eventfd usage, amba device tree support)
> - irqfd support
> 
> v1->v2 changes (Kim Phillips, Eric Auger):
> - IRQ initial support (legacy mode where eventfds are handled on
>   user side)
> - hacked dynamic instantiation
> 
> v1 (Kim Phillips):
> - initial split between PCI and platform
> - MMIO support only
> - static instantiation
> 
> Best Regards
> 
> Eric
> 
> 
> 
> 
> Eric Auger (6):
>   hw/arm/sysbus-fdt: enable vfio-calxeda-xgmac dynamic instantiation
>   kvm: rename kvm_irqchip_[add,remove]_irqfd_notifier with gsi suffix
>   kvm-all.c: add qemu_irq/gsi hash table and utility routines
>   intc: arm_gic_kvm: set the qemu_irq/gsi mapping
>   sysbus: add irq_routing_notifier
>   hw/vfio/platform: add irqfd support
> 
>  hw/arm/sysbus-fdt.c             |  72 +++++++++++++++++++++++++++
>  hw/core/sysbus.c                |   6 +++
>  hw/intc/arm_gic_kvm.c           |   6 +++
>  hw/s390x/virtio-ccw.c           |   8 +--
>  hw/vfio/pci.c                   |   6 +--
>  hw/vfio/platform.c              | 107 ++++++++++++++++++++++++++++++++++++++++
>  hw/virtio/virtio-pci.c          |   4 +-
>  include/hw/sysbus.h             |   1 +
>  include/hw/vfio/vfio-platform.h |   2 +
>  include/sysemu/kvm.h            |  11 ++++-
>  kvm-all.c                       |  44 +++++++++++++++--
>  kvm-stub.c                      |   7 +--
>  trace-events                    |   1 +
>  13 files changed, 258 insertions(+), 17 deletions(-)
> 

Re: [Qemu-devel] [RESEND PATCH v16 0/6] KVM platform device passthrough

[Eric Auger]

Hi Paolo
On 06/24/2015 11:55 AM, Paolo Bonzini wrote:
> 
> 
> On 15/06/2015 18:33, Eric Auger wrote:
>> Now that the platform-bus is instantiated in virt, the ARM sysbus-fdt
>> infrastructure, the VFIO platform device and its first Calxeda xgmac
>> specialization also are on the master, let's complete the work by
>>
>> - enabling the instantiation of the VFIO Calxeda xgmac device
>> - enabling irqfd usage, which is vital for performance.
>>
>> So the resend only relates to patches 4-9 of the v16 series.
>>
>> http://git.linaro.org/people/eric.auger/qemu.git
>> (branch vfio_integ_v16_partial_resend)
>>
>> Tested by Vikram Sethi on a SATA device featuring 2 MMIO regions.
>> Code also exercised successfully by VOSYS for AMBA derivative series.
> 
> I think I've now acked everything that's needed from me?
Thanks. what about the
[RESEND PATCH v16 5/6] sysbus: add irq_routing_notifier?

Is that one for Peter C. or anyone else?

Thans

Eric



> 
> Paolo
> 
>> Thanks
>>
>> Eric
>>
>> v15 -> v16:
>> - no code change
>> - add Vikram's T-b
>> - rebase on 0915aed
>> - remove linux header update patch file
>>
>> v14 -> v15:
>> - add Peter R-b on sysbus: add irq_routing_notifier
>> - correct g_malloc0_n usage in skeleton
>> - correct return values of reset related functions
>> - include Cornelia's patch for header update
>>
>> v13 -> v14:
>> - remove v13 9, 10, 11 patch files and replace them by a single patch file
>>   "sysbus: add irq_routing_notifier".
>> - in skeleton, fix ENAMETOOLONG sign
>> - remove VFIOINTp virtualID in "add irq assignment" patch file
>> - removed trace_vfio_platform_start_eventfd
>>
>> v12 -> v13:
>> - header update but same update was already sent by Cornelia
>> - Rework VFIO signaling & irqfd setup: restored 2-step setup featuring
>>   eventfd setup on realize and then irqfd setup on irq binding.
>> - irqfd setup now uses kvm_irqchip_add_irqfd_notifier and
>>   sysbus irq_set_hook override. This leads to the introduction of 6 patch
>>   files enabling those 2 features. Paolo advised to introduce
>>   kvm_irqchip_add_irqfd_notifier series in the VFIO one. I did the
>>   same for irq_set_hook series but if it is better I can submit it aside.
>> - above changes made possible to remove
>>   x hw/vfio/platform: add capability to start IRQ propagation"
>>   x hw/arm/virt: start VFIO IRQ propagation
>> - in sysbus-fdt.c, use platform_bus_get_mmio_addr instead of deprecated
>>   mmio[0] property. Thanks to Bharat who pointed this issue out. also
>>   cpu_to_be32 was used for size and base (Vikram input) .
>> - in skeleton misc corrections following Alex 
>>
>> v11->v12:
>> - add x-mmap property definition, without which the default value of
>>   vbasedev.allow_mmap is false, hence preventing the reg space from
>>   being mmapped.
>>
>> v10->v11:
>> - rebase onto v2.3.0-rc0 (mainly related to PCIe support in virt)
>> - add dma-coherent property for calxeda midway (fix revealed by removal
>>   of kernel-side "vfio: type1: support for ARM SMMUS with VFIO_IOMMU_TYPE1")
>> - virt modifications to start VFIO IRQ forwarding are now in a separate
>>   patch
>> - rearrange linux header exports (those are still partial exports
>>   waiting for definitive 4.1-rc0)
>> - take into account Alex Bennécomments:
>>   - use g_malloc0_n instead of g_malloc0
>>   - use block declarations when possible
>>   - rework readlink returned value treatment
>>   - use g_strlcat in place strncat
>>   - re-arrange mutex locking for multiple IRQ support (user-side handled
>>     eventfds)
>> - use g_snprintf instead of snprintf
>> - change the order of functions to avoid pre-declaration in platform.c
>> - add flags in VFIOINTp struct to detect whether the IRQ is automasked
>> - some comment rewriting
>>
>> v9->v10:
>> - rebase on "vfio: cleanup vfio_get_device error path, remove
>>   vfio_populate_device": vfio_populate_device no more called in
>>   vfio_get_device but in vfio_base_device_init
>> - update VFIO header according to vfio platform driver v13 (no AMBA)
>>
>> v8->v9:
>> - rebase on 2.2.0 and machvirt dynamic sysbus instantiation v10
>> - v8 1-11 were pulled
>> - patch files related to forwarding are moved in a seperate series since
>>   it depends on kernel series still in RFC.
>> - introduction of basic VFIO platform device split into 3 patch files to
>>   ease the review (hope it will help).
>> - add an author in platform.c
>> - add deallocation in vfio_populate_device error case
>> - add patch file doing the VFIO header sync
>> - use VFIO_DEVICE_FLAGS_PLATFORM in vfio_populate_device
>> - rename calxeda_xgmac.c into calxeda-xgmac.c
>> - sysbus-fdt: add_calxeda_midway_xgmac_fdt_node g_free in case of errors
>> - reword of linux-headers patch files
>>
>> v7->v8:
>> - rebase on v2.2.0-rc3 and integrate
>>   "Add skip_dump flag to ignore memory region during dump"
>> - KVM header evolution with subindex addition in kvm_arch_forwarded_irq
>> - split [PATCH v7 03/16] hw/vfio/pci: introduce VFIODevice into 4 patches
>> - vfio_compute_needs_reset does not return bool anymore
>> - add some comments about exposed MMIO region and IRQ in calxeda xgmac
>>   device
>> - vfio_[un]mask_irqindex renamed into vfio_[un]mask_single_irqindex
>> - rework IRQ startup: former machine init done notifier is replaced by a
>>   reset notifier. machine file passes the interrupt controller
>>   DeviceState handle (not the platform bus first irq parameter).
>> - sysbus-fdt:
>>   - move the add_fdt_node_functions array declaration between the device
>>     specific code and the generic code to avoid forward declarations of
>>     decice specific functions
>>   - rename add_basic_vfio_fdt_node into add_calxeda_midway_xgmac_fdt_node
>>     emphasizing the fact it is xgmac specific
>>
>> v6->v7:
>> - fake injection test modality removed
>> - VFIO_DEVICE_TYPE_PLATFORM only introduced with VFIO platform
>> - new helper functions to start VFIO IRQ on machine init done notifier
>>   (introduced in hw/vfio/platform: add vfio-platform support and notifier
>>   registration invoked in hw/arm/virt: add support for VFIO devices).
>>   vfio_start_irq_injection is replaced by vfio_register_irq_starter.
>>
>> v5->v6:
>> - rebase on 2.1rc5 PCI code
>> - forwarded IRQ first integraton
>> - vfio_device property renamed into host property
>> - split IRQ setup in different functions that match the 3 supported
>>   injection techniques (user handled eventfd, irqfd, forwarded IRQ):
>>   removes dynamic switch between injection methods
>> - introduce fake interrupts as a test modality:
>>   x makes possible to test multiple IRQ user-side handling.
>>   x this is a test feature only: enable to trigger a fd as if the
>>     real physical IRQ hit. No virtual IRQ is injected into the guest
>>     but handling is simulated so that the state machine can be tested
>> - user handled eventfd:
>>   x add mutex to protect IRQ state & list manipulation,
>>   x correct misleading comment in vfio_intp_interrupt.
>>   x Fix bugs using fake interrupt modality
>> - irqfd no more advertised in this patchset (handled in [3])
>> - VFIOPlatformDeviceClass becomes abstract and Calxeda xgmac device
>>   and class is re-introduced (as per v4)
>> - all DPRINTF removed in platform and replaced by trace-points
>> - corrects compilation with configure --disable-kvm
>> - simplifies the split for vfio_get_device and introduce a unique
>>   specialized function named vfio_populate_device
>> - group_list renamed into vfio_group_list
>> - hw/arm/dyn_sysbus_devtree.c currently only support vfio-calxeda-xgmac
>>   instantiation. Needs to be specialized for other VFIO devices
>> - fix 2 bugs in dyn_sysbus_devtree(reg_attr index and compat)
>>
>> v4->v5:
>> - rebase on v2.1.0 PCI code
>> - take into account Alex Williamson comments on PCI code rework
>>   - trace updates in vfio_region_write/read
>>   - remove fd from VFIORegion
>>   - get/put ckeanup
>> - bug fix: bar region's vbasedev field duly initialization
>> - misc cleanups in platform device
>> - device tree node generation removed from device and handled in
>>   hw/arm/dyn_sysbus_devtree.c
>> - remove "hw/vfio: add an example calxeda_xgmac": with removal of
>>   device tree node generation we do not have so many things to
>>   implement in that derived device yet. May be re-introduced later
>>   on if needed typically for reset/migration.
>> - no GSI routing table anymore
>>
>> v3->v4 changes (Eric Auger, Alvise Rigo)
>> - rebase on last VFIO PCI code (v2.1.0-rc0)
>> - full git history rework to ease PCI code change review
>> - mv include files in hw/vfio
>> - DPRINTF reformatting temporarily moved out
>> - support of VFIO virq (removal of resamplefd handler on user-side)
>> - integration with sysbus dynamic instantiation framwork
>> - removal of unrealize and cleanup routines until it is better
>>   understood what is really needed
>> - Support of VFIO for Amba devices should be handled in an inherited
>>   device to specialize the device tree generation (clock handle currently
>>   missing in framework however)
>> - "Always use eventfd as notifying mechanism" temporarily moved out
>> - static instantiation is not mainstream (although it remains possible)
>>   note if static instantiation is used, irqfd must be setup in machine file
>>   when virtual IRQ is known
>> - create the GSI routing table on qemu side
>>
>> v2->v3 changes (Alvise Rigo, Eric Auger):
>> - Following Alex W recommandations, further efforts to factorize the
>>   code between PCI:introduction of VFIODevice and VFIORegion
>>   as base classes
>> - unique reset handler for platform and PCI
>> - cleanup following Kim's comments
>> - multiple IRQ support mechanics should be in place although not
>>   tested
>> - Better handling of MMIO multiple regions
>> - New features and fixes by Alvise (multiple compat string, exec
>>   flag, force eventfd usage, amba device tree support)
>> - irqfd support
>>
>> v1->v2 changes (Kim Phillips, Eric Auger):
>> - IRQ initial support (legacy mode where eventfds are handled on
>>   user side)
>> - hacked dynamic instantiation
>>
>> v1 (Kim Phillips):
>> - initial split between PCI and platform
>> - MMIO support only
>> - static instantiation
>>
>> Best Regards
>>
>> Eric
>>
>>
>>
>>
>> Eric Auger (6):
>>   hw/arm/sysbus-fdt: enable vfio-calxeda-xgmac dynamic instantiation
>>   kvm: rename kvm_irqchip_[add,remove]_irqfd_notifier with gsi suffix
>>   kvm-all.c: add qemu_irq/gsi hash table and utility routines
>>   intc: arm_gic_kvm: set the qemu_irq/gsi mapping
>>   sysbus: add irq_routing_notifier
>>   hw/vfio/platform: add irqfd support
>>
>>  hw/arm/sysbus-fdt.c             |  72 +++++++++++++++++++++++++++
>>  hw/core/sysbus.c                |   6 +++
>>  hw/intc/arm_gic_kvm.c           |   6 +++
>>  hw/s390x/virtio-ccw.c           |   8 +--
>>  hw/vfio/pci.c                   |   6 +--
>>  hw/vfio/platform.c              | 107 ++++++++++++++++++++++++++++++++++++++++
>>  hw/virtio/virtio-pci.c          |   4 +-
>>  include/hw/sysbus.h             |   1 +
>>  include/hw/vfio/vfio-platform.h |   2 +
>>  include/sysemu/kvm.h            |  11 ++++-
>>  kvm-all.c                       |  44 +++++++++++++++--
>>  kvm-stub.c                      |   7 +--
>>  trace-events                    |   1 +
>>  13 files changed, 258 insertions(+), 17 deletions(-)
>>

Re: [Qemu-devel] [RESEND PATCH v16 0/6] KVM platform device passthrough

[Paolo Bonzini]

On 24/06/2015 11:59, Eric Auger wrote:
> > I think I've now acked everything that's needed from me?
> 
> Thanks. what about the
> [RESEND PATCH v16 5/6] sysbus: add irq_routing_notifier?
> 
> Is that one for Peter C. or anyone else?

There is no real maintainer, so I think Peter's Reviewed-by is enough.

Paolo

Re: [Qemu-devel] [RESEND PATCH v16 3/6] kvm-all.c: add qemu_irq/gsi hash table and utility routines

[Peter Maydell]

On 15 June 2015 at 17:33, Eric Auger <eric.auger@linaro.org> wrote:
> VFIO platform device needs to setup irqfd but it does not know the
> gsi corresponding to the device qemu_irq. This series proposes to
> store a hash table in kvm_state using the qemu_irq as key and the gsi
> as a value.
>
> kvm_irqchip_set_qemuirq_gsi allows to insert such a pair. The interrupt
> controller is supposed to use it.
>
> kvm_irqchip_[add, remove]_irqfd_notifier allows to setup/tear down
> irqfd directly from the qemu_irq.
>
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> Tested-by: Vikram Sethi <vikrams@codeaurora.org>
>
> ---
>
> v15 -> v16:
> - Added Vikram's T-b
>
> v13 -> v14:
> - correct checkpatch warning
>
> v2 -> v3 (integration into VFIO series v13):
> - rename kvm_irqchip_[add, remove]_qemuirq_irqfd_notifier into
>   kvm_irqchip_[add, remove]_irqfd_notifier. Possible since legacy
>   functions were also renamed with _gsi suffix.
>
> V1 -> v2:
> - qemu_irq get_gsi callback replaced by hash table stored in kvm
> ---
>  include/sysemu/kvm.h |  6 ++++++
>  kvm-all.c            | 37 +++++++++++++++++++++++++++++++++++++
>  2 files changed, 43 insertions(+)
>
> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> index acb3025..ba612fc 100644
> --- a/include/sysemu/kvm.h
> +++ b/include/sysemu/kvm.h
> @@ -19,6 +19,7 @@
>  #include "qemu/queue.h"
>  #include "qom/cpu.h"
>  #include "exec/memattrs.h"
> +#include "hw/irq.h"
>
>  #ifdef CONFIG_KVM
>  #include <linux/kvm.h>
> @@ -420,6 +421,11 @@ int kvm_irqchip_add_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
>                                         EventNotifier *rn, int virq);
>  int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
>                                            int virq);
> +int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n,
> +                                   EventNotifier *rn, qemu_irq irq);
> +int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n,
> +                                      qemu_irq irq);
> +void kvm_irqchip_set_qemuirq_gsi(KVMState *s, qemu_irq irq, int gsi);
>  void kvm_pc_gsi_handler(void *opaque, int n, int level);
>  void kvm_pc_setup_irq_routing(bool pci_enabled);
>  void kvm_init_irq_routing(KVMState *s);
> diff --git a/kvm-all.c b/kvm-all.c
> index 730b818..939bfc4 100644
> --- a/kvm-all.c
> +++ b/kvm-all.c
> @@ -37,6 +37,7 @@
>  #include "exec/address-spaces.h"
>  #include "qemu/event_notifier.h"
>  #include "trace.h"
> +#include "hw/irq.h"
>
>  #include "hw/boards.h"
>
> @@ -98,6 +99,7 @@ struct KVMState
>       * unsigned, and treating them as signed here can break things */
>      unsigned irq_set_ioctl;
>      unsigned int sigmask_len;
> +    GHashTable *gsimap;
>  #ifdef KVM_CAP_IRQ_ROUTING
>      struct kvm_irq_routing *irq_routes;
>      int nr_allocated_irq_routes;
> @@ -1339,6 +1341,39 @@ int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
>             false);
>  }
>
> +int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n,
> +                                   EventNotifier *rn, qemu_irq irq)
> +{
> +    gpointer key, gsi;
> +    gboolean found = g_hash_table_lookup_extended(s->gsimap, irq, &key, &gsi);
> +
> +    if (!found) {
> +        return -ENXIO;
> +    } else {
> +        return kvm_irqchip_add_irqfd_notifier_gsi(s, n, rn,
> +                                                  GPOINTER_TO_UINT(gsi));

Why do we use GPOINTER_TO_UINT() here...


> +    }
> +}
> +
> +int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n,
> +                                      qemu_irq irq)
> +{
> +    gpointer key, gsi;
> +    gboolean found = g_hash_table_lookup_extended(s->gsimap, irq, &key, &gsi);
> +
> +    if (!found) {
> +        return -ENXIO;
> +    } else {
> +        return kvm_irqchip_remove_irqfd_notifier_gsi(s, n,
> +                                                     GPOINTER_TO_INT(gsi));

...but GPOINTER_TO_INT() here?

(Aside: you don't need to put this code in an else {} clause, because
the if() part is returning anyway.)

> +    }
> +}
> +
> +void kvm_irqchip_set_qemuirq_gsi(KVMState *s, qemu_irq irq, int gsi)
> +{
> +    g_hash_table_insert(s->gsimap, irq, GINT_TO_POINTER(gsi));
> +}
> +
>  static int kvm_irqchip_create(MachineState *machine, KVMState *s)
>  {
>      int ret;
> @@ -1371,6 +1406,8 @@ static int kvm_irqchip_create(MachineState *machine, KVMState *s)
>
>      kvm_init_irq_routing(s);
>
> +    s->gsimap = g_hash_table_new(g_direct_hash, g_direct_equal);
> +
>      return 0;
>  }

thanks
-- PMM

Re: [Qemu-devel] [RESEND PATCH v16 4/6] intc: arm_gic_kvm: set the qemu_irq/gsi mapping

[Peter Maydell]

On 15 June 2015 at 17:33, Eric Auger <eric.auger@linaro.org> wrote:
> The arm_gic_kvm now calls kvm_irqchip_set_qemuirq_gsi to build
> the hash table storing qemu_irq/gsi mappings. From that point on
> irqfd can be setup directly from the qemu_irq using
> kvm_irqchip_add_irqfd_notifier.
>
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> Tested-by: Vikram Sethi <vikrams@codeaurora.org>
>
> ---
> v15 -> v16:
> - added Vikram's T-b
> - Resolve rebase conflict
>
> v2 -> v3:
> - kvm_irqchip_add_qemuirq_irqfd_notifier renamed into
>   kvm_irqchip_add_irqfd_notifier
> ---
>  hw/intc/arm_gic_kvm.c | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/hw/intc/arm_gic_kvm.c b/hw/intc/arm_gic_kvm.c
> index 2cb7d25..f56bff1 100644
> --- a/hw/intc/arm_gic_kvm.c
> +++ b/hw/intc/arm_gic_kvm.c
> @@ -570,6 +570,12 @@ static void kvm_arm_gic_realize(DeviceState *dev, Error **errp)
>       */
>      i += (GIC_INTERNAL * s->num_cpu);
>      qdev_init_gpio_in(dev, kvm_arm_gic_set_irq, i);
> +
> +    for (i = 0; i < s->num_irq - GIC_INTERNAL; i++) {
> +        qemu_irq irq = qdev_get_gpio_in(dev, i);
> +        kvm_irqchip_set_qemuirq_gsi(kvm_state, irq, i);
> +    }
> +

Is there documentation anywhere about what a "GSI" number is
for ARM? Is this part of the kernel ABI?

thanks
-- PMM

Re: [Qemu-devel] [RESEND PATCH v16 3/6] kvm-all.c: add qemu_irq/gsi hash table and utility routines

[Eric Auger]

Hi Peter,

On 06/26/2015 01:41 PM, Peter Maydell wrote:
> On 15 June 2015 at 17:33, Eric Auger <eric.auger@linaro.org> wrote:
>> VFIO platform device needs to setup irqfd but it does not know the
>> gsi corresponding to the device qemu_irq. This series proposes to
>> store a hash table in kvm_state using the qemu_irq as key and the gsi
>> as a value.
>>
>> kvm_irqchip_set_qemuirq_gsi allows to insert such a pair. The interrupt
>> controller is supposed to use it.
>>
>> kvm_irqchip_[add, remove]_irqfd_notifier allows to setup/tear down
>> irqfd directly from the qemu_irq.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> Tested-by: Vikram Sethi <vikrams@codeaurora.org>
>>
>> ---
>>
>> v15 -> v16:
>> - Added Vikram's T-b
>>
>> v13 -> v14:
>> - correct checkpatch warning
>>
>> v2 -> v3 (integration into VFIO series v13):
>> - rename kvm_irqchip_[add, remove]_qemuirq_irqfd_notifier into
>>   kvm_irqchip_[add, remove]_irqfd_notifier. Possible since legacy
>>   functions were also renamed with _gsi suffix.
>>
>> V1 -> v2:
>> - qemu_irq get_gsi callback replaced by hash table stored in kvm
>> ---
>>  include/sysemu/kvm.h |  6 ++++++
>>  kvm-all.c            | 37 +++++++++++++++++++++++++++++++++++++
>>  2 files changed, 43 insertions(+)
>>
>> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
>> index acb3025..ba612fc 100644
>> --- a/include/sysemu/kvm.h
>> +++ b/include/sysemu/kvm.h
>> @@ -19,6 +19,7 @@
>>  #include "qemu/queue.h"
>>  #include "qom/cpu.h"
>>  #include "exec/memattrs.h"
>> +#include "hw/irq.h"
>>
>>  #ifdef CONFIG_KVM
>>  #include <linux/kvm.h>
>> @@ -420,6 +421,11 @@ int kvm_irqchip_add_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
>>                                         EventNotifier *rn, int virq);
>>  int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
>>                                            int virq);
>> +int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n,
>> +                                   EventNotifier *rn, qemu_irq irq);
>> +int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n,
>> +                                      qemu_irq irq);
>> +void kvm_irqchip_set_qemuirq_gsi(KVMState *s, qemu_irq irq, int gsi);
>>  void kvm_pc_gsi_handler(void *opaque, int n, int level);
>>  void kvm_pc_setup_irq_routing(bool pci_enabled);
>>  void kvm_init_irq_routing(KVMState *s);
>> diff --git a/kvm-all.c b/kvm-all.c
>> index 730b818..939bfc4 100644
>> --- a/kvm-all.c
>> +++ b/kvm-all.c
>> @@ -37,6 +37,7 @@
>>  #include "exec/address-spaces.h"
>>  #include "qemu/event_notifier.h"
>>  #include "trace.h"
>> +#include "hw/irq.h"
>>
>>  #include "hw/boards.h"
>>
>> @@ -98,6 +99,7 @@ struct KVMState
>>       * unsigned, and treating them as signed here can break things */
>>      unsigned irq_set_ioctl;
>>      unsigned int sigmask_len;
>> +    GHashTable *gsimap;
>>  #ifdef KVM_CAP_IRQ_ROUTING
>>      struct kvm_irq_routing *irq_routes;
>>      int nr_allocated_irq_routes;
>> @@ -1339,6 +1341,39 @@ int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n,
>>             false);
>>  }
>>
>> +int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n,
>> +                                   EventNotifier *rn, qemu_irq irq)
>> +{
>> +    gpointer key, gsi;
>> +    gboolean found = g_hash_table_lookup_extended(s->gsimap, irq, &key, &gsi);
>> +
>> +    if (!found) {
>> +        return -ENXIO;
>> +    } else {
>> +        return kvm_irqchip_add_irqfd_notifier_gsi(s, n, rn,
>> +                                                  GPOINTER_TO_UINT(gsi));
> 
> Why do we use GPOINTER_TO_UINT() here...
Hum yes GPOINTER_TO_INT is in accordance with
kvm_irqchip_add_irqfd_notifier_gsi signature
> 
> 
>> +    }
>> +}
>> +
>> +int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n,
>> +                                      qemu_irq irq)
>> +{
>> +    gpointer key, gsi;
>> +    gboolean found = g_hash_table_lookup_extended(s->gsimap, irq, &key, &gsi);
>> +
>> +    if (!found) {
>> +        return -ENXIO;
>> +    } else {
>> +        return kvm_irqchip_remove_irqfd_notifier_gsi(s, n,
>> +                                                     GPOINTER_TO_INT(gsi));
> 
> ...but GPOINTER_TO_INT() here?

> 
> (Aside: you don't need to put this code in an else {} clause, because
> the if() part is returning anyway.)
Indeed.

Do you wish I re-submit all the VFIO remaining patches as v17 or only
that one?

Thanks

Eric
> 
>> +    }
>> +}
>> +
>> +void kvm_irqchip_set_qemuirq_gsi(KVMState *s, qemu_irq irq, int gsi)
>> +{
>> +    g_hash_table_insert(s->gsimap, irq, GINT_TO_POINTER(gsi));
>> +}
>> +
>>  static int kvm_irqchip_create(MachineState *machine, KVMState *s)
>>  {
>>      int ret;
>> @@ -1371,6 +1406,8 @@ static int kvm_irqchip_create(MachineState *machine, KVMState *s)
>>
>>      kvm_init_irq_routing(s);
>>
>> +    s->gsimap = g_hash_table_new(g_direct_hash, g_direct_equal);
>> +
>>      return 0;
>>  }
> 
> thanks
> -- PMM
> 

Re: [Qemu-devel] [RESEND PATCH v16 6/6] hw/vfio/platform: add irqfd support

[Peter Maydell]

On 15 June 2015 at 17:33, Eric Auger <eric.auger@linaro.org> wrote:
> This patch aims at optimizing IRQ handling using irqfd framework.
>
> Instead of handling the eventfds on user-side they are handled on
> kernel side using
> - the KVM irqfd framework,
> - the VFIO driver virqfd framework.
>
> the virtual IRQ completion is trapped at interrupt controller
> This removes the need for fast/slow path swap.
>
> Overall this brings significant performance improvements.
>
> Signed-off-by: Alvise Rigo <a.rigo@virtualopensystems.com>
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
> Tested-by: Vikram Sethi <vikrams@codeaurora.org>
>
> ---
> v15 -> v16:
> - add Vikram's T-b
>
> v13 -> v14:
> - use connect_irq_notifier
> - remove trace_vfio_platform_start_eventfd
>
> v12 -> v13:
> - setup the new mechanism for starting irqfd, based on
>   LinkPropertySetter override
> - use kvm_irqchip_[add,remove]_irqfd_notifier new functions: no need
>   to bother about gsi (hence virtualID could be removed with small
>   change in trace-events)
>
> v10 -> v11:
> - Add Alex' Reviewed-by
> - introduce kvm_accel in this patch and initialize it
>
> v5 -> v6
> - rely on kvm_irqfds_enabled() and kvm_resamplefds_enabled()
> - guard KVM code with #ifdef CONFIG_KVM
>
> v3 -> v4:
> [Alvise Rigo]
> Use of VFIO Platform driver v6 unmask/virqfd feature and removal
> of resamplefd handler. Physical IRQ unmasking is now done in
> VFIO driver.
>
> v3:
> [Eric Auger]
> initial support with resamplefd handled on QEMU side since the
> unmask was not supported on VFIO platform driver v5.
> ---
>  hw/vfio/platform.c              | 107 ++++++++++++++++++++++++++++++++++++++++
>  include/hw/vfio/vfio-platform.h |   2 +
>  trace-events                    |   1 +
>  3 files changed, 110 insertions(+)
>
> diff --git a/hw/vfio/platform.c b/hw/vfio/platform.c
> index 9382bb7..1d44085 100644
> --- a/hw/vfio/platform.c
> +++ b/hw/vfio/platform.c
> @@ -26,6 +26,7 @@
>  #include "hw/sysbus.h"
>  #include "trace.h"
>  #include "hw/platform-bus.h"
> +#include "sysemu/kvm.h"
>
>  /*
>   * Functions used whatever the injection method
> @@ -51,6 +52,7 @@ static VFIOINTp *vfio_init_intp(VFIODevice *vbasedev,
>      intp->pin = info.index;
>      intp->flags = info.flags;
>      intp->state = VFIO_IRQ_INACTIVE;
> +    intp->kvm_accel = false;
>
>      sysbus_init_irq(sbdev, &intp->qemuirq);
>
> @@ -61,6 +63,13 @@ static VFIOINTp *vfio_init_intp(VFIODevice *vbasedev,
>          error_report("vfio: Error: trigger event_notifier_init failed ");
>          return NULL;
>      }
> +    /* Get an eventfd for resample/unmask */
> +    ret = event_notifier_init(&intp->unmask, 0);
> +    if (ret) {
> +        g_free(intp);
> +        error_report("vfio: Error: resample event_notifier_init failed eoi");

This error message seems rather cryptic; what is it trying to tell the user?

> +        return NULL;
> +    }
>
>      QLIST_INSERT_HEAD(&vdev->intp_list, intp, next);
>      return intp;
> @@ -315,6 +324,95 @@ static int vfio_start_eventfd_injection(VFIOINTp *intp)
>      return ret;
>  }
>
> +/*
> + * Functions used for irqfd
> + */
> +
> +#ifdef CONFIG_KVM
> +
> +/**
> + * vfio_set_resample_eventfd - sets the resamplefd for an IRQ
> + * @intp: the IRQ struct handle
> + * programs the VFIO driver to unmask this IRQ when the
> + * intp->unmask eventfd is triggered
> + */
> +static int vfio_set_resample_eventfd(VFIOINTp *intp)
> +{
> +    VFIODevice *vbasedev = &intp->vdev->vbasedev;
> +    struct vfio_irq_set *irq_set;
> +    int argsz, ret;
> +    int32_t *pfd;
> +
> +    argsz = sizeof(*irq_set) + sizeof(*pfd);
> +    irq_set = g_malloc0(argsz);
> +    irq_set->argsz = argsz;
> +    irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_UNMASK;
> +    irq_set->index = intp->pin;
> +    irq_set->start = 0;
> +    irq_set->count = 1;
> +    pfd = (int32_t *)&irq_set->data;
> +    *pfd = event_notifier_get_fd(&intp->unmask);
> +    qemu_set_fd_handler(*pfd, NULL, NULL, NULL);
> +    ret = ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set);
> +    g_free(irq_set);
> +    if (ret < 0) {
> +        error_report("vfio: Failed to set resample eventfd: %m");
> +    }
> +    return ret;
> +}
> +
> +static void vfio_start_irqfd_injection(SysBusDevice *sbdev, qemu_irq irq)
> +{
> +    VFIOPlatformDevice *vdev = VFIO_PLATFORM_DEVICE(sbdev);
> +    VFIOINTp *intp;
> +    bool found = false;
> +
> +    QLIST_FOREACH(intp, &vdev->intp_list, next) {
> +        if (intp->qemuirq == irq) {
> +            found  = true;

Stray double-space.

> +            break;
> +        }
> +    }
> +    assert(found);
> +
> +    /* Get to a known interrupt state */
> +    qemu_set_fd_handler(event_notifier_get_fd(&intp->interrupt),
> +                        NULL, NULL, vdev);
> +
> +    vfio_mask_single_irqindex(&vdev->vbasedev, intp->pin);
> +    qemu_set_irq(intp->qemuirq, 0);
> +
> +    if (kvm_irqchip_add_irqfd_notifier(kvm_state, &intp->interrupt,
> +                                   &intp->unmask, irq) < 0) {
> +        goto fail_irqfd;
> +    }
> +
> +    if (vfio_set_trigger_eventfd(intp, NULL) < 0) {
> +        goto fail_vfio;
> +    }
> +    if (vfio_set_resample_eventfd(intp) < 0) {
> +        goto fail_vfio;
> +    }
> +
> +    /* Let'em rip */

This comment could probably be phrased more informatively :-)

> +    vfio_unmask_single_irqindex(&vdev->vbasedev, intp->pin);
> +
> +    intp->kvm_accel = true;
> +
> +    trace_vfio_platform_start_irqfd_injection(intp->pin,
> +                                     event_notifier_get_fd(&intp->interrupt),
> +                                     event_notifier_get_fd(&intp->unmask));
> +    return;
> +fail_vfio:
> +    kvm_irqchip_remove_irqfd_notifier(kvm_state, &intp->interrupt, irq);
> +fail_irqfd:
> +    vfio_start_eventfd_injection(intp);
> +    vfio_unmask_single_irqindex(&vdev->vbasedev, intp->pin);
> +    return;
> +}
> +
> +#endif /* CONFIG_KVM */
> +
>  /* VFIO skeleton */
>
>  static void vfio_platform_compute_needs_reset(VFIODevice *vbasedev)
> @@ -548,6 +646,7 @@ static void vfio_platform_realize(DeviceState *dev, Error **errp)
>  {
>      VFIOPlatformDevice *vdev = VFIO_PLATFORM_DEVICE(dev);
>      SysBusDevice *sbdev = SYS_BUS_DEVICE(dev);
> +    SysBusDeviceClass *sbc = SYS_BUS_DEVICE_GET_CLASS(dev);
>      VFIODevice *vbasedev = &vdev->vbasedev;
>      VFIOINTp *intp;
>      int i, ret;
> @@ -555,6 +654,13 @@ static void vfio_platform_realize(DeviceState *dev, Error **errp)
>      vbasedev->type = VFIO_DEVICE_TYPE_PLATFORM;
>      vbasedev->ops = &vfio_platform_ops;
>
> +#ifdef CONFIG_KVM
> +    if (kvm_irqfds_enabled() && kvm_resamplefds_enabled() &&
> +        vdev->irqfd_allowed) {
> +        sbc->connect_irq_notifier = vfio_start_irqfd_injection;
> +    }
> +#endif

You shouldn't need this CONFIG_KVM ifdef -- we deliberately provide
"always false" versions of functions like kvm_irqfds_enabled() in
sysemu/kvm.h so that we can avoid littering code with ifdefs.

It looks like your commit f41389ae3c54b failed to provide the
non-KVM version of kvm_resamplefds_enabled(), but the correct
fix is to provide it...

This implies removing the ifdefs earlier around vfio_start_irqfd_injection,
but that should be OK -- we provide stub versions of functions like
kvm_irqchip_add_irqfd_notifier() so the non-KVM code can still compile.

> +
>      trace_vfio_platform_realize(vbasedev->name, vdev->compat);
>
>      ret = vfio_base_device_init(vbasedev);
> @@ -584,6 +690,7 @@ static Property vfio_platform_dev_properties[] = {
>      DEFINE_PROP_BOOL("x-mmap", VFIOPlatformDevice, vbasedev.allow_mmap, true),
>      DEFINE_PROP_UINT32("mmap-timeout-ms", VFIOPlatformDevice,
>                         mmap_timeout, 1100),
> +    DEFINE_PROP_BOOL("x-irqfd", VFIOPlatformDevice, irqfd_allowed, true),
>      DEFINE_PROP_END_OF_LIST(),
>  };
>
> diff --git a/include/hw/vfio/vfio-platform.h b/include/hw/vfio/vfio-platform.h
> index 26b2ad6..c5cf1d7 100644
> --- a/include/hw/vfio/vfio-platform.h
> +++ b/include/hw/vfio/vfio-platform.h
> @@ -41,6 +41,7 @@ typedef struct VFIOINTp {
>      int state; /* inactive, pending, active */
>      uint8_t pin; /* index */
>      uint32_t flags; /* IRQ info flags */
> +    bool kvm_accel; /* set when QEMU bypass through KVM enabled */
>  } VFIOINTp;
>
>  /* function type for user side eventfd handler */
> @@ -57,6 +58,7 @@ typedef struct VFIOPlatformDevice {
>      uint32_t mmap_timeout; /* delay to re-enable mmaps after interrupt */
>      QEMUTimer *mmap_timer; /* allows fast-path resume after IRQ hit */
>      QemuMutex intp_mutex; /* protect the intp_list IRQ state */
> +    bool irqfd_allowed; /* debug option to force irqfd on/off */
>  } VFIOPlatformDevice;
>
>  typedef struct VFIOPlatformDeviceClass {
> diff --git a/trace-events b/trace-events
> index 6060d36..4390381 100644
> --- a/trace-events
> +++ b/trace-events
> @@ -1594,6 +1594,7 @@ vfio_platform_intp_interrupt(int pin, int fd) "Inject IRQ #%d (fd = %d)"
>  vfio_platform_intp_inject_pending_lockheld(int pin, int fd) "Inject pending IRQ #%d (fd = %d)"
>  vfio_platform_populate_interrupts(int pin, int count, int flags) "- IRQ index %d: count %d, flags=0x%x"
>  vfio_intp_interrupt_set_pending(int index) "irq %d is set PENDING"
> +vfio_platform_start_irqfd_injection(int index, int fd, int resamplefd) "IRQ index=%d, fd = %d, resamplefd = %d"
>
>  #hw/acpi/memory_hotplug.c
>  mhp_acpi_invalid_slot_selected(uint32_t slot) "0x%"PRIx32
> --
> 1.8.3.2
>

thanks
-- PMM

Re: [Qemu-devel] [RESEND PATCH v16 4/6] intc: arm_gic_kvm: set the qemu_irq/gsi mapping

[Eric Auger]

On 06/26/2015 01:43 PM, Peter Maydell wrote:
> On 15 June 2015 at 17:33, Eric Auger <eric.auger@linaro.org> wrote:
>> The arm_gic_kvm now calls kvm_irqchip_set_qemuirq_gsi to build
>> the hash table storing qemu_irq/gsi mappings. From that point on
>> irqfd can be setup directly from the qemu_irq using
>> kvm_irqchip_add_irqfd_notifier.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> Tested-by: Vikram Sethi <vikrams@codeaurora.org>
>>
>> ---
>> v15 -> v16:
>> - added Vikram's T-b
>> - Resolve rebase conflict
>>
>> v2 -> v3:
>> - kvm_irqchip_add_qemuirq_irqfd_notifier renamed into
>>   kvm_irqchip_add_irqfd_notifier
>> ---
>>  hw/intc/arm_gic_kvm.c | 6 ++++++
>>  1 file changed, 6 insertions(+)
>>
>> diff --git a/hw/intc/arm_gic_kvm.c b/hw/intc/arm_gic_kvm.c
>> index 2cb7d25..f56bff1 100644
>> --- a/hw/intc/arm_gic_kvm.c
>> +++ b/hw/intc/arm_gic_kvm.c
>> @@ -570,6 +570,12 @@ static void kvm_arm_gic_realize(DeviceState *dev, Error **errp)
>>       */
>>      i += (GIC_INTERNAL * s->num_cpu);
>>      qdev_init_gpio_in(dev, kvm_arm_gic_set_irq, i);
>> +
>> +    for (i = 0; i < s->num_irq - GIC_INTERNAL; i++) {
>> +        qemu_irq irq = qdev_get_gpio_in(dev, i);
>> +        kvm_irqchip_set_qemuirq_gsi(kvm_state, irq, i);
>> +    }
>> +
> 
> Is there documentation anywhere about what a "GSI" number is
> for ARM? Is this part of the kernel ABI?

Currently this is documented in kernel Documentation/virtual/kvm/api.txt
in the KVM_IRQFD part. With current irqfd/arm implementation this GSI
number matches an SPI index. Now we are also working on GSI routing
support which will extend that meaning. The related RFC also documents
ths gsi in the api.txt

Thanks

Eric
> 
> thanks
> -- PMM
> 

Re: [Qemu-devel] [RESEND PATCH v16 6/6] hw/vfio/platform: add irqfd support

[Eric Auger]

On 06/26/2015 01:57 PM, Peter Maydell wrote:
> On 15 June 2015 at 17:33, Eric Auger <eric.auger@linaro.org> wrote:
>> This patch aims at optimizing IRQ handling using irqfd framework.
>>
>> Instead of handling the eventfds on user-side they are handled on
>> kernel side using
>> - the KVM irqfd framework,
>> - the VFIO driver virqfd framework.
>>
>> the virtual IRQ completion is trapped at interrupt controller
>> This removes the need for fast/slow path swap.
>>
>> Overall this brings significant performance improvements.
>>
>> Signed-off-by: Alvise Rigo <a.rigo@virtualopensystems.com>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
>> Tested-by: Vikram Sethi <vikrams@codeaurora.org>
>>
>> ---
>> v15 -> v16:
>> - add Vikram's T-b
>>
>> v13 -> v14:
>> - use connect_irq_notifier
>> - remove trace_vfio_platform_start_eventfd
>>
>> v12 -> v13:
>> - setup the new mechanism for starting irqfd, based on
>>   LinkPropertySetter override
>> - use kvm_irqchip_[add,remove]_irqfd_notifier new functions: no need
>>   to bother about gsi (hence virtualID could be removed with small
>>   change in trace-events)
>>
>> v10 -> v11:
>> - Add Alex' Reviewed-by
>> - introduce kvm_accel in this patch and initialize it
>>
>> v5 -> v6
>> - rely on kvm_irqfds_enabled() and kvm_resamplefds_enabled()
>> - guard KVM code with #ifdef CONFIG_KVM
>>
>> v3 -> v4:
>> [Alvise Rigo]
>> Use of VFIO Platform driver v6 unmask/virqfd feature and removal
>> of resamplefd handler. Physical IRQ unmasking is now done in
>> VFIO driver.
>>
>> v3:
>> [Eric Auger]
>> initial support with resamplefd handled on QEMU side since the
>> unmask was not supported on VFIO platform driver v5.
>> ---
>>  hw/vfio/platform.c              | 107 ++++++++++++++++++++++++++++++++++++++++
>>  include/hw/vfio/vfio-platform.h |   2 +
>>  trace-events                    |   1 +
>>  3 files changed, 110 insertions(+)
>>
>> diff --git a/hw/vfio/platform.c b/hw/vfio/platform.c
>> index 9382bb7..1d44085 100644
>> --- a/hw/vfio/platform.c
>> +++ b/hw/vfio/platform.c
>> @@ -26,6 +26,7 @@
>>  #include "hw/sysbus.h"
>>  #include "trace.h"
>>  #include "hw/platform-bus.h"
>> +#include "sysemu/kvm.h"
>>
>>  /*
>>   * Functions used whatever the injection method
>> @@ -51,6 +52,7 @@ static VFIOINTp *vfio_init_intp(VFIODevice *vbasedev,
>>      intp->pin = info.index;
>>      intp->flags = info.flags;
>>      intp->state = VFIO_IRQ_INACTIVE;
>> +    intp->kvm_accel = false;
>>
>>      sysbus_init_irq(sbdev, &intp->qemuirq);
>>
>> @@ -61,6 +63,13 @@ static VFIOINTp *vfio_init_intp(VFIODevice *vbasedev,
>>          error_report("vfio: Error: trigger event_notifier_init failed ");
>>          return NULL;
>>      }
>> +    /* Get an eventfd for resample/unmask */
>> +    ret = event_notifier_init(&intp->unmask, 0);
>> +    if (ret) {
>> +        g_free(intp);
>> +        error_report("vfio: Error: resample event_notifier_init failed eoi");
> 
> This error message seems rather cryptic; what is it trying to tell the user?
I acknowledge  ;-)
I will replace by "vfio: Unable to init event notifier for resampling"
> 
>> +        return NULL;
>> +    }
>>
>>      QLIST_INSERT_HEAD(&vdev->intp_list, intp, next);
>>      return intp;
>> @@ -315,6 +324,95 @@ static int vfio_start_eventfd_injection(VFIOINTp *intp)
>>      return ret;
>>  }
>>
>> +/*
>> + * Functions used for irqfd
>> + */
>> +
>> +#ifdef CONFIG_KVM
>> +
>> +/**
>> + * vfio_set_resample_eventfd - sets the resamplefd for an IRQ
>> + * @intp: the IRQ struct handle
>> + * programs the VFIO driver to unmask this IRQ when the
>> + * intp->unmask eventfd is triggered
>> + */
>> +static int vfio_set_resample_eventfd(VFIOINTp *intp)
>> +{
>> +    VFIODevice *vbasedev = &intp->vdev->vbasedev;
>> +    struct vfio_irq_set *irq_set;
>> +    int argsz, ret;
>> +    int32_t *pfd;
>> +
>> +    argsz = sizeof(*irq_set) + sizeof(*pfd);
>> +    irq_set = g_malloc0(argsz);
>> +    irq_set->argsz = argsz;
>> +    irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_UNMASK;
>> +    irq_set->index = intp->pin;
>> +    irq_set->start = 0;
>> +    irq_set->count = 1;
>> +    pfd = (int32_t *)&irq_set->data;
>> +    *pfd = event_notifier_get_fd(&intp->unmask);
>> +    qemu_set_fd_handler(*pfd, NULL, NULL, NULL);
>> +    ret = ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set);
>> +    g_free(irq_set);
>> +    if (ret < 0) {
>> +        error_report("vfio: Failed to set resample eventfd: %m");
>> +    }
>> +    return ret;
>> +}
>> +
>> +static void vfio_start_irqfd_injection(SysBusDevice *sbdev, qemu_irq irq)
>> +{
>> +    VFIOPlatformDevice *vdev = VFIO_PLATFORM_DEVICE(sbdev);
>> +    VFIOINTp *intp;
>> +    bool found = false;
>> +
>> +    QLIST_FOREACH(intp, &vdev->intp_list, next) {
>> +        if (intp->qemuirq == irq) {
>> +            found  = true;
> 
> Stray double-space.
ok
> 
>> +            break;
>> +        }
>> +    }
>> +    assert(found);
>> +
>> +    /* Get to a known interrupt state */
>> +    qemu_set_fd_handler(event_notifier_get_fd(&intp->interrupt),
>> +                        NULL, NULL, vdev);
>> +
>> +    vfio_mask_single_irqindex(&vdev->vbasedev, intp->pin);
>> +    qemu_set_irq(intp->qemuirq, 0);
>> +
>> +    if (kvm_irqchip_add_irqfd_notifier(kvm_state, &intp->interrupt,
>> +                                   &intp->unmask, irq) < 0) {
>> +        goto fail_irqfd;
>> +    }
>> +
>> +    if (vfio_set_trigger_eventfd(intp, NULL) < 0) {
>> +        goto fail_vfio;
>> +    }
>> +    if (vfio_set_resample_eventfd(intp) < 0) {
>> +        goto fail_vfio;
>> +    }
>> +
>> +    /* Let'em rip */
> 
> This comment could probably be phrased more informatively :-)
sure
> 
>> +    vfio_unmask_single_irqindex(&vdev->vbasedev, intp->pin);
>> +
>> +    intp->kvm_accel = true;
>> +
>> +    trace_vfio_platform_start_irqfd_injection(intp->pin,
>> +                                     event_notifier_get_fd(&intp->interrupt),
>> +                                     event_notifier_get_fd(&intp->unmask));
>> +    return;
>> +fail_vfio:
>> +    kvm_irqchip_remove_irqfd_notifier(kvm_state, &intp->interrupt, irq);
>> +fail_irqfd:
>> +    vfio_start_eventfd_injection(intp);
>> +    vfio_unmask_single_irqindex(&vdev->vbasedev, intp->pin);
>> +    return;
>> +}
>> +
>> +#endif /* CONFIG_KVM */
>> +
>>  /* VFIO skeleton */
>>
>>  static void vfio_platform_compute_needs_reset(VFIODevice *vbasedev)
>> @@ -548,6 +646,7 @@ static void vfio_platform_realize(DeviceState *dev, Error **errp)
>>  {
>>      VFIOPlatformDevice *vdev = VFIO_PLATFORM_DEVICE(dev);
>>      SysBusDevice *sbdev = SYS_BUS_DEVICE(dev);
>> +    SysBusDeviceClass *sbc = SYS_BUS_DEVICE_GET_CLASS(dev);
>>      VFIODevice *vbasedev = &vdev->vbasedev;
>>      VFIOINTp *intp;
>>      int i, ret;
>> @@ -555,6 +654,13 @@ static void vfio_platform_realize(DeviceState *dev, Error **errp)
>>      vbasedev->type = VFIO_DEVICE_TYPE_PLATFORM;
>>      vbasedev->ops = &vfio_platform_ops;
>>
>> +#ifdef CONFIG_KVM
>> +    if (kvm_irqfds_enabled() && kvm_resamplefds_enabled() &&
>> +        vdev->irqfd_allowed) {
>> +        sbc->connect_irq_notifier = vfio_start_irqfd_injection;
>> +    }
>> +#endif
> 
> You shouldn't need this CONFIG_KVM ifdef -- we deliberately provide
> "always false" versions of functions like kvm_irqfds_enabled() in
> sysemu/kvm.h so that we can avoid littering code with ifdefs.
> 
> It looks like your commit f41389ae3c54b failed to provide the
> non-KVM version of kvm_resamplefds_enabled(), but the correct
> fix is to provide it...
OK I ignored that. I will add kvm_resamplefds_allowed in kvm-stub.c then.
> 
> This implies removing the ifdefs earlier around vfio_start_irqfd_injection,
> but that should be OK -- we provide stub versions of functions like
> kvm_irqchip_add_irqfd_notifier() so the non-KVM code can still compile.

OK will try this out.

So this answers my previous question. Will send a v17

Thanks

Eric
> 
>> +
>>      trace_vfio_platform_realize(vbasedev->name, vdev->compat);
>>
>>      ret = vfio_base_device_init(vbasedev);
>> @@ -584,6 +690,7 @@ static Property vfio_platform_dev_properties[] = {
>>      DEFINE_PROP_BOOL("x-mmap", VFIOPlatformDevice, vbasedev.allow_mmap, true),
>>      DEFINE_PROP_UINT32("mmap-timeout-ms", VFIOPlatformDevice,
>>                         mmap_timeout, 1100),
>> +    DEFINE_PROP_BOOL("x-irqfd", VFIOPlatformDevice, irqfd_allowed, true),
>>      DEFINE_PROP_END_OF_LIST(),
>>  };
>>
>> diff --git a/include/hw/vfio/vfio-platform.h b/include/hw/vfio/vfio-platform.h
>> index 26b2ad6..c5cf1d7 100644
>> --- a/include/hw/vfio/vfio-platform.h
>> +++ b/include/hw/vfio/vfio-platform.h
>> @@ -41,6 +41,7 @@ typedef struct VFIOINTp {
>>      int state; /* inactive, pending, active */
>>      uint8_t pin; /* index */
>>      uint32_t flags; /* IRQ info flags */
>> +    bool kvm_accel; /* set when QEMU bypass through KVM enabled */
>>  } VFIOINTp;
>>
>>  /* function type for user side eventfd handler */
>> @@ -57,6 +58,7 @@ typedef struct VFIOPlatformDevice {
>>      uint32_t mmap_timeout; /* delay to re-enable mmaps after interrupt */
>>      QEMUTimer *mmap_timer; /* allows fast-path resume after IRQ hit */
>>      QemuMutex intp_mutex; /* protect the intp_list IRQ state */
>> +    bool irqfd_allowed; /* debug option to force irqfd on/off */
>>  } VFIOPlatformDevice;
>>
>>  typedef struct VFIOPlatformDeviceClass {
>> diff --git a/trace-events b/trace-events
>> index 6060d36..4390381 100644
>> --- a/trace-events
>> +++ b/trace-events
>> @@ -1594,6 +1594,7 @@ vfio_platform_intp_interrupt(int pin, int fd) "Inject IRQ #%d (fd = %d)"
>>  vfio_platform_intp_inject_pending_lockheld(int pin, int fd) "Inject pending IRQ #%d (fd = %d)"
>>  vfio_platform_populate_interrupts(int pin, int count, int flags) "- IRQ index %d: count %d, flags=0x%x"
>>  vfio_intp_interrupt_set_pending(int index) "irq %d is set PENDING"
>> +vfio_platform_start_irqfd_injection(int index, int fd, int resamplefd) "IRQ index=%d, fd = %d, resamplefd = %d"
>>
>>  #hw/acpi/memory_hotplug.c
>>  mhp_acpi_invalid_slot_selected(uint32_t slot) "0x%"PRIx32
>> --
>> 1.8.3.2
>>
> 
> thanks
> -- PMM
> 

Re: [Qemu-devel] [RESEND PATCH v16 6/6] hw/vfio/platform: add irqfd support

[Alex Williamson]

On Mon, 2015-06-15 at 17:33 +0100, Eric Auger wrote:
> This patch aims at optimizing IRQ handling using irqfd framework.
> 
> Instead of handling the eventfds on user-side they are handled on
> kernel side using
> - the KVM irqfd framework,
> - the VFIO driver virqfd framework.
> 
> the virtual IRQ completion is trapped at interrupt controller
> This removes the need for fast/slow path swap.
> 
> Overall this brings significant performance improvements.
> 
> Signed-off-by: Alvise Rigo <a.rigo@virtualopensystems.com>
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
> Tested-by: Vikram Sethi <vikrams@codeaurora.org>
> 
> ---
> v15 -> v16:
> - add Vikram's T-b
> 
> v13 -> v14:
> - use connect_irq_notifier
> - remove trace_vfio_platform_start_eventfd
> 
> v12 -> v13:
> - setup the new mechanism for starting irqfd, based on
>   LinkPropertySetter override
> - use kvm_irqchip_[add,remove]_irqfd_notifier new functions: no need
>   to bother about gsi (hence virtualID could be removed with small
>   change in trace-events)
> 
> v10 -> v11:
> - Add Alex' Reviewed-by
> - introduce kvm_accel in this patch and initialize it
> 
> v5 -> v6
> - rely on kvm_irqfds_enabled() and kvm_resamplefds_enabled()
> - guard KVM code with #ifdef CONFIG_KVM
> 
> v3 -> v4:
> [Alvise Rigo]
> Use of VFIO Platform driver v6 unmask/virqfd feature and removal
> of resamplefd handler. Physical IRQ unmasking is now done in
> VFIO driver.
> 
> v3:
> [Eric Auger]
> initial support with resamplefd handled on QEMU side since the
> unmask was not supported on VFIO platform driver v5.
> ---
>  hw/vfio/platform.c              | 107 ++++++++++++++++++++++++++++++++++++++++
>  include/hw/vfio/vfio-platform.h |   2 +
>  trace-events                    |   1 +
>  3 files changed, 110 insertions(+)
> 
> diff --git a/hw/vfio/platform.c b/hw/vfio/platform.c
> index 9382bb7..1d44085 100644
> --- a/hw/vfio/platform.c
> +++ b/hw/vfio/platform.c
> @@ -26,6 +26,7 @@
>  #include "hw/sysbus.h"
>  #include "trace.h"
>  #include "hw/platform-bus.h"
> +#include "sysemu/kvm.h"
>  
>  /*
>   * Functions used whatever the injection method
> @@ -51,6 +52,7 @@ static VFIOINTp *vfio_init_intp(VFIODevice *vbasedev,
>      intp->pin = info.index;
>      intp->flags = info.flags;
>      intp->state = VFIO_IRQ_INACTIVE;
> +    intp->kvm_accel = false;
>  
>      sysbus_init_irq(sbdev, &intp->qemuirq);
>  
> @@ -61,6 +63,13 @@ static VFIOINTp *vfio_init_intp(VFIODevice *vbasedev,
>          error_report("vfio: Error: trigger event_notifier_init failed ");
>          return NULL;
>      }
> +    /* Get an eventfd for resample/unmask */
> +    ret = event_notifier_init(&intp->unmask, 0);
> +    if (ret) {
> +        g_free(intp);
> +        error_report("vfio: Error: resample event_notifier_init failed eoi");
> +        return NULL;
> +    }
>  
>      QLIST_INSERT_HEAD(&vdev->intp_list, intp, next);
>      return intp;
> @@ -315,6 +324,95 @@ static int vfio_start_eventfd_injection(VFIOINTp *intp)
>      return ret;
>  }
>  
> +/*
> + * Functions used for irqfd
> + */
> +
> +#ifdef CONFIG_KVM
> +
> +/**
> + * vfio_set_resample_eventfd - sets the resamplefd for an IRQ
> + * @intp: the IRQ struct handle
> + * programs the VFIO driver to unmask this IRQ when the
> + * intp->unmask eventfd is triggered
> + */
> +static int vfio_set_resample_eventfd(VFIOINTp *intp)
> +{
> +    VFIODevice *vbasedev = &intp->vdev->vbasedev;
> +    struct vfio_irq_set *irq_set;
> +    int argsz, ret;
> +    int32_t *pfd;
> +
> +    argsz = sizeof(*irq_set) + sizeof(*pfd);
> +    irq_set = g_malloc0(argsz);
> +    irq_set->argsz = argsz;
> +    irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_UNMASK;
> +    irq_set->index = intp->pin;
> +    irq_set->start = 0;
> +    irq_set->count = 1;
> +    pfd = (int32_t *)&irq_set->data;
> +    *pfd = event_notifier_get_fd(&intp->unmask);
> +    qemu_set_fd_handler(*pfd, NULL, NULL, NULL);
> +    ret = ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set);
> +    g_free(irq_set);
> +    if (ret < 0) {
> +        error_report("vfio: Failed to set resample eventfd: %m");
> +    }
> +    return ret;
> +}
> +
> +static void vfio_start_irqfd_injection(SysBusDevice *sbdev, qemu_irq irq)
> +{
> +    VFIOPlatformDevice *vdev = VFIO_PLATFORM_DEVICE(sbdev);
> +    VFIOINTp *intp;
> +    bool found = false;
> +
> +    QLIST_FOREACH(intp, &vdev->intp_list, next) {
> +        if (intp->qemuirq == irq) {
> +            found  = true;
> +            break;
> +        }
> +    }
> +    assert(found);

You can simply assert(intp) and get rid of found.

> +
> +    /* Get to a known interrupt state */
> +    qemu_set_fd_handler(event_notifier_get_fd(&intp->interrupt),
> +                        NULL, NULL, vdev);
> +
> +    vfio_mask_single_irqindex(&vdev->vbasedev, intp->pin);
> +    qemu_set_irq(intp->qemuirq, 0);
> +
> +    if (kvm_irqchip_add_irqfd_notifier(kvm_state, &intp->interrupt,
> +                                   &intp->unmask, irq) < 0) {
> +        goto fail_irqfd;
> +    }
> +
> +    if (vfio_set_trigger_eventfd(intp, NULL) < 0) {
> +        goto fail_vfio;
> +    }
> +    if (vfio_set_resample_eventfd(intp) < 0) {
> +        goto fail_vfio;
> +    }
> +
> +    /* Let'em rip */
> +    vfio_unmask_single_irqindex(&vdev->vbasedev, intp->pin);
> +
> +    intp->kvm_accel = true;
> +
> +    trace_vfio_platform_start_irqfd_injection(intp->pin,
> +                                     event_notifier_get_fd(&intp->interrupt),
> +                                     event_notifier_get_fd(&intp->unmask));
> +    return;
> +fail_vfio:
> +    kvm_irqchip_remove_irqfd_notifier(kvm_state, &intp->interrupt, irq);
> +fail_irqfd:
> +    vfio_start_eventfd_injection(intp);
> +    vfio_unmask_single_irqindex(&vdev->vbasedev, intp->pin);
> +    return;
> +}
> +
> +#endif /* CONFIG_KVM */
> +
>  /* VFIO skeleton */
>  
>  static void vfio_platform_compute_needs_reset(VFIODevice *vbasedev)
> @@ -548,6 +646,7 @@ static void vfio_platform_realize(DeviceState *dev, Error **errp)
>  {
>      VFIOPlatformDevice *vdev = VFIO_PLATFORM_DEVICE(dev);
>      SysBusDevice *sbdev = SYS_BUS_DEVICE(dev);
> +    SysBusDeviceClass *sbc = SYS_BUS_DEVICE_GET_CLASS(dev);
>      VFIODevice *vbasedev = &vdev->vbasedev;
>      VFIOINTp *intp;
>      int i, ret;
> @@ -555,6 +654,13 @@ static void vfio_platform_realize(DeviceState *dev, Error **errp)
>      vbasedev->type = VFIO_DEVICE_TYPE_PLATFORM;
>      vbasedev->ops = &vfio_platform_ops;
>  
> +#ifdef CONFIG_KVM
> +    if (kvm_irqfds_enabled() && kvm_resamplefds_enabled() &&
> +        vdev->irqfd_allowed) {
> +        sbc->connect_irq_notifier = vfio_start_irqfd_injection;

Should we be abstracting this to a
sysbus_register_connect_irq_notifier()?  It seems a littler personal to
be reaching in and setting it ourselves and would avoid us needing to
reference the class.

> +    }
> +#endif
> +
>      trace_vfio_platform_realize(vbasedev->name, vdev->compat);
>  
>      ret = vfio_base_device_init(vbasedev);
> @@ -584,6 +690,7 @@ static Property vfio_platform_dev_properties[] = {
>      DEFINE_PROP_BOOL("x-mmap", VFIOPlatformDevice, vbasedev.allow_mmap, true),
>      DEFINE_PROP_UINT32("mmap-timeout-ms", VFIOPlatformDevice,
>                         mmap_timeout, 1100),
> +    DEFINE_PROP_BOOL("x-irqfd", VFIOPlatformDevice, irqfd_allowed, true),
>      DEFINE_PROP_END_OF_LIST(),
>  };
>  
> diff --git a/include/hw/vfio/vfio-platform.h b/include/hw/vfio/vfio-platform.h
> index 26b2ad6..c5cf1d7 100644
> --- a/include/hw/vfio/vfio-platform.h
> +++ b/include/hw/vfio/vfio-platform.h
> @@ -41,6 +41,7 @@ typedef struct VFIOINTp {
>      int state; /* inactive, pending, active */
>      uint8_t pin; /* index */
>      uint32_t flags; /* IRQ info flags */
> +    bool kvm_accel; /* set when QEMU bypass through KVM enabled */
>  } VFIOINTp;
>  
>  /* function type for user side eventfd handler */
> @@ -57,6 +58,7 @@ typedef struct VFIOPlatformDevice {
>      uint32_t mmap_timeout; /* delay to re-enable mmaps after interrupt */
>      QEMUTimer *mmap_timer; /* allows fast-path resume after IRQ hit */
>      QemuMutex intp_mutex; /* protect the intp_list IRQ state */
> +    bool irqfd_allowed; /* debug option to force irqfd on/off */
>  } VFIOPlatformDevice;
>  
>  typedef struct VFIOPlatformDeviceClass {
> diff --git a/trace-events b/trace-events
> index 6060d36..4390381 100644
> --- a/trace-events
> +++ b/trace-events
> @@ -1594,6 +1594,7 @@ vfio_platform_intp_interrupt(int pin, int fd) "Inject IRQ #%d (fd = %d)"
>  vfio_platform_intp_inject_pending_lockheld(int pin, int fd) "Inject pending IRQ #%d (fd = %d)"
>  vfio_platform_populate_interrupts(int pin, int count, int flags) "- IRQ index %d: count %d, flags=0x%x"
>  vfio_intp_interrupt_set_pending(int index) "irq %d is set PENDING"
> +vfio_platform_start_irqfd_injection(int index, int fd, int resamplefd) "IRQ index=%d, fd = %d, resamplefd = %d"
>  
>  #hw/acpi/memory_hotplug.c
>  mhp_acpi_invalid_slot_selected(uint32_t slot) "0x%"PRIx32

Re: [Qemu-devel] [RESEND PATCH v16 6/6] hw/vfio/platform: add irqfd support

[Paolo Bonzini]

On 26/06/2015 18:21, Alex Williamson wrote:
>> > +#ifdef CONFIG_KVM
>> > +    if (kvm_irqfds_enabled() && kvm_resamplefds_enabled() &&
>> > +        vdev->irqfd_allowed) {
>> > +        sbc->connect_irq_notifier = vfio_start_irqfd_injection;
> Should we be abstracting this to a
> sysbus_register_connect_irq_notifier()?  It seems a littler personal to
> be reaching in and setting it ourselves and would avoid us needing to
> reference the class.

It's your class, so it's not too bad to touch it from that point of
view; on the other hand it's ugly to do it here nevertheless. :)

I think you should always set "sbc->connect_irq_notifier =
vfio_start_irqfd_injection" in the class_init function.  The
vfio_start_irqfd_injection function can just exit if it finds
"!kvm_enabled() || !kvm_irqfds_enabled() || !kvm_resamplefds_enabled()
|| !vdev->irqfd_allowed".

Paolo

Re: [Qemu-devel] [RESEND PATCH v16 6/6] hw/vfio/platform: add irqfd support

[Eric Auger]

On 06/26/2015 06:26 PM, Paolo Bonzini wrote:
> 
> 
> On 26/06/2015 18:21, Alex Williamson wrote:
>>>> +#ifdef CONFIG_KVM
>>>> +    if (kvm_irqfds_enabled() && kvm_resamplefds_enabled() &&
>>>> +        vdev->irqfd_allowed) {
>>>> +        sbc->connect_irq_notifier = vfio_start_irqfd_injection;
>> Should we be abstracting this to a
>> sysbus_register_connect_irq_notifier()?  It seems a littler personal to
>> be reaching in and setting it ourselves and would avoid us needing to
>> reference the class.
> 
> It's your class, so it's not too bad to touch it from that point of
> view; on the other hand it's ugly to do it here nevertheless. :)
> 
> I think you should always set "sbc->connect_irq_notifier =
> vfio_start_irqfd_injection" in the class_init function.  The
> vfio_start_irqfd_injection function can just exit if it finds
> "!kvm_enabled() || !kvm_irqfds_enabled() || !kvm_resamplefds_enabled()
> || !vdev->irqfd_allowed".
OK thanks for the guidance. Alex, are you OK with that solution. It
avoids touching the other patch

Thanks

Eric
> 
> Paolo
> 

Re: [Qemu-devel] [RESEND PATCH v16 6/6] hw/vfio/platform: add irqfd support

[Alex Williamson]

On Fri, 2015-06-26 at 18:31 +0200, Eric Auger wrote:
> On 06/26/2015 06:26 PM, Paolo Bonzini wrote:
> > 
> > 
> > On 26/06/2015 18:21, Alex Williamson wrote:
> >>>> +#ifdef CONFIG_KVM
> >>>> +    if (kvm_irqfds_enabled() && kvm_resamplefds_enabled() &&
> >>>> +        vdev->irqfd_allowed) {
> >>>> +        sbc->connect_irq_notifier = vfio_start_irqfd_injection;
> >> Should we be abstracting this to a
> >> sysbus_register_connect_irq_notifier()?  It seems a littler personal to
> >> be reaching in and setting it ourselves and would avoid us needing to
> >> reference the class.
> > 
> > It's your class, so it's not too bad to touch it from that point of
> > view; on the other hand it's ugly to do it here nevertheless. :)
> > 
> > I think you should always set "sbc->connect_irq_notifier =
> > vfio_start_irqfd_injection" in the class_init function.  The
> > vfio_start_irqfd_injection function can just exit if it finds
> > "!kvm_enabled() || !kvm_irqfds_enabled() || !kvm_resamplefds_enabled()
> > || !vdev->irqfd_allowed".
> OK thanks for the guidance. Alex, are you OK with that solution. It
> avoids touching the other patch

Yeah, I'm ok with that, it's less awkward from the class_init.  Thanks,

Alex