[Qemu-devel] [PATCH v2 0/2] spapr: introduce a new sPAPRIrq backend

[Qemu-devel] [PATCH v2 0/2] spapr: introduce a new sPAPRIrq backend

[Cédric Le Goater]

Hello,

This series adds a new sPAPRIrq backend increasing the number of
available IRQ numbers in pseries-3.1 machines. This change is an
opportunity to also fix the "ibm,pe-total-#msi" and remove the old
XICS_IRQS_SPAPR definition.

Thanks,

C.

Changes since v1:

 - merge patch 1 and 3


Cédric Le Goater (2):
  spapr: introduce a spapr_irq class 'nr_msis' attribute
  spapr: increase the size of the IRQ number space

 include/hw/ppc/spapr_irq.h |  2 ++
 include/hw/ppc/xics.h      |  2 --
 hw/ppc/spapr.c             |  1 +
 hw/ppc/spapr_irq.c         | 22 ++++++++++++++++++++--
 hw/ppc/spapr_pci.c         |  5 +++--
 5 files changed, 26 insertions(+), 6 deletions(-)

-- 
2.17.1

[Qemu-devel] [PATCH v2 1/2] spapr: introduce a spapr_irq class 'nr_msis' attribute

[Cédric Le Goater]

The number of MSI interrupts a sPAPR machine can allocate is in direct
relation with the number of interrupts of the sPAPRIrq backend. Define
statically this value at the sPAPRIrq class level and use it for the
"ibm,pe-total-#msi" property of the sPAPR PHB.

According to the PAPR specs, "ibm,pe-total-#msi" defines the maximum
number of MSIs that are available to the PE. We choose to advertise
the maximum number of MSIs that are available to the machine for
simplicity of the model and to avoid segmenting the MSI interrupt pool
which can be easily shared. If the pool limit is reached, it can be
extended dynamically.

Finally, remove XICS_IRQS_SPAPR which is now unused.

Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 include/hw/ppc/spapr_irq.h | 1 +
 include/hw/ppc/xics.h      | 2 --
 hw/ppc/spapr_irq.c         | 9 +++++++--
 hw/ppc/spapr_pci.c         | 5 +++--
 4 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/include/hw/ppc/spapr_irq.h b/include/hw/ppc/spapr_irq.h
index 0e98c4474bb2..650f810ad2aa 100644
--- a/include/hw/ppc/spapr_irq.h
+++ b/include/hw/ppc/spapr_irq.h
@@ -31,6 +31,7 @@ void spapr_irq_msi_reset(sPAPRMachineState *spapr);
 
 typedef struct sPAPRIrq {
     uint32_t    nr_irqs;
+    uint32_t    nr_msis;
 
     void (*init)(sPAPRMachineState *spapr, Error **errp);
     int (*claim)(sPAPRMachineState *spapr, int irq, bool lsi, Error **errp);
diff --git a/include/hw/ppc/xics.h b/include/hw/ppc/xics.h
index 9c2916c9b23a..9958443d1984 100644
--- a/include/hw/ppc/xics.h
+++ b/include/hw/ppc/xics.h
@@ -181,8 +181,6 @@ typedef struct XICSFabricClass {
     ICPState *(*icp_get)(XICSFabric *xi, int server);
 } XICSFabricClass;
 
-#define XICS_IRQS_SPAPR               1024
-
 void spapr_dt_xics(int nr_servers, void *fdt, uint32_t phandle);
 
 ICPState *xics_icp_get(XICSFabric *xi, int server);
diff --git a/hw/ppc/spapr_irq.c b/hw/ppc/spapr_irq.c
index 0cbb5dd39368..fe8be5f5217a 100644
--- a/hw/ppc/spapr_irq.c
+++ b/hw/ppc/spapr_irq.c
@@ -99,7 +99,7 @@ static void spapr_irq_init_xics(sPAPRMachineState *spapr, Error **errp)
 
     /* Initialize the MSI IRQ allocator. */
     if (!SPAPR_MACHINE_GET_CLASS(spapr)->legacy_irq_allocation) {
-        spapr_irq_msi_init(spapr, XICS_IRQ_BASE + nr_irqs - SPAPR_IRQ_MSI);
+        spapr_irq_msi_init(spapr, smc->irq->nr_msis);
     }
 
     if (kvm_enabled()) {
@@ -195,8 +195,13 @@ static void spapr_irq_print_info_xics(sPAPRMachineState *spapr, Monitor *mon)
     ics_pic_print_info(spapr->ics, mon);
 }
 
+#define SPAPR_IRQ_XICS_NR_IRQS     0x400
+#define SPAPR_IRQ_XICS_NR_MSIS     \
+    (XICS_IRQ_BASE + SPAPR_IRQ_XICS_NR_IRQS - SPAPR_IRQ_MSI)
+
 sPAPRIrq spapr_irq_xics = {
-    .nr_irqs     = XICS_IRQS_SPAPR,
+    .nr_irqs     = SPAPR_IRQ_XICS_NR_IRQS,
+    .nr_msis     = SPAPR_IRQ_XICS_NR_MSIS,
 
     .init        = spapr_irq_init_xics,
     .claim       = spapr_irq_claim_xics,
diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
index 6bcb4f419b6b..bb736177e76c 100644
--- a/hw/ppc/spapr_pci.c
+++ b/hw/ppc/spapr_pci.c
@@ -2121,6 +2121,7 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb,
     sPAPRTCETable *tcet;
     PCIBus *bus = PCI_HOST_BRIDGE(phb)->bus;
     sPAPRFDT s_fdt;
+    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
 
     /* Start populating the FDT */
     nodename = g_strdup_printf("pci@%" PRIx64, phb->buid);
@@ -2138,8 +2139,8 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb,
     _FDT(fdt_setprop(fdt, bus_off, "ranges", &ranges, sizeof_ranges));
     _FDT(fdt_setprop(fdt, bus_off, "reg", &bus_reg, sizeof(bus_reg)));
     _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pci-config-space-type", 0x1));
-    /* TODO: fine tune the total count of allocatable MSIs per PHB */
-    _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pe-total-#msi", XICS_IRQS_SPAPR));
+    _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pe-total-#msi",
+                          smc->irq->nr_msis));
 
     /* Dynamic DMA window */
     if (phb->ddw_enabled) {
-- 
2.17.1

[Qemu-devel] [PATCH v2 2/2] spapr: increase the size of the IRQ number space

[Cédric Le Goater]

The new layout using static IRQ number does not leave much space to
the dynamic MSI range, only 0x100 IRQ numbers. Increase the total
number of IRQS for newer machines and introduce a legacy XICS backend
for pre-3.1 machines to maintain compatibility.

For the old backend, provide a 'nr_msis' value covering the full IRQ
number space as it does not use the bitmap allocator to allocate MSI
interrupt numbers.

Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 include/hw/ppc/spapr_irq.h |  1 +
 hw/ppc/spapr.c             |  1 +
 hw/ppc/spapr_irq.c         | 15 ++++++++++++++-
 3 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/include/hw/ppc/spapr_irq.h b/include/hw/ppc/spapr_irq.h
index 650f810ad2aa..a467ce696ee4 100644
--- a/include/hw/ppc/spapr_irq.h
+++ b/include/hw/ppc/spapr_irq.h
@@ -41,6 +41,7 @@ typedef struct sPAPRIrq {
 } sPAPRIrq;
 
 extern sPAPRIrq spapr_irq_xics;
+extern sPAPRIrq spapr_irq_xics_legacy;
 
 int spapr_irq_claim(sPAPRMachineState *spapr, int irq, bool lsi, Error **errp);
 void spapr_irq_free(sPAPRMachineState *spapr, int irq, int num);
diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 4a9dd4d9bc14..eba7d60a30a7 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -3971,6 +3971,7 @@ static void spapr_machine_3_0_class_options(MachineClass *mc)
     SET_MACHINE_COMPAT(mc, SPAPR_COMPAT_3_0);
 
     smc->legacy_irq_allocation = true;
+    smc->irq = &spapr_irq_xics_legacy;
 }
 
 DEFINE_SPAPR_MACHINE(3_0, "3.0", false);
diff --git a/hw/ppc/spapr_irq.c b/hw/ppc/spapr_irq.c
index fe8be5f5217a..e77b94cc685e 100644
--- a/hw/ppc/spapr_irq.c
+++ b/hw/ppc/spapr_irq.c
@@ -195,7 +195,7 @@ static void spapr_irq_print_info_xics(sPAPRMachineState *spapr, Monitor *mon)
     ics_pic_print_info(spapr->ics, mon);
 }
 
-#define SPAPR_IRQ_XICS_NR_IRQS     0x400
+#define SPAPR_IRQ_XICS_NR_IRQS     0x1000
 #define SPAPR_IRQ_XICS_NR_MSIS     \
     (XICS_IRQ_BASE + SPAPR_IRQ_XICS_NR_IRQS - SPAPR_IRQ_MSI)
 
@@ -289,3 +289,16 @@ int spapr_irq_find(sPAPRMachineState *spapr, int num, bool align, Error **errp)
 
     return first + ics->offset;
 }
+
+#define SPAPR_IRQ_XICS_LEGACY_NR_IRQS     0x400
+
+sPAPRIrq spapr_irq_xics_legacy = {
+    .nr_irqs     = SPAPR_IRQ_XICS_LEGACY_NR_IRQS,
+    .nr_msis     = SPAPR_IRQ_XICS_LEGACY_NR_IRQS,
+
+    .init        = spapr_irq_init_xics,
+    .claim       = spapr_irq_claim_xics,
+    .free        = spapr_irq_free_xics,
+    .qirq        = spapr_qirq_xics,
+    .print_info  = spapr_irq_print_info_xics,
+};
-- 
2.17.1

Re: [Qemu-devel] [PATCH v2 1/2] spapr: introduce a spapr_irq class 'nr_msis' attribute

[Greg Kurz]

On Tue, 11 Sep 2018 07:55:02 +0200
Cédric Le Goater <clg@kaod.org> wrote:

> The number of MSI interrupts a sPAPR machine can allocate is in direct
> relation with the number of interrupts of the sPAPRIrq backend. Define
> statically this value at the sPAPRIrq class level and use it for the
> "ibm,pe-total-#msi" property of the sPAPR PHB.
> 
> According to the PAPR specs, "ibm,pe-total-#msi" defines the maximum
> number of MSIs that are available to the PE. We choose to advertise
> the maximum number of MSIs that are available to the machine for
> simplicity of the model and to avoid segmenting the MSI interrupt pool
> which can be easily shared. If the pool limit is reached, it can be
> extended dynamically.
> 
> Finally, remove XICS_IRQS_SPAPR which is now unused.
> 
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---

Looks good to me. Just one comment below.

>  include/hw/ppc/spapr_irq.h | 1 +
>  include/hw/ppc/xics.h      | 2 --
>  hw/ppc/spapr_irq.c         | 9 +++++++--
>  hw/ppc/spapr_pci.c         | 5 +++--
>  4 files changed, 11 insertions(+), 6 deletions(-)
> 
> diff --git a/include/hw/ppc/spapr_irq.h b/include/hw/ppc/spapr_irq.h
> index 0e98c4474bb2..650f810ad2aa 100644
> --- a/include/hw/ppc/spapr_irq.h
> +++ b/include/hw/ppc/spapr_irq.h
> @@ -31,6 +31,7 @@ void spapr_irq_msi_reset(sPAPRMachineState *spapr);
>  
>  typedef struct sPAPRIrq {
>      uint32_t    nr_irqs;
> +    uint32_t    nr_msis;
>  
>      void (*init)(sPAPRMachineState *spapr, Error **errp);
>      int (*claim)(sPAPRMachineState *spapr, int irq, bool lsi, Error **errp);
> diff --git a/include/hw/ppc/xics.h b/include/hw/ppc/xics.h
> index 9c2916c9b23a..9958443d1984 100644
> --- a/include/hw/ppc/xics.h
> +++ b/include/hw/ppc/xics.h
> @@ -181,8 +181,6 @@ typedef struct XICSFabricClass {
>      ICPState *(*icp_get)(XICSFabric *xi, int server);
>  } XICSFabricClass;
>  
> -#define XICS_IRQS_SPAPR               1024
> -
>  void spapr_dt_xics(int nr_servers, void *fdt, uint32_t phandle);
>  
>  ICPState *xics_icp_get(XICSFabric *xi, int server);
> diff --git a/hw/ppc/spapr_irq.c b/hw/ppc/spapr_irq.c
> index 0cbb5dd39368..fe8be5f5217a 100644
> --- a/hw/ppc/spapr_irq.c
> +++ b/hw/ppc/spapr_irq.c
> @@ -99,7 +99,7 @@ static void spapr_irq_init_xics(sPAPRMachineState *spapr, Error **errp)
>  
>      /* Initialize the MSI IRQ allocator. */
>      if (!SPAPR_MACHINE_GET_CLASS(spapr)->legacy_irq_allocation) {
> -        spapr_irq_msi_init(spapr, XICS_IRQ_BASE + nr_irqs - SPAPR_IRQ_MSI);
> +        spapr_irq_msi_init(spapr, smc->irq->nr_msis);
>      }
>  
>      if (kvm_enabled()) {
> @@ -195,8 +195,13 @@ static void spapr_irq_print_info_xics(sPAPRMachineState *spapr, Monitor *mon)
>      ics_pic_print_info(spapr->ics, mon);
>  }
>  
> +#define SPAPR_IRQ_XICS_NR_IRQS     0x400
> +#define SPAPR_IRQ_XICS_NR_MSIS     \
> +    (XICS_IRQ_BASE + SPAPR_IRQ_XICS_NR_IRQS - SPAPR_IRQ_MSI)
> +
>  sPAPRIrq spapr_irq_xics = {
> -    .nr_irqs     = XICS_IRQS_SPAPR,
> +    .nr_irqs     = SPAPR_IRQ_XICS_NR_IRQS,
> +    .nr_msis     = SPAPR_IRQ_XICS_NR_MSIS,
>  
>      .init        = spapr_irq_init_xics,
>      .claim       = spapr_irq_claim_xics,
> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
> index 6bcb4f419b6b..bb736177e76c 100644
> --- a/hw/ppc/spapr_pci.c
> +++ b/hw/ppc/spapr_pci.c
> @@ -2121,6 +2121,7 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb,
>      sPAPRTCETable *tcet;
>      PCIBus *bus = PCI_HOST_BRIDGE(phb)->bus;
>      sPAPRFDT s_fdt;
> +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
>  

It is a bit unfortunate to add another user of qdev_get_machine()...

>      /* Start populating the FDT */
>      nodename = g_strdup_printf("pci@%" PRIx64, phb->buid);
> @@ -2138,8 +2139,8 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb,
>      _FDT(fdt_setprop(fdt, bus_off, "ranges", &ranges, sizeof_ranges));
>      _FDT(fdt_setprop(fdt, bus_off, "reg", &bus_reg, sizeof(bus_reg)));
>      _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pci-config-space-type", 0x1));
> -    /* TODO: fine tune the total count of allocatable MSIs per PHB */
> -    _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pe-total-#msi", XICS_IRQS_SPAPR));
> +    _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pe-total-#msi",
> +                          smc->irq->nr_msis));
>  

... and to expose machine class internals. Since spapr_populate_pci_dt() is only
called from the core machine code, maybe have the caller to pass the number of
MSIs ?

Anyway, this can be done in a followup patch so:

Reviewed-by: Greg Kurz <groug@kaod.org>

>      /* Dynamic DMA window */
>      if (phb->ddw_enabled) {

Re: [Qemu-devel] [PATCH v2 2/2] spapr: increase the size of the IRQ number space

[Greg Kurz]

On Tue, 11 Sep 2018 07:55:03 +0200
Cédric Le Goater <clg@kaod.org> wrote:

> The new layout using static IRQ number does not leave much space to
> the dynamic MSI range, only 0x100 IRQ numbers. Increase the total
> number of IRQS for newer machines and introduce a legacy XICS backend
> for pre-3.1 machines to maintain compatibility.
> 
> For the old backend, provide a 'nr_msis' value covering the full IRQ
> number space as it does not use the bitmap allocator to allocate MSI
> interrupt numbers.
> 
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---

Reviewed-by: Greg Kurz <groug@kaod.org>

>  include/hw/ppc/spapr_irq.h |  1 +
>  hw/ppc/spapr.c             |  1 +
>  hw/ppc/spapr_irq.c         | 15 ++++++++++++++-
>  3 files changed, 16 insertions(+), 1 deletion(-)
> 
> diff --git a/include/hw/ppc/spapr_irq.h b/include/hw/ppc/spapr_irq.h
> index 650f810ad2aa..a467ce696ee4 100644
> --- a/include/hw/ppc/spapr_irq.h
> +++ b/include/hw/ppc/spapr_irq.h
> @@ -41,6 +41,7 @@ typedef struct sPAPRIrq {
>  } sPAPRIrq;
>  
>  extern sPAPRIrq spapr_irq_xics;
> +extern sPAPRIrq spapr_irq_xics_legacy;
>  
>  int spapr_irq_claim(sPAPRMachineState *spapr, int irq, bool lsi, Error **errp);
>  void spapr_irq_free(sPAPRMachineState *spapr, int irq, int num);
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index 4a9dd4d9bc14..eba7d60a30a7 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -3971,6 +3971,7 @@ static void spapr_machine_3_0_class_options(MachineClass *mc)
>      SET_MACHINE_COMPAT(mc, SPAPR_COMPAT_3_0);
>  
>      smc->legacy_irq_allocation = true;
> +    smc->irq = &spapr_irq_xics_legacy;
>  }
>  
>  DEFINE_SPAPR_MACHINE(3_0, "3.0", false);
> diff --git a/hw/ppc/spapr_irq.c b/hw/ppc/spapr_irq.c
> index fe8be5f5217a..e77b94cc685e 100644
> --- a/hw/ppc/spapr_irq.c
> +++ b/hw/ppc/spapr_irq.c
> @@ -195,7 +195,7 @@ static void spapr_irq_print_info_xics(sPAPRMachineState *spapr, Monitor *mon)
>      ics_pic_print_info(spapr->ics, mon);
>  }
>  
> -#define SPAPR_IRQ_XICS_NR_IRQS     0x400
> +#define SPAPR_IRQ_XICS_NR_IRQS     0x1000
>  #define SPAPR_IRQ_XICS_NR_MSIS     \
>      (XICS_IRQ_BASE + SPAPR_IRQ_XICS_NR_IRQS - SPAPR_IRQ_MSI)
>  
> @@ -289,3 +289,16 @@ int spapr_irq_find(sPAPRMachineState *spapr, int num, bool align, Error **errp)
>  
>      return first + ics->offset;
>  }
> +
> +#define SPAPR_IRQ_XICS_LEGACY_NR_IRQS     0x400
> +
> +sPAPRIrq spapr_irq_xics_legacy = {
> +    .nr_irqs     = SPAPR_IRQ_XICS_LEGACY_NR_IRQS,
> +    .nr_msis     = SPAPR_IRQ_XICS_LEGACY_NR_IRQS,
> +
> +    .init        = spapr_irq_init_xics,
> +    .claim       = spapr_irq_claim_xics,
> +    .free        = spapr_irq_free_xics,
> +    .qirq        = spapr_qirq_xics,
> +    .print_info  = spapr_irq_print_info_xics,
> +};

Re: [Qemu-devel] [PATCH v2 1/2] spapr: introduce a spapr_irq class 'nr_msis' attribute

[Cédric Le Goater]

On 09/11/2018 09:34 AM, Greg Kurz wrote:
> On Tue, 11 Sep 2018 07:55:02 +0200
> Cédric Le Goater <clg@kaod.org> wrote:
> 
>> The number of MSI interrupts a sPAPR machine can allocate is in direct
>> relation with the number of interrupts of the sPAPRIrq backend. Define
>> statically this value at the sPAPRIrq class level and use it for the
>> "ibm,pe-total-#msi" property of the sPAPR PHB.
>>
>> According to the PAPR specs, "ibm,pe-total-#msi" defines the maximum
>> number of MSIs that are available to the PE. We choose to advertise
>> the maximum number of MSIs that are available to the machine for
>> simplicity of the model and to avoid segmenting the MSI interrupt pool
>> which can be easily shared. If the pool limit is reached, it can be
>> extended dynamically.
>>
>> Finally, remove XICS_IRQS_SPAPR which is now unused.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
> 
> Looks good to me. Just one comment below.
> 
>>  include/hw/ppc/spapr_irq.h | 1 +
>>  include/hw/ppc/xics.h      | 2 --
>>  hw/ppc/spapr_irq.c         | 9 +++++++--
>>  hw/ppc/spapr_pci.c         | 5 +++--
>>  4 files changed, 11 insertions(+), 6 deletions(-)
>>
>> diff --git a/include/hw/ppc/spapr_irq.h b/include/hw/ppc/spapr_irq.h
>> index 0e98c4474bb2..650f810ad2aa 100644
>> --- a/include/hw/ppc/spapr_irq.h
>> +++ b/include/hw/ppc/spapr_irq.h
>> @@ -31,6 +31,7 @@ void spapr_irq_msi_reset(sPAPRMachineState *spapr);
>>  
>>  typedef struct sPAPRIrq {
>>      uint32_t    nr_irqs;
>> +    uint32_t    nr_msis;
>>  
>>      void (*init)(sPAPRMachineState *spapr, Error **errp);
>>      int (*claim)(sPAPRMachineState *spapr, int irq, bool lsi, Error **errp);
>> diff --git a/include/hw/ppc/xics.h b/include/hw/ppc/xics.h
>> index 9c2916c9b23a..9958443d1984 100644
>> --- a/include/hw/ppc/xics.h
>> +++ b/include/hw/ppc/xics.h
>> @@ -181,8 +181,6 @@ typedef struct XICSFabricClass {
>>      ICPState *(*icp_get)(XICSFabric *xi, int server);
>>  } XICSFabricClass;
>>  
>> -#define XICS_IRQS_SPAPR               1024
>> -
>>  void spapr_dt_xics(int nr_servers, void *fdt, uint32_t phandle);
>>  
>>  ICPState *xics_icp_get(XICSFabric *xi, int server);
>> diff --git a/hw/ppc/spapr_irq.c b/hw/ppc/spapr_irq.c
>> index 0cbb5dd39368..fe8be5f5217a 100644
>> --- a/hw/ppc/spapr_irq.c
>> +++ b/hw/ppc/spapr_irq.c
>> @@ -99,7 +99,7 @@ static void spapr_irq_init_xics(sPAPRMachineState *spapr, Error **errp)
>>  
>>      /* Initialize the MSI IRQ allocator. */
>>      if (!SPAPR_MACHINE_GET_CLASS(spapr)->legacy_irq_allocation) {
>> -        spapr_irq_msi_init(spapr, XICS_IRQ_BASE + nr_irqs - SPAPR_IRQ_MSI);
>> +        spapr_irq_msi_init(spapr, smc->irq->nr_msis);
>>      }
>>  
>>      if (kvm_enabled()) {
>> @@ -195,8 +195,13 @@ static void spapr_irq_print_info_xics(sPAPRMachineState *spapr, Monitor *mon)
>>      ics_pic_print_info(spapr->ics, mon);
>>  }
>>  
>> +#define SPAPR_IRQ_XICS_NR_IRQS     0x400
>> +#define SPAPR_IRQ_XICS_NR_MSIS     \
>> +    (XICS_IRQ_BASE + SPAPR_IRQ_XICS_NR_IRQS - SPAPR_IRQ_MSI)
>> +
>>  sPAPRIrq spapr_irq_xics = {
>> -    .nr_irqs     = XICS_IRQS_SPAPR,
>> +    .nr_irqs     = SPAPR_IRQ_XICS_NR_IRQS,
>> +    .nr_msis     = SPAPR_IRQ_XICS_NR_MSIS,
>>  
>>      .init        = spapr_irq_init_xics,
>>      .claim       = spapr_irq_claim_xics,
>> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
>> index 6bcb4f419b6b..bb736177e76c 100644
>> --- a/hw/ppc/spapr_pci.c
>> +++ b/hw/ppc/spapr_pci.c
>> @@ -2121,6 +2121,7 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb,
>>      sPAPRTCETable *tcet;
>>      PCIBus *bus = PCI_HOST_BRIDGE(phb)->bus;
>>      sPAPRFDT s_fdt;
>> +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
>>  
> 
> It is a bit unfortunate to add another user of qdev_get_machine()...
> 
>>      /* Start populating the FDT */
>>      nodename = g_strdup_printf("pci@%" PRIx64, phb->buid);
>> @@ -2138,8 +2139,8 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb,
>>      _FDT(fdt_setprop(fdt, bus_off, "ranges", &ranges, sizeof_ranges));
>>      _FDT(fdt_setprop(fdt, bus_off, "reg", &bus_reg, sizeof(bus_reg)));
>>      _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pci-config-space-type", 0x1));
>> -    /* TODO: fine tune the total count of allocatable MSIs per PHB */
>> -    _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pe-total-#msi", XICS_IRQS_SPAPR));
>> +    _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pe-total-#msi",
>> +                          smc->irq->nr_msis));
>>  
> 
> ... and to expose machine class internals. Since spapr_populate_pci_dt() is only
> called from the core machine code, maybe have the caller to pass the number of
> MSIs ?

yes we could add an extra parameter to spapr_populate_pci_dt()

> Anyway, this can be done in a followup patch so:
> 
> Reviewed-by: Greg Kurz <groug@kaod.org>

Thanks,

C.
> 
>>      /* Dynamic DMA window */
>>      if (phb->ddw_enabled) {
> 

Re: [Qemu-devel] [PATCH v2 1/2] spapr: introduce a spapr_irq class 'nr_msis' attribute

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 4572 bytes --]

On Tue, Sep 11, 2018 at 07:55:02AM +0200, Cédric Le Goater wrote:
> The number of MSI interrupts a sPAPR machine can allocate is in direct
> relation with the number of interrupts of the sPAPRIrq backend. Define
> statically this value at the sPAPRIrq class level and use it for the
> "ibm,pe-total-#msi" property of the sPAPR PHB.
> 
> According to the PAPR specs, "ibm,pe-total-#msi" defines the maximum
> number of MSIs that are available to the PE. We choose to advertise
> the maximum number of MSIs that are available to the machine for
> simplicity of the model and to avoid segmenting the MSI interrupt pool
> which can be easily shared. If the pool limit is reached, it can be
> extended dynamically.
> 
> Finally, remove XICS_IRQS_SPAPR which is now unused.
> 
> Signed-off-by: Cédric Le Goater <clg@kaod.org>

Applied to ppc-for-3.1, thanks.

> ---
>  include/hw/ppc/spapr_irq.h | 1 +
>  include/hw/ppc/xics.h      | 2 --
>  hw/ppc/spapr_irq.c         | 9 +++++++--
>  hw/ppc/spapr_pci.c         | 5 +++--
>  4 files changed, 11 insertions(+), 6 deletions(-)
> 
> diff --git a/include/hw/ppc/spapr_irq.h b/include/hw/ppc/spapr_irq.h
> index 0e98c4474bb2..650f810ad2aa 100644
> --- a/include/hw/ppc/spapr_irq.h
> +++ b/include/hw/ppc/spapr_irq.h
> @@ -31,6 +31,7 @@ void spapr_irq_msi_reset(sPAPRMachineState *spapr);
>  
>  typedef struct sPAPRIrq {
>      uint32_t    nr_irqs;
> +    uint32_t    nr_msis;
>  
>      void (*init)(sPAPRMachineState *spapr, Error **errp);
>      int (*claim)(sPAPRMachineState *spapr, int irq, bool lsi, Error **errp);
> diff --git a/include/hw/ppc/xics.h b/include/hw/ppc/xics.h
> index 9c2916c9b23a..9958443d1984 100644
> --- a/include/hw/ppc/xics.h
> +++ b/include/hw/ppc/xics.h
> @@ -181,8 +181,6 @@ typedef struct XICSFabricClass {
>      ICPState *(*icp_get)(XICSFabric *xi, int server);
>  } XICSFabricClass;
>  
> -#define XICS_IRQS_SPAPR               1024
> -
>  void spapr_dt_xics(int nr_servers, void *fdt, uint32_t phandle);
>  
>  ICPState *xics_icp_get(XICSFabric *xi, int server);
> diff --git a/hw/ppc/spapr_irq.c b/hw/ppc/spapr_irq.c
> index 0cbb5dd39368..fe8be5f5217a 100644
> --- a/hw/ppc/spapr_irq.c
> +++ b/hw/ppc/spapr_irq.c
> @@ -99,7 +99,7 @@ static void spapr_irq_init_xics(sPAPRMachineState *spapr, Error **errp)
>  
>      /* Initialize the MSI IRQ allocator. */
>      if (!SPAPR_MACHINE_GET_CLASS(spapr)->legacy_irq_allocation) {
> -        spapr_irq_msi_init(spapr, XICS_IRQ_BASE + nr_irqs - SPAPR_IRQ_MSI);
> +        spapr_irq_msi_init(spapr, smc->irq->nr_msis);
>      }
>  
>      if (kvm_enabled()) {
> @@ -195,8 +195,13 @@ static void spapr_irq_print_info_xics(sPAPRMachineState *spapr, Monitor *mon)
>      ics_pic_print_info(spapr->ics, mon);
>  }
>  
> +#define SPAPR_IRQ_XICS_NR_IRQS     0x400
> +#define SPAPR_IRQ_XICS_NR_MSIS     \
> +    (XICS_IRQ_BASE + SPAPR_IRQ_XICS_NR_IRQS - SPAPR_IRQ_MSI)
> +
>  sPAPRIrq spapr_irq_xics = {
> -    .nr_irqs     = XICS_IRQS_SPAPR,
> +    .nr_irqs     = SPAPR_IRQ_XICS_NR_IRQS,
> +    .nr_msis     = SPAPR_IRQ_XICS_NR_MSIS,
>  
>      .init        = spapr_irq_init_xics,
>      .claim       = spapr_irq_claim_xics,
> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
> index 6bcb4f419b6b..bb736177e76c 100644
> --- a/hw/ppc/spapr_pci.c
> +++ b/hw/ppc/spapr_pci.c
> @@ -2121,6 +2121,7 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb,
>      sPAPRTCETable *tcet;
>      PCIBus *bus = PCI_HOST_BRIDGE(phb)->bus;
>      sPAPRFDT s_fdt;
> +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
>  
>      /* Start populating the FDT */
>      nodename = g_strdup_printf("pci@%" PRIx64, phb->buid);
> @@ -2138,8 +2139,8 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb,
>      _FDT(fdt_setprop(fdt, bus_off, "ranges", &ranges, sizeof_ranges));
>      _FDT(fdt_setprop(fdt, bus_off, "reg", &bus_reg, sizeof(bus_reg)));
>      _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pci-config-space-type", 0x1));
> -    /* TODO: fine tune the total count of allocatable MSIs per PHB */
> -    _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pe-total-#msi", XICS_IRQS_SPAPR));
> +    _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pe-total-#msi",
> +                          smc->irq->nr_msis));
>  
>      /* Dynamic DMA window */
>      if (phb->ddw_enabled) {

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] [PATCH v2 1/2] spapr: introduce a spapr_irq class 'nr_msis' attribute

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 5551 bytes --]

On Tue, Sep 11, 2018 at 09:56:06AM +0200, Cédric Le Goater wrote:
> On 09/11/2018 09:34 AM, Greg Kurz wrote:
> > On Tue, 11 Sep 2018 07:55:02 +0200
> > Cédric Le Goater <clg@kaod.org> wrote:
> > 
> >> The number of MSI interrupts a sPAPR machine can allocate is in direct
> >> relation with the number of interrupts of the sPAPRIrq backend. Define
> >> statically this value at the sPAPRIrq class level and use it for the
> >> "ibm,pe-total-#msi" property of the sPAPR PHB.
> >>
> >> According to the PAPR specs, "ibm,pe-total-#msi" defines the maximum
> >> number of MSIs that are available to the PE. We choose to advertise
> >> the maximum number of MSIs that are available to the machine for
> >> simplicity of the model and to avoid segmenting the MSI interrupt pool
> >> which can be easily shared. If the pool limit is reached, it can be
> >> extended dynamically.
> >>
> >> Finally, remove XICS_IRQS_SPAPR which is now unused.
> >>
> >> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> >> ---
> > 
> > Looks good to me. Just one comment below.
> > 
> >>  include/hw/ppc/spapr_irq.h | 1 +
> >>  include/hw/ppc/xics.h      | 2 --
> >>  hw/ppc/spapr_irq.c         | 9 +++++++--
> >>  hw/ppc/spapr_pci.c         | 5 +++--
> >>  4 files changed, 11 insertions(+), 6 deletions(-)
> >>
> >> diff --git a/include/hw/ppc/spapr_irq.h b/include/hw/ppc/spapr_irq.h
> >> index 0e98c4474bb2..650f810ad2aa 100644
> >> --- a/include/hw/ppc/spapr_irq.h
> >> +++ b/include/hw/ppc/spapr_irq.h
> >> @@ -31,6 +31,7 @@ void spapr_irq_msi_reset(sPAPRMachineState *spapr);
> >>  
> >>  typedef struct sPAPRIrq {
> >>      uint32_t    nr_irqs;
> >> +    uint32_t    nr_msis;
> >>  
> >>      void (*init)(sPAPRMachineState *spapr, Error **errp);
> >>      int (*claim)(sPAPRMachineState *spapr, int irq, bool lsi, Error **errp);
> >> diff --git a/include/hw/ppc/xics.h b/include/hw/ppc/xics.h
> >> index 9c2916c9b23a..9958443d1984 100644
> >> --- a/include/hw/ppc/xics.h
> >> +++ b/include/hw/ppc/xics.h
> >> @@ -181,8 +181,6 @@ typedef struct XICSFabricClass {
> >>      ICPState *(*icp_get)(XICSFabric *xi, int server);
> >>  } XICSFabricClass;
> >>  
> >> -#define XICS_IRQS_SPAPR               1024
> >> -
> >>  void spapr_dt_xics(int nr_servers, void *fdt, uint32_t phandle);
> >>  
> >>  ICPState *xics_icp_get(XICSFabric *xi, int server);
> >> diff --git a/hw/ppc/spapr_irq.c b/hw/ppc/spapr_irq.c
> >> index 0cbb5dd39368..fe8be5f5217a 100644
> >> --- a/hw/ppc/spapr_irq.c
> >> +++ b/hw/ppc/spapr_irq.c
> >> @@ -99,7 +99,7 @@ static void spapr_irq_init_xics(sPAPRMachineState *spapr, Error **errp)
> >>  
> >>      /* Initialize the MSI IRQ allocator. */
> >>      if (!SPAPR_MACHINE_GET_CLASS(spapr)->legacy_irq_allocation) {
> >> -        spapr_irq_msi_init(spapr, XICS_IRQ_BASE + nr_irqs - SPAPR_IRQ_MSI);
> >> +        spapr_irq_msi_init(spapr, smc->irq->nr_msis);
> >>      }
> >>  
> >>      if (kvm_enabled()) {
> >> @@ -195,8 +195,13 @@ static void spapr_irq_print_info_xics(sPAPRMachineState *spapr, Monitor *mon)
> >>      ics_pic_print_info(spapr->ics, mon);
> >>  }
> >>  
> >> +#define SPAPR_IRQ_XICS_NR_IRQS     0x400
> >> +#define SPAPR_IRQ_XICS_NR_MSIS     \
> >> +    (XICS_IRQ_BASE + SPAPR_IRQ_XICS_NR_IRQS - SPAPR_IRQ_MSI)
> >> +
> >>  sPAPRIrq spapr_irq_xics = {
> >> -    .nr_irqs     = XICS_IRQS_SPAPR,
> >> +    .nr_irqs     = SPAPR_IRQ_XICS_NR_IRQS,
> >> +    .nr_msis     = SPAPR_IRQ_XICS_NR_MSIS,
> >>  
> >>      .init        = spapr_irq_init_xics,
> >>      .claim       = spapr_irq_claim_xics,
> >> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
> >> index 6bcb4f419b6b..bb736177e76c 100644
> >> --- a/hw/ppc/spapr_pci.c
> >> +++ b/hw/ppc/spapr_pci.c
> >> @@ -2121,6 +2121,7 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb,
> >>      sPAPRTCETable *tcet;
> >>      PCIBus *bus = PCI_HOST_BRIDGE(phb)->bus;
> >>      sPAPRFDT s_fdt;
> >> +    sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine());
> >>  
> > 
> > It is a bit unfortunate to add another user of qdev_get_machine()...
> > 
> >>      /* Start populating the FDT */
> >>      nodename = g_strdup_printf("pci@%" PRIx64, phb->buid);
> >> @@ -2138,8 +2139,8 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb,
> >>      _FDT(fdt_setprop(fdt, bus_off, "ranges", &ranges, sizeof_ranges));
> >>      _FDT(fdt_setprop(fdt, bus_off, "reg", &bus_reg, sizeof(bus_reg)));
> >>      _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pci-config-space-type", 0x1));
> >> -    /* TODO: fine tune the total count of allocatable MSIs per PHB */
> >> -    _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pe-total-#msi", XICS_IRQS_SPAPR));
> >> +    _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pe-total-#msi",
> >> +                          smc->irq->nr_msis));
> >>  
> > 
> > ... and to expose machine class internals. Since spapr_populate_pci_dt() is only
> > called from the core machine code, maybe have the caller to pass the number of
> > MSIs ?
> 
> yes we could add an extra parameter to spapr_populate_pci_dt()

I'd like to see this as a followup.
> 
> > Anyway, this can be done in a followup patch so:
> > 
> > Reviewed-by: Greg Kurz <groug@kaod.org>
> 
> Thanks,
> 
> C.
> > 
> >>      /* Dynamic DMA window */
> >>      if (phb->ddw_enabled) {
> > 
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] [PATCH v2 2/2] spapr: increase the size of the IRQ number space

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3004 bytes --]

On Tue, Sep 11, 2018 at 07:55:03AM +0200, Cédric Le Goater wrote:
> The new layout using static IRQ number does not leave much space to
> the dynamic MSI range, only 0x100 IRQ numbers. Increase the total
> number of IRQS for newer machines and introduce a legacy XICS backend
> for pre-3.1 machines to maintain compatibility.
> 
> For the old backend, provide a 'nr_msis' value covering the full IRQ
> number space as it does not use the bitmap allocator to allocate MSI
> interrupt numbers.
> 
> Signed-off-by: Cédric Le Goater <clg@kaod.org>

Applied to ppc-for-3.1, thanks.

> ---
>  include/hw/ppc/spapr_irq.h |  1 +
>  hw/ppc/spapr.c             |  1 +
>  hw/ppc/spapr_irq.c         | 15 ++++++++++++++-
>  3 files changed, 16 insertions(+), 1 deletion(-)
> 
> diff --git a/include/hw/ppc/spapr_irq.h b/include/hw/ppc/spapr_irq.h
> index 650f810ad2aa..a467ce696ee4 100644
> --- a/include/hw/ppc/spapr_irq.h
> +++ b/include/hw/ppc/spapr_irq.h
> @@ -41,6 +41,7 @@ typedef struct sPAPRIrq {
>  } sPAPRIrq;
>  
>  extern sPAPRIrq spapr_irq_xics;
> +extern sPAPRIrq spapr_irq_xics_legacy;
>  
>  int spapr_irq_claim(sPAPRMachineState *spapr, int irq, bool lsi, Error **errp);
>  void spapr_irq_free(sPAPRMachineState *spapr, int irq, int num);
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index 4a9dd4d9bc14..eba7d60a30a7 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -3971,6 +3971,7 @@ static void spapr_machine_3_0_class_options(MachineClass *mc)
>      SET_MACHINE_COMPAT(mc, SPAPR_COMPAT_3_0);
>  
>      smc->legacy_irq_allocation = true;
> +    smc->irq = &spapr_irq_xics_legacy;
>  }
>  
>  DEFINE_SPAPR_MACHINE(3_0, "3.0", false);
> diff --git a/hw/ppc/spapr_irq.c b/hw/ppc/spapr_irq.c
> index fe8be5f5217a..e77b94cc685e 100644
> --- a/hw/ppc/spapr_irq.c
> +++ b/hw/ppc/spapr_irq.c
> @@ -195,7 +195,7 @@ static void spapr_irq_print_info_xics(sPAPRMachineState *spapr, Monitor *mon)
>      ics_pic_print_info(spapr->ics, mon);
>  }
>  
> -#define SPAPR_IRQ_XICS_NR_IRQS     0x400
> +#define SPAPR_IRQ_XICS_NR_IRQS     0x1000
>  #define SPAPR_IRQ_XICS_NR_MSIS     \
>      (XICS_IRQ_BASE + SPAPR_IRQ_XICS_NR_IRQS - SPAPR_IRQ_MSI)
>  
> @@ -289,3 +289,16 @@ int spapr_irq_find(sPAPRMachineState *spapr, int num, bool align, Error **errp)
>  
>      return first + ics->offset;
>  }
> +
> +#define SPAPR_IRQ_XICS_LEGACY_NR_IRQS     0x400
> +
> +sPAPRIrq spapr_irq_xics_legacy = {
> +    .nr_irqs     = SPAPR_IRQ_XICS_LEGACY_NR_IRQS,
> +    .nr_msis     = SPAPR_IRQ_XICS_LEGACY_NR_IRQS,
> +
> +    .init        = spapr_irq_init_xics,
> +    .claim       = spapr_irq_claim_xics,
> +    .free        = spapr_irq_free_xics,
> +    .qirq        = spapr_qirq_xics,
> +    .print_info  = spapr_irq_print_info_xics,
> +};

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] [PATCH v2 2/2] spapr: increase the size of the IRQ number space

[Cédric Le Goater]

On 09/13/2018 04:25 AM, David Gibson wrote:
> On Tue, Sep 11, 2018 at 07:55:03AM +0200, Cédric Le Goater wrote:
>> The new layout using static IRQ number does not leave much space to
>> the dynamic MSI range, only 0x100 IRQ numbers. Increase the total
>> number of IRQS for newer machines and introduce a legacy XICS backend
>> for pre-3.1 machines to maintain compatibility.
>>
>> For the old backend, provide a 'nr_msis' value covering the full IRQ
>> number space as it does not use the bitmap allocator to allocate MSI
>> interrupt numbers.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> 
> Applied to ppc-for-3.1, thanks.

I think we are ready for Xive now ?

The patchset is organized as below. The patches tagged v4 have not changed but the others have and a resend will be needed. 

* Device models for Source, Router, EQs, Presenter, Controller :

  ppc/xive: introduce a XIVE interrupt source model (v4)
  ppc/xive: add support for the LSI interrupt sources (v4)
  ppc/xive: introduce the XiveFabric interface (v4)
  ppc/xive: introduce the XiveRouter model (v4)
  ppc/xive: introduce the XIVE Event Queues (v4)
  ppc/xive: add support for the EQ Event State buffers (v4)
  ppc/xive: introduce the XIVE interrupt thread context (v4)
  ppc/xive: introduce a simplified XIVE presenter (v4)
  ppc/xive: notify the CPU when the interrupt priority is more privileged (v4)
  spapr/xive: introduce a XIVE interrupt controller (v4)
  spapr/xive: use the VCPU id as a VP identifier (v5)

* Integration in the sPAPR machine (we can add pnv also)

  spapr: initialize VSMT before initializing the IRQ backend  (v5)
  spapr: introdude a new machine IRQ backend for XIVE  (v5)
  spapr: add hcalls support for the XIVE exploitation interrupt mode  (v5)
  spapr: add device tree support for the XIVE exploitation mode  (v5)
  spapr: allocate the interrupt thread context under the CPU core  (v5)
  spapr: add a 'pseries-3.1-xive' machine type (v5)


* KVM support (KVM XIVE device interfaces)

  spapr: add classes for the XIVE models (v5+)
  target/ppc/kvm: add Linux KVM definitions for XIVE (v5+)
  spapr/xive: add models for KVM support (v5+)
  
* KVM migration (more KVM XIVE device interfaces)

  spapr/xive: add migration support for KVM (v5+)
  spapr: fix XICS migration  (v5+)


Greg is giving it some tests on TCG and now KVM as XIVE is a building 
block for the OpenCAPI passthrough. 

Thanks,

C.