[Qemu-devel] [PATCH for-4.1 0/2] spapr/xics: Last minute fixes

[Qemu-devel] [PATCH for-4.1 0/2] spapr/xics: Last minute fixes

[Greg Kurz]

KVM on POWER9 doesn't use the XIVE VP space optimally. This currently
limits the number of VMs we can start to 127. Starting with the 128th
one, KVM fails to create the XIVE or the XICS-on-XIVE device and we
go through the fallback paths in QEMU.

The XICS error path still has an issue that prevents the guest to do
interrupts, even after QEMU fell back on XICS emulation. This is
fixed with patch 2.

Patch 1 is just a _trivial_ improvement of the warning that gets
emited when falling back to emulated IC. Feel free to apply it to
4.2 or even to drop it if you don't want it in 4.1.

--
Greg

---

Greg Kurz (2):
      spapr/irq: Inform the user when falling back to emulated IC
      xics/kvm: Fix fallback to emulated XICS


 hw/intc/xics_kvm.c |   11 -----------
 hw/ppc/spapr_irq.c |    1 +
 2 files changed, 1 insertion(+), 11 deletions(-)

[Qemu-devel] [PATCH for-4.1 1/2] spapr/irq: Inform the user when falling back to emulated IC

[Greg Kurz]

Just to give an indication to the user that the error condition is
handled and how.

Reported-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com>
Signed-off-by: Greg Kurz <groug@kaod.org>
---
 hw/ppc/spapr_irq.c |    1 +
 1 file changed, 1 insertion(+)

diff --git a/hw/ppc/spapr_irq.c b/hw/ppc/spapr_irq.c
index ff3df0bbd8cf..d07aed8ca9f9 100644
--- a/hw/ppc/spapr_irq.c
+++ b/hw/ppc/spapr_irq.c
@@ -86,6 +86,7 @@ static void spapr_irq_init_kvm(SpaprMachineState *spapr,
          * emulated mode
          */
         error_prepend(&local_err, "kernel_irqchip allowed but unavailable: ");
+        error_append_hint(&local_err, "Falling back to kernel-irqchip=off\n");
         warn_report_err(local_err);
     }
 }

[Qemu-devel] [PATCH for-4.1 2/2] xics/kvm: Fix fallback to emulated XICS

[Greg Kurz]

Commit 4812f2615288 tried to fix rollback path of xics_kvm_connect() but
it isn't enough. If we fail to create the KVM device, the guest fails
to boot later on with:

[    0.010817] pci 0000:00:00.0: Adding to iommu group 0
[    0.010863] irq: unknown-1 didn't like hwirq-0x1200 to VIRQ17 mapping (rc=-22)
[    0.010923] pci 0000:00:01.0: Adding to iommu group 0
[    0.010968] irq: unknown-1 didn't like hwirq-0x1201 to VIRQ17 mapping (rc=-22)
[    0.011543] EEH: No capable adapters found
[    0.011597] irq: unknown-1 didn't like hwirq-0x1000 to VIRQ17 mapping (rc=-22)
[    0.011651] audit: type=2000 audit(1563977526.000:1): state=initialized audit_enabled=0 res=1
[    0.011703] ------------[ cut here ]------------
[    0.011729] event-sources: Unable to allocate interrupt number for /event-sources/epow-events
[    0.011776] WARNING: CPU: 0 PID: 1 at arch/powerpc/platforms/pseries/event_sources.c:34 request_event_sources_irqs+0xbc/0x150
[    0.011828] Modules linked in:
[    0.011850] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.1.17-300.fc30.ppc64le #1
[    0.011886] NIP:  c0000000000d4fac LR: c0000000000d4fa8 CTR: c0000000018f0000
[    0.011923] REGS: c00000001e4c38d0 TRAP: 0700   Not tainted  (5.1.17-300.fc30.ppc64le)
[    0.011966] MSR:  8000000002029033 <SF,VEC,EE,ME,IR,DR,RI,LE>  CR: 28000284  XER: 20040000
[    0.012012] CFAR: c00000000011b42c IRQMASK: 0
[    0.012012] GPR00: c0000000000d4fa8 c00000001e4c3b60 c0000000015fc400 0000000000000051
[    0.012012] GPR04: 0000000000000001 0000000000000000 0000000000000081 772d6576656e7473
[    0.012012] GPR08: 000000001edf0000 c0000000014d4830 c0000000014d4830 6e6576652f20726f
[    0.012012] GPR12: 0000000000000000 c0000000018f0000 c000000000010bf0 0000000000000000
[    0.012012] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    0.012012] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    0.012012] GPR24: 0000000000000000 0000000000000000 c000000000ebbf00 c0000000000d5570
[    0.012012] GPR28: c000000000ebc008 c00000001fff8248 0000000000000000 0000000000000000
[    0.012372] NIP [c0000000000d4fac] request_event_sources_irqs+0xbc/0x150
[    0.012409] LR [c0000000000d4fa8] request_event_sources_irqs+0xb8/0x150
[    0.012445] Call Trace:
[    0.012462] [c00000001e4c3b60] [c0000000000d4fa8] request_event_sources_irqs+0xb8/0x150 (unreliable)
[    0.012513] [c00000001e4c3bf0] [c000000001042848] __machine_initcall_pseries_init_ras_IRQ+0xc8/0xf8
[    0.012563] [c00000001e4c3c20] [c000000000010810] do_one_initcall+0x60/0x254
[    0.012611] [c00000001e4c3cf0] [c000000001024538] kernel_init_freeable+0x35c/0x444
[    0.012655] [c00000001e4c3db0] [c000000000010c14] kernel_init+0x2c/0x148
[    0.012693] [c00000001e4c3e20] [c00000000000bdc4] ret_from_kernel_thread+0x5c/0x78
[    0.012736] Instruction dump:
[    0.012759] 38a00000 7c7f1b78 7f64db78 2c1f0000 2fbf0000 78630020 4180002c 409effa8
[    0.012805] 7fa4eb78 7f43d378 48046421 60000000 <0fe00000> 3bde0001 2c1e0010 7fde07b4
[    0.012851] ---[ end trace aa5785707323fad3 ]---

This happens because QEMU fell back on XICS emulation but didn't unregister
the RTAS calls from KVM. The emulated RTAS calls are hence never called and
the KVM ones return an error to the guest since the KVM device is absent.

The sanity checks in xics_kvm_disconnect() are abusive since we're freeing
the KVM device. Simply drop them.

Fixes: 4812f2615288 "xics/kvm: Add proper rollback to xics_kvm_init()"
Signed-off-by: Greg Kurz <groug@kaod.org>
---
 hw/intc/xics_kvm.c |   11 -----------
 1 file changed, 11 deletions(-)

diff --git a/hw/intc/xics_kvm.c b/hw/intc/xics_kvm.c
index 2df1f3e92c7e..65c35f90f9af 100644
--- a/hw/intc/xics_kvm.c
+++ b/hw/intc/xics_kvm.c
@@ -430,17 +430,6 @@ fail:
 
 void xics_kvm_disconnect(SpaprMachineState *spapr, Error **errp)
 {
-    /* The KVM XICS device is not in use */
-    if (kernel_xics_fd == -1) {
-        return;
-    }
-
-    if (!kvm_enabled() || !kvm_check_extension(kvm_state, KVM_CAP_IRQ_XICS)) {
-        error_setg(errp,
-                   "KVM and IRQ_XICS capability must be present for KVM XICS device");
-        return;
-    }
-
     /*
      * Only on P9 using the XICS-on XIVE KVM device:
      *

Re: [Qemu-devel] [PATCH for-4.1 1/2] spapr/irq: Inform the user when falling back to emulated IC

[Cédric Le Goater]

On 24/07/2019 18:57, Greg Kurz wrote:
> Just to give an indication to the user that the error condition is
> handled and how.
> 
> Reported-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com>
> Signed-off-by: Greg Kurz <groug@kaod.org>



Reviewed-by: Cédric Le Goater <clg@kaod.org>

Thanks,

C.

> ---
>  hw/ppc/spapr_irq.c |    1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/hw/ppc/spapr_irq.c b/hw/ppc/spapr_irq.c
> index ff3df0bbd8cf..d07aed8ca9f9 100644
> --- a/hw/ppc/spapr_irq.c
> +++ b/hw/ppc/spapr_irq.c
> @@ -86,6 +86,7 @@ static void spapr_irq_init_kvm(SpaprMachineState *spapr,
>           * emulated mode
>           */
>          error_prepend(&local_err, "kernel_irqchip allowed but unavailable: ");
> +        error_append_hint(&local_err, "Falling back to kernel-irqchip=off\n");
>          warn_report_err(local_err);
>      }
>  }
> 

Re: [Qemu-devel] [PATCH for-4.1 2/2] xics/kvm: Fix fallback to emulated XICS

[Cédric Le Goater]

On 24/07/2019 18:57, Greg Kurz wrote:
> Commit 4812f2615288 tried to fix rollback path of xics_kvm_connect() but
> it isn't enough. If we fail to create the KVM device, the guest fails
> to boot later on with:
> 
> [    0.010817] pci 0000:00:00.0: Adding to iommu group 0
> [    0.010863] irq: unknown-1 didn't like hwirq-0x1200 to VIRQ17 mapping (rc=-22)
> [    0.010923] pci 0000:00:01.0: Adding to iommu group 0
> [    0.010968] irq: unknown-1 didn't like hwirq-0x1201 to VIRQ17 mapping (rc=-22)
> [    0.011543] EEH: No capable adapters found
> [    0.011597] irq: unknown-1 didn't like hwirq-0x1000 to VIRQ17 mapping (rc=-22)
> [    0.011651] audit: type=2000 audit(1563977526.000:1): state=initialized audit_enabled=0 res=1
> [    0.011703] ------------[ cut here ]------------
> [    0.011729] event-sources: Unable to allocate interrupt number for /event-sources/epow-events
> [    0.011776] WARNING: CPU: 0 PID: 1 at arch/powerpc/platforms/pseries/event_sources.c:34 request_event_sources_irqs+0xbc/0x150
> [    0.011828] Modules linked in:
> [    0.011850] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.1.17-300.fc30.ppc64le #1
> [    0.011886] NIP:  c0000000000d4fac LR: c0000000000d4fa8 CTR: c0000000018f0000
> [    0.011923] REGS: c00000001e4c38d0 TRAP: 0700   Not tainted  (5.1.17-300.fc30.ppc64le)
> [    0.011966] MSR:  8000000002029033 <SF,VEC,EE,ME,IR,DR,RI,LE>  CR: 28000284  XER: 20040000
> [    0.012012] CFAR: c00000000011b42c IRQMASK: 0
> [    0.012012] GPR00: c0000000000d4fa8 c00000001e4c3b60 c0000000015fc400 0000000000000051
> [    0.012012] GPR04: 0000000000000001 0000000000000000 0000000000000081 772d6576656e7473
> [    0.012012] GPR08: 000000001edf0000 c0000000014d4830 c0000000014d4830 6e6576652f20726f
> [    0.012012] GPR12: 0000000000000000 c0000000018f0000 c000000000010bf0 0000000000000000
> [    0.012012] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [    0.012012] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [    0.012012] GPR24: 0000000000000000 0000000000000000 c000000000ebbf00 c0000000000d5570
> [    0.012012] GPR28: c000000000ebc008 c00000001fff8248 0000000000000000 0000000000000000
> [    0.012372] NIP [c0000000000d4fac] request_event_sources_irqs+0xbc/0x150
> [    0.012409] LR [c0000000000d4fa8] request_event_sources_irqs+0xb8/0x150
> [    0.012445] Call Trace:
> [    0.012462] [c00000001e4c3b60] [c0000000000d4fa8] request_event_sources_irqs+0xb8/0x150 (unreliable)
> [    0.012513] [c00000001e4c3bf0] [c000000001042848] __machine_initcall_pseries_init_ras_IRQ+0xc8/0xf8
> [    0.012563] [c00000001e4c3c20] [c000000000010810] do_one_initcall+0x60/0x254
> [    0.012611] [c00000001e4c3cf0] [c000000001024538] kernel_init_freeable+0x35c/0x444
> [    0.012655] [c00000001e4c3db0] [c000000000010c14] kernel_init+0x2c/0x148
> [    0.012693] [c00000001e4c3e20] [c00000000000bdc4] ret_from_kernel_thread+0x5c/0x78
> [    0.012736] Instruction dump:
> [    0.012759] 38a00000 7c7f1b78 7f64db78 2c1f0000 2fbf0000 78630020 4180002c 409effa8
> [    0.012805] 7fa4eb78 7f43d378 48046421 60000000 <0fe00000> 3bde0001 2c1e0010 7fde07b4
> [    0.012851] ---[ end trace aa5785707323fad3 ]---
> 
> This happens because QEMU fell back on XICS emulation but didn't unregister
> the RTAS calls from KVM. The emulated RTAS calls are hence never called and
> the KVM ones return an error to the guest since the KVM device is absent.
> 
> The sanity checks in xics_kvm_disconnect() are abusive since we're freeing
> the KVM device. Simply drop them.
> 
> Fixes: 4812f2615288 "xics/kvm: Add proper rollback to xics_kvm_init()"
> Signed-off-by: Greg Kurz <groug@kaod.org>



Reviewed-by: Cédric Le Goater <clg@kaod.org>

Thanks,

C.

> ---
>  hw/intc/xics_kvm.c |   11 -----------
>  1 file changed, 11 deletions(-)
> 
> diff --git a/hw/intc/xics_kvm.c b/hw/intc/xics_kvm.c
> index 2df1f3e92c7e..65c35f90f9af 100644
> --- a/hw/intc/xics_kvm.c
> +++ b/hw/intc/xics_kvm.c
> @@ -430,17 +430,6 @@ fail:
>  
>  void xics_kvm_disconnect(SpaprMachineState *spapr, Error **errp)
>  {
> -    /* The KVM XICS device is not in use */
> -    if (kernel_xics_fd == -1) {
> -        return;
> -    }
> -
> -    if (!kvm_enabled() || !kvm_check_extension(kvm_state, KVM_CAP_IRQ_XICS)) {
> -        error_setg(errp,
> -                   "KVM and IRQ_XICS capability must be present for KVM XICS device");
> -        return;
> -    }
> -
>      /*
>       * Only on P9 using the XICS-on XIVE KVM device:
>       *
> 

Re: [Qemu-devel] [PATCH for-4.1 0/2] spapr/xics: Last minute fixes

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 981 bytes --]

On Wed, Jul 24, 2019 at 06:57:09PM +0200, Greg Kurz wrote:
> KVM on POWER9 doesn't use the XIVE VP space optimally. This currently
> limits the number of VMs we can start to 127. Starting with the 128th
> one, KVM fails to create the XIVE or the XICS-on-XIVE device and we
> go through the fallback paths in QEMU.
> 
> The XICS error path still has an issue that prevents the guest to do
> interrupts, even after QEMU fell back on XICS emulation. This is
> fixed with patch 2.
> 
> Patch 1 is just a _trivial_ improvement of the warning that gets
> emited when falling back to emulated IC. Feel free to apply it to
> 4.2 or even to drop it if you don't want it in 4.1.

Applied to ppc-for-4.1.  Finding a time to do a PR will be
interesting, but I'll see what I can do

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]