Xen-unstable: Regression (on AMD) host boot stuck on (XEN) testing the IO APIC.......................

Xen-unstable: Regression (on AMD) host boot stuck on (XEN) testing the IO APIC.......................

[Sander Eikelenboom]

Hi Jan / Andrew,

I just tried xen-unstable of today(last commit 7645640d6ff1791733c43f71590c6e2976cfa549)
and it fails to boot on my AMD machine, it gets stuck on:
"(XEN) testing the IO APIC......................." which takes forever.

The current tree failing is: commit 7645640d6ff1791733c43f71590c6e2976cfa549
My last working tree when everything booted fine was: xen_changeset: Mon Aug 11 15:13:04 2014 +0200 git:f093fcf-dirty   

Since i use the special options at boot (ivrs_ioapic[6]=00:14.0 ivrs_hpet[0]=00:14.0),
I also tried booting without these, but that gets stuck at the same point.

Due to time constraints i haven't been able to bisected it further.

--
Sander

 __  __            _  _    ____                     _        _     _      
 \ \/ /___ _ __   | || |  | ___|    _   _ _ __  ___| |_ __ _| |__ | | ___ 
  \  // _ \ '_ \  | || |_ |___ \ __| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
  /  \  __/ | | | |__   _| ___) |__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_|    |_|(_)____/    \__,_|_| |_|___/\__\__,_|_.__/|_|\___|
                                                                          
(XEN) Xen version 4.5-unstable (root@dyndns.org) (gcc-4.7.real (Debian 4.7.2-5) 4.7.2) debug=y Mon Aug 25 17:50:33 CEST 2014
(XEN) Latest ChangeSet: Fri Aug 22 14:34:46 2014 +0200 git:7645640-dirty
(XEN) Bootloader: GRUB 1.99-27+deb7u2
(XEN) Command line: dom0_mem=1536M,max:1536M loglvl=all loglvl_guest=all console_timestamps=datems vga=gfx-1280x1024x32 cpuidle cpufreq=xen debug lapic=debug apic_verbosity=debug apic=debug iommu=on,verbose,debug,amd-iommu-debug ivrs_ioapic[6]=00:14.0 ivrs_hpet[0]=00:14.0 com1=38400,8n1 console=vga,com1
(XEN) Video information:
(XEN)  VGA is graphics mode 1280x1024, 32 bpp
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 2 MBR signatures
(XEN)  Found 2 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 0000000000099400 (usable)
(XEN)  0000000000099400 - 00000000000a0000 (reserved)
(XEN)  00000000000e4000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000009ff90000 (usable)
(XEN)  000000009ff90000 - 000000009ff9e000 (ACPI data)
(XEN)  000000009ff9e000 - 000000009ffe0000 (ACPI NVS)
(XEN)  000000009ffe0000 - 00000000a0000000 (reserved)
(XEN)  00000000ffe00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000560000000 (usable)
(XEN) ACPI: RSDP 000FB100, 0014 (r0 ACPIAM)
(XEN) ACPI: RSDT 9FF90000, 0048 (r1 MSI    OEMSLIC  20100913 MSFT       97)
(XEN) ACPI: FACP 9FF90200, 0084 (r1 7640MS A7640100 20100913 MSFT       97)
(XEN) ACPI: DSDT 9FF905E0, 9427 (r1  A7640 A7640100      100 INTL 20051117)
(XEN) ACPI: FACS 9FF9E000, 0040
(XEN) ACPI: APIC 9FF90390, 0088 (r1 7640MS A7640100 20100913 MSFT       97)
(XEN) ACPI: MCFG 9FF90420, 003C (r1 7640MS OEMMCFG  20100913 MSFT       97)
(XEN) ACPI: SLIC 9FF90460, 0176 (r1 MSI    OEMSLIC  20100913 MSFT       97)
(XEN) ACPI: OEMB 9FF9E040, 0072 (r1 7640MS A7640100 20100913 MSFT       97)
(XEN) ACPI: SRAT 9FF9A5E0, 0108 (r3 AMD    FAM_F_10        2 AMD         1)
(XEN) ACPI: HPET 9FF9A6F0, 0038 (r1 7640MS OEMHPET  20100913 MSFT       97)
(XEN) ACPI: IVRS 9FF9A730, 0110 (r1  AMD     RD890S   202031 AMD         0)
(XEN) ACPI: SSDT 9FF9A840, 0DA4 (r1 A M I  POWERNOW        1 AMD         1)
(XEN) System RAM: 20479MB (20970660kB)
(XEN) SRAT: PXM 0 -> APIC 0 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 1 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 2 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 3 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 4 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 5 -> Node 0
(XEN) SRAT: Node 0 PXM 0 0-a0000
(XEN) SRAT: Node 0 PXM 0 100000-a0000000
(XEN) SRAT: Node 0 PXM 0 100000000-560000000
(XEN) NUMA: Allocated memnodemap from 55d0b4000 - 55d0ba000
(XEN) NUMA: Using 8 for the hash shift.
(XEN) Domain heap initialised
(XEN) vesafb: framebuffer at 0xd0000000, mapped to 0xffff82c000201000, using 6144k, total 16384k
(XEN) vesafb: mode is 1280x1024x32, linelength=5120, font 8x16
(XEN) vesafb: Truecolor: size=0:8:8:8, shift=0:16:8:0
(XEN) found SMP MP-table at 000ff780
(XEN) DMI present.
(XEN) APIC boot state is 'xapic'
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x808
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:804,1:0], pm1x_evt[1:800,1:0]
(XEN) ACPI:             wakeup_vec[9ff9e00c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 0:10 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
(XEN) Processor #1 0:10 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
(XEN) Processor #2 0:10 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
(XEN) Processor #3 0:10 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled)
(XEN) Processor #4 0:10 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x05] enabled)
(XEN) Processor #5 0:10 APIC version 16
(XEN) ACPI: IOAPIC (id[0x06] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 6, version 33, address 0xfec00000, GSI 0-23
(XEN) ACPI: IOAPIC (id[0x07] address[0xfec20000] gsi_base[24])
(XEN) IOAPIC[1]: apic_id 7, version 33, address 0xfec20000, GSI 24-55
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 2 I/O APICs
(XEN) ACPI: HPET id: 0x8300 base: 0xfed00000
(XEN) ERST table was not found
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 6 CPUs (0 hotplug CPUs)
(XEN) mapped APIC to ffff82cfffdfb000 (fee00000)
(XEN) mapped IOAPIC to ffff82cfffdfa000 (fec00000)
(XEN) mapped IOAPIC to ffff82cfffdf9000 (fec20000)
(XEN) IRQ limits: 56 GSI, 1112 MSI/MSI-X
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3200.156 MHz processor.
(XEN) Initing memory sharing.
(XEN) AMD Fam10h machine check reporting enabled
(XEN) alt table ffff82d0802d49b0 -> ffff82d0802d59d0
(XEN) PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff
(XEN) PCI: Not using MCFG for segment 0000 bus 00-ff
(XEN) AMD-Vi: Found MSI capability block at 0x54
(XEN) AMD-Vi: ACPI Table:
(XEN) AMD-Vi:  Signature IVRS
(XEN) AMD-Vi:  Length 0x110
(XEN) AMD-Vi:  Revision 0x1
(XEN) AMD-Vi:  CheckSum 0xeb
(XEN) AMD-Vi:  OEM_Id AMD  
(XEN) AMD-Vi:  OEM_Table_Id RD890S
(XEN) AMD-Vi:  OEM_Revision 0x202031
(XEN) AMD-Vi:  Creator_Id AMD 
(XEN) AMD-Vi:  Creator_Revision 0
(XEN) AMD-Vi: IVRS Block: type 0x10 flags 0x3e len 0xe0 id 0x2
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0 -> 0x2
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x10 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0xf00 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0xf00 -> 0xf01
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x18 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0xe00 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0xe00 -> 0xe01
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x28 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xd00 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x30 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xc00 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x48 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xb00 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x50 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa00 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x58 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x900 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x900 -> 0x901
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x60 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x500 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x608 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x800 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x610 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x700 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x68 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x400 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x88 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x90 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x90 -> 0x92
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x98 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x98 -> 0x9a
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa0 flags 0xd7
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa2 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa3 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa4 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x43 id 0x300 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x300 -> 0x3ff alias 0xa4
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa5 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa8 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa9 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x100 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0xb0 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0xb0 -> 0xb2
(XEN) AMD-Vi: IVHD Device Entry: type 0 id 0 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0xd7
(XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x2 handle 0
(XEN) AMD-Vi: IVHD: Command line override present for HPET 0 (IVRS: 0 devID 0000:00:14.0)
(XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0
(XEN) AMD-Vi: IVHD Special: 0000:00:00.1 variety 0x1 handle 0x7
(XEN) AMD-Vi: Disabled HAP memory map sharing with IOMMU
(XEN) AMD-Vi: IOMMU 0 Enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Getting VERSION: 80050010
(XEN) Getting VERSION: 80050010
(XEN) Getting ID: 0
(XEN) Getting LVT0: 700
(XEN) Getting LVT1: 400
(XEN) enabled ExtINT on CPU#0
(XEN) ESR value before enabling vector: 0x4  after: 0
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) init IO_APIC IRQs
(XEN)  IO-APIC (apicid-pin) 6-0, 6-16, 6-17, 6-18, 6-19, 6-20, 6-21, 6-22, 6-23, 7-0, 7-1, 7-2, 7-3, 7-4, 7-5, 7-6, 7-7, 7-8, 7-9, 7-10, 7-11, 7-12, 7-13, 7-14, 7-15, 7-16, 7-17, 7-18, 7-19, 7-20, 7-21, 7-22, 7-23, 7-24, 7-25, 7-26, 7-27, 7-28, 7-29, 7-30, 7-31 not connected.
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) number of MP IRQ sources: 15.
(XEN) number of IO-APIC #6 registers: 24.
(XEN) number of IO-APIC #7 registers: 32.
(XEN) testing the IO APIC.......................





-- 
Best regards,
 Sander                          mailto:linux@eikelenboom.it

Re: Xen-unstable: Regression (on AMD) host boot stuck on (XEN) testing the IO APIC.......................

[Andrew Cooper]

On 25/08/14 18:12, Sander Eikelenboom wrote:
> Hi Jan / Andrew,
>
> I just tried xen-unstable of today(last commit 7645640d6ff1791733c43f71590c6e2976cfa549)
> and it fails to boot on my AMD machine, it gets stuck on:
> "(XEN) testing the IO APIC......................." which takes forever.
>
> The current tree failing is: commit 7645640d6ff1791733c43f71590c6e2976cfa549
> My last working tree when everything booted fine was: xen_changeset: Mon Aug 11 15:13:04 2014 +0200 git:f093fcf-dirty   
>
> Since i use the special options at boot (ivrs_ioapic[6]=00:14.0 ivrs_hpet[0]=00:14.0),
> I also tried booting without these, but that gets stuck at the same point.
>
> Due to time constraints i haven't been able to bisected it further.
>
> --
> Sander

Hmm - Does reverting e13b3203990706db1313ec2aadd9a30b249ee793 fix the issue?

That is the only change in that area, but softirqs should be fine even
by that point on boot.

~Andrew

Re: Xen-unstable: Regression (on AMD) host boot stuck on (XEN) testing the IO APIC.......................

[Sander Eikelenboom]

Monday, August 25, 2014, 7:54:05 PM, you wrote:

> On 25/08/14 18:12, Sander Eikelenboom wrote:
>> Hi Jan / Andrew,
>>
>> I just tried xen-unstable of today(last commit 7645640d6ff1791733c43f71590c6e2976cfa549)
>> and it fails to boot on my AMD machine, it gets stuck on:
>> "(XEN) testing the IO APIC......................." which takes forever.
>>
>> The current tree failing is: commit 7645640d6ff1791733c43f71590c6e2976cfa549
>> My last working tree when everything booted fine was: xen_changeset: Mon Aug 11 15:13:04 2014 +0200 git:f093fcf-dirty   
>>
>> Since i use the special options at boot (ivrs_ioapic[6]=00:14.0 ivrs_hpet[0]=00:14.0),
>> I also tried booting without these, but that gets stuck at the same point.
>>
>> Due to time constraints i haven't been able to bisected it further.
>>
>> --
>> Sander

> Hmm - Does reverting e13b3203990706db1313ec2aadd9a30b249ee793 fix the issue?

> That is the only change in that area, but softirqs should be fine even
> by that point on boot.

> ~Andrew

Hi Andrew,

Hmm i completely overlooked that commit (it's quite evident when you look at the 
commitdiff :-) )

Just tested reverting that single commit and that indeed lets the machine boot again.

--
Sander

Re: Xen-unstable: Regression (on AMD) host boot stuck on (XEN) testing the IO APIC.......................

[Jan Beulich]

>>> On 25.08.14 at 20:25, <linux@eikelenboom.it> wrote:
> Monday, August 25, 2014, 7:54:05 PM, you wrote:
>> Hmm - Does reverting e13b3203990706db1313ec2aadd9a30b249ee793 fix the issue?
> 
> Hmm i completely overlooked that commit (it's quite evident when you look at 
> the 
> commitdiff :-) )
> 
> Just tested reverting that single commit and that indeed lets the machine 
> boot again.

Considering how early this is, the only theory I have is that
rcu_pending() may end up constantly returning true, thus always
causing RCU_SOFTIRQ to get raised, not allowing the loop in
__do_softirq() to ever exit. If that's the case, we seem to have
two options:
- pass into __print_IO_APIC() whether it's being called from a key
  handler
- only call process_pending_softirqs() when system_state >=
  SYS_STATE_smp_boot

Personally I'd prefer the former option.

But Sander, a fundamental question: Why would you _always_
boot your system with "apic=debug" and those various other
debugging options? They are, as you may guess, for debugging,
not for day to day use. (That said I'm glad you have this in place
for this particular case, as it allowed spotting a regression early.)

Jan

Re: Xen-unstable: Regression (on AMD) host boot stuck on (XEN) testing the IO APIC.......................

[Jan Beulich]

>>> On 26.08.14 at 09:34, <JBeulich@suse.com> wrote:
> Considering how early this is, the only theory I have is that
> rcu_pending() may end up constantly returning true, thus always
> causing RCU_SOFTIRQ to get raised, not allowing the loop in
> __do_softirq() to ever exit. If that's the case, we seem to have
> two options:
> - pass into __print_IO_APIC() whether it's being called from a key
>   handler
> - only call process_pending_softirqs() when system_state >=
>   SYS_STATE_smp_boot
> 
> Personally I'd prefer the former option.

So here's a patch implementing that (the issue clearly isn't limited
to AMD).

Jan

x86/IO-APIC: don't process softirqs during early boot

Commit e13b320399 ("x86/irq: process softirqs in irq keyhandlers")
made this unconditional, but the boot time use of __print_IO_APIC()
(when "apic_verbosity=debug" was given) can't tolerate that.

Reported-by: Sander Eikelenboom <linux@eikelenboom.it>
Signed-off-by: Jan Beulich <jbeulich@suse.com>

--- a/xen/arch/x86/io_apic.c
+++ b/xen/arch/x86/io_apic.c
@@ -1092,7 +1092,7 @@ static inline void UNEXPECTED_IO_APIC(vo
 {
 }
 
-static void /*__init*/ __print_IO_APIC(void)
+static void /*__init*/ __print_IO_APIC(bool_t boot)
 {
     int apic, i;
     union IO_APIC_reg_00 reg_00;
@@ -1113,7 +1113,8 @@ static void /*__init*/ __print_IO_APIC(v
     printk(KERN_INFO "testing the IO APIC.......................\n");
 
     for (apic = 0; apic < nr_ioapics; apic++) {
-        process_pending_softirqs();
+        if ( !boot )
+            process_pending_softirqs();
 
         if (!nr_ioapic_entries[apic])
             continue;
@@ -1219,7 +1220,7 @@ static void /*__init*/ __print_IO_APIC(v
     for (i = 0; i < nr_irqs_gsi; i++) {
         struct irq_pin_list *entry = irq_2_pin + i;
 
-        if ( !(i & 0x1f) )
+        if ( !boot && !(i & 0x1f) )
             process_pending_softirqs();
 
         if (entry->pin < 0)
@@ -1242,12 +1243,12 @@ static void /*__init*/ __print_IO_APIC(v
 static void __init print_IO_APIC(void)
 {
     if (apic_verbosity != APIC_QUIET)
-        __print_IO_APIC();
+        __print_IO_APIC(1);
 }
 
 static void _print_IO_APIC_keyhandler(unsigned char key)
 {
-    __print_IO_APIC();
+    __print_IO_APIC(0);
 }
 static struct keyhandler print_IO_APIC_keyhandler = {
     .diagnostic = 1,

Re: Xen-unstable: Regression (on AMD) host boot stuck on (XEN) testing the IO APIC.......................

[Sander Eikelenboom]

Tuesday, August 26, 2014, 9:34:30 AM, you wrote:

>>>> On 25.08.14 at 20:25, <linux@eikelenboom.it> wrote:
>> Monday, August 25, 2014, 7:54:05 PM, you wrote:
>>> Hmm - Does reverting e13b3203990706db1313ec2aadd9a30b249ee793 fix the issue?
>> 
>> Hmm i completely overlooked that commit (it's quite evident when you look at 
>> the 
>> commitdiff :-) )
>> 
>> Just tested reverting that single commit and that indeed lets the machine 
>> boot again.

> Considering how early this is, the only theory I have is that
> rcu_pending() may end up constantly returning true, thus always
> causing RCU_SOFTIRQ to get raised, not allowing the loop in
> __do_softirq() to ever exit. If that's the case, we seem to have
> two options:
> - pass into __print_IO_APIC() whether it's being called from a key
>   handler
> - only call process_pending_softirqs() when system_state >=
>   SYS_STATE_smp_boot

> Personally I'd prefer the former option.

> But Sander, a fundamental question: Why would you _always_
> boot your system with "apic=debug" and those various other
> debugging options? They are, as you may guess, for debugging,
> not for day to day use. (That said I'm glad you have this in place
> for this particular case, as it allowed spotting a regression early.)

> Jan

Hmm just a left over from previous endeavours :-)

Don't know if they also incur a (large) performance penalty ?
(they primarily seemed to add some debug info to the log)

Re: Xen-unstable: Regression (on AMD) host boot stuck on (XEN) testing the IO APIC.......................

[Andrew Cooper]

On 26/08/14 08:45, Jan Beulich wrote:
>>>> On 26.08.14 at 09:34, <JBeulich@suse.com> wrote:
>> Considering how early this is, the only theory I have is that
>> rcu_pending() may end up constantly returning true, thus always
>> causing RCU_SOFTIRQ to get raised, not allowing the loop in
>> __do_softirq() to ever exit. If that's the case, we seem to have
>> two options:
>> - pass into __print_IO_APIC() whether it's being called from a key
>>   handler
>> - only call process_pending_softirqs() when system_state >=
>>   SYS_STATE_smp_boot
>>
>> Personally I'd prefer the former option.
> So here's a patch implementing that (the issue clearly isn't limited
> to AMD).
>
> Jan
>
> x86/IO-APIC: don't process softirqs during early boot
>
> Commit e13b320399 ("x86/irq: process softirqs in irq keyhandlers")
> made this unconditional, but the boot time use of __print_IO_APIC()
> (when "apic_verbosity=debug" was given) can't tolerate that.
>
> Reported-by: Sander Eikelenboom <linux@eikelenboom.it>
> Signed-off-by: Jan Beulich <jbeulich@suse.com>

Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>

>
> --- a/xen/arch/x86/io_apic.c
> +++ b/xen/arch/x86/io_apic.c
> @@ -1092,7 +1092,7 @@ static inline void UNEXPECTED_IO_APIC(vo
>  {
>  }
>  
> -static void /*__init*/ __print_IO_APIC(void)
> +static void /*__init*/ __print_IO_APIC(bool_t boot)
>  {
>      int apic, i;
>      union IO_APIC_reg_00 reg_00;
> @@ -1113,7 +1113,8 @@ static void /*__init*/ __print_IO_APIC(v
>      printk(KERN_INFO "testing the IO APIC.......................\n");
>  
>      for (apic = 0; apic < nr_ioapics; apic++) {
> -        process_pending_softirqs();
> +        if ( !boot )
> +            process_pending_softirqs();
>  
>          if (!nr_ioapic_entries[apic])
>              continue;
> @@ -1219,7 +1220,7 @@ static void /*__init*/ __print_IO_APIC(v
>      for (i = 0; i < nr_irqs_gsi; i++) {
>          struct irq_pin_list *entry = irq_2_pin + i;
>  
> -        if ( !(i & 0x1f) )
> +        if ( !boot && !(i & 0x1f) )
>              process_pending_softirqs();
>  
>          if (entry->pin < 0)
> @@ -1242,12 +1243,12 @@ static void /*__init*/ __print_IO_APIC(v
>  static void __init print_IO_APIC(void)
>  {
>      if (apic_verbosity != APIC_QUIET)
> -        __print_IO_APIC();
> +        __print_IO_APIC(1);
>  }
>  
>  static void _print_IO_APIC_keyhandler(unsigned char key)
>  {
> -    __print_IO_APIC();
> +    __print_IO_APIC(0);
>  }
>  static struct keyhandler print_IO_APIC_keyhandler = {
>      .diagnostic = 1,
>
>
>

Re: Xen-unstable: Regression (on AMD) host boot stuck on (XEN) testing the IO APIC.......................

[Sander Eikelenboom]

Tuesday, August 26, 2014, 11:32:10 AM, you wrote:

> On 26/08/14 08:45, Jan Beulich wrote:
>>>>> On 26.08.14 at 09:34, <JBeulich@suse.com> wrote:
>>> Considering how early this is, the only theory I have is that
>>> rcu_pending() may end up constantly returning true, thus always
>>> causing RCU_SOFTIRQ to get raised, not allowing the loop in
>>> __do_softirq() to ever exit. If that's the case, we seem to have
>>> two options:
>>> - pass into __print_IO_APIC() whether it's being called from a key
>>>   handler
>>> - only call process_pending_softirqs() when system_state >=
>>>   SYS_STATE_smp_boot
>>>
>>> Personally I'd prefer the former option.
>> So here's a patch implementing that (the issue clearly isn't limited
>> to AMD).
>>
>> Jan
>>
>> x86/IO-APIC: don't process softirqs during early boot
>>
>> Commit e13b320399 ("x86/irq: process softirqs in irq keyhandlers")
>> made this unconditional, but the boot time use of __print_IO_APIC()
>> (when "apic_verbosity=debug" was given) can't tolerate that.
>>
>> Reported-by: Sander Eikelenboom <linux@eikelenboom.it>
>> Signed-off-by: Jan Beulich <jbeulich@suse.com>

> Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>

And Tested-by:  Sander Eikelenboom <linux@eikelenboom.it>

Thanks again ! 

>>
>> --- a/xen/arch/x86/io_apic.c
>> +++ b/xen/arch/x86/io_apic.c
>> @@ -1092,7 +1092,7 @@ static inline void UNEXPECTED_IO_APIC(vo
>>  {
>>  }
>>  
>> -static void /*__init*/ __print_IO_APIC(void)
>> +static void /*__init*/ __print_IO_APIC(bool_t boot)
>>  {
>>      int apic, i;
>>      union IO_APIC_reg_00 reg_00;
>> @@ -1113,7 +1113,8 @@ static void /*__init*/ __print_IO_APIC(v
>>      printk(KERN_INFO "testing the IO APIC.......................\n");
>>  
>>      for (apic = 0; apic < nr_ioapics; apic++) {
>> -        process_pending_softirqs();
>> +        if ( !boot )
>> +            process_pending_softirqs();
>>  
>>          if (!nr_ioapic_entries[apic])
>>              continue;
>> @@ -1219,7 +1220,7 @@ static void /*__init*/ __print_IO_APIC(v
>>      for (i = 0; i < nr_irqs_gsi; i++) {
>>          struct irq_pin_list *entry = irq_2_pin + i;
>>  
>> -        if ( !(i & 0x1f) )
>> +        if ( !boot && !(i & 0x1f) )
>>              process_pending_softirqs();
>>  
>>          if (entry->pin < 0)
>> @@ -1242,12 +1243,12 @@ static void /*__init*/ __print_IO_APIC(v
>>  static void __init print_IO_APIC(void)
>>  {
>>      if (apic_verbosity != APIC_QUIET)
>> -        __print_IO_APIC();
>> +        __print_IO_APIC(1);
>>  }
>>  
>>  static void _print_IO_APIC_keyhandler(unsigned char key)
>>  {
>> -    __print_IO_APIC();
>> +    __print_IO_APIC(0);
>>  }
>>  static struct keyhandler print_IO_APIC_keyhandler = {
>>      .diagnostic = 1,
>>
>>
>>