arch/x86/kernel/apic/apic.c: calibrate_APIC_clock() soft hangs when PIC is not configured by BIOS before kernel is launched.

arch/x86/kernel/apic/apic.c: calibrate_APIC_clock() soft hangs when PIC is not configured by BIOS before kernel is launched.

[Kirkendall, Garrett]

Sorry, resending as plain text for linux-kernel@vger.kernel.org

I am trying to boot a UEFI BIOS with minimal legacy hardware support.  The Linux kernel soft hangs when the PIC is not configured by the BIOS because it is using IOAPIC.  Hopefully, this provides enough information.

Observed under Ubuntu Server Linux 18.04 LTS with kernel 4.15.0, and with kernel compiled from source tag v5.0

Where it hangs:
Soft hang occurs in calibrate_APIC_clock(): https://github.com/torvalds/linux/blob/v5.0/arch/x86/kernel/apic/apic.c#L805
specific location of soft hang waiting for interrupts: https://github.com/torvalds/linux/blob/v5.0/arch/x86/kernel/apic/apic.c#L854


How it gets to the hang:
If 8259A PIC is not configured before kernel is launched, HPET IRQ 0 registration fails because probe_8259A returns PIC as not available and therefore interrupt descriptors 0-15 are not allocated.  This happens when BIOS does not configure 8259A PIC because it uses IOAPIC.

This sequence prevents allocating interrupts 0-15 unless PIC is configured before kernel starts.  legacy_pic.init = init_8259A is not called before early_irq_init():
  early_irq_init(): https://github.com/torvalds/linux/blob/v5.0/init/main.c#L642
    initcnt = arch_probe_nr_irqs(): https://github.com/torvalds/linux/blob/v5.0/kernel/irq/irqdesc.c#L512
      return legacy_pic->probe(): https://github.com/torvalds/linux/blob/v5.0/arch/x86/kernel/apic/vector.c#L656
        default_legacy_pic.probe: https://github.com/torvalds/linux/blob/v5.0/arch/x86/kernel/i8259.c#L418
          probe_8259A(): https://github.com/torvalds/linux/blob/v5.0/arch/x86/kernel/i8259.c#L301
    interrupt 0-15 descriptors not allocated and prevents IOAPIC interrupts 0-15:  https://github.com/torvalds/linux/blob/v5.0/kernel/irq/irqdesc.c#L525

In this call is where init_8259A is called and PIC is initialized.  However, interrupt descriptors for 0-15 were not allocated in early_irq_init() sequence, so descriptors are not available later.
  init_IRQ() : https://github.com/torvalds/linux/blob/v5.0/init/main.c#L643

This sequence tries to register the HPET to irq0, but irq0 descriptor is not allocated by early_irq_init() sequence:
  late_time_init(): https://github.com/torvalds/linux/blob/v5.0/init/main.c#L703
    late_time_init = x86_late_time_init: https://github.com/torvalds/linux/blob/v5.0/arch/x86/kernel/time.c#L107
      x86_init.timers.timer_init(): https://github.com/torvalds/linux/blob/v5.0/arch/x86/kernel/time.c#L92
        x86_init.timers.timer_init = hpet_time_init: https://github.com/torvalds/linux/blob/v5.0/arch/x86/kernel/x86_init.c#L75
          setup_default_timer_irq(): https://github.com/torvalds/linux/blob/v5.0/arch/x86/kernel/time.c#L83
            if (setup_irq(0, &irq0)): https://github.com/torvalds/linux/blob/v5.0/arch/x86/kernel/time.c#L78

This gets called at some point after the above sequences, I couldn't track it all the way back to main.c easily:
  x86_init.timers.setup_percpu_clockev = setup_boot_APIC_clock: https://github.com/torvalds/linux/blob/v5.0/arch/x86/kernel/x86_init.c#L74
    https://github.com/torvalds/linux/blob/v5.0/arch/x86/kernel/apic/apic.c#L961
      *** soft hang in calibrate_APIC_clock(): https://github.com/torvalds/linux/blob/v5.0/arch/x86/kernel/apic/apic.c#L854



Why I directly emailed people:
$ perl scripts/get_maintainer.pl arch/x86/kernel/i8259.c:
Nicolai Stange <mailto:nstange@suse.de> (commit_signer:1/1=100%,authored:1/1=100%,added_lines:1/1=100%)
Thomas Gleixner <mailto:tglx@linutronix.de> (commit_signer:1/1=100%)

$ perl scripts/get_maintainer.pl arch/x86/kernel/time.c
Thomas Gleixner <mailto:tglx@linutronix.de> (commit_signer:4/4=100%,authored:2/4=50%,added_lines:22/24=92%)
Andy Lutomirski <mailto:luto@kernel.org> (commit_signer:2/4=50%)
Nathan Chancellor <mailto:natechancellor@gmail.com> (commit_signer:1/4=25%,authored:1/4=25%,removed_lines:1/1=100%)
Nicolai Stange <mailto:nstange@suse.de> (commit_signer:1/4=25%,authored:1/4=25%)

GARRETT KIRKENDALL
SMTS Firmware Engineer | CTE
7171 Southwest Parkway, Austin, TX 78735 USA 
AMD   https://www.facebook.com/AMD  |  http://www.amd.com/

Re: arch/x86/kernel/apic/apic.c: calibrate_APIC_clock() soft hangs when PIC is not configured by BIOS before kernel is launched.

[Thomas Gleixner]

[-- Attachment #1: Type: text/plain, Size: 841 bytes --]

On Thu, 9 May 2019, Kirkendall, Garrett wrote:
> I am trying to boot a UEFI BIOS with minimal legacy hardware support. 
> The Linux kernel soft hangs when the PIC is not configured by the BIOS
> because it is using IOAPIC.  Hopefully, this provides enough information.
>
> Soft hang occurs in calibrate_APIC_clock():

...

> If 8259A PIC is not configured before kernel is launched, HPET IRQ 0
> registration fails because probe_8259A returns PIC as not available and
> therefore interrupt descriptors 0-15 are not allocated.  This happens
> when BIOS does not configure 8259A PIC because it uses IOAPIC.

Right. Works as designed.

There is not much we can do at that point, unless your platform has other
means to provide the TSC frequency (cpuid or MSR) along with the bus
frequency which is fed into the local apic timer.

Thanks,

	tglx

RE: arch/x86/kernel/apic/apic.c: calibrate_APIC_clock() soft hangs when PIC is not configured by BIOS before kernel is launched.

[Kirkendall, Garrett]

1.  Is it correct to probe the 8259 before it is initialized by the kernel?  The 8259 will not respond properly to the probe unless it is properly initialized.
2.  Should IOAPIC interrupts 0-15 require the legacy PIC be available and initialized by the BIOS?
2.  The kernel will not boot if there is no legacy 8259 PIC even if all the other factors stated are provided.

I want to understand why a preinitialized 8259 is a requirement for a system configured to use the IOAPIC?

GARRETT KIRKENDALL
SMTS Firmware Engineer | CTE
7171 Southwest Parkway, Austin, TX 78735 USA 
AMD   facebook  |  amd.com

-----Original Message-----
From: Thomas Gleixner <tglx@linutronix.de> 
Sent: Thursday, May 9, 2019 8:32 AM
To: Kirkendall, Garrett <Garrett.Kirkendall@amd.com>
Cc: nstange@suse.de; luto@kernel.org; natechancellor@gmail.com; x86@kernel.org; linux-kernel@vger.kernel.org
Subject: Re: arch/x86/kernel/apic/apic.c: calibrate_APIC_clock() soft hangs when PIC is not configured by BIOS before kernel is launched.

[CAUTION: External Email]

On Thu, 9 May 2019, Kirkendall, Garrett wrote:
> I am trying to boot a UEFI BIOS with minimal legacy hardware support.
> The Linux kernel soft hangs when the PIC is not configured by the BIOS 
> because it is using IOAPIC.  Hopefully, this provides enough information.
>
> Soft hang occurs in calibrate_APIC_clock():

...

> If 8259A PIC is not configured before kernel is launched, HPET IRQ 0 
> registration fails because probe_8259A returns PIC as not available 
> and therefore interrupt descriptors 0-15 are not allocated.  This 
> happens when BIOS does not configure 8259A PIC because it uses IOAPIC.

Right. Works as designed.

There is not much we can do at that point, unless your platform has other means to provide the TSC frequency (cpuid or MSR) along with the bus frequency which is fed into the local apic timer.

Thanks,

        tglx

RE: arch/x86/kernel/apic/apic.c: calibrate_APIC_clock() soft hangs when PIC is not configured by BIOS before kernel is launched.

[Thomas Gleixner]

Garrett,

On Thu, 9 May 2019, Kirkendall, Garrett wrote:

  A: Because it messes up the order in which people normally read text.
  Q: Why is top-posting such a bad thing?
  A: Top-posting.
  Q: What is the most annoying thing in e-mail?

  A: No.
  Q: Should I include quotations after my reply?

  http://daringfireball.net/2007/07/on_top

Also please break the lines around 78 chars.

> 1.  Is it correct to probe the 8259 before it is initialized by the
>     kernel?  The 8259 will not respond properly to the probe unless it is
>     properly initialized.

So far the kernel relied on the BIOS to initialize 8259

> 2.  Should IOAPIC interrupts 0-15 require the legacy PIC be available and
>     initialized by the BIOS?

Unless the platform explicitely states that there is no legacy PIC, which
is true for some MID and HV guest systems. 

> 2.  The kernel will not boot if there is no legacy 8259 PIC even if all
>     the other factors stated are provided.

Hmm? what other factors?

> I want to understand why a preinitialized 8259 is a requirement for a
> system configured to use the IOAPIC?

Mostly historical reasons and the whole PIC/IOAPIC thing has been a fragile
nightmare forever, so I'm reluctant to do any extra work there which might
break older machines.

Thanks,

	tglx