kdump/kexec on EFI-enabled x2apic platforms

kdump/kexec on EFI-enabled x2apic platforms

[Jack Steiner]

All -

I just started debugging kdump/kexec on our UV platform and
have run into some problems. I suspect others have encountered these
same or similar problems. Any help would be appreciated.



Our platform uses EFI boot. It is Nehalem based & has a large number of cpus.
The BIOS enables x2apic mode and the kernel runs with interrupt remapping enabled.
Note that some apicids have more than 8 bits - x2apic mode is required.


I am able to successfully kexec the dump kernel but run into several problems.

	- because the initial kernel boots using EFI, BIOS does not build the legacy
	  tables that are required to locate the RSDP using the legacy method in
	  acpi_find_root_pointer(). (When booting with EFI, acpi_find_root_pointer() is
	  not used. The ACPI tables are found from pointers in EFI tables.)

	- it appears that kdump/kexec intentionally boots the kdump kernel
	  in a mode does does enable efi mode. (Am I correct here???)
	  This avoids the issues with EFI virtual mode. However, the result
	  is that ACPI tables are not found. From the dump kernel:

	  	ACPI Error: A valid RSDP was not found (20090903/tbxfroot-222)

	- Because ACPI tables are not found, the dump kernel does not transition
	  into x2apic mode. The hardware, however, is still in x2apic mode from the
	  initial kernel. Later in the boot of the dump kernel, read_apic_id()
	  tried to read memory-mapped apic registers instead of the MSRs that are
	  used in x2apic mode. This is not allowed & the dump kernel panics with:

		[    0.000000]  [<ffffffff81b52195>] early_idt_handler+0x55/0x68
		[    0.000000]  [<ffffffff8101f393>] ? native_apic_mem_read+0x3/0x10
		[    0.000000]  [<ffffffff8101a3c6>] ? read_apic_id+0x16/0x30
		[    0.000000]  [<ffffffff81b5f857>] init_apic_mappings+0xe7/0x137
		[    0.000000]  [<ffffffff81b559fd>] setup_arch+0x900/0xc33
		[    0.000000]  [<ffffffff81b52bae>] start_kernel+0x6f/0x4a1



I checked an Intel Nehalem whitebox using the Intel BIOS. The dump kernel does not
find the RDSP but the initial kernel does not enable x2apic mode either (possibly
because of an old BIOS - not sure).  As a result, the dump kernel does not hit
the panic shown above. The kdump kernel successfully boots w/o having discovered
ACPI tables.


How should I proceed?
	- should I be running the dump kernel with EFI mode enabled?
	- should I be fixing the issues with x2apic mode in a non-EFI dump kernel?
	- or should BIOS be building the tables necessary to support both EFI & non-EFI boot.


--- jack


Jack Steiner (steiner@sgi.com)		SGI - Silicon Graphics, Inc.


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kdump/kexec on EFI-enabled x2apic platforms

[Eric W. Biederman]

Jack Steiner <steiner@sgi.com> writes:

> All -
>
> I just started debugging kdump/kexec on our UV platform and
> have run into some problems. I suspect others have encountered these
> same or similar problems. Any help would be appreciated.
>
>
>
> Our platform uses EFI boot. It is Nehalem based & has a large number of cpus.
> The BIOS enables x2apic mode and the kernel runs with interrupt remapping enabled.
> Note that some apicids have more than 8 bits - x2apic mode is required.
>
>
> I am able to successfully kexec the dump kernel but run into several problems.
>
> 	- because the initial kernel boots using EFI, BIOS does not build the legacy
> 	  tables that are required to locate the RSDP using the legacy method in
> 	  acpi_find_root_pointer(). (When booting with EFI, acpi_find_root_pointer() is
> 	  not used. The ACPI tables are found from pointers in EFI tables.)

Ouch.  EFI tables are a major pain to use because they are not 32/64 clean.  Thus
making them unreasonably difficult to work with.

I believe the boot loader should be passing in the location of acpi tables instead
of expecting the kernel to wade through EFI tables.

> 	- it appears that kdump/kexec intentionally boots the kdump kernel
> 	  in a mode does does enable efi mode. (Am I correct here???)
> 	  This avoids the issues with EFI virtual mode. However, the result
> 	  is that ACPI tables are not found. From the dump kernel:
>
> 	  	ACPI Error: A valid RSDP was not found (20090903/tbxfroot-222)

My personal opinion is that EFI virtual mode is a mistake.  We should not have
any interactions with the EFI bios of sufficient frequency that we need an
efficient virtual mapping.


> 	- Because ACPI tables are not found, the dump kernel does not transition
> 	  into x2apic mode. The hardware, however, is still in x2apic mode from the
> 	  initial kernel. 

Hmm.  That is a bug of many flavors.  We should have transitions back into 
i8259 legacy mode, before calling kdump.  Then regardless of what happen
before we ran if the hardware is x2apic capable we should force the hardware
into the mode we want it, not just assume x2apic mode is off by default.

>	  Later in the boot of the dump kernel, read_apic_id()
> 	  tried to read memory-mapped apic registers instead of the MSRs that are
> 	  used in x2apic mode. This is not allowed & the dump kernel panics with:
>
> 		[    0.000000]  [<ffffffff81b52195>] early_idt_handler+0x55/0x68
> 		[    0.000000]  [<ffffffff8101f393>] ? native_apic_mem_read+0x3/0x10
> 		[    0.000000]  [<ffffffff8101a3c6>] ? read_apic_id+0x16/0x30
> 		[    0.000000]  [<ffffffff81b5f857>] init_apic_mappings+0xe7/0x137
> 		[    0.000000]  [<ffffffff81b559fd>] setup_arch+0x900/0xc33
> 		[    0.000000]  [<ffffffff81b52bae>] start_kernel+0x6f/0x4a1
>
>
>
> I checked an Intel Nehalem whitebox using the Intel BIOS. The dump kernel does not
> find the RDSP but the initial kernel does not enable x2apic mode either (possibly
> because of an old BIOS - not sure).  As a result, the dump kernel does not hit
> the panic shown above. The kdump kernel successfully boots w/o having discovered
> ACPI tables.
>
>
> How should I proceed?
> 	- should I be running the dump kernel with EFI mode enabled?

We can not interact with the BIOS/EFI in kdump in general so finding a way
to make it work without interacting with the BIOS seems to be a proper path.

> 	- should I be fixing the issues with x2apic mode in a non-EFI dump kernel?

Definitely.

> 	- or should BIOS be building the tables necessary to support both EFI & non-EFI boot.

That too.  Unless it is impossible it is good to be able to avoid EFI nonsense.

Eric

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kdump/kexec on EFI-enabled x2apic platforms

[Jack Steiner]

On Mon, Mar 29, 2010 at 03:13:05PM -0700, Eric W. Biederman wrote:
> Jack Steiner <steiner@sgi.com> writes:
> 

Thnaks for the quick reply. I am still digging thru the issues but am making
good progress.

See comments below.


> > All -
> >
> > I just started debugging kdump/kexec on our UV platform and
> > have run into some problems. I suspect others have encountered these
> > same or similar problems. Any help would be appreciated.
> >
> >
> >
> > Our platform uses EFI boot. It is Nehalem based & has a large number of cpus.
> > The BIOS enables x2apic mode and the kernel runs with interrupt remapping enabled.
> > Note that some apicids have more than 8 bits - x2apic mode is required.
> >
> >
> > I am able to successfully kexec the dump kernel but run into several problems.
> >
> > 	- because the initial kernel boots using EFI, BIOS does not build the legacy
> > 	  tables that are required to locate the RSDP using the legacy method in
> > 	  acpi_find_root_pointer(). (When booting with EFI, acpi_find_root_pointer() is
> > 	  not used. The ACPI tables are found from pointers in EFI tables.)
> 
> Ouch.  EFI tables are a major pain to use because they are not 32/64 clean.  Thus
> making them unreasonably difficult to work with.
> 
> I believe the boot loader should be passing in the location of acpi tables instead
> of expecting the kernel to wade through EFI tables.

I made a temporary fix to the BIOS to build the legacy table used by
acpi_find_root_pointer() to locate the ACPI tables. I tested this on a system simulator &
it seems to work. ACPI tables are discovered and parsed correctly - at least for the
current BIOS & hardware configuration. I'll try this on real hardware later but
don't expect any problems.


> 
> > 	- it appears that kdump/kexec intentionally boots the kdump kernel
> > 	  in a mode does does enable efi mode. (Am I correct here???)
> > 	  This avoids the issues with EFI virtual mode. However, the result
> > 	  is that ACPI tables are not found. From the dump kernel:
> >
> > 	  	ACPI Error: A valid RSDP was not found (20090903/tbxfroot-222)
> 
> My personal opinion is that EFI virtual mode is a mistake.  We should not have
> any interactions with the EFI bios of sufficient frequency that we need an
> efficient virtual mapping.

Agree. The EFI virtual mode support was poorly architected as far as kdump/kexec is
concerned. We had a lot of problem on IA64, too.


> 
> 
> > 	- Because ACPI tables are not found, the dump kernel does not transition
> > 	  into x2apic mode. The hardware, however, is still in x2apic mode from the
> > 	  initial kernel. 
> 
> Hmm.  That is a bug of many flavors.  We should have transitions back into 
> i8259 legacy mode, before calling kdump.  Then regardless of what happen
> before we ran if the hardware is x2apic capable we should force the hardware
> into the mode we want it, not just assume x2apic mode is off by default.

It is not possible to transition back into non-x2apic mode. The initial transition
INTO x2apic mode is done by the BIOS - not the OS. X2apic mode is required because
apic ids are greater than 8 bits. Fortunately, now that we discover ACPI tables, the
OS is handling x2apic mode correctly.


With the above changes, I can now successfully boot to single user mode (simulator)
in the dump kernel. I may still have a few issues with lack-of EFI support but I think
I can work thru them. I did not see any problems on the simulator but might
hit them on hardware.


The only issue that still needs to be resolved (that I know of) is the memmap. The
E820 table in the boot_param block only supports 128 entries. Additional entries
are passed in the EFI memmap but this is missing w/o enabling EFI.

I suspect there are other problems waiting to be discovered but at least I'm making
progress.

Thanks for the help.

--- jack

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kdump/kexec on EFI-enabled x2apic platforms

[David N. Lombard]

On Mon, Mar 29, 2010 at 01:01:04PM -0700, Jack Steiner wrote:
> All -
> 
> I just started debugging kdump/kexec on our UV platform and
> have run into some problems. I suspect others have encountered these
> same or similar problems. Any help would be appreciated.
> 
> 
> 
> Our platform uses EFI boot. It is Nehalem based & has a large number of cpus.
> The BIOS enables x2apic mode and the kernel runs with interrupt remapping enabled.
> Note that some apicids have more than 8 bits - x2apic mode is required.

What are you using as a boot loader?  I've tested kexec with elilo 3.8 and
got good results.  But, that was early last year, so I'll need to try it out
with production hardware and the current firmware and sw stack.  I'll try to
get to that this week, and let you know the results.

-- 
David N. Lombard, Intel, Irvine, CA
I do not speak for Intel Corporation; all comments are strictly my own.

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kdump/kexec on EFI-enabled x2apic platforms

[Jack Steiner]

On Wed, Mar 31, 2010 at 03:24:50PM -0700, David N. Lombard wrote:
> On Mon, Mar 29, 2010 at 01:01:04PM -0700, Jack Steiner wrote:
> > All -
> > 
> > I just started debugging kdump/kexec on our UV platform and
> > have run into some problems. I suspect others have encountered these
> > same or similar problems. Any help would be appreciated.
> > 
> > 
> > 
> > Our platform uses EFI boot. It is Nehalem based & has a large number of cpus.
> > The BIOS enables x2apic mode and the kernel runs with interrupt remapping enabled.
> > Note that some apicids have more than 8 bits - x2apic mode is required.
> 
> What are you using as a boot loader?  I've tested kexec with elilo 3.8 and
> got good results.  But, that was early last year, so I'll need to try it out
> with production hardware and the current firmware and sw stack.  I'll try to
> get to that this week, and let you know the results.

We use:
	elilo-3.12-0.3.6

AFAICT, the following are the problem areas:

	- nehalem platform using a BIOS that puts RSDP at a location that
	  is not scanned by acpi_find_root_pointer() looking for ACPI tables

	  	[    0.000000] ACPI: RSDP 000000007b7fe014 00024 (v02 INTEL )
		[    0.000000] ACPI: XSDT 000000007b7fe120 0005C (v01 INTEL  TIANO    00000000      01000013)
		[    0.000000] ACPI: FACP 000000007b7fc000 000F4 (v03 INTEL  TIANO    00000000 MSFT 01000013)
		...

	  Booting with EFI works because ACPI tables are pointed to by the EFI tables. However,
	  the kdump kernel does not enable EFI and the OS uses an alternate method
	  to locate the ACPI tables. The alternate method fails.

	  	NOTE: with a hacked BIOS that puts the RSDP at 0x9ec00, ACPI tables
		are correctly discovered & booting the kdump kernel is successful (except
		that some memory is missing (see next item)).


	- number of memory blocks exceeds the size supported by the E820 table. (I'm
	  still learning - memory is missing from the kdump kernel memory maps. Does
	  this affect the memory dumpped from the kernel that crashed???)


	- x2apic enabled by the BIOS & interrupt remapping enabled by the OS (this
	  may no longer be an issue but caused problems earlier)



Appreciate the help....

--- jack

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec