[RFD] efi assisted kdump

[RFD] efi assisted kdump

[Dave Young]

Hi,

Kdump has several limitations currently such as kdump kernel reboot will bypass
device shutdown path so device drivers should reset during initialization.

* One of such problem we encounter now is the iommu
issue, 1st kernel's on fly DMA requests cause 2nd kernel hang. There's some effort
on this area, Zhenhua Li posted patches to resolve the intel iommu issue which is
under review. But that is just one case there's other possible problems in the future.

* There's no serial console on most machines in the market especially for desktop
machines and laptops. kms enabled kernel need drm layer driver for framebuffer
console, after kernel crashing we need a console to see the 2nd kernel output
ideally a serial console because we can not switch back to VGA mode.

ppc64 has a feature "firmware assisted kdump", see below documentation:
Documentation/powerpc/firmware-assisted-dump.txt

In case UEFI machines I wonder if we can do similar things, basic idea is doing
minimum thing based on original kdump process.

kernel reserve crashkernel memory during early boot
 -> user (kdump service) save necessary informations in some way so that second
    kernel boot can access ie. efi runtime variables or in reserved memory
    * crashkernel memory ranges infomation
    * collect infomations and save elf notes for 2nd kernel vmcore initialization
    -> crashing
       * call efi warm reboot
       -> 2nd kernel boot
          -> efi stub checking if this is a warm reboot and if this is a crashed boot
             -> if this is not a crashed boot then
                    do nothing more than original logic
                else
                    boot kernel with limited memory which was reserved in previous kernel
                    -> 2nd kernel boot
                       vmcore intializing

There's several things I need to ask for help from EFI gurus to see if it is doable:

* Make sure in case EFI warm reboot the memory of previous kernel can be retained.

* How to determine if kernel is boot with EFI warm reboot in stub

* What is the right way to pass informations from 1st kernel to 2nd kernel.
  Is it ok for saving something with efi variables? 

* Other possible problems what I missed

Thanks
Dave

[RFD] efi assisted kdump

[Dave Young]

Hi,

Kdump has several limitations currently such as kdump kernel reboot will bypass
device shutdown path so device drivers should reset during initialization.

* One of such problem we encounter now is the iommu
issue, 1st kernel's on fly DMA requests cause 2nd kernel hang. There's some effort
on this area, Zhenhua Li posted patches to resolve the intel iommu issue which is
under review. But that is just one case there's other possible problems in the future.

* There's no serial console on most machines in the market especially for desktop
machines and laptops. kms enabled kernel need drm layer driver for framebuffer
console, after kernel crashing we need a console to see the 2nd kernel output
ideally a serial console because we can not switch back to VGA mode.

ppc64 has a feature "firmware assisted kdump", see below documentation:
Documentation/powerpc/firmware-assisted-dump.txt

In case UEFI machines I wonder if we can do similar things, basic idea is doing
minimum thing based on original kdump process.

kernel reserve crashkernel memory during early boot
 -> user (kdump service) save necessary informations in some way so that second
    kernel boot can access ie. efi runtime variables or in reserved memory
    * crashkernel memory ranges infomation
    * collect infomations and save elf notes for 2nd kernel vmcore initialization
    -> crashing
       * call efi warm reboot
       -> 2nd kernel boot
          -> efi stub checking if this is a warm reboot and if this is a crashed boot
             -> if this is not a crashed boot then
                    do nothing more than original logic
                else
                    boot kernel with limited memory which was reserved in previous kernel
                    -> 2nd kernel boot
                       vmcore intializing

There's several things I need to ask for help from EFI gurus to see if it is doable:

* Make sure in case EFI warm reboot the memory of previous kernel can be retained.

* How to determine if kernel is boot with EFI warm reboot in stub

* What is the right way to pass informations from 1st kernel to 2nd kernel.
  Is it ok for saving something with efi variables? 

* Other possible problems what I missed

Thanks
Dave

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [RFD] efi assisted kdump

[Petr Tesarik]

On Sat, 24 Jan 2015 21:26:37 +0800
Dave Young <dyoung@redhat.com> wrote:

> Hi,
> 
> Kdump has several limitations currently such as kdump kernel reboot will bypass
> device shutdown path so device drivers should reset during initialization.
> 
>[...]
> 
> ppc64 has a feature "firmware assisted kdump", see below documentation:
> Documentation/powerpc/firmware-assisted-dump.txt

Hi Dave,

while I'm no expert on either UEFI or IBM POWER, I'd like to warn you
that fadump (firmware-assisted dump) on PPC is not quite optimal in its
current form. One of the things that have always irritated me are
excessive RAM requirements.

The problem is that there is only one reboot in fadump - after saving
the dump, the secondary kernel discards the saved area and continues
booting as usual. However, many kernel structuers must be already
allocated at that point, e.g. the memmap array(s), but they are sized
by the total RAM, not the limited amount available to the 2nd kernel.

This issue is not so bad on a typical IBM POWER system, because these
machines tend to be partitioned into many small LPARs, where each has
relatively little RAM. But on a machine with 6TB RAM, you'll have to
reserve approx. 84G merely for the memmap array on the secondary
kernel's boot, which looks just a bit wasteful...

Maybe the two approaches can be combined somehow, so that the
kdump system uses kexec() to transition into the production system, but
that only works if a normal kexec does not have any of the issues
you're trying to address.

Just my two cents,
Petr Tesarik

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [RFD] efi assisted kdump

[Vivek Goyal]

On Sat, Jan 24, 2015 at 09:26:37PM +0800, Dave Young wrote:
> Hi,
> 
> Kdump has several limitations currently such as kdump kernel reboot will bypass
> device shutdown path so device drivers should reset during initialization.
> 
> * One of such problem we encounter now is the iommu
> issue, 1st kernel's on fly DMA requests cause 2nd kernel hang. There's some effort
> on this area, Zhenhua Li posted patches to resolve the intel iommu issue which is
> under review. But that is just one case there's other possible problems in the future.
> 
> * There's no serial console on most machines in the market especially for desktop
> machines and laptops. kms enabled kernel need drm layer driver for framebuffer
> console, after kernel crashing we need a console to see the 2nd kernel output
> ideally a serial console because we can not switch back to VGA mode.
> 
> ppc64 has a feature "firmware assisted kdump", see below documentation:
> Documentation/powerpc/firmware-assisted-dump.txt
> 
> In case UEFI machines I wonder if we can do similar things, basic idea is doing
> minimum thing based on original kdump process.
> 
> kernel reserve crashkernel memory during early boot
>  -> user (kdump service) save necessary informations in some way so that second
>     kernel boot can access ie. efi runtime variables or in reserved memory
>     * crashkernel memory ranges infomation
>     * collect infomations and save elf notes for 2nd kernel vmcore initialization

I think anything related to vmcore initilization can be in memory
somewhere and should not be a problem.

But we do have to think about anything which is passed in bootparams or
command line to second kenrnel with current mechanism, how will it be
passed to second kernel.

Thanks
Vivek

Re: [RFD] efi assisted kdump

[Vivek Goyal]

On Sat, Jan 24, 2015 at 09:26:37PM +0800, Dave Young wrote:
> Hi,
> 
> Kdump has several limitations currently such as kdump kernel reboot will bypass
> device shutdown path so device drivers should reset during initialization.
> 
> * One of such problem we encounter now is the iommu
> issue, 1st kernel's on fly DMA requests cause 2nd kernel hang. There's some effort
> on this area, Zhenhua Li posted patches to resolve the intel iommu issue which is
> under review. But that is just one case there's other possible problems in the future.
> 
> * There's no serial console on most machines in the market especially for desktop
> machines and laptops. kms enabled kernel need drm layer driver for framebuffer
> console, after kernel crashing we need a console to see the 2nd kernel output
> ideally a serial console because we can not switch back to VGA mode.
> 
> ppc64 has a feature "firmware assisted kdump", see below documentation:
> Documentation/powerpc/firmware-assisted-dump.txt
> 
> In case UEFI machines I wonder if we can do similar things, basic idea is doing
> minimum thing based on original kdump process.
> 
> kernel reserve crashkernel memory during early boot
>  -> user (kdump service) save necessary informations in some way so that second
>     kernel boot can access ie. efi runtime variables or in reserved memory
>     * crashkernel memory ranges infomation
>     * collect infomations and save elf notes for 2nd kernel vmcore initialization

I think anything related to vmcore initilization can be in memory
somewhere and should not be a problem.

But we do have to think about anything which is passed in bootparams or
command line to second kenrnel with current mechanism, how will it be
passed to second kernel.

Thanks
Vivek

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [RFD] efi assisted kdump

[Matt Fleming]

On Sat, 24 Jan, at 09:26:37PM, Dave Young wrote:
> 
> There's several things I need to ask for help from EFI gurus to see if it is doable:
> 
> * Make sure in case EFI warm reboot the memory of previous kernel can be retained.
 
The only way to reserve memory across a reboot is with EFI capsules.

> * How to determine if kernel is boot with EFI warm reboot in stub

The presence of a capsule would indicate whether we were invoked from a
crash handler.
 
> * What is the right way to pass informations from 1st kernel to 2nd kernel.
>   Is it ok for saving something with efi variables? 

EFI capsules would the be most natural mechanism, but you could
conceivably do it with EFI variables (there's an upper limit on the size
of variable data, though).

> * Other possible problems what I missed

Support for runtime EFI capsules was rather spotty last time I looked.
Things may have moved on, but it's definitely something to watch out
for.

There have been patches on linux-efi in recent months for making use of
EFI capsules,

  https://lkml.kernel.org/r/1412692886-25306-1-git-send-email-matt-HNK1S37rvNbeXh+fF434Mdi2O/JbrIOy@public.gmane.org
  https://lkml.kernel.org/r/1414984030-13859-1-git-send-email-hock.leong.kweh-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

I could take a look at resubmitting the first series above if that would
be useful for doing kexec across a reset.

-- 
Matt Fleming, Intel Open Source Technology Center

Re: [RFD] efi assisted kdump

[Matt Fleming]

On Sat, 24 Jan, at 09:26:37PM, Dave Young wrote:
> 
> There's several things I need to ask for help from EFI gurus to see if it is doable:
> 
> * Make sure in case EFI warm reboot the memory of previous kernel can be retained.
 
The only way to reserve memory across a reboot is with EFI capsules.

> * How to determine if kernel is boot with EFI warm reboot in stub

The presence of a capsule would indicate whether we were invoked from a
crash handler.
 
> * What is the right way to pass informations from 1st kernel to 2nd kernel.
>   Is it ok for saving something with efi variables? 

EFI capsules would the be most natural mechanism, but you could
conceivably do it with EFI variables (there's an upper limit on the size
of variable data, though).

> * Other possible problems what I missed

Support for runtime EFI capsules was rather spotty last time I looked.
Things may have moved on, but it's definitely something to watch out
for.

There have been patches on linux-efi in recent months for making use of
EFI capsules,

  https://lkml.kernel.org/r/1412692886-25306-1-git-send-email-matt@console-pimps.org
  https://lkml.kernel.org/r/1414984030-13859-1-git-send-email-hock.leong.kweh@intel.com

I could take a look at resubmitting the first series above if that would
be useful for doing kexec across a reset.

-- 
Matt Fleming, Intel Open Source Technology Center

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [RFD] efi assisted kdump

[Dave Young]

Hi, Petr

On 01/24/15 at 05:03pm, Petr Tesarik wrote:
> On Sat, 24 Jan 2015 21:26:37 +0800
> Dave Young <dyoung@redhat.com> wrote:
> 
> > Hi,
> > 
> > Kdump has several limitations currently such as kdump kernel reboot will bypass
> > device shutdown path so device drivers should reset during initialization.
> > 
> >[...]
> > 
> > ppc64 has a feature "firmware assisted kdump", see below documentation:
> > Documentation/powerpc/firmware-assisted-dump.txt
> 
> Hi Dave,
> 
> while I'm no expert on either UEFI or IBM POWER, I'd like to warn you
> that fadump (firmware-assisted dump) on PPC is not quite optimal in its
> current form. One of the things that have always irritated me are
> excessive RAM requirements.
> 
> The problem is that there is only one reboot in fadump - after saving
> the dump, the secondary kernel discards the saved area and continues
> booting as usual. However, many kernel structuers must be already
> allocated at that point, e.g. the memmap array(s), but they are sized
> by the total RAM, not the limited amount available to the 2nd kernel.

Thanks for the comment.

I'm not sure I understand the RAM requirements you mentioned. I think you
are worrying about freeing oldmem to be used by 2nd kernel.

But we do not need to do same as power we can just reboot another time
as long as vmcore is saved. The main advantage is the capture kernel can
boot with all devices being reset. 

> 
> This issue is not so bad on a typical IBM POWER system, because these
> machines tend to be partitioned into many small LPARs, where each has
> relatively little RAM. But on a machine with 6TB RAM, you'll have to
> reserve approx. 84G merely for the memmap array on the secondary
> kernel's boot, which looks just a bit wasteful...
> 
> Maybe the two approaches can be combined somehow, so that the
> kdump system uses kexec() to transition into the production system, but
> that only works if a normal kexec does not have any of the issues
> you're trying to address.
> 
> Just my two cents,
> Petr Tesarik
> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [RFD] efi assisted kdump

[Dave Young]

On 01/26/15 at 09:08am, Vivek Goyal wrote:
> On Sat, Jan 24, 2015 at 09:26:37PM +0800, Dave Young wrote:
> > Hi,
> > 
> > Kdump has several limitations currently such as kdump kernel reboot will bypass
> > device shutdown path so device drivers should reset during initialization.
> > 
> > * One of such problem we encounter now is the iommu
> > issue, 1st kernel's on fly DMA requests cause 2nd kernel hang. There's some effort
> > on this area, Zhenhua Li posted patches to resolve the intel iommu issue which is
> > under review. But that is just one case there's other possible problems in the future.
> > 
> > * There's no serial console on most machines in the market especially for desktop
> > machines and laptops. kms enabled kernel need drm layer driver for framebuffer
> > console, after kernel crashing we need a console to see the 2nd kernel output
> > ideally a serial console because we can not switch back to VGA mode.
> > 
> > ppc64 has a feature "firmware assisted kdump", see below documentation:
> > Documentation/powerpc/firmware-assisted-dump.txt
> > 
> > In case UEFI machines I wonder if we can do similar things, basic idea is doing
> > minimum thing based on original kdump process.
> > 
> > kernel reserve crashkernel memory during early boot
> >  -> user (kdump service) save necessary informations in some way so that second
> >     kernel boot can access ie. efi runtime variables or in reserved memory
> >     * crashkernel memory ranges infomation
> >     * collect infomations and save elf notes for 2nd kernel vmcore initialization
> 
> I think anything related to vmcore initilization can be in memory
> somewhere and should not be a problem.
> 
> But we do have to think about anything which is passed in bootparams or
> command line to second kenrnel with current mechanism, how will it be
> passed to second kernel.

Hi, Vivek,

Since we are rebooting via efi maybe we can just choose another boot entry
in grub.cfg with the proper cmdline?

Thanks
Dave 

Re: [RFD] efi assisted kdump

[Dave Young]

On 01/26/15 at 09:08am, Vivek Goyal wrote:
> On Sat, Jan 24, 2015 at 09:26:37PM +0800, Dave Young wrote:
> > Hi,
> > 
> > Kdump has several limitations currently such as kdump kernel reboot will bypass
> > device shutdown path so device drivers should reset during initialization.
> > 
> > * One of such problem we encounter now is the iommu
> > issue, 1st kernel's on fly DMA requests cause 2nd kernel hang. There's some effort
> > on this area, Zhenhua Li posted patches to resolve the intel iommu issue which is
> > under review. But that is just one case there's other possible problems in the future.
> > 
> > * There's no serial console on most machines in the market especially for desktop
> > machines and laptops. kms enabled kernel need drm layer driver for framebuffer
> > console, after kernel crashing we need a console to see the 2nd kernel output
> > ideally a serial console because we can not switch back to VGA mode.
> > 
> > ppc64 has a feature "firmware assisted kdump", see below documentation:
> > Documentation/powerpc/firmware-assisted-dump.txt
> > 
> > In case UEFI machines I wonder if we can do similar things, basic idea is doing
> > minimum thing based on original kdump process.
> > 
> > kernel reserve crashkernel memory during early boot
> >  -> user (kdump service) save necessary informations in some way so that second
> >     kernel boot can access ie. efi runtime variables or in reserved memory
> >     * crashkernel memory ranges infomation
> >     * collect infomations and save elf notes for 2nd kernel vmcore initialization
> 
> I think anything related to vmcore initilization can be in memory
> somewhere and should not be a problem.
> 
> But we do have to think about anything which is passed in bootparams or
> command line to second kenrnel with current mechanism, how will it be
> passed to second kernel.

Hi, Vivek,

Since we are rebooting via efi maybe we can just choose another boot entry
in grub.cfg with the proper cmdline?

Thanks
Dave 

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [RFD] efi assisted kdump

[Dave Young]

Hi, Matt

Thanks for the feedback.

On 01/26/15 at 04:39pm, Matt Fleming wrote:
> On Sat, 24 Jan, at 09:26:37PM, Dave Young wrote:
> > 
> > There's several things I need to ask for help from EFI gurus to see if it is doable:
> > 
> > * Make sure in case EFI warm reboot the memory of previous kernel can be retained.
>  
> The only way to reserve memory across a reboot is with EFI capsules.

I know your capsules patchset, but I did not look into the details.

Do you means the old memory will be not reliable in case booting without capsules
even for warm reboot?

> 
> > * How to determine if kernel is boot with EFI warm reboot in stub
> 
> The presence of a capsule would indicate whether we were invoked from a
> crash handler.

Ok, another problem is for efi capsule handling, can we do it in efi stub, or I
in a efi application, which stage is the doable?

>  
> > * What is the right way to pass informations from 1st kernel to 2nd kernel.
> >   Is it ok for saving something with efi variables? 
> 
> EFI capsules would the be most natural mechanism, but you could
> conceivably do it with EFI variables (there's an upper limit on the size
> of variable data, though).

Will think about it after I checking the capsule machanisms details.

> 
> > * Other possible problems what I missed
> 
> Support for runtime EFI capsules was rather spotty last time I looked.
> Things may have moved on, but it's definitely something to watch out
> for.
> 
> There have been patches on linux-efi in recent months for making use of
> EFI capsules,
> 
>   https://lkml.kernel.org/r/1412692886-25306-1-git-send-email-matt-HNK1S37rvNbeXh+fF434Mdi2O/JbrIOy@public.gmane.org
>   https://lkml.kernel.org/r/1414984030-13859-1-git-send-email-hock.leong.kweh-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org
> 
> I could take a look at resubmitting the first series above if that would
> be useful for doing kexec across a reset.

Thanks a lot. I will take a look at how capsules works firstly.
Dave

Re: [RFD] efi assisted kdump

[Dave Young]

Hi, Matt

Thanks for the feedback.

On 01/26/15 at 04:39pm, Matt Fleming wrote:
> On Sat, 24 Jan, at 09:26:37PM, Dave Young wrote:
> > 
> > There's several things I need to ask for help from EFI gurus to see if it is doable:
> > 
> > * Make sure in case EFI warm reboot the memory of previous kernel can be retained.
>  
> The only way to reserve memory across a reboot is with EFI capsules.

I know your capsules patchset, but I did not look into the details.

Do you means the old memory will be not reliable in case booting without capsules
even for warm reboot?

> 
> > * How to determine if kernel is boot with EFI warm reboot in stub
> 
> The presence of a capsule would indicate whether we were invoked from a
> crash handler.

Ok, another problem is for efi capsule handling, can we do it in efi stub, or I
in a efi application, which stage is the doable?

>  
> > * What is the right way to pass informations from 1st kernel to 2nd kernel.
> >   Is it ok for saving something with efi variables? 
> 
> EFI capsules would the be most natural mechanism, but you could
> conceivably do it with EFI variables (there's an upper limit on the size
> of variable data, though).

Will think about it after I checking the capsule machanisms details.

> 
> > * Other possible problems what I missed
> 
> Support for runtime EFI capsules was rather spotty last time I looked.
> Things may have moved on, but it's definitely something to watch out
> for.
> 
> There have been patches on linux-efi in recent months for making use of
> EFI capsules,
> 
>   https://lkml.kernel.org/r/1412692886-25306-1-git-send-email-matt@console-pimps.org
>   https://lkml.kernel.org/r/1414984030-13859-1-git-send-email-hock.leong.kweh@intel.com
> 
> I could take a look at resubmitting the first series above if that would
> be useful for doing kexec across a reset.

Thanks a lot. I will take a look at how capsules works firstly.
Dave

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [RFD] efi assisted kdump

[Petr Tesarik]

On Tue, 27 Jan 2015 14:21:41 +0800
Dave Young <dyoung@redhat.com> wrote:

> Hi, Petr
> 
> On 01/24/15 at 05:03pm, Petr Tesarik wrote:
> > On Sat, 24 Jan 2015 21:26:37 +0800
> > Dave Young <dyoung@redhat.com> wrote:
> > 
> > > Hi,
> > > 
> > > Kdump has several limitations currently such as kdump kernel reboot will bypass
> > > device shutdown path so device drivers should reset during initialization.
> > > 
> > >[...]
> > > 
> > > ppc64 has a feature "firmware assisted kdump", see below documentation:
> > > Documentation/powerpc/firmware-assisted-dump.txt
> > 
> > Hi Dave,
> > 
> > while I'm no expert on either UEFI or IBM POWER, I'd like to warn you
> > that fadump (firmware-assisted dump) on PPC is not quite optimal in its
> > current form. One of the things that have always irritated me are
> > excessive RAM requirements.
> > 
> > The problem is that there is only one reboot in fadump - after saving
> > the dump, the secondary kernel discards the saved area and continues
> > booting as usual. However, many kernel structuers must be already
> > allocated at that point, e.g. the memmap array(s), but they are sized
> > by the total RAM, not the limited amount available to the 2nd kernel.
> 
> Thanks for the comment.
> 
> I'm not sure I understand the RAM requirements you mentioned. I think you
> are worrying about freeing oldmem to be used by 2nd kernel.
> 
> But we do not need to do same as power we can just reboot another time
> as long as vmcore is saved. The main advantage is the capture kernel can
> boot with all devices being reset. 

Yes, that's true. If you're fine with rebooting the machine twice (once
for the dump-taking and once for bringing up the normal system), then
there's no concern. The IBM folks apparently wanted to save some
downtime associated with going through the boot-up sequence. In fact,
you could kexec back into the normal system instead of rebooting, if
kexec can be used.

All I wanted to say is that fadump has its own issues, and we'd better
not repeat them with UEFI-based dumps. ;-)

Petr Tesarik

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec