Firmware assisted dump support in dracut

Firmware assisted dump support in dracut

[Mahesh J Salgaonkar]

Hi Harald,

Last year I worked on adding Firmware assisted dump (fadump) support in
PowerLinux (ppc64). This feature is now accepted upstream Linux kernel and the
documentation is available at http://lwn.net/Articles/488132/

Like kdump, fadump also exports the memory dump through /proc/vmcore in ELF
format. This enables us to reuse the existing kdump infrastructure for dump
capture and filtering. However, unlike kdump, fadump does not use kexec-based
approach, instead it depends on Power firmware to preserve the memory dump and
reboot into new kernel. This is what happens in fadump after crash:

1. At the crash, kernel informs power firmware that kernel has crashed.
2. Firmware takes the control and reboots the entire system preserving only the
memory (resets all other devices).
3. The reboot follows the normal booting process (non-kexec).
4. The boot loader loads the default kernel and initrd from /boot

I am working on integrating fadump with existing kdump infrastructure. The
current kdump infrastructure builds a separate initrd (whenever there is a
change detected in kdump config file /etc/kdump.conf) which then gets loaded
into memory by kexec tool for use by kdump kernel. But, in the fadump approach,
the second kernel (after crash) always use the default (OS built) initramfs.
Hence, to support fadump, change is required to introduce dump capturing steps
in default initramfs itself. Hence the possible approaches I am looking into
are:

1. Rebuild the default initramfs every time when there is a change detected in
kdump config file (/etc/kdump.conf)
   This approach would modify existing initramfs in place. I did work on this
   approach by enhancing mkdumprd (tool from kexec-tools package) to extract
   and rebuild default initramfs with dump capturing steps.  After discussing
   with Vivek Goyal, he suggested that better approach to add code to dracut
   for rebuilding boot initramfs instead of enhancing mkdumprd.  This means
   introducing a new dracut module that will be responsible for introducing
   dump capture steps inside the rebuilt initramfs by pulling required modules
   e.g. ssh, nfs etc. depending on /etc/kdump.conf

   Now the question is whether it is possible for dracut to rebuild boot
   initramfs in place? if Yes, is there any issues in rebuilding of boot
   initramfs everytime when there is a change in /etc/kdump.conf?

2. Make dracut tool fadump aware and it will build fadump aware initrd (default
OS initrd) during kernel install. Once built, this initrd should never be
rebuilt again. Which means the default initramfs will contain vmcore capture
steps. This approach would pack all possible dracut modules (nfs, ssh, bonding
etc.) so that the default initrd will be capable of supporting all possible
dump target. Hence this approach will bloat the initramfs. On my Power test
system, the size of the default initrd generated without any changes is 16M.
With the above approach this size jumps to 27M.

Since both the above approaches would require changes to dracut package, I would
like to know your views on above approaches.

Thanks,
-Mahesh.
-- 
Mahesh J Salgaonkar

Re: Firmware assisted dump support in dracut

[Harald Hoyer]

Am 28.11.2012 17:12, schrieb Mahesh J Salgaonkar:
> Hi Harald,
> 
> Last year I worked on adding Firmware assisted dump (fadump) support in
> PowerLinux (ppc64). This feature is now accepted upstream Linux kernel and the
> documentation is available at http://lwn.net/Articles/488132/
> 
> Like kdump, fadump also exports the memory dump through /proc/vmcore in ELF
> format. This enables us to reuse the existing kdump infrastructure for dump
> capture and filtering. However, unlike kdump, fadump does not use kexec-based
> approach, instead it depends on Power firmware to preserve the memory dump and
> reboot into new kernel. This is what happens in fadump after crash:
> 
> 1. At the crash, kernel informs power firmware that kernel has crashed.
> 2. Firmware takes the control and reboots the entire system preserving only the
> memory (resets all other devices).
> 3. The reboot follows the normal booting process (non-kexec).
> 4. The boot loader loads the default kernel and initrd from /boot
> 
> I am working on integrating fadump with existing kdump infrastructure. The
> current kdump infrastructure builds a separate initrd (whenever there is a
> change detected in kdump config file /etc/kdump.conf) which then gets loaded
> into memory by kexec tool for use by kdump kernel. But, in the fadump approach,
> the second kernel (after crash) always use the default (OS built) initramfs.
> Hence, to support fadump, change is required to introduce dump capturing steps
> in default initramfs itself. Hence the possible approaches I am looking into
> are:
> 
> 1. Rebuild the default initramfs every time when there is a change detected in
> kdump config file (/etc/kdump.conf)
>    This approach would modify existing initramfs in place. I did work on this
>    approach by enhancing mkdumprd (tool from kexec-tools package) to extract
>    and rebuild default initramfs with dump capturing steps.  After discussing
>    with Vivek Goyal, he suggested that better approach to add code to dracut
>    for rebuilding boot initramfs instead of enhancing mkdumprd.  This means
>    introducing a new dracut module that will be responsible for introducing
>    dump capture steps inside the rebuilt initramfs by pulling required modules
>    e.g. ssh, nfs etc. depending on /etc/kdump.conf
> 
>    Now the question is whether it is possible for dracut to rebuild boot
>    initramfs in place? if Yes, is there any issues in rebuilding of boot
>    initramfs everytime when there is a change in /etc/kdump.conf?

dracut is called by /sbin/new-kernel-package. dracut is not a service, which is
run on every boot. So, no, dracut can't watch files and build an initramfs on
itsself.

Of course such a service could be created and create the initramfs on shutdown,
if any of the files, which should be watched, have changed.

This of course is a dangerous automatism, which might as well lead to a
unbootable system. Further steps (with grubby) would have to make sure, the last
bootable entry is still there, along with the last kernel version.

I was already thinking about creating such a monster, but this would involve to
reinvent the current /sbin/new-kernel-package / grubby infrastructure, which has
to be done anyway in the next years. The first step was to create a common
configuration file format for all the bootloaders, which they parse and display.

http://harald.fedorapeople.org/downloads/boot-unification.pdf
A simple boot manager parsing the config layout as a reference implementation:
http://freedesktop.org/wiki/Software/gummiboot

Next step is to patch grub2 to parse those config files and display the menu
entries.

Next step is to enhance /sbin/new-kernel-package and obsolete grubby.

Then, I think we are ready to create such a service.

> 
> 2. Make dracut tool fadump aware and it will build fadump aware initrd (default
> OS initrd) during kernel install. Once built, this initrd should never be
> rebuilt again. Which means the default initramfs will contain vmcore capture
> steps. This approach would pack all possible dracut modules (nfs, ssh, bonding
> etc.) so that the default initrd will be capable of supporting all possible
> dump target. Hence this approach will bloat the initramfs. On my Power test
> system, the size of the default initrd generated without any changes is 16M.
> With the above approach this size jumps to 27M.

Yeah, this is crazy.

> 
> Since both the above approaches would require changes to dracut package, I would
> like to know your views on above approaches.
> 
> Thanks,
> -Mahesh.
> 

Re: Firmware assisted dump support in dracut

[Vivek Goyal]

On Thu, Nov 29, 2012 at 11:14:48AM +0100, Harald Hoyer wrote:

[..]
> >    Now the question is whether it is possible for dracut to rebuild boot
> >    initramfs in place? if Yes, is there any issues in rebuilding of boot
> >    initramfs everytime when there is a change in /etc/kdump.conf?
> 
> dracut is called by /sbin/new-kernel-package. dracut is not a service, which is
> run on every boot. So, no, dracut can't watch files and build an initramfs on
> itsself.
> 
> Of course such a service could be created and create the initramfs on shutdown,
> if any of the files, which should be watched, have changed.
> 
> This of course is a dangerous automatism, which might as well lead to a
> unbootable system. Further steps (with grubby) would have to make sure, the last
> bootable entry is still there, along with the last kernel version.
> 
> I was already thinking about creating such a monster, but this would involve to
> reinvent the current /sbin/new-kernel-package / grubby infrastructure, which has
> to be done anyway in the next years. The first step was to create a common
> configuration file format for all the bootloaders, which they parse and display.
> 
> http://harald.fedorapeople.org/downloads/boot-unification.pdf
> A simple boot manager parsing the config layout as a reference implementation:
> http://freedesktop.org/wiki/Software/gummiboot
> 
> Next step is to patch grub2 to parse those config files and display the menu
> entries.
> 
> Next step is to enhance /sbin/new-kernel-package and obsolete grubby.
> 
> Then, I think we are ready to create such a service.

Once we create the infrastructure where we can regenerate the
initramfs, then kdump service should be able to detect change in
files and call relevant hook (new-kernel-package or whatever).

So important point here seems to be that yes we will regenrate initramfs
but we need to do it in such a way so that we retain the old bootable
kernel/initramfs around in case new initramfs fails to boot.

So this is like installing new kernel execpt the fact that there is no
new kernel. It is old kernel but new initramfs. 

> 
> > 
> > 2. Make dracut tool fadump aware and it will build fadump aware initrd (default
> > OS initrd) during kernel install. Once built, this initrd should never be
> > rebuilt again. Which means the default initramfs will contain vmcore capture
> > steps. This approach would pack all possible dracut modules (nfs, ssh, bonding
> > etc.) so that the default initrd will be capable of supporting all possible
> > dump target. Hence this approach will bloat the initramfs. On my Power test
> > system, the size of the default initrd generated without any changes is 16M.
> > With the above approach this size jumps to 27M.
> 
> Yeah, this is crazy.

In the long term being able to regenerate initramfs in response to kdump
configuration changes should prove to be a better way of doing things.

Thanks
Vivek

Re: Firmware assisted dump support in dracut

[Mahesh Jagannath Salgaonkar]

On 11/29/2012 03:44 PM, Harald Hoyer wrote:
> Am 28.11.2012 17:12, schrieb Mahesh J Salgaonkar:
>> Hi Harald,
>>
>> Last year I worked on adding Firmware assisted dump (fadump) support in
>> PowerLinux (ppc64). This feature is now accepted upstream Linux kernel and the
>> documentation is available at http://lwn.net/Articles/488132/
>>
>> Like kdump, fadump also exports the memory dump through /proc/vmcore in ELF
>> format. This enables us to reuse the existing kdump infrastructure for dump
>> capture and filtering. However, unlike kdump, fadump does not use kexec-based
>> approach, instead it depends on Power firmware to preserve the memory dump and
>> reboot into new kernel. This is what happens in fadump after crash:
>>
>> 1. At the crash, kernel informs power firmware that kernel has crashed.
>> 2. Firmware takes the control and reboots the entire system preserving only the
>> memory (resets all other devices).
>> 3. The reboot follows the normal booting process (non-kexec).
>> 4. The boot loader loads the default kernel and initrd from /boot
>>
>> I am working on integrating fadump with existing kdump infrastructure. The
>> current kdump infrastructure builds a separate initrd (whenever there is a
>> change detected in kdump config file /etc/kdump.conf) which then gets loaded
>> into memory by kexec tool for use by kdump kernel. But, in the fadump approach,
>> the second kernel (after crash) always use the default (OS built) initramfs.
>> Hence, to support fadump, change is required to introduce dump capturing steps
>> in default initramfs itself. Hence the possible approaches I am looking into
>> are:
>>
>> 1. Rebuild the default initramfs every time when there is a change detected in
>> kdump config file (/etc/kdump.conf)
>>    This approach would modify existing initramfs in place. I did work on this
>>    approach by enhancing mkdumprd (tool from kexec-tools package) to extract
>>    and rebuild default initramfs with dump capturing steps.  After discussing
>>    with Vivek Goyal, he suggested that better approach to add code to dracut
>>    for rebuilding boot initramfs instead of enhancing mkdumprd.  This means
>>    introducing a new dracut module that will be responsible for introducing
>>    dump capture steps inside the rebuilt initramfs by pulling required modules
>>    e.g. ssh, nfs etc. depending on /etc/kdump.conf
>>
>>    Now the question is whether it is possible for dracut to rebuild boot
>>    initramfs in place? if Yes, is there any issues in rebuilding of boot
>>    initramfs everytime when there is a change in /etc/kdump.conf?
> 
> dracut is called by /sbin/new-kernel-package. dracut is not a service, which is
> run on every boot. So, no, dracut can't watch files and build an initramfs on
> itsself.
> 

Normally user would restart kdump service after changing kdump config
file. Idea is to invoke dracut through kdump service, as it already
watches and detect changes in kdump config files.

I was wondering if dracut allow us to insert/install additional dracut
modules to already existing initramfs? OR does it rebuild initramfs all
over again?

> Of course such a service could be created and create the initramfs on shutdown,
> if any of the files, which should be watched, have changed.
> 
> This of course is a dangerous automatism, which might as well lead to a
> unbootable system. Further steps (with grubby) would have to make sure, the last
> bootable entry is still there, along with the last kernel version.
> 
> I was already thinking about creating such a monster, but this would involve to
> reinvent the current /sbin/new-kernel-package / grubby infrastructure, which has
> to be done anyway in the next years. The first step was to create a common
> configuration file format for all the bootloaders, which they parse and display.
> 
> http://harald.fedorapeople.org/downloads/boot-unification.pdf
> A simple boot manager parsing the config layout as a reference implementation:
> http://freedesktop.org/wiki/Software/gummiboot
> 
> Next step is to patch grub2 to parse those config files and display the menu
> entries.

I think once such a infrastructure exists, things would be much easier.
Also, would it be possible to make such a infrastructure OS crash aware,
so that it can load different initramfs in crash scenario and default
initramfs in a normal boot process. I am not sure how practical this
idea would be to implement.

> 
> Next step is to enhance /sbin/new-kernel-package and obsolete grubby.
> 
> Then, I think we are ready to create such a service.
> 
>>
>> 2. Make dracut tool fadump aware and it will build fadump aware initrd (default
>> OS initrd) during kernel install. Once built, this initrd should never be
>> rebuilt again. Which means the default initramfs will contain vmcore capture
>> steps. This approach would pack all possible dracut modules (nfs, ssh, bonding
>> etc.) so that the default initrd will be capable of supporting all possible
>> dump target. Hence this approach will bloat the initramfs. On my Power test
>> system, the size of the default initrd generated without any changes is 16M.
>> With the above approach this size jumps to 27M.
> 
> Yeah, this is crazy.
> 
>>
>> Since both the above approaches would require changes to dracut package, I would
>> like to know your views on above approaches.
>>
>> Thanks,
>> -Mahesh.
>>
> 
> --
> To unsubscribe from this list: send the line "unsubscribe initramfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Firmware assisted dump support in dracut

[Mahesh Jagannath Salgaonkar]

On 12/06/2012 11:51 AM, Mahesh Jagannath Salgaonkar wrote:
> On 11/29/2012 03:44 PM, Harald Hoyer wrote:
>> Am 28.11.2012 17:12, schrieb Mahesh J Salgaonkar:
>>> Hi Harald,
>>>
>>> Last year I worked on adding Firmware assisted dump (fadump) support in
>>> PowerLinux (ppc64). This feature is now accepted upstream Linux kernel and the
>>> documentation is available at http://lwn.net/Articles/488132/
>>>
>>> Like kdump, fadump also exports the memory dump through /proc/vmcore in ELF
>>> format. This enables us to reuse the existing kdump infrastructure for dump
>>> capture and filtering. However, unlike kdump, fadump does not use kexec-based
>>> approach, instead it depends on Power firmware to preserve the memory dump and
>>> reboot into new kernel. This is what happens in fadump after crash:
>>>
>>> 1. At the crash, kernel informs power firmware that kernel has crashed.
>>> 2. Firmware takes the control and reboots the entire system preserving only the
>>> memory (resets all other devices).
>>> 3. The reboot follows the normal booting process (non-kexec).
>>> 4. The boot loader loads the default kernel and initrd from /boot
>>>
>>> I am working on integrating fadump with existing kdump infrastructure. The
>>> current kdump infrastructure builds a separate initrd (whenever there is a
>>> change detected in kdump config file /etc/kdump.conf) which then gets loaded
>>> into memory by kexec tool for use by kdump kernel. But, in the fadump approach,
>>> the second kernel (after crash) always use the default (OS built) initramfs.
>>> Hence, to support fadump, change is required to introduce dump capturing steps
>>> in default initramfs itself. Hence the possible approaches I am looking into
>>> are:
>>>
>>> 1. Rebuild the default initramfs every time when there is a change detected in
>>> kdump config file (/etc/kdump.conf)
>>>    This approach would modify existing initramfs in place. I did work on this
>>>    approach by enhancing mkdumprd (tool from kexec-tools package) to extract
>>>    and rebuild default initramfs with dump capturing steps.  After discussing
>>>    with Vivek Goyal, he suggested that better approach to add code to dracut
>>>    for rebuilding boot initramfs instead of enhancing mkdumprd.  This means
>>>    introducing a new dracut module that will be responsible for introducing
>>>    dump capture steps inside the rebuilt initramfs by pulling required modules
>>>    e.g. ssh, nfs etc. depending on /etc/kdump.conf
>>>
>>>    Now the question is whether it is possible for dracut to rebuild boot
>>>    initramfs in place? if Yes, is there any issues in rebuilding of boot
>>>    initramfs everytime when there is a change in /etc/kdump.conf?
>>
>> dracut is called by /sbin/new-kernel-package. dracut is not a service, which is
>> run on every boot. So, no, dracut can't watch files and build an initramfs on
>> itsself.
>>
> 
> Normally user would restart kdump service after changing kdump config
> file. Idea is to invoke dracut through kdump service, as it already
> watches and detect changes in kdump config files.
> 
> I was wondering if dracut allow us to insert/install additional dracut
> modules to already existing initramfs? OR does it rebuild initramfs all
> over again?

Sorry for restarting this discussion very late. I would like to know how
safe is to rebuild kernel's default (boot) initramfs for an already
installed kernel ?

Also, Does dracut provides any of following mechanism?
a) Mechanism where dracut can detect what options were used during first
build for a given (exsisting) initramfs. (This mechanism may help one to
regenerate similar initramfs with additional dracut modules.)

OR

b) Add additional modules to an existing initramfs instead of
regenerating again. Is this possible at all?

Thanks,
-Mahesh.

> 
>> Of course such a service could be created and create the initramfs on shutdown,
>> if any of the files, which should be watched, have changed.
>>
>> This of course is a dangerous automatism, which might as well lead to a
>> unbootable system. Further steps (with grubby) would have to make sure, the last
>> bootable entry is still there, along with the last kernel version.
>>
>> I was already thinking about creating such a monster, but this would involve to
>> reinvent the current /sbin/new-kernel-package / grubby infrastructure, which has
>> to be done anyway in the next years. The first step was to create a common
>> configuration file format for all the bootloaders, which they parse and display.
>>
>> http://harald.fedorapeople.org/downloads/boot-unification.pdf
>> A simple boot manager parsing the config layout as a reference implementation:
>> http://freedesktop.org/wiki/Software/gummiboot
>>
>> Next step is to patch grub2 to parse those config files and display the menu
>> entries.
> 
> I think once such a infrastructure exists, things would be much easier.
> Also, would it be possible to make such a infrastructure OS crash aware,
> so that it can load different initramfs in crash scenario and default
> initramfs in a normal boot process. I am not sure how practical this
> idea would be to implement.
> 
>>
>> Next step is to enhance /sbin/new-kernel-package and obsolete grubby.
>>
>> Then, I think we are ready to create such a service.
>>
>>>
>>> 2. Make dracut tool fadump aware and it will build fadump aware initrd (default
>>> OS initrd) during kernel install. Once built, this initrd should never be
>>> rebuilt again. Which means the default initramfs will contain vmcore capture
>>> steps. This approach would pack all possible dracut modules (nfs, ssh, bonding
>>> etc.) so that the default initrd will be capable of supporting all possible
>>> dump target. Hence this approach will bloat the initramfs. On my Power test
>>> system, the size of the default initrd generated without any changes is 16M.
>>> With the above approach this size jumps to 27M.
>>
>> Yeah, this is crazy.
>>
>>>
>>> Since both the above approaches would require changes to dracut package, I would
>>> like to know your views on above approaches.
>>>
>>> Thanks,
>>> -Mahesh.
>>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe initramfs" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 
> --
> To unsubscribe from this list: send the line "unsubscribe initramfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Firmware assisted dump support in dracut

[Harald Hoyer]

On 05/30/2013 07:17 AM, Mahesh Jagannath Salgaonkar wrote:
> On 12/06/2012 11:51 AM, Mahesh Jagannath Salgaonkar wrote:
>> On 11/29/2012 03:44 PM, Harald Hoyer wrote:
>>> Am 28.11.2012 17:12, schrieb Mahesh J Salgaonkar:
>>>> Hi Harald,
>>>>
>>>> Last year I worked on adding Firmware assisted dump (fadump) support in
>>>> PowerLinux (ppc64). This feature is now accepted upstream Linux kernel and the
>>>> documentation is available at http://lwn.net/Articles/488132/
>>>>
>>>> Like kdump, fadump also exports the memory dump through /proc/vmcore in ELF
>>>> format. This enables us to reuse the existing kdump infrastructure for dump
>>>> capture and filtering. However, unlike kdump, fadump does not use kexec-based
>>>> approach, instead it depends on Power firmware to preserve the memory dump and
>>>> reboot into new kernel. This is what happens in fadump after crash:
>>>>
>>>> 1. At the crash, kernel informs power firmware that kernel has crashed.
>>>> 2. Firmware takes the control and reboots the entire system preserving only the
>>>> memory (resets all other devices).
>>>> 3. The reboot follows the normal booting process (non-kexec).
>>>> 4. The boot loader loads the default kernel and initrd from /boot
>>>>
>>>> I am working on integrating fadump with existing kdump infrastructure. The
>>>> current kdump infrastructure builds a separate initrd (whenever there is a
>>>> change detected in kdump config file /etc/kdump.conf) which then gets loaded
>>>> into memory by kexec tool for use by kdump kernel. But, in the fadump approach,
>>>> the second kernel (after crash) always use the default (OS built) initramfs.
>>>> Hence, to support fadump, change is required to introduce dump capturing steps
>>>> in default initramfs itself. Hence the possible approaches I am looking into
>>>> are:
>>>>
>>>> 1. Rebuild the default initramfs every time when there is a change detected in
>>>> kdump config file (/etc/kdump.conf)
>>>>    This approach would modify existing initramfs in place. I did work on this
>>>>    approach by enhancing mkdumprd (tool from kexec-tools package) to extract
>>>>    and rebuild default initramfs with dump capturing steps.  After discussing
>>>>    with Vivek Goyal, he suggested that better approach to add code to dracut
>>>>    for rebuilding boot initramfs instead of enhancing mkdumprd.  This means
>>>>    introducing a new dracut module that will be responsible for introducing
>>>>    dump capture steps inside the rebuilt initramfs by pulling required modules
>>>>    e.g. ssh, nfs etc. depending on /etc/kdump.conf
>>>>
>>>>    Now the question is whether it is possible for dracut to rebuild boot
>>>>    initramfs in place? if Yes, is there any issues in rebuilding of boot
>>>>    initramfs everytime when there is a change in /etc/kdump.conf?
>>>
>>> dracut is called by /sbin/new-kernel-package. dracut is not a service, which is
>>> run on every boot. So, no, dracut can't watch files and build an initramfs on
>>> itsself.
>>>
>>
>> Normally user would restart kdump service after changing kdump config
>> file. Idea is to invoke dracut through kdump service, as it already
>> watches and detect changes in kdump config files.
>>
>> I was wondering if dracut allow us to insert/install additional dracut
>> modules to already existing initramfs? OR does it rebuild initramfs all
>> over again?
> 
> Sorry for restarting this discussion very late. I would like to know how
> safe is to rebuild kernel's default (boot) initramfs for an already
> installed kernel ?
> 
> Also, Does dracut provides any of following mechanism?
> a) Mechanism where dracut can detect what options were used during first
> build for a given (exsisting) initramfs. (This mechanism may help one to
> regenerate similar initramfs with additional dracut modules.)

currently dracut only stores which modules were used to generate the image in
usr/lib/dracut/modules.txt

But yes, you are right. Would be nice to save all the options and have a
mechanism to regenerate it with those.

> 
> OR
> 
> b) Add additional modules to an existing initramfs instead of
> regenerating again. Is this possible at all?

You can always generate a second cpio image with additional files and either
concatenate both images or specify those two images in the bootloader.

OR

unpack the initramfs, add the file, (maybe run depmod, if kernel driver added),
and repack.

Re: Firmware assisted dump support in dracut

[Vivek Goyal]

On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:

[..]
> > Sorry for restarting this discussion very late. I would like to know how
> > safe is to rebuild kernel's default (boot) initramfs for an already
> > installed kernel ?
> > 
> > Also, Does dracut provides any of following mechanism?
> > a) Mechanism where dracut can detect what options were used during first
> > build for a given (exsisting) initramfs. (This mechanism may help one to
> > regenerate similar initramfs with additional dracut modules.)
> 
> currently dracut only stores which modules were used to generate the image in
> usr/lib/dracut/modules.txt
> 
> But yes, you are right. Would be nice to save all the options and have a
> mechanism to regenerate it with those.
> 

I am not sure how well it will work in the context of kdump.  kdump
options and original options might conflict. kdump might drop some of
its own cmdline options in initramfs which might not make any sense
in regular boot.

Kdump will specify additional mount points. IIUC, then we will try
to bring up those targets in regular boot too and mount at user
specified mount point. And later init services might get confused
when respective daemon tries to bring up same targets again.

Thanks
Vivek

Re: Firmware assisted dump support in dracut

[Mahesh Jagannath Salgaonkar]

On 05/30/2013 07:53 PM, Vivek Goyal wrote:
> On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:
> 
> [..]
>>> Sorry for restarting this discussion very late. I would like to know how
>>> safe is to rebuild kernel's default (boot) initramfs for an already
>>> installed kernel ?
>>>
>>> Also, Does dracut provides any of following mechanism?
>>> a) Mechanism where dracut can detect what options were used during first
>>> build for a given (exsisting) initramfs. (This mechanism may help one to
>>> regenerate similar initramfs with additional dracut modules.)
>>
>> currently dracut only stores which modules were used to generate the image in
>> usr/lib/dracut/modules.txt
>>
>> But yes, you are right. Would be nice to save all the options and have a
>> mechanism to regenerate it with those.
>>
> 
> I am not sure how well it will work in the context of kdump.  kdump
> options and original options might conflict. kdump might drop some of
> its own cmdline options in initramfs which might not make any sense
> in regular boot.
> 
> Kdump will specify additional mount points. IIUC, then we will try
> to bring up those targets in regular boot too and mount at user
> specified mount point. And later init services might get confused
> when respective daemon tries to bring up same targets again.

I think we should be able differentiate between regular boot and boot
after crash by checking existence of '/proc/vmcore' file, and do the
kdump specific configs/mount if this file exists. In regular boot we can
mute the kdump code path.

> 
> Thanks
> Vivek
> 

Re: Firmware assisted dump support in dracut

[Mahesh Jagannath Salgaonkar]

On 05/30/2013 12:19 PM, Harald Hoyer wrote:
> On 05/30/2013 07:17 AM, Mahesh Jagannath Salgaonkar wrote:
>> On 12/06/2012 11:51 AM, Mahesh Jagannath Salgaonkar wrote:
>>> On 11/29/2012 03:44 PM, Harald Hoyer wrote:
>>>> Am 28.11.2012 17:12, schrieb Mahesh J Salgaonkar:
>>>>> Hi Harald,
>>>>>
>>>>> Last year I worked on adding Firmware assisted dump (fadump) support in
>>>>> PowerLinux (ppc64). This feature is now accepted upstream Linux kernel and the
>>>>> documentation is available at http://lwn.net/Articles/488132/
>>>>>
>>>>> Like kdump, fadump also exports the memory dump through /proc/vmcore in ELF
>>>>> format. This enables us to reuse the existing kdump infrastructure for dump
>>>>> capture and filtering. However, unlike kdump, fadump does not use kexec-based
>>>>> approach, instead it depends on Power firmware to preserve the memory dump and
>>>>> reboot into new kernel. This is what happens in fadump after crash:
>>>>>
>>>>> 1. At the crash, kernel informs power firmware that kernel has crashed.
>>>>> 2. Firmware takes the control and reboots the entire system preserving only the
>>>>> memory (resets all other devices).
>>>>> 3. The reboot follows the normal booting process (non-kexec).
>>>>> 4. The boot loader loads the default kernel and initrd from /boot
>>>>>
>>>>> I am working on integrating fadump with existing kdump infrastructure. The
>>>>> current kdump infrastructure builds a separate initrd (whenever there is a
>>>>> change detected in kdump config file /etc/kdump.conf) which then gets loaded
>>>>> into memory by kexec tool for use by kdump kernel. But, in the fadump approach,
>>>>> the second kernel (after crash) always use the default (OS built) initramfs.
>>>>> Hence, to support fadump, change is required to introduce dump capturing steps
>>>>> in default initramfs itself. Hence the possible approaches I am looking into
>>>>> are:
>>>>>
>>>>> 1. Rebuild the default initramfs every time when there is a change detected in
>>>>> kdump config file (/etc/kdump.conf)
>>>>>    This approach would modify existing initramfs in place. I did work on this
>>>>>    approach by enhancing mkdumprd (tool from kexec-tools package) to extract
>>>>>    and rebuild default initramfs with dump capturing steps.  After discussing
>>>>>    with Vivek Goyal, he suggested that better approach to add code to dracut
>>>>>    for rebuilding boot initramfs instead of enhancing mkdumprd.  This means
>>>>>    introducing a new dracut module that will be responsible for introducing
>>>>>    dump capture steps inside the rebuilt initramfs by pulling required modules
>>>>>    e.g. ssh, nfs etc. depending on /etc/kdump.conf
>>>>>
>>>>>    Now the question is whether it is possible for dracut to rebuild boot
>>>>>    initramfs in place? if Yes, is there any issues in rebuilding of boot
>>>>>    initramfs everytime when there is a change in /etc/kdump.conf?
>>>>
>>>> dracut is called by /sbin/new-kernel-package. dracut is not a service, which is
>>>> run on every boot. So, no, dracut can't watch files and build an initramfs on
>>>> itsself.
>>>>
>>>
>>> Normally user would restart kdump service after changing kdump config
>>> file. Idea is to invoke dracut through kdump service, as it already
>>> watches and detect changes in kdump config files.
>>>
>>> I was wondering if dracut allow us to insert/install additional dracut
>>> modules to already existing initramfs? OR does it rebuild initramfs all
>>> over again?
>>
>> Sorry for restarting this discussion very late. I would like to know how
>> safe is to rebuild kernel's default (boot) initramfs for an already
>> installed kernel ?
>>
>> Also, Does dracut provides any of following mechanism?
>> a) Mechanism where dracut can detect what options were used during first
>> build for a given (exsisting) initramfs. (This mechanism may help one to
>> regenerate similar initramfs with additional dracut modules.)
> 
> currently dracut only stores which modules were used to generate the image in
> usr/lib/dracut/modules.txt
> 
> But yes, you are right. Would be nice to save all the options and have a
> mechanism to regenerate it with those.
> 
>>
>> OR
>>
>> b) Add additional modules to an existing initramfs instead of
>> regenerating again. Is this possible at all?
> 
> You can always generate a second cpio image with additional files and either
> concatenate both images or specify those two images in the bootloader.

Is it possible to specify two images to bootloader for a single kernel
entry? Does grub2 support it?

> 
> OR
> 
> unpack the initramfs, add the file, (maybe run depmod, if kernel driver added),
> and repack.
> 

Re: Firmware assisted dump support in dracut

[Vivek Goyal]

On Tue, Jun 04, 2013 at 01:50:19PM +0530, Mahesh Jagannath Salgaonkar wrote:
> On 05/30/2013 07:53 PM, Vivek Goyal wrote:
> > On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:
> > 
> > [..]
> >>> Sorry for restarting this discussion very late. I would like to know how
> >>> safe is to rebuild kernel's default (boot) initramfs for an already
> >>> installed kernel ?
> >>>
> >>> Also, Does dracut provides any of following mechanism?
> >>> a) Mechanism where dracut can detect what options were used during first
> >>> build for a given (exsisting) initramfs. (This mechanism may help one to
> >>> regenerate similar initramfs with additional dracut modules.)
> >>
> >> currently dracut only stores which modules were used to generate the image in
> >> usr/lib/dracut/modules.txt
> >>
> >> But yes, you are right. Would be nice to save all the options and have a
> >> mechanism to regenerate it with those.
> >>
> > 
> > I am not sure how well it will work in the context of kdump.  kdump
> > options and original options might conflict. kdump might drop some of
> > its own cmdline options in initramfs which might not make any sense
> > in regular boot.
> > 
> > Kdump will specify additional mount points. IIUC, then we will try
> > to bring up those targets in regular boot too and mount at user
> > specified mount point. And later init services might get confused
> > when respective daemon tries to bring up same targets again.
> 
> I think we should be able differentiate between regular boot and boot
> after crash by checking existence of '/proc/vmcore' file, and do the
> kdump specific configs/mount if this file exists. In regular boot we can
> mute the kdump code path.

dracut takes --mount option and that option doesn not differentiate
between environment.

One can always create new options to mount certain things only in
kdump environment. I am not sure how much sense does it make to try
to special case everything related to kdump in dracut.

Thanks
Vivek

Re: Firmware assisted dump support in dracut

[Vivek Goyal]

On Tue, Jun 04, 2013 at 01:50:19PM +0530, Mahesh Jagannath Salgaonkar wrote:
> On 05/30/2013 07:53 PM, Vivek Goyal wrote:
> > On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:
> > 
> > [..]
> >>> Sorry for restarting this discussion very late. I would like to know how
> >>> safe is to rebuild kernel's default (boot) initramfs for an already
> >>> installed kernel ?
> >>>
> >>> Also, Does dracut provides any of following mechanism?
> >>> a) Mechanism where dracut can detect what options were used during first
> >>> build for a given (exsisting) initramfs. (This mechanism may help one to
> >>> regenerate similar initramfs with additional dracut modules.)
> >>
> >> currently dracut only stores which modules were used to generate the image in
> >> usr/lib/dracut/modules.txt
> >>
> >> But yes, you are right. Would be nice to save all the options and have a
> >> mechanism to regenerate it with those.
> >>
> > 
> > I am not sure how well it will work in the context of kdump.  kdump
> > options and original options might conflict. kdump might drop some of
> > its own cmdline options in initramfs which might not make any sense
> > in regular boot.
> > 
> > Kdump will specify additional mount points. IIUC, then we will try
> > to bring up those targets in regular boot too and mount at user
> > specified mount point. And later init services might get confused
> > when respective daemon tries to bring up same targets again.
> 
> I think we should be able differentiate between regular boot and boot
> after crash by checking existence of '/proc/vmcore' file, and do the
> kdump specific configs/mount if this file exists. In regular boot we can
> mute the kdump code path.

Also what about kernel command line? Kernel command line can be very
specific to kdump environment. It is atleast in x86. I am hoping that's
not the case with firmware assisted dump.

Also can you point me to some writeup of fadump again. I want to refresh
what fadump is doing and how firmware is assisting and how does it make
better than regular kdump. 

I suspect it is very similar to s390 firmware dump (zfcpdump). Does
s390 kdump or s390 zfcpdump reboot after saving dump or they have
mechanism to continue to boot and hotplug memory after saving dump. I
am wondering how do they do it.

CCing michael.

Vivek

Re: Firmware assisted dump support in dracut

[Vivek Goyal]

On Tue, Jun 04, 2013 at 10:14:03AM -0400, Vivek Goyal wrote:
> On Tue, Jun 04, 2013 at 01:50:19PM +0530, Mahesh Jagannath Salgaonkar wrote:
> > On 05/30/2013 07:53 PM, Vivek Goyal wrote:
> > > On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:
> > > 
> > > [..]
> > >>> Sorry for restarting this discussion very late. I would like to know how
> > >>> safe is to rebuild kernel's default (boot) initramfs for an already
> > >>> installed kernel ?
> > >>>
> > >>> Also, Does dracut provides any of following mechanism?
> > >>> a) Mechanism where dracut can detect what options were used during first
> > >>> build for a given (exsisting) initramfs. (This mechanism may help one to
> > >>> regenerate similar initramfs with additional dracut modules.)
> > >>
> > >> currently dracut only stores which modules were used to generate the image in
> > >> usr/lib/dracut/modules.txt
> > >>
> > >> But yes, you are right. Would be nice to save all the options and have a
> > >> mechanism to regenerate it with those.
> > >>
> > > 
> > > I am not sure how well it will work in the context of kdump.  kdump
> > > options and original options might conflict. kdump might drop some of
> > > its own cmdline options in initramfs which might not make any sense
> > > in regular boot.
> > > 
> > > Kdump will specify additional mount points. IIUC, then we will try
> > > to bring up those targets in regular boot too and mount at user
> > > specified mount point. And later init services might get confused
> > > when respective daemon tries to bring up same targets again.
> > 
> > I think we should be able differentiate between regular boot and boot
> > after crash by checking existence of '/proc/vmcore' file, and do the
> > kdump specific configs/mount if this file exists. In regular boot we can
> > mute the kdump code path.
> 
> Also what about kernel command line? Kernel command line can be very
> specific to kdump environment. It is atleast in x86. I am hoping that's
> not the case with firmware assisted dump.

If first kernel and kdump kernel command line are same in fadump and
boot environment is same, I think it is worth exploring the option
of saving kdump from kdump service when system boots back. That will
do away with any need to tweak initramfs.

It just requires more memory to be reserved so that system can boot
, mount root, run services and then save dump. I am hoping that firmware
can set aside enough memory to (1G?), to copy the memory contents and
boot the kernel in that 1G.

Also does fadump work well with new mmap() interface /proc/vmcore is
going to support. Patches are in -mm now. s390 zfcpdump is going to
face some issues and michael is working on resolving these. I suspect
that fadump will run into similar issues if firmware is copying memory
contents somewhere else.

Thanks
Vivek

Re: Firmware assisted dump support in dracut

[Michael Holzheu]

On Tue, 4 Jun 2013 10:14:03 -0400
Vivek Goyal <vgoyal-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> On Tue, Jun 04, 2013 at 01:50:19PM +0530, Mahesh Jagannath Salgaonkar
> wrote:
> > On 05/30/2013 07:53 PM, Vivek Goyal wrote:
> > > On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:
> > > 
> > > [..]
> > >>> Sorry for restarting this discussion very late. I would like to
> > >>> know how safe is to rebuild kernel's default (boot) initramfs
> > >>> for an already installed kernel ?
> > >>>
> > >>> Also, Does dracut provides any of following mechanism?
> > >>> a) Mechanism where dracut can detect what options were used
> > >>> during first build for a given (exsisting) initramfs. (This
> > >>> mechanism may help one to regenerate similar initramfs with
> > >>> additional dracut modules.)
> > >>
> > >> currently dracut only stores which modules were used to generate
> > >> the image in usr/lib/dracut/modules.txt
> > >>
> > >> But yes, you are right. Would be nice to save all the options
> > >> and have a mechanism to regenerate it with those.
> > >>
> > > 
> > > I am not sure how well it will work in the context of kdump.
> > > kdump options and original options might conflict. kdump might
> > > drop some of its own cmdline options in initramfs which might not
> > > make any sense in regular boot.
> > > 
> > > Kdump will specify additional mount points. IIUC, then we will try
> > > to bring up those targets in regular boot too and mount at user
> > > specified mount point. And later init services might get confused
> > > when respective daemon tries to bring up same targets again.
> > 
> > I think we should be able differentiate between regular boot and
> > boot after crash by checking existence of '/proc/vmcore' file, and
> > do the kdump specific configs/mount if this file exists. In regular
> > boot we can mute the kdump code path.
> 
> Also what about kernel command line? Kernel command line can be very
> specific to kdump environment. It is atleast in x86. I am hoping
> that's not the case with firmware assisted dump.
> 
> Also can you point me to some writeup of fadump again. I want to
> refresh what fadump is doing and how firmware is assisting and how
> does it make better than regular kdump. 
> 
> I suspect it is very similar to s390 firmware dump (zfcpdump). Does
> s390 kdump or s390 zfcpdump reboot after saving dump or they have
> mechanism to continue to boot and hotplug memory after saving dump. I
> am wondering how do they do it.

Hello Vivek,

Currently s390 zfcpdump and kdump both reboot the production system
after dump.

But we also had discussed the option to write the dump in the background
while the production system already starts.

Best Regards,
Michael

Re: Firmware assisted dump support in dracut

[Andrey Borzenkov]

В Tue, 04 Jun 2013 14:04:51 +0530
Mahesh Jagannath Salgaonkar <mahesh-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> пишет:

> On 05/30/2013 12:19 PM, Harald Hoyer wrote:
> > On 05/30/2013 07:17 AM, Mahesh Jagannath Salgaonkar wrote:
> >> On 12/06/2012 11:51 AM, Mahesh Jagannath Salgaonkar wrote:
> >>> On 11/29/2012 03:44 PM, Harald Hoyer wrote:
> >>>> Am 28.11.2012 17:12, schrieb Mahesh J Salgaonkar:
> >>>>> Hi Harald,
> >>>>>
> >>>>> Last year I worked on adding Firmware assisted dump (fadump) support in
> >>>>> PowerLinux (ppc64). This feature is now accepted upstream Linux kernel and the
> >>>>> documentation is available at http://lwn.net/Articles/488132/
> >>>>>
> >>>>> Like kdump, fadump also exports the memory dump through /proc/vmcore in ELF
> >>>>> format. This enables us to reuse the existing kdump infrastructure for dump
> >>>>> capture and filtering. However, unlike kdump, fadump does not use kexec-based
> >>>>> approach, instead it depends on Power firmware to preserve the memory dump and
> >>>>> reboot into new kernel. This is what happens in fadump after crash:
> >>>>>
> >>>>> 1. At the crash, kernel informs power firmware that kernel has crashed.
> >>>>> 2. Firmware takes the control and reboots the entire system preserving only the
> >>>>> memory (resets all other devices).
> >>>>> 3. The reboot follows the normal booting process (non-kexec).
> >>>>> 4. The boot loader loads the default kernel and initrd from /boot
> >>>>>
> >>>>> I am working on integrating fadump with existing kdump infrastructure. The
> >>>>> current kdump infrastructure builds a separate initrd (whenever there is a
> >>>>> change detected in kdump config file /etc/kdump.conf) which then gets loaded
> >>>>> into memory by kexec tool for use by kdump kernel. But, in the fadump approach,
> >>>>> the second kernel (after crash) always use the default (OS built) initramfs.
> >>>>> Hence, to support fadump, change is required to introduce dump capturing steps
> >>>>> in default initramfs itself. Hence the possible approaches I am looking into
> >>>>> are:
> >>>>>
> >>>>> 1. Rebuild the default initramfs every time when there is a change detected in
> >>>>> kdump config file (/etc/kdump.conf)
> >>>>>    This approach would modify existing initramfs in place. I did work on this
> >>>>>    approach by enhancing mkdumprd (tool from kexec-tools package) to extract
> >>>>>    and rebuild default initramfs with dump capturing steps.  After discussing
> >>>>>    with Vivek Goyal, he suggested that better approach to add code to dracut
> >>>>>    for rebuilding boot initramfs instead of enhancing mkdumprd.  This means
> >>>>>    introducing a new dracut module that will be responsible for introducing
> >>>>>    dump capture steps inside the rebuilt initramfs by pulling required modules
> >>>>>    e.g. ssh, nfs etc. depending on /etc/kdump.conf
> >>>>>
> >>>>>    Now the question is whether it is possible for dracut to rebuild boot
> >>>>>    initramfs in place? if Yes, is there any issues in rebuilding of boot
> >>>>>    initramfs everytime when there is a change in /etc/kdump.conf?
> >>>>
> >>>> dracut is called by /sbin/new-kernel-package. dracut is not a service, which is
> >>>> run on every boot. So, no, dracut can't watch files and build an initramfs on
> >>>> itsself.
> >>>>
> >>>
> >>> Normally user would restart kdump service after changing kdump config
> >>> file. Idea is to invoke dracut through kdump service, as it already
> >>> watches and detect changes in kdump config files.
> >>>
> >>> I was wondering if dracut allow us to insert/install additional dracut
> >>> modules to already existing initramfs? OR does it rebuild initramfs all
> >>> over again?
> >>
> >> Sorry for restarting this discussion very late. I would like to know how
> >> safe is to rebuild kernel's default (boot) initramfs for an already
> >> installed kernel ?
> >>
> >> Also, Does dracut provides any of following mechanism?
> >> a) Mechanism where dracut can detect what options were used during first
> >> build for a given (exsisting) initramfs. (This mechanism may help one to
> >> regenerate similar initramfs with additional dracut modules.)
> > 
> > currently dracut only stores which modules were used to generate the image in
> > usr/lib/dracut/modules.txt
> > 
> > But yes, you are right. Would be nice to save all the options and have a
> > mechanism to regenerate it with those.
> > 
> >>
> >> OR
> >>
> >> b) Add additional modules to an existing initramfs instead of
> >> regenerating again. Is this possible at all?
> > 
> > You can always generate a second cpio image with additional files and either
> > concatenate both images or specify those two images in the bootloader.
> 
> Is it possible to specify two images to bootloader for a single kernel
> entry? Does grub2 support it?
> 


Yes, it does. You can use

initrd file1 file2 ...

And it will concatenate files. It aligns files on 4 bytes boundary, not
sure whether it can be a problem.

> > 
> > OR
> > 
> > unpack the initramfs, add the file, (maybe run depmod, if kernel driver added),
> > and repack.
> > 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe initramfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Firmware assisted dump support in dracut

[Mahesh Jagannath Salgaonkar]

On 06/04/2013 07:44 PM, Vivek Goyal wrote:
> On Tue, Jun 04, 2013 at 01:50:19PM +0530, Mahesh Jagannath Salgaonkar wrote:
>> On 05/30/2013 07:53 PM, Vivek Goyal wrote:
>>> On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:
>>>
>>> [..]
>>>>> Sorry for restarting this discussion very late. I would like to know how
>>>>> safe is to rebuild kernel's default (boot) initramfs for an already
>>>>> installed kernel ?
>>>>>
>>>>> Also, Does dracut provides any of following mechanism?
>>>>> a) Mechanism where dracut can detect what options were used during first
>>>>> build for a given (exsisting) initramfs. (This mechanism may help one to
>>>>> regenerate similar initramfs with additional dracut modules.)
>>>>
>>>> currently dracut only stores which modules were used to generate the image in
>>>> usr/lib/dracut/modules.txt
>>>>
>>>> But yes, you are right. Would be nice to save all the options and have a
>>>> mechanism to regenerate it with those.
>>>>
>>>
>>> I am not sure how well it will work in the context of kdump.  kdump
>>> options and original options might conflict. kdump might drop some of
>>> its own cmdline options in initramfs which might not make any sense
>>> in regular boot.
>>>
>>> Kdump will specify additional mount points. IIUC, then we will try
>>> to bring up those targets in regular boot too and mount at user
>>> specified mount point. And later init services might get confused
>>> when respective daemon tries to bring up same targets again.
>>
>> I think we should be able differentiate between regular boot and boot
>> after crash by checking existence of '/proc/vmcore' file, and do the
>> kdump specific configs/mount if this file exists. In regular boot we can
>> mute the kdump code path
> 
> Also what about kernel command line? Kernel command line can be very
> specific to kdump environment. It is atleast in x86. I am hoping that's
> not the case with firmware assisted dump.

In fadump case, kernel command line is same as first kernel.

> 
> Also can you point me to some writeup of fadump again. I want to refresh
> what fadump is doing and how firmware is assisting and how does it make
> better than regular kdump.

It's at Documentation/powerpc/firmware-assisted-dump.txt in upstream
kernel.

> 
> I suspect it is very similar to s390 firmware dump (zfcpdump). Does
> s390 kdump or s390 zfcpdump reboot after saving dump or they have
> mechanism to continue to boot and hotplug memory after saving dump. I
> am wondering how do they do it.
> 
> CCing michael.
> 
> Vivek
> 

Re: Firmware assisted dump support in dracut

[Mahesh Jagannath Salgaonkar]

On 06/04/2013 07:51 PM, Vivek Goyal wrote:
> On Tue, Jun 04, 2013 at 10:14:03AM -0400, Vivek Goyal wrote:
>> On Tue, Jun 04, 2013 at 01:50:19PM +0530, Mahesh Jagannath Salgaonkar wrote:
>>> On 05/30/2013 07:53 PM, Vivek Goyal wrote:
>>>> On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:
>>>>
>>>> [..]
>>>>>> Sorry for restarting this discussion very late. I would like to know how
>>>>>> safe is to rebuild kernel's default (boot) initramfs for an already
>>>>>> installed kernel ?
>>>>>>
>>>>>> Also, Does dracut provides any of following mechanism?
>>>>>> a) Mechanism where dracut can detect what options were used during first
>>>>>> build for a given (exsisting) initramfs. (This mechanism may help one to
>>>>>> regenerate similar initramfs with additional dracut modules.)
>>>>>
>>>>> currently dracut only stores which modules were used to generate the image in
>>>>> usr/lib/dracut/modules.txt
>>>>>
>>>>> But yes, you are right. Would be nice to save all the options and have a
>>>>> mechanism to regenerate it with those.
>>>>>
>>>>
>>>> I am not sure how well it will work in the context of kdump.  kdump
>>>> options and original options might conflict. kdump might drop some of
>>>> its own cmdline options in initramfs which might not make any sense
>>>> in regular boot.
>>>>
>>>> Kdump will specify additional mount points. IIUC, then we will try
>>>> to bring up those targets in regular boot too and mount at user
>>>> specified mount point. And later init services might get confused
>>>> when respective daemon tries to bring up same targets again.
>>>
>>> I think we should be able differentiate between regular boot and boot
>>> after crash by checking existence of '/proc/vmcore' file, and do the
>>> kdump specific configs/mount if this file exists. In regular boot we can
>>> mute the kdump code path.
>>
>> Also what about kernel command line? Kernel command line can be very
>> specific to kdump environment. It is atleast in x86. I am hoping that's
>> not the case with firmware assisted dump.
> 
> If first kernel and kdump kernel command line are same in fadump and
> boot environment is same, I think it is worth exploring the option
> of saving kdump from kdump service when system boots back. That will
> do away with any need to tweak initramfs.

This needs larger memory to be reserved. We may need to make sure that
kdump service is invoked at very early stage before other services kicks
in which may start requesting for memory and run into OOM issue. If we
can make sure that no other services are invoked when kdump service is
saving dump then it will help to fix the size of reserved memory. The
systemd infrastructure provides aggressive parallelization capabilities.
I am not expert on systemd but will need to explore whether with systemd
is this possible.

> 
> It just requires more memory to be reserved so that system can boot
> , mount root, run services and then save dump. I am hoping that firmware
> can set aside enough memory to (1G?), to copy the memory contents and
> boot the kernel in that 1G.

It's the OS who reserves the memory from the available system RAM and
provides it to firmware to use. This means production kernel will have
set aside this memory for firmware to use.

> 
> Also does fadump work well with new mmap() interface /proc/vmcore is
> going to support. Patches are in -mm now. s390 zfcpdump is going to
> face some issues and michael is working on resolving these. I suspect
> that fadump will run into similar issues if firmware is copying memory
> contents somewhere else.

Will need to check this.

> 
> Thanks
> Vivek
> 

Re: Firmware assisted dump support in dracut

[Vivek Goyal]

On Wed, Jun 05, 2013 at 07:14:27PM +0530, Mahesh Jagannath Salgaonkar wrote:
> On 06/04/2013 07:51 PM, Vivek Goyal wrote:
> > On Tue, Jun 04, 2013 at 10:14:03AM -0400, Vivek Goyal wrote:
> >> On Tue, Jun 04, 2013 at 01:50:19PM +0530, Mahesh Jagannath Salgaonkar wrote:
> >>> On 05/30/2013 07:53 PM, Vivek Goyal wrote:
> >>>> On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:
> >>>>
> >>>> [..]
> >>>>>> Sorry for restarting this discussion very late. I would like to know how
> >>>>>> safe is to rebuild kernel's default (boot) initramfs for an already
> >>>>>> installed kernel ?
> >>>>>>
> >>>>>> Also, Does dracut provides any of following mechanism?
> >>>>>> a) Mechanism where dracut can detect what options were used during first
> >>>>>> build for a given (exsisting) initramfs. (This mechanism may help one to
> >>>>>> regenerate similar initramfs with additional dracut modules.)
> >>>>>
> >>>>> currently dracut only stores which modules were used to generate the image in
> >>>>> usr/lib/dracut/modules.txt
> >>>>>
> >>>>> But yes, you are right. Would be nice to save all the options and have a
> >>>>> mechanism to regenerate it with those.
> >>>>>
> >>>>
> >>>> I am not sure how well it will work in the context of kdump.  kdump
> >>>> options and original options might conflict. kdump might drop some of
> >>>> its own cmdline options in initramfs which might not make any sense
> >>>> in regular boot.
> >>>>
> >>>> Kdump will specify additional mount points. IIUC, then we will try
> >>>> to bring up those targets in regular boot too and mount at user
> >>>> specified mount point. And later init services might get confused
> >>>> when respective daemon tries to bring up same targets again.
> >>>
> >>> I think we should be able differentiate between regular boot and boot
> >>> after crash by checking existence of '/proc/vmcore' file, and do the
> >>> kdump specific configs/mount if this file exists. In regular boot we can
> >>> mute the kdump code path.
> >>
> >> Also what about kernel command line? Kernel command line can be very
> >> specific to kdump environment. It is atleast in x86. I am hoping that's
> >> not the case with firmware assisted dump.
> > 
> > If first kernel and kdump kernel command line are same in fadump and
> > boot environment is same, I think it is worth exploring the option
> > of saving kdump from kdump service when system boots back. That will
> > do away with any need to tweak initramfs.
> 
> This needs larger memory to be reserved. We may need to make sure that
> kdump service is invoked at very early stage before other services kicks
> in which may start requesting for memory and run into OOM issue. If we
> can make sure that no other services are invoked when kdump service is
> saving dump then it will help to fix the size of reserved memory. The
> systemd infrastructure provides aggressive parallelization capabilities.
> I am not expert on systemd but will need to explore whether with systemd
> is this possible.

I think we can create a new service for saving dump and make some of
the systemd services dependent on start of this service. 

Can't integrate with existing kdump service as that one can start
later and can take time in regenerating initramfs and holding back other
services for that duration will not make sense.

By creating a seprate service, delay will be introduced only when there
s dump to be saved. Before this we will have bring up all networking
and storage in the system so that we can save dump to intended target.

I think same service can be used to save dump from inside initramfs
too. (As systemd is now started inside initramfs too). 

Other appraoch is dumping through initramfs and if you do things
conditionally (if dump is available), then any dracut parameter
used by kdump will have to have a conditional version too. Something
like

mount=
mount.kdump_only=

mount.kdump_only will kick in only dump is available. Also we need to
make sure that effects of all kdump_only options are undone before
root is mounted or we create nice handover mechanism to systemd. I think
there handover mechanism already between initramfs and systemd for complex
storage targets like multipathd.

I will let harald comment on what does he think of kdump_only variant
options.

Thanks
Vivek

Re: Firmware assisted dump support in dracut

[Vivek Goyal]

On Wed, Jun 05, 2013 at 10:55:04AM -0400, Vivek Goyal wrote:

[..]
> > > If first kernel and kdump kernel command line are same in fadump and
> > > boot environment is same, I think it is worth exploring the option
> > > of saving kdump from kdump service when system boots back. That will
> > > do away with any need to tweak initramfs.
> > 
> > This needs larger memory to be reserved. We may need to make sure that
> > kdump service is invoked at very early stage before other services kicks
> > in which may start requesting for memory and run into OOM issue. If we
> > can make sure that no other services are invoked when kdump service is
> > saving dump then it will help to fix the size of reserved memory. The
> > systemd infrastructure provides aggressive parallelization capabilities.
> > I am not expert on systemd but will need to explore whether with systemd
> > is this possible.
> 
> I think we can create a new service for saving dump and make some of
> the systemd services dependent on start of this service. 
> 
> Can't integrate with existing kdump service as that one can start
> later and can take time in regenerating initramfs and holding back other
> services for that duration will not make sense.
> 
> By creating a seprate service, delay will be introduced only when there
> s dump to be saved. Before this we will have bring up all networking
> and storage in the system so that we can save dump to intended target.
> 
> I think same service can be used to save dump from inside initramfs
> too. (As systemd is now started inside initramfs too). 
> 
> Other appraoch is dumping through initramfs and if you do things
> conditionally (if dump is available), then any dracut parameter
> used by kdump will have to have a conditional version too. Something
> like
> 
> mount=
> mount.kdump_only=
> 
> mount.kdump_only will kick in only dump is available. Also we need to
> make sure that effects of all kdump_only options are undone before
> root is mounted or we create nice handover mechanism to systemd. I think
> there handover mechanism already between initramfs and systemd for complex
> storage targets like multipathd.
> 
> I will let harald comment on what does he think of kdump_only variant
> options.

Thinking more about it, I think from kdump point of view, it is better
to be able to dump from initrafs. We don't want different mechanism
for different architecture which will result in more confusion and
possibly code duplication.

Thanks
Vivek

Re: Firmware assisted dump support in dracut

[Mahesh Jagannath Salgaonkar]

On 06/05/2013 09:23 PM, Vivek Goyal wrote:
> On Wed, Jun 05, 2013 at 10:55:04AM -0400, Vivek Goyal wrote:
> 
> [..]
>>>> If first kernel and kdump kernel command line are same in fadump and
>>>> boot environment is same, I think it is worth exploring the option
>>>> of saving kdump from kdump service when system boots back. That will
>>>> do away with any need to tweak initramfs.
>>>
>>> This needs larger memory to be reserved. We may need to make sure that
>>> kdump service is invoked at very early stage before other services kicks
>>> in which may start requesting for memory and run into OOM issue. If we
>>> can make sure that no other services are invoked when kdump service is
>>> saving dump then it will help to fix the size of reserved memory. The
>>> systemd infrastructure provides aggressive parallelization capabilities.
>>> I am not expert on systemd but will need to explore whether with systemd
>>> is this possible.
>>
>> I think we can create a new service for saving dump and make some of
>> the systemd services dependent on start of this service. 
>>
>> Can't integrate with existing kdump service as that one can start
>> later and can take time in regenerating initramfs and holding back other
>> services for that duration will not make sense.
>>
>> By creating a seprate service, delay will be introduced only when there
>> s dump to be saved. Before this we will have bring up all networking
>> and storage in the system so that we can save dump to intended target.
>>
>> I think same service can be used to save dump from inside initramfs
>> too. (As systemd is now started inside initramfs too). 
>>
>> Other appraoch is dumping through initramfs and if you do things
>> conditionally (if dump is available), then any dracut parameter
>> used by kdump will have to have a conditional version too. Something
>> like
>>
>> mount=
>> mount.kdump_only=

Agree. The kdump_only options can be used to store the kdump related
required configs (e.g. lvm configs, network configs etc.) in a separate
directory and add a pre-udev hook that can check if dump is available
and move these configs to their original paths so that the normal
initramfs flow will do its job as per configs.

>>
>> mount.kdump_only will kick in only dump is available. Also we need to
>> make sure that effects of all kdump_only options are undone before
>> root is mounted or we create nice handover mechanism to systemd. I think

Agree.

>> there handover mechanism already between initramfs and systemd for complex
>> storage targets like multipathd.
>>
>> I will let harald comment on what does he think of kdump_only variant
>> options.
> 
> Thinking more about it, I think from kdump point of view, it is better
> to be able to dump from initrafs. We don't want different mechanism
> for different architecture which will result in more confusion and
> possibly code duplication.
> 

Agree. And if we rely on dracut to rebuild the initramfs (with original
dracut arguments) we can be sure that we wont run into issues like
unbootable system.

Thanks,
-Mahesh.