[RFC] autokdump - automated kdump testsuite

[RFC] autokdump - automated kdump testsuite

[CAI Qian]

I plan to release an automated kdump testsuite that will be
focus on testing kernel and the crash utility. It should work
for all major distros since it will use none of distro-specific
stuff, and also support different arches including x86, ARM,
PPC64 and s390x.

It does the following:
1) check if there is a memory reserved for kdump. If not,
   reserve the memory and reboot the system.
2) once the system is back, load kexec on panic and
   prepare a separate initramfs that including needed
   modules to load a local filesystem and necessary utilities
   in order to analyse /proc/vmcore in the 2nd kernel.
3) trigger the system crash using methods like sysrq-c, NMI,
   and panic_on_hung_task etc.
4) in the 2nd kernel, mount a filesystem and use the crash
   utility to analyse /proc/vmcore. Then, gather the analyse
   logs, serial console output, dmesg etc into the filesystem.
5) reboot back into the 1st kernel.

implementation:
It will setup a daemon to handle reboots.

plan:
I might also to test the makedumpfile all together later.
   CAI Qian

Re: [RFC] autokdump - automated kdump testsuite

[Vivek Goyal]

On Fri, Sep 19, 2014 at 05:52:25AM -0400, CAI Qian wrote:
> I plan to release an automated kdump testsuite that will be

So will this be a standalone test suit? Can it be merged with
something already existing say, LTP.

> focus on testing kernel and the crash utility. It should work
> for all major distros since it will use none of distro-specific
> stuff, and also support different arches including x86, ARM,
> PPC64 and s390x.
> 
> It does the following:
> 1) check if there is a memory reserved for kdump. If not,
>    reserve the memory and reboot the system.
> 2) once the system is back, load kexec on panic and
>    prepare a separate initramfs that including needed
>    modules to load a local filesystem and necessary utilities

So you will write logic to prepare custom initramfs or will rely
on dracut or some other utility for that.

>    in order to analyse /proc/vmcore in the 2nd kernel.
> 3) trigger the system crash using methods like sysrq-c, NMI,
>    and panic_on_hung_task etc.
> 4) in the 2nd kernel, mount a filesystem and use the crash
>    utility to analyse /proc/vmcore. Then, gather the analyse
>    logs, serial console output, dmesg etc into the filesystem.

Why not save core and boot back in first kernel and then analyze. 

Trying to work directly with /proc/vmcore does not test makedumfile
which everybody uses. Also it will require more memory to be reserved
and packing crash and debug vmlinux into initramfs.

I think being able to test makedumpfile also is the key here.

Thanks
Vivek 
> 5) reboot back into the 1st kernel.
> 
> implementation:
> It will setup a daemon to handle reboots.
> 
> plan:
> I might also to test the makedumpfile all together later.
>    CAI Qian

Re: [RFC] autokdump - automated kdump testsuite

[Vivek Goyal]

On Fri, Sep 19, 2014 at 09:22:36AM -0400, Vivek Goyal wrote:
> On Fri, Sep 19, 2014 at 05:52:25AM -0400, CAI Qian wrote:
> > I plan to release an automated kdump testsuite that will be
> 
> So will this be a standalone test suit? Can it be merged with
> something already existing say, LTP.

Also how about calling it just kdumptest. autokdump feels more
like as if there have been improvements to kdump mechanism and
there is improved method named "autokdump".

kdumptest is more intuitive and it being "auto" is more or less
implied.

Thanks
Vivek

Re: [RFC] autokdump - automated kdump testsuite

[CAI Qian]

----- Original Message -----
> From: "Vivek Goyal" <vgoyal@redhat.com>
> To: "CAI Qian" <caiqian@redhat.com>
> Cc: "linux-kernel" <linux-kernel@vger.kernel.org>, "ltp-list" <ltp-list@lists.sourceforge.net>, "crash-utility"
> <crash-utility@redhat.com>, "kexec" <kexec@lists.infradead.org>, "kexec kdump redhat mailing list"
> <kexec-kdump-list@redhat.com>
> Sent: Friday, September 19, 2014 9:22:36 PM
> Subject: Re: [RFC] autokdump - automated kdump testsuite
> 
> On Fri, Sep 19, 2014 at 05:52:25AM -0400, CAI Qian wrote:
> > I plan to release an automated kdump testsuite that will be
> 
> So will this be a standalone test suit? Can it be merged with
> something already existing say, LTP.
Yes, it is likely to be standalone. It won't make use of the LTP
API, and the LTP kdump test suite is outdated, so there is no
benefit to continue working over there.
> 
> > focus on testing kernel and the crash utility. It should work
> > for all major distros since it will use none of distro-specific
> > stuff, and also support different arches including x86, ARM,
> > PPC64 and s390x.
> > 
> > It does the following:
> > 1) check if there is a memory reserved for kdump. If not,
> >    reserve the memory and reboot the system.
> > 2) once the system is back, load kexec on panic and
> >    prepare a separate initramfs that including needed
> >    modules to load a local filesystem and necessary utilities
> 
> So you will write logic to prepare custom initramfs or will rely
> on dracut or some other utility for that.
I'll probably prepare custom initramfs for the sake of simplicity.
> 
> >    in order to analyse /proc/vmcore in the 2nd kernel.
> > 3) trigger the system crash using methods like sysrq-c, NMI,
> >    and panic_on_hung_task etc.
> > 4) in the 2nd kernel, mount a filesystem and use the crash
> >    utility to analyse /proc/vmcore. Then, gather the analyse
> >    logs, serial console output, dmesg etc into the filesystem.
> 
> Why not save core and boot back in first kernel and then analyze.
> 
> Trying to work directly with /proc/vmcore does not test makedumfile
> which everybody uses. Also it will require more memory to be reserved
> and packing crash and debug vmlinux into initramfs.
The additional memory for vmlinux and the crash utility is predictable
and manageable, so it can just ask 256M memory reserved before running
the program. On the other hand, it is not usually feasible to ask
the systems under testing has enough available disk spaces bigger than
the memory size.
   CAI Qian
> 
> I think being able to test makedumpfile also is the key here.
> 
> Thanks
> Vivek
> > 5) reboot back into the 1st kernel.
> > 
> > implementation:
> > It will setup a daemon to handle reboots.
> > 
> > plan:
> > I might also to test the makedumpfile all together later.
> >    CAI Qian
> 

Re: [RFC] autokdump - automated kdump testsuite

[Vivek Goyal]

On Mon, Sep 22, 2014 at 09:00:00AM -0400, CAI Qian wrote:
> 
> 
> ----- Original Message -----
> > From: "Vivek Goyal" <vgoyal@redhat.com>
> > To: "CAI Qian" <caiqian@redhat.com>
> > Cc: "linux-kernel" <linux-kernel@vger.kernel.org>, "ltp-list" <ltp-list@lists.sourceforge.net>, "crash-utility"
> > <crash-utility@redhat.com>, "kexec" <kexec@lists.infradead.org>, "kexec kdump redhat mailing list"
> > <kexec-kdump-list@redhat.com>
> > Sent: Friday, September 19, 2014 9:22:36 PM
> > Subject: Re: [RFC] autokdump - automated kdump testsuite
> > 
> > On Fri, Sep 19, 2014 at 05:52:25AM -0400, CAI Qian wrote:
> > > I plan to release an automated kdump testsuite that will be
> > 
> > So will this be a standalone test suit? Can it be merged with
> > something already existing say, LTP.
> Yes, it is likely to be standalone. It won't make use of the LTP
> API, and the LTP kdump test suite is outdated, so there is no
> benefit to continue working over there.

So why make it standalone and not replace the old LTP kdump test suite
with this new one?

> > 
> > > focus on testing kernel and the crash utility. It should work
> > > for all major distros since it will use none of distro-specific
> > > stuff, and also support different arches including x86, ARM,
> > > PPC64 and s390x.
> > > 
> > > It does the following:
> > > 1) check if there is a memory reserved for kdump. If not,
> > >    reserve the memory and reboot the system.
> > > 2) once the system is back, load kexec on panic and
> > >    prepare a separate initramfs that including needed
> > >    modules to load a local filesystem and necessary utilities
> > 
> > So you will write logic to prepare custom initramfs or will rely
> > on dracut or some other utility for that.
> I'll probably prepare custom initramfs for the sake of simplicity.

Well, preparing custom initramfs will become very tricky. We used
to do that and finally we switched to dracut.

Why not simply let the respective service on the host do this job and
test only makes sure that kdump service is running. It feels little
out of place that a test is generating custom initramfs. 

> > 
> > >    in order to analyse /proc/vmcore in the 2nd kernel.
> > > 3) trigger the system crash using methods like sysrq-c, NMI,
> > >    and panic_on_hung_task etc.
> > > 4) in the 2nd kernel, mount a filesystem and use the crash
> > >    utility to analyse /proc/vmcore. Then, gather the analyse
> > >    logs, serial console output, dmesg etc into the filesystem.
> > 
> > Why not save core and boot back in first kernel and then analyze.
> > 
> > Trying to work directly with /proc/vmcore does not test makedumfile
> > which everybody uses. Also it will require more memory to be reserved
> > and packing crash and debug vmlinux into initramfs.
> The additional memory for vmlinux and the crash utility is predictable
> and manageable, so it can just ask 256M memory reserved before running
> the program. On the other hand, it is not usually feasible to ask
> the systems under testing has enough available disk spaces bigger than
> the memory size.

makedumpfile will reduce the vmcore file size to few hundreds of mega
bytes on most of the systems. Especially, this is just a test, so 
system will be lightly loaded and vmcore will be small after filtering.

If there is not enough space, test fails, period. I don't think there
is any need to try to circumvent that and try to run crash in initramfs.
And in the process we don't test makedumpfile which is very imporatnt
component of this whole process.

IMHO, just rely on "systemctl start kdump" to generate and load custom
initramfs and save filtered vmcore to root fs by default and alanyze
vmcore post reboot. That will keep things simple.

Thanks
Vivek

Re: [RFC] autokdump - automated kdump testsuite

[CAI Qian]

----- Original Message -----
> From: "Vivek Goyal" <vgoyal@redhat.com>
> To: "CAI Qian" <caiqian@redhat.com>
> Cc: "ltp-list" sourceforge.net>, "kexec kdump redhat mailing list" <kexec-kdump-list@redhat.com>,
> "kexec" <kexec@lists.infradead.org>, "linux-kernel" vger.kernel.org>, "crash-utility"
> redhat.com>
> Sent: Monday, September 22, 2014 10:47:13 PM
> Subject: Re: [RFC] autokdump - automated kdump testsuite
> 
> On Mon, Sep 22, 2014 at 09:00:00AM -0400, CAI Qian wrote:
> > 
> > 
> > ----- Original Message -----
> > > From: "Vivek Goyal" <vgoyal@redhat.com>
> > > To: "CAI Qian" <caiqian@redhat.com>
> > > Cc: "linux-kernel" vger.kernel.org>, "ltp-list"
> > > sourceforge.net>, "crash-utility"
> > > redhat.com>, "kexec" <kexec@lists.infradead.org>, "kexec
> > > kdump redhat mailing list"
> > > <kexec-kdump-list@redhat.com>
> > > Sent: Friday, September 19, 2014 9:22:36 PM
> > > Subject: Re: [RFC] autokdump - automated kdump testsuite
> > > 
> > > On Fri, Sep 19, 2014 at 05:52:25AM -0400, CAI Qian wrote:
> > > > I plan to release an automated kdump testsuite that will be
> > > 
> > > So will this be a standalone test suit? Can it be merged with
> > > something already existing say, LTP.
> > Yes, it is likely to be standalone. It won't make use of the LTP
> > API, and the LTP kdump test suite is outdated, so there is no
> > benefit to continue working over there.
> 
> So why make it standalone and not replace the old LTP kdump test suite
> with this new one?
It will be totally a rewrite from scratch, and will have no direct relationship
with the rest of the LTP.
> 
> > > 
> > > > focus on testing kernel and the crash utility. It should work
> > > > for all major distros since it will use none of distro-specific
> > > > stuff, and also support different arches including x86, ARM,
> > > > PPC64 and s390x.
> > > > 
> > > > It does the following:
> > > > 1) check if there is a memory reserved for kdump. If not,
> > > >    reserve the memory and reboot the system.
> > > > 2) once the system is back, load kexec on panic and
> > > >    prepare a separate initramfs that including needed
> > > >    modules to load a local filesystem and necessary utilities
> > > 
> > > So you will write logic to prepare custom initramfs or will rely
> > > on dracut or some other utility for that.
> > I'll probably prepare custom initramfs for the sake of simplicity.
> 
> Well, preparing custom initramfs will become very tricky. We used
> to do that and finally we switched to dracut.
> 
> Why not simply let the respective service on the host do this job and
> test only makes sure that kdump service is running. It feels little
> out of place that a test is generating custom initramfs.
Because not every distro will have a kdump service like Fedora.
> 
> > > 
> > > >    in order to analyse /proc/vmcore in the 2nd kernel.
> > > > 3) trigger the system crash using methods like sysrq-c, NMI,
> > > >    and panic_on_hung_task etc.
> > > > 4) in the 2nd kernel, mount a filesystem and use the crash
> > > >    utility to analyse /proc/vmcore. Then, gather the analyse
> > > >    logs, serial console output, dmesg etc into the filesystem.
> > > 
> > > Why not save core and boot back in first kernel and then analyze.
> > > 
> > > Trying to work directly with /proc/vmcore does not test makedumfile
> > > which everybody uses. Also it will require more memory to be reserved
> > > and packing crash and debug vmlinux into initramfs.
> > The additional memory for vmlinux and the crash utility is predictable
> > and manageable, so it can just ask 256M memory reserved before running
> > the program. On the other hand, it is not usually feasible to ask
> > the systems under testing has enough available disk spaces bigger than
> > the memory size.
> 
> makedumpfile will reduce the vmcore file size to few hundreds of mega
> bytes on most of the systems. Especially, this is just a test, so
> system will be lightly loaded and vmcore will be small after filtering.
It probably actually have test cases to heavily loaded the memory before
dumping.
> 
> If there is not enough space, test fails, period. I don't think there
> is any need to try to circumvent that and try to run crash in initramfs.
> And in the process we don't test makedumpfile which is very imporatnt
> component of this whole process.
Test makedumpfile is in plan. Tests failed because of lack of disk space
is a testsuite design problem, and especially problematic on those large
memory systems that we had seen more and more those days.
> 
> IMHO, just rely on "systemctl start kdump" to generate and load custom
> initramfs and save filtered vmcore to root fs by default and alanyze
> vmcore post reboot. That will keep things simple.
Again, not every major distro has that.
   CAI Qian
> 
> Thanks
> Vivek
> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec
> 

Re: [RFC] autokdump - automated kdump testsuite

[Vivek Goyal]

On Mon, Sep 22, 2014 at 10:53:55PM -0400, CAI Qian wrote:

[..]
> > Why not simply let the respective service on the host do this job and
> > test only makes sure that kdump service is running. It feels little
> > out of place that a test is generating custom initramfs.
> Because not every distro will have a kdump service like Fedora.

So which distro does not have a service? Do we know this or we are
assuming that distributions don't have a service to load/unload
kdump kernel.

[..]
> > makedumpfile will reduce the vmcore file size to few hundreds of mega
> > bytes on most of the systems. Especially, this is just a test, so
> > system will be lightly loaded and vmcore will be small after filtering.
> It probably actually have test cases to heavily loaded the memory before
> dumping.

Your original proposal does not take care of this case either. A test
case could be heavily used memory and if user does not have sufficient
memory to save core, so be it. That test will fail.

Thanks
Vivek

Re: [RFC] autokdump - automated kdump testsuite

[Dave Young]

Hi,

On 09/19/14 at 05:52am, CAI Qian wrote:
> I plan to release an automated kdump testsuite that will be
> focus on testing kernel and the crash utility. It should work
> for all major distros since it will use none of distro-specific
> stuff, and also support different arches including x86, ARM,
> PPC64 and s390x.

It's a good idea to create a kdump test project. Maybe you can consider to
test kexec as well.

> 
> It does the following:
> 1) check if there is a memory reserved for kdump. If not,
>    reserve the memory and reboot the system.
> 2) once the system is back, load kexec on panic and
>    prepare a separate initramfs that including needed
>    modules to load a local filesystem and necessary utilities
>    in order to analyse /proc/vmcore in the 2nd kernel.
> 3) trigger the system crash using methods like sysrq-c, NMI,
>    and panic_on_hung_task etc.
> 4) in the 2nd kernel, mount a filesystem and use the crash
>    utility to analyse /proc/vmcore. Then, gather the analyse
>    logs, serial console output, dmesg etc into the filesystem.
> 5) reboot back into the 1st kernel.
> 
> implementation:
> It will setup a daemon to handle reboots.
> 
> plan:
> I might also to test the makedumpfile all together later.

For makedumpfile, IMO you can add several test cases and collect the dump
time, vmcore size data as test results besides of the dump status.

Thanks
Dave

Re: [RFC] autokdump - automated kdump testsuite

[Dave Young]

On 09/19/14 at 05:52am, CAI Qian wrote:
> I plan to release an automated kdump testsuite that will be
> focus on testing kernel and the crash utility. It should work
> for all major distros since it will use none of distro-specific
> stuff, and also support different arches including x86, ARM,
> PPC64 and s390x.
> 
> It does the following:
> 1) check if there is a memory reserved for kdump. If not,
>    reserve the memory and reboot the system.
> 2) once the system is back, load kexec on panic and
>    prepare a separate initramfs that including needed
>    modules to load a local filesystem and necessary utilities
>    in order to analyse /proc/vmcore in the 2nd kernel.
> 3) trigger the system crash using methods like sysrq-c, NMI,
>    and panic_on_hung_task etc.
> 4) in the 2nd kernel, mount a filesystem and use the crash
>    utility to analyse /proc/vmcore. Then, gather the analyse
>    logs, serial console output, dmesg etc into the filesystem.
> 5) reboot back into the 1st kernel.
> 
> implementation:
> It will setup a daemon to handle reboots.
> 
> plan:
> I might also to test the makedumpfile all together later.
>    CAI Qian
> 

Hi, Cai

I noticed you cced redhat internal lists, you'd better to remove
them from cc list so that people do not get confused by the auto reply.

Thanks
Dave