Re: How to get the crash dump if system hangs?

[Muni Sekhar <munisekharrms@gmail.com>]

On Fri, Oct 11, 2019 at 7:33 PM Muni Sekhar <munisekharrms@gmail.com> wrote:
>
> On Fri, Oct 11, 2019 at 12:01 AM Kees Cook <keescook@chromium.org> wrote:
> >
> > On Thu, Oct 10, 2019 at 10:45:21PM +0530, Muni Sekhar wrote:
> > > I'm using Ubuntu distro, ran "reboot" command but after reboot
> > > console-ramoops-0 is not present in /sys/fs/pstore
> >
> > Hmpf. Well, then, I guess your boot firmware is pretty aggressive about
> > wiping RAM across boots. :(

After UEFI boot mode set Now I see that ramoops is working fine.

To validate this, I simulated the crash using the following command.

# echo c > /proc/sysrq-trigger


Then my system got rebooted automatically. After reboot , initially no
files present in pstore.

$ ls -ltr /sys/fs/pstore/

total 0


$ sudo modprobe ramoops mem_size=1048576 ecc=1 mem_address=0x3ff00000
console_size=16384 ftrace_size=16384 pmsg_size=16384 record_size=32768
mem_type=1 dump_oops=1


After loading the ramoops module, I see it generates dmesg and console logs.

$ ls -ltr /sys/fs/pstore/

total 0

-r--r--r-- 1 root root 54522 Oct 24 13:27 dmesg-ramoops-0

-r--r--r-- 1 root root 54604 Oct 24 13:27 dmesg-ramoops-1

-r--r--r-- 1 root root  3641 Oct 24 13:30 console-ramoops-0


I repeated it for many times and verified that it works consistently.

I’ve a actual test case where my system gets frozen  so have no
software control. I executed this test case and as expected my system
has frozen and recovered it by powering it on(cold boot?) and then
loaded the ramoops but this time no files present in /sys/fs/pstore.

Any idea, why it works for ‘simulating the crash’ and not in actual
hang scenario? The only difference is , In actual hang case it needs a
manual hard reboot.

If you restart a PC in cold(hard) boot, is it possible to see the RAM
memory(previous boot) still? I really I don’t know how it works.

So, is there a  way to automatically reboot the Linux system when it
freezes? I set “kernel.softlockup_panic = 1, kernel.unknown_nmi_panic
= 1, kernel.softlockup_all_cpu_backtrace = 1, kernel.panic = 1,
kernel.panic_on_io_nmi = 1, kernel.panic_on_oops = 1,
kernel.panic_on_stackoverflow = 1, kernel.panic_on_unrecovered_nmi =
1”, but it does not helped to reboot when it freezes.


> >
> > There was a patch set to store to disk, but I haven't seen a recent
> > version of it, if you want to go that route[1].
> >
> > -Kees
> >
> > [1] https://lore.kernel.org/lkml/1551922630-27548-1-git-send-email-liaoweixiong@allwinnertech.com/
> Thanks I will check it out.
>
> While loading ramoops I see "persistent_ram: uncorrectable error in
> header", is this harmful?
>
> [  270.864969] ramoops: using module parameters
> [  270.866651] persistent_ram: uncorrectable error in header
> [  270.867252] persistent_ram: uncorrectable error in header
> [  270.867728] persistent_ram: uncorrectable error in header
> [  270.868067] persistent_ram: uncorrectable error in header
> [  270.868492] persistent_ram: uncorrectable error in header
> [  270.868839] persistent_ram: uncorrectable error in header
> [  270.869209] persistent_ram: uncorrectable error in header
> [  270.869681] persistent_ram: uncorrectable error in header
> [  270.870026] persistent_ram: uncorrectable error in header
> [  270.870430] persistent_ram: uncorrectable error in header
> [  270.870774] persistent_ram: uncorrectable error in header
> [  270.871110] persistent_ram: uncorrectable error in header
> [  270.871687] persistent_ram: uncorrectable error in header
> [  270.872055] persistent_ram: uncorrectable error in header
> [  270.872567] persistent_ram: uncorrectable error in header
> [  270.872910] persistent_ram: uncorrectable error in header
> [  270.873243] persistent_ram: uncorrectable error in header
> [  270.873592] persistent_ram: uncorrectable error in header
> [  270.873932] persistent_ram: uncorrectable error in header
> [  270.874267] persistent_ram: uncorrectable error in header
> [  270.874614] persistent_ram: uncorrectable error in header
> [  270.874958] persistent_ram: uncorrectable error in header
> [  270.875300] persistent_ram: uncorrectable error in header
> [  270.875686] persistent_ram: uncorrectable error in header
> [  270.876028] persistent_ram: uncorrectable error in header
> [  270.876462] persistent_ram: uncorrectable error in header
> [  270.876808] persistent_ram: uncorrectable error in header
> [  270.877144] persistent_ram: uncorrectable error in header
> [  270.877519] persistent_ram: uncorrectable error in header
> [  270.877860] persistent_ram: uncorrectable error in header
> [  270.878199] persistent_ram: uncorrectable error in header
> [  270.878565] persistent_ram: uncorrectable error in header
> [  270.878916] persistent_ram: uncorrectable error in header
> [  270.881129] console [pstore-1] enabled
> [  270.884817] pstore: Registered ramoops as persistent store backend
> [  270.885232] ramoops: attached 0x100000@0x3ff00000, ecc: 16/0
>
> >
> > --
> > Kees Cook
>
>
>
> --
> Thanks,
> Sekhar



-- 
Thanks,
Sekhar