Re: How to get the crash dump if system hangs?

From: Muni Sekhar <munisekharrms@gmail.com>
To: Kees Cook <keescook@chromium.org>
Cc: kernel-hardening@lists.openwall.com
Subject: Re: How to get the crash dump if system hangs?
Date: Thu, 10 Oct 2019 22:45:21 +0530	[thread overview]
Message-ID: <CAHhAz+j9oaAY9_sn16J2c=U+iidZKu3mp0pRpPZAvu4dJPetkg@mail.gmail.com> (raw)
In-Reply-To: <201910100950.5179A62E2@keescook>

On Thu, Oct 10, 2019 at 10:26 PM Kees Cook <keescook@chromium.org> wrote:
>
> On Thu, Oct 10, 2019 at 09:19:26PM +0530, Muni Sekhar wrote:
> > Later I booted with “memmap=1M!1023M ramoops.mem_size=1048576
> > ramoops.ecc=1 ramoops.mem_address=0x3ff00000
> > ramoops.console_size=16384 ramoops.ftrace_size=16384
> > ramoops.pmsg_size=16384 ramoops.record_size=32768 ramoops.mem_type=1
> > ramoops.dump_oops=1”
> >
> > After reboot, In dmesg I see the following lines:
> >
> > [    0.373084] pstore: Registered ramoops as persistent store backend
> > [    0.373266] ramoops: attached 0x100000@0x3ff00000, ecc: 16/0
> >
> > # cat /proc/iomem | grep "System RAM"
> > 00001000-0009d7ff : System RAM
> > 00100000-1fffffff : System RAM
> > 20100000-3fefffff : System RAM
> > 3ff00000-3fffffff : Persistent RAM
> > 40000000-b937dfff : System RAM
> > b9ba6000-b9ba6fff : System RAM
> > b9be9000-b9d5dfff : System RAM
> > b9ffa000-b9ffffff : System RAM
> > 100000000-13fffffff : System RAM
> >
> > I noticed Persistent RAM, not Persistent Memory (legacy). What is the
> > difference between these two?
>
> I think this might just be a difference is kernel versions and the
> string reported here. As long as it's not "System RAM" it should be
> available for pstore.
>
> > I could not find any file in /sys/fs/pstore after warm boot. Even
> > tried to trigger the crash by running “echo c > /proc/sysrq-trigger”
> > and then rebooted  the system manually. After system boots up, I could
> > not find dmesg-ramoops-N file in /sys/fs/pstore, even I could not find
> > any file in /sys/fs/pstore directory.
> >
> > Am I missing anything?
>
> Silly question: has the pstore filesystem been mounted there?
Yes, pstore is mounted.
# mount | grep pstore
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)

>
> $ mount | grep pstore
> pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
>
> If so, try a warm reboot and you should have at least the prior boot's
> console output in /sys/fs/pstore/console-ramoops-0
I'm using Ubuntu distro, ran "reboot" command but after reboot
console-ramoops-0 is not present in /sys/fs/pstore
>
> If you don't, I'm not sure what's happening. You may want to try a newer
> kernel (I see you've also go the old ramoops dmesg reporting about ecc.)
>
> Here's my dmesg...
>
> # dmesg | egrep -i 'pstore|ramoops'
> ...
> [    1.004376] ramoops: using module parameters
> [    1.010837] ramoops: uncorrectable error in header
> [    1.163014] printk: console [pstore-1] enabled
> [    1.164476] pstore: Registered ramoops as persistent store backend
> [    1.165028] ramoops: using 0x100000@0x440000000, ecc: 16
> [    4.610229] pstore: Using crash dump compression: deflate
>
Here is my dmesg:
# dmesg | egrep -i 'pstore|ramoops'
[    0.274931] ramoops: using module parameters
[    0.369885] console [pstore-1] enabled
[    0.372306] pstore: Registered ramoops as persistent store backend
[    0.372504] ramoops: attached 0x100000@0x3ff00000, ecc: 16/0

> If a warm boot works and cold boot doesn't, then it looks like your
> hardware wipes enough of RAM (or loses refresh for long enough) that
> even the ECC can't repair it, in which case pstore isn't going to work.
> :(
>
> --
> Kees Cook

-- 
Thanks,
Sekhar