Re: How to get the crash dump if system hangs?

From: Kees Cook <keescook@chromium.org>
To: Muni Sekhar <munisekharrms@gmail.com>
Cc: kernel-hardening@lists.openwall.com
Subject: Re: How to get the crash dump if system hangs?
Date: Thu, 10 Oct 2019 09:56:55 -0700	[thread overview]
Message-ID: <201910100950.5179A62E2@keescook> (raw)
In-Reply-To: <CAHhAz+iUOum7EV1g9W=vFHZ0kq9US7L4CJFX4=QbSExrgBX7yg@mail.gmail.com>

On Thu, Oct 10, 2019 at 09:19:26PM +0530, Muni Sekhar wrote:
> Later I booted with “memmap=1M!1023M ramoops.mem_size=1048576
> ramoops.ecc=1 ramoops.mem_address=0x3ff00000
> ramoops.console_size=16384 ramoops.ftrace_size=16384
> ramoops.pmsg_size=16384 ramoops.record_size=32768 ramoops.mem_type=1
> ramoops.dump_oops=1”
> 
> After reboot, In dmesg I see the following lines:
> 
> [    0.373084] pstore: Registered ramoops as persistent store backend
> [    0.373266] ramoops: attached 0x100000@0x3ff00000, ecc: 16/0
> 
> # cat /proc/iomem | grep "System RAM"
> 00001000-0009d7ff : System RAM
> 00100000-1fffffff : System RAM
> 20100000-3fefffff : System RAM
> 3ff00000-3fffffff : Persistent RAM
> 40000000-b937dfff : System RAM
> b9ba6000-b9ba6fff : System RAM
> b9be9000-b9d5dfff : System RAM
> b9ffa000-b9ffffff : System RAM
> 100000000-13fffffff : System RAM
> 
> I noticed Persistent RAM, not Persistent Memory (legacy). What is the
> difference between these two?

I think this might just be a difference is kernel versions and the
string reported here. As long as it's not "System RAM" it should be
available for pstore.

> I could not find any file in /sys/fs/pstore after warm boot. Even
> tried to trigger the crash by running “echo c > /proc/sysrq-trigger”
> and then rebooted  the system manually. After system boots up, I could
> not find dmesg-ramoops-N file in /sys/fs/pstore, even I could not find
> any file in /sys/fs/pstore directory.
> 
> Am I missing anything?

Silly question: has the pstore filesystem been mounted there?

$ mount | grep pstore
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)

If so, try a warm reboot and you should have at least the prior boot's
console output in /sys/fs/pstore/console-ramoops-0

If you don't, I'm not sure what's happening. You may want to try a newer
kernel (I see you've also go the old ramoops dmesg reporting about ecc.)

Here's my dmesg...

# dmesg | egrep -i 'pstore|ramoops'
...
[    1.004376] ramoops: using module parameters
[    1.010837] ramoops: uncorrectable error in header
[    1.163014] printk: console [pstore-1] enabled
[    1.164476] pstore: Registered ramoops as persistent store backend
[    1.165028] ramoops: using 0x100000@0x440000000, ecc: 16
[    4.610229] pstore: Using crash dump compression: deflate

If a warm boot works and cold boot doesn't, then it looks like your
hardware wipes enough of RAM (or loses refresh for long enough) that
even the ECC can't repair it, in which case pstore isn't going to work.
:(

-- 
Kees Cook