* How to get the crash dump if system hangs?
@ 2019-09-25 20:17 Muni Sekhar
2019-09-30 23:51 ` Kees Cook
0 siblings, 1 reply; 16+ messages in thread
From: Muni Sekhar @ 2019-09-25 20:17 UTC (permalink / raw)
To: kernel-hardening; +Cc: Kees Cook
Hi All,
I looked at the available tests with "cat
/sys/kernel/debug/provoke-crash/DIRECT", from this I’d like to know
which test causes system hang? I could not find any test case for
deadlock, is any reason for this?
I’m having a Linux system, I’m seeing it gets hung during certain
tests. When it hung, it does not even respond for SYSRQ button, only
way to recover is power-button-only. Does no response for SYSRQ
button means kernel crashed?
After reboot I looked at the kern.log and most of the times it has
“^@^@^@^ ...“ line just before reboot. Can someone clarify me what the
kernel log entry “^@^@^@^ ...“ means? I suspect kernel is crashed, but
it does give any crashdump in kern.log.
Later I enabled the kernel crash dump(sudo apt install
linux-crashdump) and rerun the test but still nothing copied to the
disk(/var/crash/). I don’t have onboard serial port in my machine, so
I tried get the crash dump via netconsole, but this method also does
not able to catch the crash dump.
Can someone help me how to debug in this scenario?
And I'd like to know what other options available to get the crash
dump? Can someone please clarify me on this?
Also , does the crash dump fails if incase deadlock occurs?
Any help will be greatly appreciated.
--
Thanks,
Sekhar
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: How to get the crash dump if system hangs?
2019-09-25 20:17 How to get the crash dump if system hangs? Muni Sekhar
@ 2019-09-30 23:51 ` Kees Cook
2019-10-03 16:48 ` Muni Sekhar
0 siblings, 1 reply; 16+ messages in thread
From: Kees Cook @ 2019-09-30 23:51 UTC (permalink / raw)
To: Muni Sekhar; +Cc: kernel-hardening
On Thu, Sep 26, 2019 at 01:47:00AM +0530, Muni Sekhar wrote:
> I looked at the available tests with "cat
> /sys/kernel/debug/provoke-crash/DIRECT", from this I’d like to know
> which test causes system hang? I could not find any test case for
> deadlock, is any reason for this?
The various *LOCKUP tests will hang a CPU or task (though SPINLOCKUP
needs to be called twice). You could keep calling HARDLOCKUP until
you're out of CPUs, for example. :)
What kind of deadlock do you want to test?
> I’m having a Linux system, I’m seeing it gets hung during certain
> tests. When it hung, it does not even respond for SYSRQ button, only
> way to recover is power-button-only. Does no response for SYSRQ
> button means kernel crashed?
That's an impressive hang! :(
> After reboot I looked at the kern.log and most of the times it has
> “^@^@^@^ ...“ line just before reboot. Can someone clarify me what the
> kernel log entry “^@^@^@^ ...“ means? I suspect kernel is crashed, but
> it does give any crashdump in kern.log.
That's a zero byte. I would suggest using something like pstore to
capture this in RAM instead of hoping it makes it to disk.
> Later I enabled the kernel crash dump(sudo apt install
> linux-crashdump) and rerun the test but still nothing copied to the
> disk(/var/crash/). I don’t have onboard serial port in my machine, so
> I tried get the crash dump via netconsole, but this method also does
> not able to catch the crash dump.
>
> Can someone help me how to debug in this scenario?
>
> And I'd like to know what other options available to get the crash
> dump? Can someone please clarify me on this?
>
> Also , does the crash dump fails if incase deadlock occurs?
>
> Any help will be greatly appreciated.
If you really need to hard-power your system to get it back, pstore may
only work if you're really quick and likely enable software ECC.
--
Kees Cook
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: How to get the crash dump if system hangs?
2019-09-30 23:51 ` Kees Cook
@ 2019-10-03 16:48 ` Muni Sekhar
2019-10-03 21:36 ` Kees Cook
0 siblings, 1 reply; 16+ messages in thread
From: Muni Sekhar @ 2019-10-03 16:48 UTC (permalink / raw)
To: Kees Cook; +Cc: kernel-hardening
On Tue, Oct 1, 2019 at 5:21 AM Kees Cook <keescook@chromium.org> wrote:
>
> On Thu, Sep 26, 2019 at 01:47:00AM +0530, Muni Sekhar wrote:
> > I looked at the available tests with "cat
> > /sys/kernel/debug/provoke-crash/DIRECT", from this I’d like to know
> > which test causes system hang? I could not find any test case for
> > deadlock, is any reason for this?
>
> The various *LOCKUP tests will hang a CPU or task (though SPINLOCKUP
> needs to be called twice). You could keep calling HARDLOCKUP until
> you're out of CPUs, for example. :)
>
> What kind of deadlock do you want to test?
I'm looking for a test where crash dump fails.
>
> > I’m having a Linux system, I’m seeing it gets hung during certain
> > tests. When it hung, it does not even respond for SYSRQ button, only
> > way to recover is power-button-only. Does no response for SYSRQ
> > button means kernel crashed?
>
> That's an impressive hang! :(
>
> > After reboot I looked at the kern.log and most of the times it has
> > “^@^@^@^ ...“ line just before reboot. Can someone clarify me what the
> > kernel log entry “^@^@^@^ ...“ means? I suspect kernel is crashed, but
> > it does give any crashdump in kern.log.
>
> That's a zero byte. I would suggest using something like pstore to
> capture this in RAM instead of hoping it makes it to disk.
>
> > Later I enabled the kernel crash dump(sudo apt install
> > linux-crashdump) and rerun the test but still nothing copied to the
> > disk(/var/crash/). I don’t have onboard serial port in my machine, so
> > I tried get the crash dump via netconsole, but this method also does
> > not able to catch the crash dump.
> >
> > Can someone help me how to debug in this scenario?
> >
> > And I'd like to know what other options available to get the crash
> > dump? Can someone please clarify me on this?
> >
> > Also , does the crash dump fails if incase deadlock occurs?
> >
> > Any help will be greatly appreciated.
>
> If you really need to hard-power your system to get it back, pstore may
> only work if you're really quick and likely enable software ECC.
Thanks a lot for letting me know about pstore, will try this option.
It will be helpful if you can share some pointers on 'how to enable
software ECC'?
>
> --
> Kees Cook
--
Thanks,
Sekhar
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: How to get the crash dump if system hangs?
2019-10-03 16:48 ` Muni Sekhar
@ 2019-10-03 21:36 ` Kees Cook
2019-10-10 15:49 ` Muni Sekhar
0 siblings, 1 reply; 16+ messages in thread
From: Kees Cook @ 2019-10-03 21:36 UTC (permalink / raw)
To: Muni Sekhar; +Cc: kernel-hardening
On Thu, Oct 03, 2019 at 10:18:48PM +0530, Muni Sekhar wrote:
> Thanks a lot for letting me know about pstore, will try this option.
> It will be helpful if you can share some pointers on 'how to enable
> software ECC'?
When I boot with pstore, I use a bunch of command line arguments to test
all its feature:
ramoops.mem_size=1048576
ramoops.ecc=1
ramoops.mem_address=0x440000000
ramoops.console_size=16384
ramoops.ftrace_size=16384
ramoops.pmsg_size=16384
ramoops.record_size=32768
but I'm using pmem driver to reserve the 1MB of memory at 0x440000000.
To do a RAM reservation on a regular system, you'll need to do something
like boot with:
memmap=1M!1023M
which says, reserve 1MB of memory at the 1023M offset. So this depends
on how much physical memory you have, etc, but you'll be able to see the
reservation after booting in /proc/iomem. e.g. for me, before:
...
00100000-bffd9fff : System RAM
...
with memmap:
...
00100000-3fefffff : System RAM
3ff00000-3fffffff : Persistent Memory (legacy)
40000000-bffd9fff : System RAM
...
So in that example, the address you'd want is 0x3ff00000
memmap=1M!1023M
ramoops.mem_size=1048576
ramoops.ecc=1
ramoops.mem_address=0x3ff00000
ramoops.console_size=16384
ramoops.ftrace_size=16384
ramoops.pmsg_size=16384
ramoops.record_size=32768
In dmesg you should see:
[ 0.868818] pstore: Registered ramoops as persistent store backend
[ 0.869713] ramoops: using 0x100000@0x3ff00000, ecc: 16
And if that address lines up with the "Persistent Memory (legacy)" line
in /proc/iomem you should be good to go.
Just mount /sys/fs/pstore and see if the console dump updates between
warm boots, then try some cold boots, see if the ECC works, etc.
Good luck!
--
Kees Cook
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: How to get the crash dump if system hangs?
2019-10-03 21:36 ` Kees Cook
@ 2019-10-10 15:49 ` Muni Sekhar
2019-10-10 16:56 ` Kees Cook
0 siblings, 1 reply; 16+ messages in thread
From: Muni Sekhar @ 2019-10-10 15:49 UTC (permalink / raw)
To: Kees Cook; +Cc: kernel-hardening
On Fri, Oct 4, 2019 at 3:06 AM Kees Cook <keescook@chromium.org> wrote:
>
> On Thu, Oct 03, 2019 at 10:18:48PM +0530, Muni Sekhar wrote:
> > Thanks a lot for letting me know about pstore, will try this option.
> > It will be helpful if you can share some pointers on 'how to enable
> > software ECC'?
>
> When I boot with pstore, I use a bunch of command line arguments to test
> all its feature:
>
> ramoops.mem_size=1048576
> ramoops.ecc=1
> ramoops.mem_address=0x440000000
> ramoops.console_size=16384
> ramoops.ftrace_size=16384
> ramoops.pmsg_size=16384
> ramoops.record_size=32768
>
> but I'm using pmem driver to reserve the 1MB of memory at 0x440000000.
>
> To do a RAM reservation on a regular system, you'll need to do something
> like boot with:
>
> memmap=1M!1023M
>
> which says, reserve 1MB of memory at the 1023M offset. So this depends
> on how much physical memory you have, etc, but you'll be able to see the
> reservation after booting in /proc/iomem. e.g. for me, before:
>
> ...
> 00100000-bffd9fff : System RAM
> ...
>
> with memmap:
>
> ...
> 00100000-3fefffff : System RAM
> 3ff00000-3fffffff : Persistent Memory (legacy)
> 40000000-bffd9fff : System RAM
> ...
>
> So in that example, the address you'd want is 0x3ff00000
>
> memmap=1M!1023M
> ramoops.mem_size=1048576
> ramoops.ecc=1
> ramoops.mem_address=0x3ff00000
> ramoops.console_size=16384
> ramoops.ftrace_size=16384
> ramoops.pmsg_size=16384
> ramoops.record_size=32768
>
> In dmesg you should see:
>
> [ 0.868818] pstore: Registered ramoops as persistent store backend
> [ 0.869713] ramoops: using 0x100000@0x3ff00000, ecc: 16
>
> And if that address lines up with the "Persistent Memory (legacy)" line
> in /proc/iomem you should be good to go.
>
> Just mount /sys/fs/pstore and see if the console dump updates between
> warm boots, then try some cold boots, see if the ECC works, etc.
>
> Good luck!
>
> --
> Kees Cook
Thanks for the answers.
My kernel is configured with following .config options:
CONFIG_EFI_VARS_PSTORE=y
# CONFIG_EFI_VARS_PSTORE_DEFAULT_DISABLE is not set
CONFIG_PSTORE=y
CONFIG_PSTORE_CONSOLE=y
CONFIG_PSTORE_PMSG=y
# CONFIG_PSTORE_FTRACE is not set
CONFIG_PSTORE_RAM=y
Before RAM reservation, I see the following in /proc/iomem :
# cat iomem | grep "System RAM"
00001000-0009d7ff : System RAM
00100000-1fffffff : System RAM
20100000-b937dfff : System RAM
b9ba6000-b9ba6fff : System RAM
b9be9000-b9d5dfff : System RAM
b9ffa000-b9ffffff : System RAM
100000000-13fffffff : System RAM
Later I booted with “memmap=1M!1023M ramoops.mem_size=1048576
ramoops.ecc=1 ramoops.mem_address=0x3ff00000
ramoops.console_size=16384 ramoops.ftrace_size=16384
ramoops.pmsg_size=16384 ramoops.record_size=32768 ramoops.mem_type=1
ramoops.dump_oops=1”
After reboot, In dmesg I see the following lines:
[ 0.373084] pstore: Registered ramoops as persistent store backend
[ 0.373266] ramoops: attached 0x100000@0x3ff00000, ecc: 16/0
# cat /proc/iomem | grep "System RAM"
00001000-0009d7ff : System RAM
00100000-1fffffff : System RAM
20100000-3fefffff : System RAM
3ff00000-3fffffff : Persistent RAM
40000000-b937dfff : System RAM
b9ba6000-b9ba6fff : System RAM
b9be9000-b9d5dfff : System RAM
b9ffa000-b9ffffff : System RAM
100000000-13fffffff : System RAM
I noticed Persistent RAM, not Persistent Memory (legacy). What is the
difference between these two?
I could not find any file in /sys/fs/pstore after warm boot. Even
tried to trigger the crash by running “echo c > /proc/sysrq-trigger”
and then rebooted the system manually. After system boots up, I could
not find dmesg-ramoops-N file in /sys/fs/pstore, even I could not find
any file in /sys/fs/pstore directory.
Am I missing anything?
--
Thanks,
Sekhar
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: How to get the crash dump if system hangs?
2019-10-10 15:49 ` Muni Sekhar
@ 2019-10-10 16:56 ` Kees Cook
2019-10-10 17:15 ` Muni Sekhar
0 siblings, 1 reply; 16+ messages in thread
From: Kees Cook @ 2019-10-10 16:56 UTC (permalink / raw)
To: Muni Sekhar; +Cc: kernel-hardening
On Thu, Oct 10, 2019 at 09:19:26PM +0530, Muni Sekhar wrote:
> Later I booted with “memmap=1M!1023M ramoops.mem_size=1048576
> ramoops.ecc=1 ramoops.mem_address=0x3ff00000
> ramoops.console_size=16384 ramoops.ftrace_size=16384
> ramoops.pmsg_size=16384 ramoops.record_size=32768 ramoops.mem_type=1
> ramoops.dump_oops=1”
>
> After reboot, In dmesg I see the following lines:
>
> [ 0.373084] pstore: Registered ramoops as persistent store backend
> [ 0.373266] ramoops: attached 0x100000@0x3ff00000, ecc: 16/0
>
> # cat /proc/iomem | grep "System RAM"
> 00001000-0009d7ff : System RAM
> 00100000-1fffffff : System RAM
> 20100000-3fefffff : System RAM
> 3ff00000-3fffffff : Persistent RAM
> 40000000-b937dfff : System RAM
> b9ba6000-b9ba6fff : System RAM
> b9be9000-b9d5dfff : System RAM
> b9ffa000-b9ffffff : System RAM
> 100000000-13fffffff : System RAM
>
> I noticed Persistent RAM, not Persistent Memory (legacy). What is the
> difference between these two?
I think this might just be a difference is kernel versions and the
string reported here. As long as it's not "System RAM" it should be
available for pstore.
> I could not find any file in /sys/fs/pstore after warm boot. Even
> tried to trigger the crash by running “echo c > /proc/sysrq-trigger”
> and then rebooted the system manually. After system boots up, I could
> not find dmesg-ramoops-N file in /sys/fs/pstore, even I could not find
> any file in /sys/fs/pstore directory.
>
> Am I missing anything?
Silly question: has the pstore filesystem been mounted there?
$ mount | grep pstore
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
If so, try a warm reboot and you should have at least the prior boot's
console output in /sys/fs/pstore/console-ramoops-0
If you don't, I'm not sure what's happening. You may want to try a newer
kernel (I see you've also go the old ramoops dmesg reporting about ecc.)
Here's my dmesg...
# dmesg | egrep -i 'pstore|ramoops'
...
[ 1.004376] ramoops: using module parameters
[ 1.010837] ramoops: uncorrectable error in header
[ 1.163014] printk: console [pstore-1] enabled
[ 1.164476] pstore: Registered ramoops as persistent store backend
[ 1.165028] ramoops: using 0x100000@0x440000000, ecc: 16
[ 4.610229] pstore: Using crash dump compression: deflate
If a warm boot works and cold boot doesn't, then it looks like your
hardware wipes enough of RAM (or loses refresh for long enough) that
even the ECC can't repair it, in which case pstore isn't going to work.
:(
--
Kees Cook
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: How to get the crash dump if system hangs?
2019-10-10 16:56 ` Kees Cook
@ 2019-10-10 17:15 ` Muni Sekhar
2019-10-10 18:31 ` Kees Cook
0 siblings, 1 reply; 16+ messages in thread
From: Muni Sekhar @ 2019-10-10 17:15 UTC (permalink / raw)
To: Kees Cook; +Cc: kernel-hardening
On Thu, Oct 10, 2019 at 10:26 PM Kees Cook <keescook@chromium.org> wrote:
>
> On Thu, Oct 10, 2019 at 09:19:26PM +0530, Muni Sekhar wrote:
> > Later I booted with “memmap=1M!1023M ramoops.mem_size=1048576
> > ramoops.ecc=1 ramoops.mem_address=0x3ff00000
> > ramoops.console_size=16384 ramoops.ftrace_size=16384
> > ramoops.pmsg_size=16384 ramoops.record_size=32768 ramoops.mem_type=1
> > ramoops.dump_oops=1”
> >
> > After reboot, In dmesg I see the following lines:
> >
> > [ 0.373084] pstore: Registered ramoops as persistent store backend
> > [ 0.373266] ramoops: attached 0x100000@0x3ff00000, ecc: 16/0
> >
> > # cat /proc/iomem | grep "System RAM"
> > 00001000-0009d7ff : System RAM
> > 00100000-1fffffff : System RAM
> > 20100000-3fefffff : System RAM
> > 3ff00000-3fffffff : Persistent RAM
> > 40000000-b937dfff : System RAM
> > b9ba6000-b9ba6fff : System RAM
> > b9be9000-b9d5dfff : System RAM
> > b9ffa000-b9ffffff : System RAM
> > 100000000-13fffffff : System RAM
> >
> > I noticed Persistent RAM, not Persistent Memory (legacy). What is the
> > difference between these two?
>
> I think this might just be a difference is kernel versions and the
> string reported here. As long as it's not "System RAM" it should be
> available for pstore.
>
> > I could not find any file in /sys/fs/pstore after warm boot. Even
> > tried to trigger the crash by running “echo c > /proc/sysrq-trigger”
> > and then rebooted the system manually. After system boots up, I could
> > not find dmesg-ramoops-N file in /sys/fs/pstore, even I could not find
> > any file in /sys/fs/pstore directory.
> >
> > Am I missing anything?
>
> Silly question: has the pstore filesystem been mounted there?
Yes, pstore is mounted.
# mount | grep pstore
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
>
> $ mount | grep pstore
> pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
>
> If so, try a warm reboot and you should have at least the prior boot's
> console output in /sys/fs/pstore/console-ramoops-0
I'm using Ubuntu distro, ran "reboot" command but after reboot
console-ramoops-0 is not present in /sys/fs/pstore
>
> If you don't, I'm not sure what's happening. You may want to try a newer
> kernel (I see you've also go the old ramoops dmesg reporting about ecc.)
>
> Here's my dmesg...
>
> # dmesg | egrep -i 'pstore|ramoops'
> ...
> [ 1.004376] ramoops: using module parameters
> [ 1.010837] ramoops: uncorrectable error in header
> [ 1.163014] printk: console [pstore-1] enabled
> [ 1.164476] pstore: Registered ramoops as persistent store backend
> [ 1.165028] ramoops: using 0x100000@0x440000000, ecc: 16
> [ 4.610229] pstore: Using crash dump compression: deflate
>
Here is my dmesg:
# dmesg | egrep -i 'pstore|ramoops'
[ 0.274931] ramoops: using module parameters
[ 0.369885] console [pstore-1] enabled
[ 0.372306] pstore: Registered ramoops as persistent store backend
[ 0.372504] ramoops: attached 0x100000@0x3ff00000, ecc: 16/0
> If a warm boot works and cold boot doesn't, then it looks like your
> hardware wipes enough of RAM (or loses refresh for long enough) that
> even the ECC can't repair it, in which case pstore isn't going to work.
> :(
>
> --
> Kees Cook
--
Thanks,
Sekhar
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: How to get the crash dump if system hangs?
2019-10-10 17:15 ` Muni Sekhar
@ 2019-10-10 18:31 ` Kees Cook
2019-10-11 14:03 ` Muni Sekhar
0 siblings, 1 reply; 16+ messages in thread
From: Kees Cook @ 2019-10-10 18:31 UTC (permalink / raw)
To: Muni Sekhar; +Cc: kernel-hardening
On Thu, Oct 10, 2019 at 10:45:21PM +0530, Muni Sekhar wrote:
> I'm using Ubuntu distro, ran "reboot" command but after reboot
> console-ramoops-0 is not present in /sys/fs/pstore
Hmpf. Well, then, I guess your boot firmware is pretty aggressive about
wiping RAM across boots. :(
There was a patch set to store to disk, but I haven't seen a recent
version of it, if you want to go that route[1].
-Kees
[1] https://lore.kernel.org/lkml/1551922630-27548-1-git-send-email-liaoweixiong@allwinnertech.com/
--
Kees Cook
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: How to get the crash dump if system hangs?
2019-10-10 18:31 ` Kees Cook
@ 2019-10-11 14:03 ` Muni Sekhar
2019-10-25 2:10 ` Muni Sekhar
0 siblings, 1 reply; 16+ messages in thread
From: Muni Sekhar @ 2019-10-11 14:03 UTC (permalink / raw)
To: Kees Cook; +Cc: kernel-hardening
On Fri, Oct 11, 2019 at 12:01 AM Kees Cook <keescook@chromium.org> wrote:
>
> On Thu, Oct 10, 2019 at 10:45:21PM +0530, Muni Sekhar wrote:
> > I'm using Ubuntu distro, ran "reboot" command but after reboot
> > console-ramoops-0 is not present in /sys/fs/pstore
>
> Hmpf. Well, then, I guess your boot firmware is pretty aggressive about
> wiping RAM across boots. :(
>
> There was a patch set to store to disk, but I haven't seen a recent
> version of it, if you want to go that route[1].
>
> -Kees
>
> [1] https://lore.kernel.org/lkml/1551922630-27548-1-git-send-email-liaoweixiong@allwinnertech.com/
Thanks I will check it out.
While loading ramoops I see "persistent_ram: uncorrectable error in
header", is this harmful?
[ 270.864969] ramoops: using module parameters
[ 270.866651] persistent_ram: uncorrectable error in header
[ 270.867252] persistent_ram: uncorrectable error in header
[ 270.867728] persistent_ram: uncorrectable error in header
[ 270.868067] persistent_ram: uncorrectable error in header
[ 270.868492] persistent_ram: uncorrectable error in header
[ 270.868839] persistent_ram: uncorrectable error in header
[ 270.869209] persistent_ram: uncorrectable error in header
[ 270.869681] persistent_ram: uncorrectable error in header
[ 270.870026] persistent_ram: uncorrectable error in header
[ 270.870430] persistent_ram: uncorrectable error in header
[ 270.870774] persistent_ram: uncorrectable error in header
[ 270.871110] persistent_ram: uncorrectable error in header
[ 270.871687] persistent_ram: uncorrectable error in header
[ 270.872055] persistent_ram: uncorrectable error in header
[ 270.872567] persistent_ram: uncorrectable error in header
[ 270.872910] persistent_ram: uncorrectable error in header
[ 270.873243] persistent_ram: uncorrectable error in header
[ 270.873592] persistent_ram: uncorrectable error in header
[ 270.873932] persistent_ram: uncorrectable error in header
[ 270.874267] persistent_ram: uncorrectable error in header
[ 270.874614] persistent_ram: uncorrectable error in header
[ 270.874958] persistent_ram: uncorrectable error in header
[ 270.875300] persistent_ram: uncorrectable error in header
[ 270.875686] persistent_ram: uncorrectable error in header
[ 270.876028] persistent_ram: uncorrectable error in header
[ 270.876462] persistent_ram: uncorrectable error in header
[ 270.876808] persistent_ram: uncorrectable error in header
[ 270.877144] persistent_ram: uncorrectable error in header
[ 270.877519] persistent_ram: uncorrectable error in header
[ 270.877860] persistent_ram: uncorrectable error in header
[ 270.878199] persistent_ram: uncorrectable error in header
[ 270.878565] persistent_ram: uncorrectable error in header
[ 270.878916] persistent_ram: uncorrectable error in header
[ 270.881129] console [pstore-1] enabled
[ 270.884817] pstore: Registered ramoops as persistent store backend
[ 270.885232] ramoops: attached 0x100000@0x3ff00000, ecc: 16/0
>
> --
> Kees Cook
--
Thanks,
Sekhar
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: How to get the crash dump if system hangs?
2019-10-11 14:03 ` Muni Sekhar
@ 2019-10-25 2:10 ` Muni Sekhar
2019-10-28 19:22 ` Kees Cook
0 siblings, 1 reply; 16+ messages in thread
From: Muni Sekhar @ 2019-10-25 2:10 UTC (permalink / raw)
To: Kees Cook; +Cc: kernel-hardening
On Fri, Oct 11, 2019 at 7:33 PM Muni Sekhar <munisekharrms@gmail.com> wrote:
>
> On Fri, Oct 11, 2019 at 12:01 AM Kees Cook <keescook@chromium.org> wrote:
> >
> > On Thu, Oct 10, 2019 at 10:45:21PM +0530, Muni Sekhar wrote:
> > > I'm using Ubuntu distro, ran "reboot" command but after reboot
> > > console-ramoops-0 is not present in /sys/fs/pstore
> >
> > Hmpf. Well, then, I guess your boot firmware is pretty aggressive about
> > wiping RAM across boots. :(
After UEFI boot mode set Now I see that ramoops is working fine.
To validate this, I simulated the crash using the following command.
# echo c > /proc/sysrq-trigger
Then my system got rebooted automatically. After reboot , initially no
files present in pstore.
$ ls -ltr /sys/fs/pstore/
total 0
$ sudo modprobe ramoops mem_size=1048576 ecc=1 mem_address=0x3ff00000
console_size=16384 ftrace_size=16384 pmsg_size=16384 record_size=32768
mem_type=1 dump_oops=1
After loading the ramoops module, I see it generates dmesg and console logs.
$ ls -ltr /sys/fs/pstore/
total 0
-r--r--r-- 1 root root 54522 Oct 24 13:27 dmesg-ramoops-0
-r--r--r-- 1 root root 54604 Oct 24 13:27 dmesg-ramoops-1
-r--r--r-- 1 root root 3641 Oct 24 13:30 console-ramoops-0
I repeated it for many times and verified that it works consistently.
I’ve a actual test case where my system gets frozen so have no
software control. I executed this test case and as expected my system
has frozen and recovered it by powering it on(cold boot?) and then
loaded the ramoops but this time no files present in /sys/fs/pstore.
Any idea, why it works for ‘simulating the crash’ and not in actual
hang scenario? The only difference is , In actual hang case it needs a
manual hard reboot.
If you restart a PC in cold(hard) boot, is it possible to see the RAM
memory(previous boot) still? I really I don’t know how it works.
So, is there a way to automatically reboot the Linux system when it
freezes? I set “kernel.softlockup_panic = 1, kernel.unknown_nmi_panic
= 1, kernel.softlockup_all_cpu_backtrace = 1, kernel.panic = 1,
kernel.panic_on_io_nmi = 1, kernel.panic_on_oops = 1,
kernel.panic_on_stackoverflow = 1, kernel.panic_on_unrecovered_nmi =
1”, but it does not helped to reboot when it freezes.
> >
> > There was a patch set to store to disk, but I haven't seen a recent
> > version of it, if you want to go that route[1].
> >
> > -Kees
> >
> > [1] https://lore.kernel.org/lkml/1551922630-27548-1-git-send-email-liaoweixiong@allwinnertech.com/
> Thanks I will check it out.
>
> While loading ramoops I see "persistent_ram: uncorrectable error in
> header", is this harmful?
>
> [ 270.864969] ramoops: using module parameters
> [ 270.866651] persistent_ram: uncorrectable error in header
> [ 270.867252] persistent_ram: uncorrectable error in header
> [ 270.867728] persistent_ram: uncorrectable error in header
> [ 270.868067] persistent_ram: uncorrectable error in header
> [ 270.868492] persistent_ram: uncorrectable error in header
> [ 270.868839] persistent_ram: uncorrectable error in header
> [ 270.869209] persistent_ram: uncorrectable error in header
> [ 270.869681] persistent_ram: uncorrectable error in header
> [ 270.870026] persistent_ram: uncorrectable error in header
> [ 270.870430] persistent_ram: uncorrectable error in header
> [ 270.870774] persistent_ram: uncorrectable error in header
> [ 270.871110] persistent_ram: uncorrectable error in header
> [ 270.871687] persistent_ram: uncorrectable error in header
> [ 270.872055] persistent_ram: uncorrectable error in header
> [ 270.872567] persistent_ram: uncorrectable error in header
> [ 270.872910] persistent_ram: uncorrectable error in header
> [ 270.873243] persistent_ram: uncorrectable error in header
> [ 270.873592] persistent_ram: uncorrectable error in header
> [ 270.873932] persistent_ram: uncorrectable error in header
> [ 270.874267] persistent_ram: uncorrectable error in header
> [ 270.874614] persistent_ram: uncorrectable error in header
> [ 270.874958] persistent_ram: uncorrectable error in header
> [ 270.875300] persistent_ram: uncorrectable error in header
> [ 270.875686] persistent_ram: uncorrectable error in header
> [ 270.876028] persistent_ram: uncorrectable error in header
> [ 270.876462] persistent_ram: uncorrectable error in header
> [ 270.876808] persistent_ram: uncorrectable error in header
> [ 270.877144] persistent_ram: uncorrectable error in header
> [ 270.877519] persistent_ram: uncorrectable error in header
> [ 270.877860] persistent_ram: uncorrectable error in header
> [ 270.878199] persistent_ram: uncorrectable error in header
> [ 270.878565] persistent_ram: uncorrectable error in header
> [ 270.878916] persistent_ram: uncorrectable error in header
> [ 270.881129] console [pstore-1] enabled
> [ 270.884817] pstore: Registered ramoops as persistent store backend
> [ 270.885232] ramoops: attached 0x100000@0x3ff00000, ecc: 16/0
>
> >
> > --
> > Kees Cook
>
>
>
> --
> Thanks,
> Sekhar
--
Thanks,
Sekhar
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: How to get the crash dump if system hangs?
2019-10-25 2:10 ` Muni Sekhar
@ 2019-10-28 19:22 ` Kees Cook
0 siblings, 0 replies; 16+ messages in thread
From: Kees Cook @ 2019-10-28 19:22 UTC (permalink / raw)
To: Muni Sekhar; +Cc: kernel-hardening
On Fri, Oct 25, 2019 at 07:40:58AM +0530, Muni Sekhar wrote:
> After loading the ramoops module, I see it generates dmesg and console logs.
Excellent!
> I’ve a actual test case where my system gets frozen so have no
> software control. I executed this test case and as expected my system
> has frozen and recovered it by powering it on(cold boot?) and then
> loaded the ramoops but this time no files present in /sys/fs/pstore.
I wonder if you could use a hardware watchdog driver of some kind to
trigger the soft reboot?
> If you restart a PC in cold(hard) boot, is it possible to see the RAM
> memory(previous boot) still? I really I don’t know how it works.
It depends a lot on your chipset and RAM. It sounds like your system
very quickly wipes its RAM contents on a cold reset.
> So, is there a way to automatically reboot the Linux system when it
> freezes? I set “kernel.softlockup_panic = 1, kernel.unknown_nmi_panic
> = 1, kernel.softlockup_all_cpu_backtrace = 1, kernel.panic = 1,
> kernel.panic_on_io_nmi = 1, kernel.panic_on_oops = 1,
> kernel.panic_on_stackoverflow = 1, kernel.panic_on_unrecovered_nmi =
> 1”, but it does not helped to reboot when it freezes.
See if Documentation/nmi_watchdog.txt helps?
Good luck!
--
Kees Cook
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: How to get the crash dump if system hangs?
@ 2019-10-20 19:01 youling257
2019-10-21 8:39 ` Lukas Odzioba
0 siblings, 1 reply; 16+ messages in thread
From: youling257 @ 2019-10-20 19:01 UTC (permalink / raw)
To: keescook; +Cc: kernel-hardening, munisekharrms
I don't know my ramoops.mem_address, please help me.
what is ramoops.mem_address?
# cat /proc/iomem
00000000-00000fff : Reserved
00001000-0008efff : System RAM
0008f000-0008ffff : ACPI Non-volatile Storage
00090000-0009dfff : System RAM
0009e000-0009ffff : Reserved
000a0000-000bffff : PCI Bus 0000:00
000c0000-000dffff : PCI Bus 0000:00
000e0000-000fffff : PCI Bus 0000:00
000f0000-000fffff : System ROM
00100000-1fffffff : System RAM
20000000-201fffff : Reserved
20000000-201fffff : 80860F28:00
20200000-779c5fff : System RAM
4c800000-4d401150 : Kernel code
4d401151-4dd1fb3f : Kernel data
4e212000-4e3fffff : Kernel bss
779c6000-77a12fff : Reserved
77a13000-77a1bfff : Unknown E820 type
77a1c000-77afffff : Reserved
77b00000-7bb6dfff : System RAM
7bb6e000-7bbcdfff : Unknown E820 type
7bbce000-7bdcdfff : Reserved
7bdce000-7bdcefff : System RAM
7bdcf000-7bdcffff : Reserved
7bdd0000-7bdd1fff : System RAM
7bdd2000-7bdd2fff : Reserved
7bdd3000-7c7fffff : System RAM
7c800000-7cbfffff : RAM buffer
7cc00000-7cc3afff : System RAM
7cc3b000-7cc47fff : Reserved
7cc48000-7ccb7fff : ACPI Non-volatile Storage
7ccb8000-7ccf7fff : ACPI Tables
7ccf8000-7ccfffff : System RAM
7cd00000-7cefffff : RAM buffer
7cf00000-7ef00000 : Reserved
7cf00001-7ef00000 : PCI Bus 0000:00
7cf00001-7eeffffe : Graphics Stolen Memory
7ef00001-7fefffff : RAM buffer
7ff00000-7ff00fff : MSFT0101:00
7ff01000-7fffffff : RAM buffer
80000000-90affffe : PCI Bus 0000:00
80000000-8fffffff : 0000:00:02.0
90000000-903fffff : 0000:00:02.0
901e5000-901e5fff : hdmi-lpe-audio-mmio
90400000-905fffff : dwc_usb3
90400000-905fffff : 0000:00:16.0
9040c100-905fffff : dwc3.3.auto
90800000-908fffff : 0000:00:1a.0
90900000-909fffff : 0000:00:1a.0
90a00000-90a0ffff : 0000:00:14.0
90a00000-90a0ffff : xhci-hcd
90a2c000-90a2cfff : 0000:00:16.0
90b00000-90b00fff : 80860F28:00
90b01000-90b01fff : INT33BB:00
90b01000-90b01fff : INT33BB:00
90b04000-90b07fff : INTL9C60:01
90b04000-90b07fff : INTL9C60:01
90b08000-90b08fff : 80860F41:00
90b08000-90b08fff : 80860F41:00
90b0a000-90b0afff : 80860F41:01
90b0a000-90b0afff : 80860F41:01
90b0c000-90b0cfff : 80860F41:02
90b0c000-90b0cfff : 80860F41:02
90b0e000-90b0efff : 80860F41:03
90b0e000-90b0efff : 80860F41:03
90b10000-90b10fff : 80860F41:04
90b10000-90b10fff : 80860F41:04
90b13000-90b13fff : 80860F09:00
90b13000-90b13fff : 80860F09:00
90b14000-90b17fff : INTL9C60:00
90b14000-90b17fff : INTL9C60:00
90b19000-90b19fff : 80860F0A:00
90b19000-90b1901f : serial
90b1b000-90b1bfff : 80860F0A:01
90b1b000-90b1b01f : serial
90b1d000-90b1dfff : 80860F14:02
90b1d000-90b1dfff : 80860F14:02
90b1f000-90b1ffff : 80860F14:00
90b1f000-90b1ffff : 80860F14:00
90c00000-90ffffff : PCI Bus 0000:00
91000000-911fffff : 80860F28:00
e00000d0-e00000db : INT33BD:00
e00f8000-e00f8fff : Reserved
fec00000-fec003ff : IOAPIC 0
fed00000-fed003ff : HPET 0
fed00000-fed003ff : PNP0103:00
fed01000-fed01fff : Reserved
fed01000-fed011ff : intel-spi
fed03008-fed0300c : iTCO_wdt.2.auto
fed05000-fed057ff : INT3401:00
fed0c000-fed0cfff : INT33FC:00
fed0c000-fed0cfff : INT33FC:00
fed0d000-fed0dfff : INT33FC:01
fed0d000-fed0dfff : INT33FC:01
fed0e000-fed0efff : INT33FC:02
fed0e000-fed0efff : INT33FC:02
fed40000-fed40fff : PCI Bus 0000:00
fee00000-fee00fff : Local APIC
ff000000-ffffffff : INT0800:00
ffd00000-ffffffff : Reserved
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: How to get the crash dump if system hangs?
2019-10-20 19:01 youling257
@ 2019-10-21 8:39 ` Lukas Odzioba
2019-10-21 10:22 ` youling 257
0 siblings, 1 reply; 16+ messages in thread
From: Lukas Odzioba @ 2019-10-21 8:39 UTC (permalink / raw)
To: youling257; +Cc: keescook, kernel-hardening, munisekharrms
youling257 <youling257@gmail.com> wrote:
>
> I don't know my ramoops.mem_address, please help me.
>
> what is ramoops.mem_address?
It is a Linux kernel parameter, see documentation below:
https://www.kernel.org/doc/Documentation/admin-guide/ramoops.rst
It requires memory which can hold data between reboots, so i'm not
sure how it will suit your case.
Thanks,
Lukas
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: How to get the crash dump if system hangs?
2019-10-21 8:39 ` Lukas Odzioba
@ 2019-10-21 10:22 ` youling 257
2019-11-02 5:42 ` youling 257
0 siblings, 1 reply; 16+ messages in thread
From: youling 257 @ 2019-10-21 10:22 UTC (permalink / raw)
To: Lukas Odzioba; +Cc: keescook, kernel-hardening, munisekharrms
[-- Attachment #1: Type: text/plain, Size: 707 bytes --]
When add cmdline memmap=1M!2047M, the iomem will be 7ef00001-7fefffff : RAM
buffer 7ff00000-7fffffff : Persistent Memory (legacy) 7ff00000-7ff00fff :
MSFT0101:00,
so ramoops.mem_address=0x7ff00000.
Lukas Odzioba <lukas.odzioba@gmail.com> 于 2019年10月21日周一 下午4:39写道：
> youling257 <youling257@gmail.com> wrote:
> >
> > I don't know my ramoops.mem_address, please help me.
> >
> > what is ramoops.mem_address?
>
> It is a Linux kernel parameter, see documentation below:
> https://www.kernel.org/doc/Documentation/admin-guide/ramoops.rst
>
> It requires memory which can hold data between reboots, so i'm not
> sure how it will suit your case.
>
> Thanks,
> Lukas
>
[-- Attachment #2: Type: text/html, Size: 1282 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: How to get the crash dump if system hangs?
2019-10-21 10:22 ` youling 257
@ 2019-11-02 5:42 ` youling 257
0 siblings, 0 replies; 16+ messages in thread
From: youling 257 @ 2019-11-02 5:42 UTC (permalink / raw)
To: Lukas Odzioba; +Cc: Kees Cook, kernel-hardening, munisekharrms
On my v891w, i add "memmap=1M!2047M ramoops.mem_size=1048576
ramoops.ecc=1 ramoops.mem_address=0x7ff00000
ramoops.console_size=16384 ramoops.ftrace_size=16384
ramoops.pmsg_size=16384 ramoops.record_size=32768" boot parameter,
[ 0.483935] printk: console [pstore-1] enabled
[ 0.484034] pstore: Registered ramoops as persistent store backend
[ 0.484121] ramoops: using 0x100000@0x7ff00000, ecc: 16
But on my ezpad 6 m4, i add "memmap=1M!4095M ramoops.mem_size=1048576
ramoops.ecc=1 ramoops.mem_address=0xfff00000
ramoops.console_size=16384 ramoops.ftrace_size=16384
ramoops.pmsg_size=16384 ramoops.record_size=32768" boot parameter, it
can't boot, stop at "boot command list", no anyting happen, no dmesg.
I test boot parameter one by one, just add ramoops.mem_size=1048576,
will cause can't boot.
youling 257 <youling257@gmail.com> 于2019年10月21日周一 下午6:22写道：
>
> When add cmdline memmap=1M!2047M, the iomem will be 7ef00001-7fefffff : RAM buffer 7ff00000-7fffffff : Persistent Memory (legacy) 7ff00000-7ff00fff : MSFT0101:00,
> so ramoops.mem_address=0x7ff00000.
>
> Lukas Odzioba <lukas.odzioba@gmail.com> 于 2019年10月21日周一 下午4:39写道：
>>
>> youling257 <youling257@gmail.com> wrote:
>> >
>> > I don't know my ramoops.mem_address, please help me.
>> >
>> > what is ramoops.mem_address?
>>
>> It is a Linux kernel parameter, see documentation below:
>> https://www.kernel.org/doc/Documentation/admin-guide/ramoops.rst
>>
>> It requires memory which can hold data between reboots, so i'm not
>> sure how it will suit your case.
>>
>> Thanks,
>> Lukas
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: How to get the crash dump if system hangs?
@ 2019-10-20 19:31 youling257
0 siblings, 0 replies; 16+ messages in thread
From: youling257 @ 2019-10-20 19:31 UTC (permalink / raw)
To: keescook; +Cc: kernel-hardening, munisekharrms
I don't know my ramoops.mem_address, please help me.
what is ramoops.mem_address?
# cat /proc/iomem
00000000-00000fff : Reserved
00001000-0008efff : System RAM
0008f000-0008ffff : ACPI Non-volatile Storage
00090000-0009dfff : System RAM
0009e000-0009ffff : Reserved
000a0000-000bffff : PCI Bus 0000:00
000c0000-000dffff : PCI Bus 0000:00
000e0000-000fffff : PCI Bus 0000:00
000f0000-000fffff : System ROM
00100000-1fffffff : System RAM
20000000-201fffff : Reserved
20000000-201fffff : 80860F28:00
20200000-779c5fff : System RAM
4c800000-4d401150 : Kernel code
4d401151-4dd1fb3f : Kernel data
4e212000-4e3fffff : Kernel bss
779c6000-77a12fff : Reserved
77a13000-77a1bfff : Unknown E820 type
77a1c000-77afffff : Reserved
77b00000-7bb6dfff : System RAM
7bb6e000-7bbcdfff : Unknown E820 type
7bbce000-7bdcdfff : Reserved
7bdce000-7bdcefff : System RAM
7bdcf000-7bdcffff : Reserved
7bdd0000-7bdd1fff : System RAM
7bdd2000-7bdd2fff : Reserved
7bdd3000-7c7fffff : System RAM
7c800000-7cbfffff : RAM buffer
7cc00000-7cc3afff : System RAM
7cc3b000-7cc47fff : Reserved
7cc48000-7ccb7fff : ACPI Non-volatile Storage
7ccb8000-7ccf7fff : ACPI Tables
7ccf8000-7ccfffff : System RAM
7cd00000-7cefffff : RAM buffer
7cf00000-7ef00000 : Reserved
7cf00001-7ef00000 : PCI Bus 0000:00
7cf00001-7eeffffe : Graphics Stolen Memory
7ef00001-7fefffff : RAM buffer
7ff00000-7ff00fff : MSFT0101:00
7ff01000-7fffffff : RAM buffer
80000000-90affffe : PCI Bus 0000:00
80000000-8fffffff : 0000:00:02.0
90000000-903fffff : 0000:00:02.0
901e5000-901e5fff : hdmi-lpe-audio-mmio
90400000-905fffff : dwc_usb3
90400000-905fffff : 0000:00:16.0
9040c100-905fffff : dwc3.3.auto
90800000-908fffff : 0000:00:1a.0
90900000-909fffff : 0000:00:1a.0
90a00000-90a0ffff : 0000:00:14.0
90a00000-90a0ffff : xhci-hcd
90a2c000-90a2cfff : 0000:00:16.0
90b00000-90b00fff : 80860F28:00
90b01000-90b01fff : INT33BB:00
90b01000-90b01fff : INT33BB:00
90b04000-90b07fff : INTL9C60:01
90b04000-90b07fff : INTL9C60:01
90b08000-90b08fff : 80860F41:00
90b08000-90b08fff : 80860F41:00
90b0a000-90b0afff : 80860F41:01
90b0a000-90b0afff : 80860F41:01
90b0c000-90b0cfff : 80860F41:02
90b0c000-90b0cfff : 80860F41:02
90b0e000-90b0efff : 80860F41:03
90b0e000-90b0efff : 80860F41:03
90b10000-90b10fff : 80860F41:04
90b10000-90b10fff : 80860F41:04
90b13000-90b13fff : 80860F09:00
90b13000-90b13fff : 80860F09:00
90b14000-90b17fff : INTL9C60:00
90b14000-90b17fff : INTL9C60:00
90b19000-90b19fff : 80860F0A:00
90b19000-90b1901f : serial
90b1b000-90b1bfff : 80860F0A:01
90b1b000-90b1b01f : serial
90b1d000-90b1dfff : 80860F14:02
90b1d000-90b1dfff : 80860F14:02
90b1f000-90b1ffff : 80860F14:00
90b1f000-90b1ffff : 80860F14:00
90c00000-90ffffff : PCI Bus 0000:00
91000000-911fffff : 80860F28:00
e00000d0-e00000db : INT33BD:00
e00f8000-e00f8fff : Reserved
fec00000-fec003ff : IOAPIC 0
fed00000-fed003ff : HPET 0
fed00000-fed003ff : PNP0103:00
fed01000-fed01fff : Reserved
fed01000-fed011ff : intel-spi
fed03008-fed0300c : iTCO_wdt.2.auto
fed05000-fed057ff : INT3401:00
fed0c000-fed0cfff : INT33FC:00
fed0c000-fed0cfff : INT33FC:00
fed0d000-fed0dfff : INT33FC:01
fed0d000-fed0dfff : INT33FC:01
fed0e000-fed0efff : INT33FC:02
fed0e000-fed0efff : INT33FC:02
fed40000-fed40fff : PCI Bus 0000:00
fee00000-fee00fff : Local APIC
ff000000-ffffffff : INT0800:00
ffd00000-ffffffff : Reserved
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2019-11-02 5:42 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-25 20:17 How to get the crash dump if system hangs? Muni Sekhar
2019-09-30 23:51 ` Kees Cook
2019-10-03 16:48 ` Muni Sekhar
2019-10-03 21:36 ` Kees Cook
2019-10-10 15:49 ` Muni Sekhar
2019-10-10 16:56 ` Kees Cook
2019-10-10 17:15 ` Muni Sekhar
2019-10-10 18:31 ` Kees Cook
2019-10-11 14:03 ` Muni Sekhar
2019-10-25 2:10 ` Muni Sekhar
2019-10-28 19:22 ` Kees Cook
2019-10-20 19:01 youling257
2019-10-21 8:39 ` Lukas Odzioba
2019-10-21 10:22 ` youling 257
2019-11-02 5:42 ` youling 257
2019-10-20 19:31 youling257
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).