Re: How to get the crash dump if system hangs?

From: Muni Sekhar <munisekharrms@gmail.com>
To: Kees Cook <keescook@chromium.org>
Cc: kernel-hardening@lists.openwall.com
Subject: Re: How to get the crash dump if system hangs?
Date: Thu, 3 Oct 2019 22:18:48 +0530	[thread overview]
Message-ID: <CAHhAz+jyZmLBsFBxLG_XmZRBrprrxa49T+07NhcrsH4Yi6jp6A@mail.gmail.com> (raw)
In-Reply-To: <201909301645.5FA44A4@keescook>

On Tue, Oct 1, 2019 at 5:21 AM Kees Cook <keescook@chromium.org> wrote:
>
> On Thu, Sep 26, 2019 at 01:47:00AM +0530, Muni Sekhar wrote:
> > I looked at the available tests with "cat
> > /sys/kernel/debug/provoke-crash/DIRECT", from this I’d like to know
> > which test causes system hang? I could not find any test case for
> > deadlock, is any reason for this?
>
> The various *LOCKUP tests will hang a CPU or task (though SPINLOCKUP
> needs to be called twice). You could keep calling HARDLOCKUP until
> you're out of CPUs, for example. :)
>
> What kind of deadlock do you want to test?
I'm looking for a test where crash dump fails.

>
> > I’m having a Linux system, I’m seeing it gets hung during certain
> > tests. When it hung, it does not even respond for SYSRQ button, only
> > way to recover is power-button-only.  Does no response for SYSRQ
> > button means kernel crashed?
>
> That's an impressive hang! :(
>
> > After reboot I looked at the kern.log and most of the times it has
> > “^@^@^@^ ...“ line just before reboot. Can someone clarify me what the
> > kernel log entry “^@^@^@^ ...“ means? I suspect kernel is crashed, but
> > it does give any crashdump in kern.log.
>
> That's a zero byte. I would suggest using something like pstore to
> capture this in RAM instead of hoping it makes it to disk.
>
> > Later I enabled the kernel crash dump(sudo apt install
> > linux-crashdump) and rerun the test but still nothing copied to the
> > disk(/var/crash/). I don’t have onboard serial port in my machine, so
> > I tried get the crash dump via netconsole, but this method also does
> > not able to catch the crash dump.
> >
> > Can someone help me how to debug in this scenario?
> >
> > And I'd like to know what other options available to get the crash
> > dump? Can someone please clarify me on this?
> >
> > Also , does the crash dump fails if incase deadlock occurs?
> >
> > Any help will be greatly appreciated.
>
> If you really need to hard-power your system to get it back, pstore may
> only work if you're really quick and likely enable software ECC.
Thanks a lot for letting me know about pstore, will try this option.
It will be helpful if you can share some pointers on 'how to enable
software ECC'?
>
> --
> Kees Cook

-- 
Thanks,
Sekhar