Magic Sysrq key option ... What is the option to record the boot logs to my hard disk before i issue a reboot command ?

Magic Sysrq key option ... What is the option to record the boot logs to my hard disk before i issue a reboot command ?

[AIAMUZZ]

Hi,

I have this nagging and frustrating boot freeze i often face on my
Deepin OS boot ... Deepin OS i think uses 'journalctl' to record logs
on its system.

'journalctl' however seems to record boot logs ONLY for successful
boots ... boot logs for instances of boot freezes/hangs that are
issued with a ... 'Ctrl+Alt+Del' ... key combination or the ... Magic
Sysrq 'b' ... key combination to get out of the frozen/hung machine
state are just not aved in the 'journalctl' log ...


Is there any option using Magic Sysrq that will enable me to record
the failed/frozen/hung boot information to a file for troubleshooting
before i reboot the system using the option 'b' ?


If not ... Isn't it a good idea to have such an option added to Magic
Sysrq options, that can save/record the boot/system logs upto that
point, until just before we restart the machines ?

It would really help to have such an option ... an option to record
and save the machine state(logs) before we reboot the system using the
'b' option of Magic Sysrq !!!

thanks for looking

Re: Magic Sysrq key option ... What is the option to record the boot logs to my hard disk before i issue a reboot command ?

[Theodore Y. Ts'o]

On Mon, Nov 19, 2018 at 08:51:17PM +0530, AIAMUZZ wrote:
> Hi,
> 
> I have this nagging and frustrating boot freeze i often face on my
> Deepin OS boot ... Deepin OS i think uses 'journalctl' to record logs
> on its system.
> 
> 'journalctl' however seems to record boot logs ONLY for successful
> boots ... boot logs for instances of boot freezes/hangs that are
> issued with a ... 'Ctrl+Alt+Del' ... key combination or the ... Magic
> Sysrq 'b' ... key combination to get out of the frozen/hung machine
> state are just not aved in the 'journalctl' log ...
> 
> Is there any option using Magic Sysrq that will enable me to record
> the failed/frozen/hung boot information to a file for troubleshooting
> before i reboot the system using the option 'b' ?
> 
> If not ... Isn't it a good idea to have such an option added to Magic
> Sysrq options, that can save/record the boot/system logs upto that
> point, until just before we restart the machines ?

This is a hard problem to solve, because there's no place to store the
information, at least not in the general case.  The problem is on an
unsuccessful boot, the root file system may not have been mounted yet.
Heck, the storage devices might not have been probed at all!

If your hardware has a place to store dmesg output across reboots (via
one of the CONFIG_PSTORE_* kernel configuration options) then this
would be an easy problem --- in fact, it would be the default even
with out needing a magic sysrq to request it.  The problem is that
most x86 devices do not have hardware capable of supporting
CONFIG_PSTORE.  If you have a custom BIOS which doesn't clear memory
across a warm reset, that would make things easy.  Unless, unless you
are a big cloud company using custom hardware and/or a custom BIOS,
life is much more difficult.  :-(

One alternative solution you can use is to simply use a serial
console, and have a another computer monitoring the output from the
serial console.  This will allow you to see all of the kernel messages
during early bootup and afterwards.  So this is also useful for when
your X server dies and your system mysteriously reboot; either
CONFIG_PSTORE_* hardware or using a serial console might allow you to
see the kernel oops message which took out your desktop machine.  (Or
in the case of servers, you can cross connect your servers' serial
ports so that one server is monitoring another server's serial
consoles.)

Cheers,

					- Ted

Re: Magic Sysrq key option ... What is the option to record the boot logs to my hard disk before i issue a reboot command ?

[Randy Dunlap]

On 11/19/18 10:48 AM, Theodore Y. Ts'o wrote:
> On Mon, Nov 19, 2018 at 08:51:17PM +0530, AIAMUZZ wrote:
>> Hi,
>>
>> I have this nagging and frustrating boot freeze i often face on my
>> Deepin OS boot ... Deepin OS i think uses 'journalctl' to record logs
>> on its system.
>>
>> 'journalctl' however seems to record boot logs ONLY for successful
>> boots ... boot logs for instances of boot freezes/hangs that are
>> issued with a ... 'Ctrl+Alt+Del' ... key combination or the ... Magic
>> Sysrq 'b' ... key combination to get out of the frozen/hung machine
>> state are just not aved in the 'journalctl' log ...
>>
>> Is there any option using Magic Sysrq that will enable me to record
>> the failed/frozen/hung boot information to a file for troubleshooting
>> before i reboot the system using the option 'b' ?
>>
>> If not ... Isn't it a good idea to have such an option added to Magic
>> Sysrq options, that can save/record the boot/system logs upto that
>> point, until just before we restart the machines ?
> 
> This is a hard problem to solve, because there's no place to store the
> information, at least not in the general case.  The problem is on an
> unsuccessful boot, the root file system may not have been mounted yet.
> Heck, the storage devices might not have been probed at all!
> 
> If your hardware has a place to store dmesg output across reboots (via
> one of the CONFIG_PSTORE_* kernel configuration options) then this
> would be an easy problem --- in fact, it would be the default even
> with out needing a magic sysrq to request it.  The problem is that
> most x86 devices do not have hardware capable of supporting
> CONFIG_PSTORE.  If you have a custom BIOS which doesn't clear memory
> across a warm reset, that would make things easy.  Unless, unless you
> are a big cloud company using custom hardware and/or a custom BIOS,
> life is much more difficult.  :-(
> 
> One alternative solution you can use is to simply use a serial
> console, and have a another computer monitoring the output from the
> serial console.  This will allow you to see all of the kernel messages
> during early bootup and afterwards.  So this is also useful for when
> your X server dies and your system mysteriously reboot; either
> CONFIG_PSTORE_* hardware or using a serial console might allow you to
> see the kernel oops message which took out your desktop machine.  (Or
> in the case of servers, you can cross connect your servers' serial
> ports so that one server is monitoring another server's serial
> consoles.)
> 
> Cheers,

Yes, all of that.
Having some kind of pstore on x86 would be wonderful.

kexec/kdump used to be an option also.  I haven't tried it lately.

-- 
~Randy

Re: Magic Sysrq key option ... What is the option to record the boot logs to my hard disk before i issue a reboot command ?

[Theodore Y. Ts'o]

On Mon, Nov 19, 2018 at 11:31:21AM -0800, Randy Dunlap wrote:
> 
> Yes, all of that.
> Having some kind of pstore on x86 would be wonderful.
> 
> kexec/kdump used to be an option also.  I haven't tried it lately.

Sure, but kexec/kdump won't work to debug a boot failure during early
boot.  You need to be able to get the root file system mounted at the
very least, and usually kexec/kdump doesn't get set up until the init
scripts are running.

Fortunately, usually only kernel developers need to debug early boot
failures, and most of us can usually use serial consoles (for those of
us who can't just cheat and use VM's.  :-)

       	     	  	    	       - Ted

Re: Magic Sysrq key option ... What is the option to record the boot logs to my hard disk before i issue a reboot command ?

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2706 bytes --]

On Mon 2018-11-19 13:48:03, Theodore Y. Ts'o wrote:
> On Mon, Nov 19, 2018 at 08:51:17PM +0530, AIAMUZZ wrote:
> > Hi,
> > 
> > I have this nagging and frustrating boot freeze i often face on my
> > Deepin OS boot ... Deepin OS i think uses 'journalctl' to record logs
> > on its system.
> > 
> > 'journalctl' however seems to record boot logs ONLY for successful
> > boots ... boot logs for instances of boot freezes/hangs that are
> > issued with a ... 'Ctrl+Alt+Del' ... key combination or the ... Magic
> > Sysrq 'b' ... key combination to get out of the frozen/hung machine
> > state are just not aved in the 'journalctl' log ...
> > 
> > Is there any option using Magic Sysrq that will enable me to record
> > the failed/frozen/hung boot information to a file for troubleshooting
> > before i reboot the system using the option 'b' ?
> > 
> > If not ... Isn't it a good idea to have such an option added to Magic
> > Sysrq options, that can save/record the boot/system logs upto that
> > point, until just before we restart the machines ?
> 
> This is a hard problem to solve, because there's no place to store the
> information, at least not in the general case.  The problem is on an
> unsuccessful boot, the root file system may not have been mounted yet.
> Heck, the storage devices might not have been probed at all!
> 
> If your hardware has a place to store dmesg output across reboots (via
> one of the CONFIG_PSTORE_* kernel configuration options) then this
> would be an easy problem --- in fact, it would be the default even
> with out needing a magic sysrq to request it.  The problem is that
> most x86 devices do not have hardware capable of supporting
> CONFIG_PSTORE.  If you have a custom BIOS which doesn't clear memory
> across a warm reset, that would make things easy.  Unless, unless you
> are a big cloud company using custom hardware and/or a custom BIOS,
> life is much more difficult.  :-(
> 
> One alternative solution you can use is to simply use a serial
> console, and have a another computer monitoring the output from the
> serial console.  This will allow you to see all of the kernel

Serial ports are not very common these days :-(.

Ethernets are, and we have netconsole, maybe that can be used?

(Actually, I wish we had sysrq key "configure netconsole up and
re-send the dmesg buffer". Could be enabled all the time, and would
eliminate frustration of looking at blinking capslock LED and thinking
"I wish I knew what is in the buffer at the moment"....) 

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]