How to printk synchronously

How to printk synchronously

[aleave]

[-- Attachment #1.1: Type: text/plain, Size: 671 bytes --]

Hi everyone,

I'm modifying kernel code and running into system dead-lock issues. I have printk statements to try to retrieve info about the freeze after I do a hard shutdown, but printk is asynchronous so /var/log/kern.log only shows NULL character gibberish instead (^@^@^@^@). Is there a way to make printk synchronous? I have looked at people using [printk_sync](https://lore.kernel.org/patchwork/patch/671535/) and [printk.synchronous](https://stackoverflow.com/questions/26413750/how-to-avoid-printk-log-dropping-in-linux-kernel) variables as recently as 2017, but I don't see any reference to these anymore in the kernel version I have (5.4).

Thanks for the help.

[-- Attachment #1.2: Type: text/html, Size: 1016 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: How to printk synchronously

[Lucas Tanure]

Hi,

What about ftrace ? Documentation/trace/ftrace.txt

Thanks
Lucas

On Tue, Nov 26, 2019 at 8:08 PM aleave <aleave@protonmail.com> wrote:
>
> Hi everyone,
>
> I'm modifying kernel code and running into system dead-lock issues. I have printk statements to try to retrieve info about the freeze after I do a hard shutdown, but printk is asynchronous so /var/log/kern.log only shows NULL character gibberish instead (^@^@^@^@). Is there a way to make printk synchronous? I have looked at people using printk_sync and printk.synchronous variables as recently as 2017, but I don't see any reference to these anymore in the kernel version I have (5.4).
>
> Thanks for the help.
>
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: How to printk synchronously

[Valdis Klētnieks]

[-- Attachment #1.1: Type: text/plain, Size: 882 bytes --]

On Tue, 26 Nov 2019 22:31:08 +0000, Lucas Tanure said:
> Hi,
>
> What about ftrace ? Documentation/trace/ftrace.txt

That won't help - his ^@^@^@ is a result of the system stopping and no longer
writing to disk, so his logfile has blocks allocated to it but not yet written
to.

Using ftrace will have the same problem, if his kernel is locking up and not
syncing to disk.

A better approach would be to use netconsole to send all the console output to
another machine, or serial console if that's an option.

Though I have to wonder how he's determining it's a deadlock issue rather than
a panic that's just plain stopping the system - building the kernel with
lockdep support should reveal the issue before a lockup. And if it *is* a
deadlock, there should be sufficient info in the watchdog logging (assuming the
hardware has some sort of watchdog support, as most systems do).


[-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: How to printk synchronously

[Muni Sekhar]

On Wed, Nov 27, 2019 at 2:48 PM Valdis Klētnieks
<valdis.kletnieks@vt.edu> wrote:
>
> On Tue, 26 Nov 2019 22:31:08 +0000, Lucas Tanure said:
> > Hi,
> >
> > What about ftrace ? Documentation/trace/ftrace.txt
>
> That won't help - his ^@^@^@ is a result of the system stopping and no longer
> writing to disk, so his logfile has blocks allocated to it but not yet written
> to.
>
> Using ftrace will have the same problem, if his kernel is locking up and not
> syncing to disk.
I too noticed this kind of behavior i.e. kernel locks up and won’t
respond for SYSRQ key(even it is enabled), just system freezes. Only
way to recover is hard reboot, after reboot I looked at the kernel log
and most of the times it has ^@^@^@.

My system is installed with crashdump tools and it captures the
crashed Linux kernel dump for soft\hard lockups triggered via
/sys/kernel/debug/provoke-crash/DIRECT file, but the crashdump tool
not helped in the original scenario.

Apart from soft\hard lockups, are there any other scenarios kernel
locks up and won’t respond for SYSRQ key? Is there any kernel patch I
can apply to get the stack trace of each CPU before it stops?

I enabled “kernel.softlockup_panic = 1” and “kernel.hardlockup_panic =
1” variables, but just system freezes without panic, so can we say
this fault neither soft lockup nor hard lockup?

Does machine check exceptions result the system to freeze? If so how
to capture the dump\log for this scenario?

I build the kernel with lockdep too, but does not show any possible
lockups. Netconsole also not helped.


>
> A better approach would be to use netconsole to send all the console output to
> another machine, or serial console if that's an option.
>
> Though I have to wonder how he's determining it's a deadlock issue rather than
> a panic that's just plain stopping the system - building the kernel with
> lockdep support should reveal the issue before a lockup. And if it *is* a
> deadlock, there should be sufficient info in the watchdog logging (assuming the
> hardware has some sort of watchdog support, as most systems do).
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies



-- 
Thanks,
Sekhar

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies