* How to printk synchronously @ 2019-11-26 20:07 aleave 2019-11-26 22:31 ` Lucas Tanure 0 siblings, 1 reply; 4+ messages in thread From: aleave @ 2019-11-26 20:07 UTC (permalink / raw) To: kernelnewbies [-- Attachment #1.1: Type: text/plain, Size: 671 bytes --] Hi everyone, I'm modifying kernel code and running into system dead-lock issues. I have printk statements to try to retrieve info about the freeze after I do a hard shutdown, but printk is asynchronous so /var/log/kern.log only shows NULL character gibberish instead (^@^@^@^@). Is there a way to make printk synchronous? I have looked at people using [printk_sync](https://lore.kernel.org/patchwork/patch/671535/) and [printk.synchronous](https://stackoverflow.com/questions/26413750/how-to-avoid-printk-log-dropping-in-linux-kernel) variables as recently as 2017, but I don't see any reference to these anymore in the kernel version I have (5.4). Thanks for the help. [-- Attachment #1.2: Type: text/html, Size: 1016 bytes --] [-- Attachment #2: Type: text/plain, Size: 170 bytes --] _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: How to printk synchronously 2019-11-26 20:07 How to printk synchronously aleave @ 2019-11-26 22:31 ` Lucas Tanure 2019-11-27 9:18 ` Valdis Klētnieks 0 siblings, 1 reply; 4+ messages in thread From: Lucas Tanure @ 2019-11-26 22:31 UTC (permalink / raw) To: aleave; +Cc: kernelnewbies Hi, What about ftrace ? Documentation/trace/ftrace.txt Thanks Lucas On Tue, Nov 26, 2019 at 8:08 PM aleave <aleave@protonmail.com> wrote: > > Hi everyone, > > I'm modifying kernel code and running into system dead-lock issues. I have printk statements to try to retrieve info about the freeze after I do a hard shutdown, but printk is asynchronous so /var/log/kern.log only shows NULL character gibberish instead (^@^@^@^@). Is there a way to make printk synchronous? I have looked at people using printk_sync and printk.synchronous variables as recently as 2017, but I don't see any reference to these anymore in the kernel version I have (5.4). > > Thanks for the help. > > > _______________________________________________ > Kernelnewbies mailing list > Kernelnewbies@kernelnewbies.org > https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: How to printk synchronously 2019-11-26 22:31 ` Lucas Tanure @ 2019-11-27 9:18 ` Valdis Klētnieks 2019-11-28 6:12 ` Muni Sekhar 0 siblings, 1 reply; 4+ messages in thread From: Valdis Klētnieks @ 2019-11-27 9:18 UTC (permalink / raw) To: tanure; +Cc: kernelnewbies, aleave [-- Attachment #1.1: Type: text/plain, Size: 882 bytes --] On Tue, 26 Nov 2019 22:31:08 +0000, Lucas Tanure said: > Hi, > > What about ftrace ? Documentation/trace/ftrace.txt That won't help - his ^@^@^@ is a result of the system stopping and no longer writing to disk, so his logfile has blocks allocated to it but not yet written to. Using ftrace will have the same problem, if his kernel is locking up and not syncing to disk. A better approach would be to use netconsole to send all the console output to another machine, or serial console if that's an option. Though I have to wonder how he's determining it's a deadlock issue rather than a panic that's just plain stopping the system - building the kernel with lockdep support should reveal the issue before a lockup. And if it *is* a deadlock, there should be sufficient info in the watchdog logging (assuming the hardware has some sort of watchdog support, as most systems do). [-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --] [-- Attachment #2: Type: text/plain, Size: 170 bytes --] _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: How to printk synchronously 2019-11-27 9:18 ` Valdis Klētnieks @ 2019-11-28 6:12 ` Muni Sekhar 0 siblings, 0 replies; 4+ messages in thread From: Muni Sekhar @ 2019-11-28 6:12 UTC (permalink / raw) To: Valdis Klētnieks; +Cc: aleave, tanure, kernelnewbies On Wed, Nov 27, 2019 at 2:48 PM Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote: > > On Tue, 26 Nov 2019 22:31:08 +0000, Lucas Tanure said: > > Hi, > > > > What about ftrace ? Documentation/trace/ftrace.txt > > That won't help - his ^@^@^@ is a result of the system stopping and no longer > writing to disk, so his logfile has blocks allocated to it but not yet written > to. > > Using ftrace will have the same problem, if his kernel is locking up and not > syncing to disk. I too noticed this kind of behavior i.e. kernel locks up and won’t respond for SYSRQ key(even it is enabled), just system freezes. Only way to recover is hard reboot, after reboot I looked at the kernel log and most of the times it has ^@^@^@. My system is installed with crashdump tools and it captures the crashed Linux kernel dump for soft\hard lockups triggered via /sys/kernel/debug/provoke-crash/DIRECT file, but the crashdump tool not helped in the original scenario. Apart from soft\hard lockups, are there any other scenarios kernel locks up and won’t respond for SYSRQ key? Is there any kernel patch I can apply to get the stack trace of each CPU before it stops? I enabled “kernel.softlockup_panic = 1” and “kernel.hardlockup_panic = 1” variables, but just system freezes without panic, so can we say this fault neither soft lockup nor hard lockup? Does machine check exceptions result the system to freeze? If so how to capture the dump\log for this scenario? I build the kernel with lockdep too, but does not show any possible lockups. Netconsole also not helped. > > A better approach would be to use netconsole to send all the console output to > another machine, or serial console if that's an option. > > Though I have to wonder how he's determining it's a deadlock issue rather than > a panic that's just plain stopping the system - building the kernel with > lockdep support should reveal the issue before a lockup. And if it *is* a > deadlock, there should be sufficient info in the watchdog logging (assuming the > hardware has some sort of watchdog support, as most systems do). > > _______________________________________________ > Kernelnewbies mailing list > Kernelnewbies@kernelnewbies.org > https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies -- Thanks, Sekhar _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2019-11-28 6:13 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-11-26 20:07 How to printk synchronously aleave 2019-11-26 22:31 ` Lucas Tanure 2019-11-27 9:18 ` Valdis Klētnieks 2019-11-28 6:12 ` Muni Sekhar
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).