Kernel Newbies archive on lore.kernel.org
 help / color / Atom feed
* How to printk synchronously
@ 2019-11-26 20:07 aleave
  2019-11-26 22:31 ` Lucas Tanure
  0 siblings, 1 reply; 4+ messages in thread
From: aleave @ 2019-11-26 20:07 UTC (permalink / raw)
  To: kernelnewbies

[-- Attachment #1.1: Type: text/plain, Size: 671 bytes --]

Hi everyone,

I'm modifying kernel code and running into system dead-lock issues. I have printk statements to try to retrieve info about the freeze after I do a hard shutdown, but printk is asynchronous so /var/log/kern.log only shows NULL character gibberish instead (^@^@^@^@). Is there a way to make printk synchronous? I have looked at people using [printk_sync](https://lore.kernel.org/patchwork/patch/671535/) and [printk.synchronous](https://stackoverflow.com/questions/26413750/how-to-avoid-printk-log-dropping-in-linux-kernel) variables as recently as 2017, but I don't see any reference to these anymore in the kernel version I have (5.4).

Thanks for the help.

[-- Attachment #1.2: Type: text/html, Size: 1016 bytes --]

<div>Hi everyone,<br></div><div><br></div><div>I'm modifying kernel code and running into system dead-lock issues. I have printk statements to try to retrieve info about the freeze after I do a hard shutdown, but printk is asynchronous so /var/log/kern.log only shows NULL character gibberish instead (^@^@^@^@). Is there a way to make printk synchronous? I have looked at people using <a rel="nofollow" title="https://lore.kernel.org/patchwork/patch/671535/" target="_blank" href="https://lore.kernel.org/patchwork/patch/671535/">printk_sync</a> and <a rel="nofollow" title="https://stackoverflow.com/questions/26413750/how-to-avoid-printk-log-dropping-in-linux-kernel" target="_blank" href="https://stackoverflow.com/questions/26413750/how-to-avoid-printk-log-dropping-in-linux-kernel">printk.synchronous</a> variables as recently as 2017, but I don't see any reference to these anymore in the kernel version I have (5.4).&nbsp;<br></div><div><br></div><div>Thanks for the help.</div><div><br></div><div><br></div>

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: How to printk synchronously
  2019-11-26 20:07 How to printk synchronously aleave
@ 2019-11-26 22:31 ` Lucas Tanure
  2019-11-27  9:18   ` Valdis Klētnieks
  0 siblings, 1 reply; 4+ messages in thread
From: Lucas Tanure @ 2019-11-26 22:31 UTC (permalink / raw)
  To: aleave; +Cc: kernelnewbies

Hi,

What about ftrace ? Documentation/trace/ftrace.txt

Thanks
Lucas

On Tue, Nov 26, 2019 at 8:08 PM aleave <aleave@protonmail.com> wrote:
>
> Hi everyone,
>
> I'm modifying kernel code and running into system dead-lock issues. I have printk statements to try to retrieve info about the freeze after I do a hard shutdown, but printk is asynchronous so /var/log/kern.log only shows NULL character gibberish instead (^@^@^@^@). Is there a way to make printk synchronous? I have looked at people using printk_sync and printk.synchronous variables as recently as 2017, but I don't see any reference to these anymore in the kernel version I have (5.4).
>
> Thanks for the help.
>
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: How to printk synchronously
  2019-11-26 22:31 ` Lucas Tanure
@ 2019-11-27  9:18   ` Valdis Klētnieks
  2019-11-28  6:12     ` Muni Sekhar
  0 siblings, 1 reply; 4+ messages in thread
From: Valdis Klētnieks @ 2019-11-27  9:18 UTC (permalink / raw)
  To: tanure; +Cc: kernelnewbies, aleave

[-- Attachment #1.1: Type: text/plain, Size: 882 bytes --]

On Tue, 26 Nov 2019 22:31:08 +0000, Lucas Tanure said:
> Hi,
>
> What about ftrace ? Documentation/trace/ftrace.txt

That won't help - his ^@^@^@ is a result of the system stopping and no longer
writing to disk, so his logfile has blocks allocated to it but not yet written
to.

Using ftrace will have the same problem, if his kernel is locking up and not
syncing to disk.

A better approach would be to use netconsole to send all the console output to
another machine, or serial console if that's an option.

Though I have to wonder how he's determining it's a deadlock issue rather than
a panic that's just plain stopping the system - building the kernel with
lockdep support should reveal the issue before a lockup. And if it *is* a
deadlock, there should be sufficient info in the watchdog logging (assuming the
hardware has some sort of watchdog support, as most systems do).


[-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: How to printk synchronously
  2019-11-27  9:18   ` Valdis Klētnieks
@ 2019-11-28  6:12     ` Muni Sekhar
  0 siblings, 0 replies; 4+ messages in thread
From: Muni Sekhar @ 2019-11-28  6:12 UTC (permalink / raw)
  To: Valdis Klētnieks; +Cc: aleave, tanure, kernelnewbies

On Wed, Nov 27, 2019 at 2:48 PM Valdis Klētnieks
<valdis.kletnieks@vt.edu> wrote:
>
> On Tue, 26 Nov 2019 22:31:08 +0000, Lucas Tanure said:
> > Hi,
> >
> > What about ftrace ? Documentation/trace/ftrace.txt
>
> That won't help - his ^@^@^@ is a result of the system stopping and no longer
> writing to disk, so his logfile has blocks allocated to it but not yet written
> to.
>
> Using ftrace will have the same problem, if his kernel is locking up and not
> syncing to disk.
I too noticed this kind of behavior i.e. kernel locks up and won’t
respond for SYSRQ key(even it is enabled), just system freezes. Only
way to recover is hard reboot, after reboot I looked at the kernel log
and most of the times it has ^@^@^@.

My system is installed with crashdump tools and it captures the
crashed Linux kernel dump for soft\hard lockups triggered via
/sys/kernel/debug/provoke-crash/DIRECT file, but the crashdump tool
not helped in the original scenario.

Apart from soft\hard lockups, are there any other scenarios kernel
locks up and won’t respond for SYSRQ key? Is there any kernel patch I
can apply to get the stack trace of each CPU before it stops?

I enabled “kernel.softlockup_panic = 1” and “kernel.hardlockup_panic =
1” variables, but just system freezes without panic, so can we say
this fault neither soft lockup nor hard lockup?

Does machine check exceptions result the system to freeze? If so how
to capture the dump\log for this scenario?

I build the kernel with lockdep too, but does not show any possible
lockups. Netconsole also not helped.


>
> A better approach would be to use netconsole to send all the console output to
> another machine, or serial console if that's an option.
>
> Though I have to wonder how he's determining it's a deadlock issue rather than
> a panic that's just plain stopping the system - building the kernel with
> lockdep support should reveal the issue before a lockup. And if it *is* a
> deadlock, there should be sufficient info in the watchdog logging (assuming the
> hardware has some sort of watchdog support, as most systems do).
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies



-- 
Thanks,
Sekhar

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, back to index

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-26 20:07 How to printk synchronously aleave
2019-11-26 22:31 ` Lucas Tanure
2019-11-27  9:18   ` Valdis Klētnieks
2019-11-28  6:12     ` Muni Sekhar

Kernel Newbies archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/kernelnewbies/0 kernelnewbies/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 kernelnewbies kernelnewbies/ https://lore.kernel.org/kernelnewbies \
		kernelnewbies@kernelnewbies.org
	public-inbox-index kernelnewbies

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernelnewbies.kernelnewbies


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git