From: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
To: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Cc: Steven Rostedt <rostedt@goodmis.org>,
Petr Mladek <pmladek@suse.com>,
Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
Dmitriy Vyukov <dvyukov@google.com>,
Alexander Potapenko <glider@google.com>,
Fengguang Wu <fengguang.wu@intel.com>,
Josh Poimboeuf <jpoimboe@redhat.com>,
LKML <linux-kernel@vger.kernel.org>,
Andrew Morton <akpm@linux-foundation.org>,
linux-mm@kvack.org, Ingo Molnar <mingo@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
Will Deacon <will.deacon@arm.com>
Subject: Re: [PATCH 3/3] lockdep: Use line-buffered printk() for lockdep messages.
Date: Mon, 26 Nov 2018 13:34:45 +0900 [thread overview]
Message-ID: <20181126043445.GB540@jagdpanzerIV> (raw)
In-Reply-To: <6422717f-db27-8ba8-1183-ccb9f0400fc3@i-love.sakura.ne.jp>
On (11/24/18 09:24), Tetsuo Handa wrote:
> >> Steven told me on Plumbers conference that even few initial
> >> characters saved him a day few times.
> >
> > Yes, and that has happened more than once. I would reboot and retest
> > code that is crashing, and due to a triple fault, the machine would
> > reboot because of some race, and the little output I get from the
> > console would help tremendously.
> >
> > Remember, debugging the kernel is a lot like forensics, especially when
> > it's from a customer's site. You look at all the evidence that you can
> > get, and sometimes it's just 10 characters in the output that gives you
> > an idea of where things went wrong. I'm really not liking the buffering
> > idea because of this.
>
> Then, we should not enforce buffering in a way that requires modification of
> printk() callers. That is, we should not ask printk() callers to use their
> private buffer. What we can do is to enable/disable line buffering inside
> printk() depending on the problem the user wants to debug.
Right; overall I tend to agree with what you guys are saying and
I like Petr's "I am more and more wondering if the buffered printk
is worth the effort" comment; and I like Steven's comment on flushes;
and admire Tetsuo's efforts.
I think that printk_seq_buf/printk_buffer was never going to replace
pr_cont() and I never liked the idea. The printk_safe proposal for
lockdep had one OK thing about it - it would pass our normal marshaling
before it would reach the buffering stage. Which means - no buffering
for people who "detest" printk buffering.
This looks better:
printk->vprint_func->{early_printk/printk_safe/vprintk_emit}->buffering
Than this:
pr_buffer->buffering->vprintk_func->{early_printk/printk_safe/vprintk_emit}
Another thing is - printk seq_buf/printk_buffer doesn't really solve any
problem. People, who can use seq_buf/char buf[256]/etc. buffering, already
can do so; people who cannot - won't switch to a new buffering printk
anyway.
The bad thing about printk_safe proposal is that it's per-CPU; which
is OK for some paths (like lockdep), but not OK in general (e.g. OOM).
IMO, try_buffered_printk() attempts to solve the problem at the right
place - printk. And it does not break our normal marshaling, so we don't
"fix" printk users and we keep people, who does vprintk_func->early_printk
thing, happy. So I don't dislike try_buffered_printk() approach. And unlike
before, now we are talking about a single line buffering.
If we'd walk this way, I would prefer to NOT introduce any structs and
any new code, or any new "split and log_store() in the middle" rules. Just
a bunch of "struct cont" buffers:
static struct cont conts[N];
and cont_add()/cont_flush() to handle pr_cont, with all the flushes it
does; but on a per-context basis.
conts[0] should serve as a fallback cont buffer, in case if there are
no available cont buffers left. flush_on_panic() is still miserable,
for sure; probably we can do something about it.
Or... Instead.
We can just leave pr_cont() alone for now. And make it possible to
reconstruct messages - IOW, inject some info to printk messages. We
do this at Samsung (inject CPU number at the beginning of every
message. `cat serial.0 | grep "\[1\]"` to grep for all messages from
CPU1). Probably this would be the simplest thing.
> Also, we should allow disabling "struct cont" depending on the problem (in
> order to allow flushing the 10 characters in the "cont" buffer).
>
> By the way, is the comment
>
> /*
> * Continuation lines are buffered, and not committed to the record buffer
> * until the line is complete, or a race forces it. The line fragments
> * though, are printed immediately to the consoles to ensure everything has
> * reached the console in case of a kernel crash.
> */
printk does not do this anymore; you are right.
> appropriate despite we don't call cont_flush() upon a kernel crash?
I tend to count on flush_on_panic more than on a "last moment"
pr_cont->cont_flush(), which was guaranteed to happen immediately
only with early_con.
A kernel crash usually has enough pr_emerg/printk-s to force cont flush.
Even pr_info() will do. We look at the loglevel much later; so even
messages which never make it to the consoles still flush cont buffer.
-ss
next prev parent reply other threads:[~2018-11-26 4:34 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-02 13:31 [PATCH v6 1/3] printk: Add line-buffered printk() API Tetsuo Handa
2018-11-02 13:31 ` [PATCH 2/3] mm: Use line-buffered printk() for show_free_areas() Tetsuo Handa
2018-11-07 14:07 ` Petr Mladek
2018-11-02 13:31 ` [PATCH 3/3] lockdep: Use line-buffered printk() for lockdep messages Tetsuo Handa
2018-11-02 13:36 ` Peter Zijlstra
2018-11-03 2:00 ` Tetsuo Handa
2018-11-06 8:38 ` Petr Mladek
2018-11-06 9:05 ` Sergey Senozhatsky
2018-11-06 12:57 ` Petr Mladek
2018-11-06 9:56 ` Tetsuo Handa
2018-11-07 15:19 ` Petr Mladek
2018-11-08 4:45 ` Sergey Senozhatsky
2018-11-08 11:37 ` Tetsuo Handa
2018-11-09 6:12 ` Sergey Senozhatsky
2018-11-09 9:55 ` Tetsuo Handa
2018-11-09 15:43 ` Petr Mladek
2018-11-10 2:42 ` Tetsuo Handa
2018-11-23 12:46 ` Petr Mladek
2018-11-23 13:12 ` Tetsuo Handa
2018-11-23 15:56 ` Steven Rostedt
2018-11-24 0:24 ` Tetsuo Handa
2018-11-26 4:34 ` Sergey Senozhatsky [this message]
2018-11-28 13:29 ` David Laight
2018-11-29 10:09 ` Tetsuo Handa
2018-11-30 16:01 ` Petr Mladek
2018-11-10 8:52 ` Tetsuo Handa
2018-11-23 12:52 ` Petr Mladek
2018-11-09 14:08 ` Linus Torvalds
2018-11-09 14:42 ` Steven Rostedt
2018-11-08 11:53 ` Petr Mladek
2018-11-08 12:44 ` Sergey Senozhatsky
2018-11-08 14:21 ` Sergey Senozhatsky
2018-11-09 9:54 ` Tetsuo Handa
2018-11-02 14:40 ` [PATCH v6 1/3] printk: Add line-buffered printk() API Matthew Wilcox
2018-11-03 1:55 ` Tetsuo Handa
2018-11-02 18:12 ` kbuild test robot
2018-11-06 14:35 ` Sergey Senozhatsky
2018-11-07 10:21 ` Petr Mladek
2018-11-07 12:54 ` Tetsuo Handa
2018-11-08 2:21 ` Sergey Senozhatsky
2018-11-08 11:24 ` Petr Mladek
2018-11-08 11:46 ` Tetsuo Handa
2018-11-08 12:30 ` Sergey Senozhatsky
2018-11-09 14:10 ` Petr Mladek
2018-11-12 7:59 ` Sergey Senozhatsky
2018-11-12 10:42 ` Tetsuo Handa
2018-11-17 10:14 ` Tetsuo Handa
2018-11-07 10:52 ` Tetsuo Handa
2018-11-07 11:01 ` David Laight
2018-11-07 12:00 ` Petr Mladek
2018-11-07 11:45 ` Petr Mladek
2018-11-08 2:30 ` Sergey Senozhatsky
2018-11-07 13:41 ` Petr Mladek
2018-11-07 14:06 ` Petr Mladek
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181126043445.GB540@jagdpanzerIV \
--to=sergey.senozhatsky.work@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=dvyukov@google.com \
--cc=fengguang.wu@intel.com \
--cc=glider@google.com \
--cc=jpoimboe@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mingo@redhat.com \
--cc=penguin-kernel@i-love.sakura.ne.jp \
--cc=peterz@infradead.org \
--cc=pmladek@suse.com \
--cc=rostedt@goodmis.org \
--cc=sergey.senozhatsky@gmail.com \
--cc=torvalds@linux-foundation.org \
--cc=will.deacon@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).