From: "J. Avila" <elavila@google.com>
To: John Ogness <john.ogness@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>,
Petr Mladek <pmladek@suse.com>,
Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>,
Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
Steven Rostedt <rostedt@goodmis.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Andrea Parri <parri.andrea@gmail.com>,
Thomas Gleixner <tglx@linutronix.de>,
Paul McKenney <paulmck@kernel.org>,
Saravana Kannan <saravanak@google.com>,
kexec@lists.infradead.org, linux-kernel@vger.kernel.org
Subject: Re: Issue in dmesg time with lockless ring buffer
Date: Mon, 25 Jan 2021 16:00:05 -0800 [thread overview]
Message-ID: <CAGFReeNEf7a4W4hEx5bD+v0qsdszrgPfh1Sa-B-2HeaAY5natg@mail.gmail.com> (raw)
In-Reply-To: <87im7l2lcr.fsf@jogness.linutronix.de>
Hello,
This dmesg uses /dev/kmsg; we've verified that we don't see this long
dmesg time when reading from syslog (via dmesg -S).
We've also tried testing this with logging daemons disabled as well as
within initrd - both result in similar behavior.
If it's relevant, this was done on a toybox shell.
Thanks,
Avila
On Mon, Jan 25, 2021 at 5:32 AM John Ogness <john.ogness@linutronix.de> wrote:
>
> On 2021-01-22, "J. Avila" <elavila@google.com> wrote:
> > When doing some internal testing on a 5.10.4 kernel, we found that the
> > time taken for dmesg seemed to increase from the order of milliseconds
> > to the order of seconds when the dmesg size approached the ~1.2MB
> > limit. After doing some digging, we found that by reverting all of the
> > patches in printk/ up to and including
> > 896fbe20b4e2333fb55cc9b9b783ebcc49eee7c7 ("use the lockless
> > ringbuffer"), we were able to once more see normal dmesg times.
> >
> > This kernel had no meaningful diffs in the printk/ dir when compared
> > to Linus' tree. This behavior was consistently reproducible using the
> > following steps:
> >
> > 1) In one shell, run "time dmesg > /dev/null"
> > 2) In another, constantly write to /dev/kmsg
> >
> > Within ~5 minutes, we saw that dmesg times increased to 1 second, only
> > increasing further from there. Is this a known issue?
>
> The last couple days I have tried to reproduce this issue with no
> success.
>
> Is your dmesg using /dev/kmsg or syslog() to read the buffer?
>
> Are there any syslog daemons or systemd running? Perhaps you can run
> your test within an initrd to see if this effect is still visible?
>
> John Ogness
next prev parent reply other threads:[~2021-01-26 0:01 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-07-09 13:23 [PATCH v5 0/4] printk: replace ringbuffer John Ogness
2020-07-09 13:23 ` [PATCH v5 1/4] crash: add VMCOREINFO macro to define offset in a struct declared by typedef John Ogness
2020-07-09 13:23 ` [PATCH v5 2/4] printk: add lockless ringbuffer John Ogness
2020-07-10 8:43 ` Sergey Senozhatsky
2020-07-10 8:47 ` Sergey Senozhatsky
2020-07-09 13:23 ` [PATCH v5 3/4] Revert "printk: lock/unlock console only for new logbuf entries" John Ogness
2020-07-09 13:23 ` [PATCH v5 4/4] printk: use the lockless ringbuffer John Ogness
2020-07-14 5:26 ` [printk] 96d5c61b1f: stress-ng.timerfd.ops_per_sec 8.3% improvement kernel test robot
2020-07-18 12:10 ` [PATCH v5 4/4] printk: use the lockless ringbuffer Marco Elver
2020-07-19 3:43 ` Sergey Senozhatsky
2020-07-20 6:43 ` Marco Elver
2020-07-20 8:22 ` Andy Shevchenko
2020-07-20 8:34 ` Sergey Senozhatsky
2020-07-20 8:41 ` Sergey Senozhatsky
2020-07-20 9:40 ` Marco Elver
2020-07-20 10:01 ` Dmitry Vyukov
2020-07-20 10:20 ` John Ogness
2020-07-20 10:32 ` Marco Elver
2020-08-11 20:47 ` Nick Desaulniers
2020-08-12 16:40 ` John Ogness
2021-01-22 23:52 ` Issue in dmesg time with lockless ring buffer J. Avila
2021-01-25 13:32 ` John Ogness
2021-01-26 0:00 ` J. Avila [this message]
2021-01-29 1:46 ` J. Avila
2020-07-10 8:28 ` [PATCH v5 0/4] printk: replace ringbuffer Petr Mladek
2020-07-13 2:23 ` Sergey Senozhatsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAGFReeNEf7a4W4hEx5bD+v0qsdszrgPfh1Sa-B-2HeaAY5natg@mail.gmail.com \
--to=elavila@google.com \
--cc=gregkh@linuxfoundation.org \
--cc=john.ogness@linutronix.de \
--cc=kexec@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=parri.andrea@gmail.com \
--cc=paulmck@kernel.org \
--cc=peterz@infradead.org \
--cc=pmladek@suse.com \
--cc=rostedt@goodmis.org \
--cc=saravanak@google.com \
--cc=sergey.senozhatsky.work@gmail.com \
--cc=sergey.senozhatsky@gmail.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).