From: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: Tejun Heo <tj@kernel.org>,
Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>,
Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
Petr Mladek <pmladek@suse.com>, Jan Kara <jack@suse.cz>,
Andrew Morton <akpm@linux-foundation.org>,
Peter Zijlstra <peterz@infradead.org>,
Rafael Wysocki <rjw@rjwysocki.net>, Pavel Machek <pavel@ucw.cz>,
Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>,
linux-kernel@vger.kernel.org
Subject: Re: [RFC][PATCHv6 00/12] printk: introduce printing kernel thread
Date: Fri, 15 Dec 2017 15:52:05 +0900 [thread overview]
Message-ID: <20171215065205.GB468@jagdpanzerIV> (raw)
In-Reply-To: <20171215050607.GC11199@jagdpanzerIV>
On (12/15/17 14:06), Sergey Senozhatsky wrote:
[..]
> > Where do we do the above? And has this been proven to be an issue?
>
> um... hundreds of cases.
>
> deep-stack spin_lock_irqsave() lockup reports from multiple CPUs (3 cpus)
> happening at the same moment + NMI backtraces from all the CPUs (more
> than 3 cpus) that follows the lockups, over not-so-fast serial console.
> exactly the bug report I received two days ago. so which one of the CPUs
> here is a good candidate to successfully emit all of the pending logbuf
> entries? none. all of them either have local IRQs disabled, or dump_stack()
> from either backtrace IPI or backtrace NMI (depending on the configuration).
and, Steven, one more thing. wondering what's your opinion.
suppose we have consoe_owner hand off enabled, 1 non-atomic CPU doing
printk-s and several atomic CPUs doing printk-s. Is proposed hand off
scheme really useful in this case? CPUs will now
a) print their lines (a potentially slow call_console_drivers())
and
b) spin in vprintk_emit on console_owner with local IRQs disabled
waiting for either non-atomic printk CPU or another atomic CPU
to finish printing its line (call_console_drivers()) and to hand
off printing. so current CPU, after busy-waiting for foreign CPU's
call_console_drivers(), will go and do his own call_console_drivers().
which, time-wise, simply doubles (roughly) the amount of time that
CPU spends in printk()->console_unlock(). agreed?
if we previously could have a case when non-atomic printk CPU would
grab the console_sem and print all atomic printk CPUs messages first,
and then its own messages, thus atomic printk CPUs would have just
log_store(), now we will have CPUs to call_console_driver() and to
spin on console_sem owner waiting for call_console_driver() on a foreign
CPU [not all of them: it's one CPU doing the print out and one CPU
spinning console_owner. but overall I think all CPUs will experience
that spin on console_sem waiting for call_console_driver() and then do
its own call_console_driver()].
even two CPUs case is not so simple anymore. see below.
- first, assume one CPU is atomic and one is non-atomic.
- second, assume that both CPUs are atomic CPUs, and go thought it again.
CPU0 CPU1
printk() printk()
log_store()
log_store()
console_unlock()
set console_owner
sees console_owner
sets console_waiter
spin
call_console_drivers()
sees console_waiter
break
printk()
log_store()
console_unlock()
set console_owner
sees console_owner
sets console_waiter
spin
call_console_drivers()
sees console_waiter
break
printk()
log_store()
console_unlock()
set console_owner
sees console_owner
sets console_waiter
spin
call_console_drivers()
sees console_waiter
break
printk()
log_store()
console_unlock()
set console_owner
sees console_owner
sets console_waiter
spin
....
that "wait for call_console_drivers() on another CPU and then do
its own call_console_drivers()" pattern does look dangerous. the
benefit of hand-off is really fragile sometimes, isn't it?
-ss
next prev parent reply other threads:[~2017-12-15 6:52 UTC|newest]
Thread overview: 79+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-12-04 13:48 [RFC][PATCHv6 00/12] printk: introduce printing kernel thread Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 01/12] printk: move printk_pending out of per-cpu Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 02/12] printk: introduce printing kernel thread Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 03/12] printk: consider watchdogs thresholds for offloading Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 04/12] printk: add sync printk_emergency API Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 05/12] printk: enable printk offloading Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 06/12] PM: switch between printk emergency modes Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 07/12] printk: register syscore notifier Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 08/12] printk: force printk_kthread to offload printing Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 09/12] printk: do not cond_resched() when we can offload Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 10/12] printk: move offloading logic to per-cpu Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 11/12] printk: add offloading watchdog API Sergey Senozhatsky
2017-12-04 13:48 ` [RFC][PATCHv6 12/12] printk: improve printk offloading mechanism Sergey Senozhatsky
2017-12-04 13:53 ` [PATCH 0/4] printk: offloading testing module/trace events Sergey Senozhatsky
2017-12-04 13:53 ` [PATCH 1/4] printk/lib: add offloading trace events and test_printk module Sergey Senozhatsky
2017-12-04 13:53 ` [PATCH 2/4] printk/lib: simulate slow consoles Sergey Senozhatsky
2017-12-04 13:53 ` [PATCH 3/4] printk: add offloading takeover traces Sergey Senozhatsky
2017-12-04 13:53 ` [PATCH 4/4] printk: add task name and CPU to console messages Sergey Senozhatsky
2017-12-14 14:27 ` [RFC][PATCHv6 00/12] printk: introduce printing kernel thread Petr Mladek
2017-12-14 14:39 ` Sergey Senozhatsky
2017-12-15 15:55 ` Steven Rostedt
2017-12-14 15:25 ` Tejun Heo
2017-12-14 17:55 ` Steven Rostedt
2017-12-14 18:11 ` Tejun Heo
2017-12-14 18:21 ` Steven Rostedt
2017-12-22 0:09 ` Tejun Heo
2017-12-22 4:19 ` Steven Rostedt
2017-12-28 6:48 ` Sergey Senozhatsky
2017-12-28 10:07 ` Sergey Senozhatsky
2017-12-29 13:59 ` Tetsuo Handa
2017-12-31 1:44 ` Sergey Senozhatsky
2018-01-09 20:06 ` Tejun Heo
2018-01-09 22:08 ` Tetsuo Handa
2018-01-09 22:17 ` Tejun Heo
2018-01-11 11:14 ` Tetsuo Handa
2018-01-09 22:08 ` Steven Rostedt
2018-01-09 22:17 ` Tejun Heo
2018-01-09 22:47 ` Steven Rostedt
2018-01-09 22:53 ` Tejun Heo
2018-01-10 7:18 ` Steven Rostedt
2018-01-10 14:04 ` Tejun Heo
2017-12-15 2:10 ` Sergey Senozhatsky
2017-12-15 3:18 ` Steven Rostedt
2017-12-15 5:06 ` Sergey Senozhatsky
2017-12-15 6:52 ` Sergey Senozhatsky [this message]
2017-12-15 15:39 ` Steven Rostedt
2017-12-15 8:31 ` Petr Mladek
2017-12-15 8:42 ` Sergey Senozhatsky
2017-12-15 9:08 ` Petr Mladek
2017-12-15 15:47 ` Steven Rostedt
2017-12-18 9:36 ` Sergey Senozhatsky
2017-12-18 10:36 ` Sergey Senozhatsky
2017-12-18 12:35 ` Sergey Senozhatsky
2017-12-18 13:51 ` Petr Mladek
2017-12-18 13:31 ` Petr Mladek
2017-12-18 13:39 ` Sergey Senozhatsky
2017-12-18 14:13 ` Petr Mladek
2017-12-18 17:46 ` Steven Rostedt
2017-12-19 1:03 ` Sergey Senozhatsky
2017-12-19 1:08 ` Steven Rostedt
2017-12-19 1:24 ` Sergey Senozhatsky
2017-12-19 2:03 ` Steven Rostedt
2017-12-19 2:46 ` Sergey Senozhatsky
2017-12-19 3:38 ` Steven Rostedt
2017-12-19 4:58 ` Sergey Senozhatsky
2017-12-19 14:40 ` Steven Rostedt
2017-12-20 7:46 ` Sergey Senozhatsky
2017-12-19 14:31 ` Michal Hocko
2017-12-20 7:10 ` Sergey Senozhatsky
2017-12-20 12:06 ` Tetsuo Handa
2017-12-21 6:52 ` Sergey Senozhatsky
2017-12-19 4:36 ` Sergey Senozhatsky
2017-12-18 14:10 ` Petr Mladek
2017-12-19 1:09 ` Sergey Senozhatsky
2017-12-15 15:42 ` Steven Rostedt
2017-12-15 15:19 ` Steven Rostedt
2017-12-19 0:52 ` Sergey Senozhatsky
2017-12-19 1:03 ` Steven Rostedt
2018-01-05 2:54 ` Sergey Senozhatsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20171215065205.GB468@jagdpanzerIV \
--to=sergey.senozhatsky.work@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=jack@suse.cz \
--cc=linux-kernel@vger.kernel.org \
--cc=pavel@ucw.cz \
--cc=penguin-kernel@I-love.SAKURA.ne.jp \
--cc=peterz@infradead.org \
--cc=pmladek@suse.com \
--cc=rjw@rjwysocki.net \
--cc=rostedt@goodmis.org \
--cc=sergey.senozhatsky@gmail.com \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).