From: Petr Mladek <pmladek@suse.com>
To: Vladislav Levenetz <vlevenetz@mm-sol.com>
Cc: Viresh Kumar <viresh.kumar@linaro.org>, Jan Kara <jack@suse.cz>,
Andrew Morton <akpm@linux-foundation.org>,
Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
Jan Kara <jack@suse.com>, Tejun Heo <tj@kernel.org>,
Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
Byungchul Park <byungchul.park@lge.com>,
Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Subject: Re: [PATCH v10 1/2] printk: Make printk() completely async
Date: Tue, 16 Aug 2016 11:04:30 +0200 [thread overview]
Message-ID: <20160816090430.GK13300@pathway.suse.cz> (raw)
In-Reply-To: <57B1D12A.6030106@mm-sol.com>
On Mon 2016-08-15 17:26:50, Vladislav Levenetz wrote:
> On 08/12/2016 12:44 PM, Petr Mladek wrote:
> >But I was curious if we could hit a printk from the wake_up_process().
> >The change above causes using the fair scheduler and there is
> >the following call chain [*]
> >
> > vprintk_emit()
> > -> wake_up_process()
> > -> try_to_wake_up()
> > -> ttwu_queue()
> > -> ttwu_do_activate()
> > -> ttwu_activate()
> > -> activate_task()
> > -> enqueue_task()
> > -> enqueue_task_fair() via p->sched_class->enqueue_task
> > -> cfs_rq_of()
> > -> task_of()
> > -> WARN_ON_ONCE(!entity_is_task(se))
> >
> >We should never trigger this because printk_kthread is a task.
> >But what if the date gets inconsistent?
> >
> >Then there is the following chain:
> >
> > vprintk_emit()
> > -> wake_up_process()
> > -> try_to_wake_up()
> > -> ttwu_queue()
> > -> ttwu_do_activate()
> > -> ttwu_activate()
> > -> activate_task()
> > -> enqueue_task()
> > -> enqueue_task_fair() via p->sched_class->enqueue_task
> > ->hrtick_update()
> > -> hrtick_start_fair()
> > -> WARN_ON(task_rq(p) != rq)
> >
> >This looks like another paranoid consistency check that might be
> >triggered when the scheduler gets messed.
> >
> >I see few possible solutions:
> >
> >1. Replace the WARN_ONs by printk_deferred().
> >
> > This is the usual solution but it would make debugging less convenient.
> >
> >
> >2. Force synchronous printk inside WARN()/BUG() macros.
> >
> > This would make sense even from other reasons. These are printed
> > when the system is in a strange state. There is no guarantee that
> > the printk_kthread will get scheduled.
> >
> >
> >3. Force printk_deferred() inside WARN()/BUG() macros via the per-CPU
> > printk_func.
> >
> > It might be elegant. But we do not want this outside the scheduler
> > code. Therefore we would need special variants of WARN_*_SCHED()
> > BUG_*_SCHED() macros.
> >
> >
> >I personally prefer the 2nd solution. What do you think about it,
> >please?
> >
> >
> >Best Regards,
> >Petr
>
> Hi Petr,
>
> Sorry with for the late reply.
No problem.
> Hitting a WARN()/BUG() from wake_up calls will lead to a deadlock if
> only a single CPU is running.
I think that the deadlock might happen also with more CPUs if
the async_printk() is enabled. I mean:
printk_emit()
wake_up_process()
try_to_wake_up()
raw_spin_lock_irqsave(&p->pi_lock, flags) !!!!
ttwu_queue()
ttwu_do_activate()
ttwu_activate()
activate_task()
enqueue_task()
enqueue_task_fair() via p->sched_class->enqueue_task
hrtick_update()
hrtick_start_fair()
WARN_ON(task_rq(p) != rq)
printk()
vprintk_emit()
wake_up_process()
try_to_wake_up()
raw_spin_lock_irqsave(&p->pi_lock,
flags)
There is a deadlock because p->pi_lock is already taken by
the first try_to_wake_up().
By other words, I think that the single running CPU was only
a symptom but it was not the root cause of the deadlock.
> We already had such a situation with system suspend. During a
> specific test on our device sometimes we hit a WARN from the time
> keeping core. (Cannot recall which one exactly. Viresh have it) from
> a printk wake_up path during system suspend and with already only
> one CPU running.
> So we were forced to make printing synchronous in the suspend path
> prior disabling all non-boot cpu's.
>
> Your solution number 2) sounds reasonable to me.
Good.
> I'm wondering if we could hit a WARN()/BUG() somewhere from the fair
> scheduler like the example you made for the RT sched?
Unfortunately, it looks like. The example above actually is from
the fair scheduler.
Best Regards,
Petr
next prev parent reply other threads:[~2016-08-16 9:04 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-04-04 16:57 [PATCH v10 0/2] printk: Make printk() completely async Sergey Senozhatsky
2016-04-04 16:57 ` [PATCH v10 1/2] " Sergey Senozhatsky
2016-04-04 22:51 ` Andrew Morton
2016-04-05 5:17 ` Sergey Senozhatsky
2016-04-05 7:39 ` Sergey Senozhatsky
2016-04-06 0:19 ` Sergey Senozhatsky
2016-04-06 8:27 ` Jan Kara
2016-04-07 9:48 ` Sergey Senozhatsky
2016-04-07 12:08 ` Sergey Senozhatsky
2016-04-07 13:15 ` Jan Kara
2016-08-10 21:17 ` Viresh Kumar
2016-08-12 9:44 ` Petr Mladek
2016-08-15 14:26 ` Vladislav Levenetz
2016-08-16 9:04 ` Petr Mladek [this message]
2016-08-18 2:27 ` Sergey Senozhatsky
2016-08-18 9:33 ` Petr Mladek
2016-08-18 9:51 ` Sergey Senozhatsky
2016-08-18 10:56 ` Petr Mladek
2016-08-19 6:32 ` Sergey Senozhatsky
2016-08-19 9:54 ` Petr Mladek
2016-08-19 19:00 ` Jan Kara
2016-08-20 5:24 ` Sergey Senozhatsky
2016-08-22 4:15 ` Sergey Senozhatsky
2016-08-23 12:19 ` Petr Mladek
2016-08-24 1:33 ` Sergey Senozhatsky
2016-08-25 21:10 ` Petr Mladek
2016-08-26 1:56 ` Sergey Senozhatsky
2016-08-26 8:20 ` Sergey Senozhatsky
2016-08-30 9:29 ` Petr Mladek
2016-08-31 2:31 ` Sergey Senozhatsky
2016-08-31 9:38 ` Petr Mladek
2016-08-31 12:52 ` Sergey Senozhatsky
2016-09-01 8:58 ` Petr Mladek
2016-09-02 7:58 ` Sergey Senozhatsky
2016-09-02 15:15 ` Petr Mladek
2016-09-06 7:16 ` Sergey Senozhatsky
2016-08-23 13:03 ` Petr Mladek
2016-08-23 13:48 ` Petr Mladek
2016-04-04 16:57 ` [PATCH v10 2/2] printk: Make wake_up_klogd_work_func() async Sergey Senozhatsky
2016-08-23 3:32 [PATCH v10 1/2] printk: Make printk() completely async Andreas Mohr
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160816090430.GK13300@pathway.suse.cz \
--to=pmladek@suse.com \
--cc=akpm@linux-foundation.org \
--cc=byungchul.park@lge.com \
--cc=gregkh@linuxfoundation.org \
--cc=jack@suse.com \
--cc=jack@suse.cz \
--cc=linux-kernel@vger.kernel.org \
--cc=penguin-kernel@i-love.sakura.ne.jp \
--cc=sergey.senozhatsky.work@gmail.com \
--cc=sergey.senozhatsky@gmail.com \
--cc=tj@kernel.org \
--cc=viresh.kumar@linaro.org \
--cc=vlevenetz@mm-sol.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).