From: Ram Muthiah <rammuthiah@google.com>
To: Quentin Perret <qperret@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>,
linux-kernel@vger.kernel.org, aaron.lwe@gmail.com,
valentin.schneider@arm.com, mingo@kernel.org, pauld@redhat.com,
jdesfossez@digitalocean.com, naravamudan@digitalocean.com,
vincent.guittot@linaro.org, dietmar.eggemann@arm.com,
juri.lelli@redhat.com, rostedt@goodmis.org, bsegall@google.com,
mgorman@suse.de, kernel-team@android.com, john.stultz@linaro.org
Subject: Re: NULL pointer dereference in pick_next_task_fair
Date: Wed, 30 Oct 2019 15:50:00 -0700 [thread overview]
Message-ID: <CA+CXyWsoW8ann52pcR66ejRmjJ=4QmoaHTRVhb3=ohe0ZDnm-A@mail.gmail.com> (raw)
In-Reply-To: <20191029115000.GA11194@google.com>
On Tue, Oct 29, 2019 at 4:50 AM 'Quentin Perret' via kernel-team
<kernel-team@android.com> wrote:
>
> On Tuesday 29 Oct 2019 at 12:34:11 (+0100), Peter Zijlstra wrote:
> > On Mon, Oct 28, 2019 at 05:46:03PM +0000, Quentin Perret wrote:
> > > The issue is very transient and relatively hard to reproduce.
> > >
> > > After digging a bit, the offending commit seems to be:
> > >
> > > 67692435c411 ("sched: Rework pick_next_task() slow-path")
> > >
> > > By 'offending' I mean that reverting it makes the issue go away. The
> > > issue comes from the fact that pick_next_entity() returns a NULL se in
> > > the 'simple' path of pick_next_task_fair(), which causes obvious
> > > problems in the subsequent call to set_next_entity().
> > >
> > > I'll dig more, but if anybody understands the issue in the meatime feel
> > > free to send me a patch to try out :)
> >
> > Can you please see if this makes any difference?
> >
> > ---
> > kernel/sched/core.c | 6 ++++--
> > kernel/sched/fair.c | 2 +-
> > kernel/sched/idle.c | 3 +--
> > 3 files changed, 6 insertions(+), 5 deletions(-)
> >
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index 7880f4f64d0e..abd2d4f80381 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -3922,8 +3922,10 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
> > goto restart;
> >
> > /* Assumes fair_sched_class->next == idle_sched_class */
> > - if (unlikely(!p))
> > - p = idle_sched_class.pick_next_task(rq, prev, rf);
> > + if (unlikely(!p)) {
> > + prev->sched_class->put_prev_task(rq, prev, rf);
> > + p = idle_sched_class.pick_next_task(rq, NULL, NULL);
> > + }
> >
> > return p;
> > }
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 83ab35e2374f..2aad94bb7165 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -6820,7 +6820,7 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf
> > simple:
> > #endif
> > if (prev)
> > - put_prev_task(rq, prev);
> > + prev->sched_class->put_prev_task(rq, prev, rf);
> >
> > do {
> > se = pick_next_entity(cfs_rq, NULL);
> > diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
> > index 8dad5aa600ea..e8dfc84f375a 100644
> > --- a/kernel/sched/idle.c
> > +++ b/kernel/sched/idle.c
> > @@ -390,8 +390,7 @@ pick_next_task_idle(struct rq *rq, struct task_struct *prev, struct rq_flags *rf
> > {
> > struct task_struct *next = rq->idle;
> >
> > - if (prev)
> > - put_prev_task(rq, prev);
> > + WARN_ON_ONCE(prev || rf);
> >
> > set_next_task_idle(rq, next);
> >
> >
>
> Unfortunately the issue went into hiding this morning and I struggle to
> reproduce it (this is turning bordeline nightmare now TBH).
>
> I'll try the patch once my reproducer is fixed :/
>
> Thank you very much for the help,
> Quentin
>
> --
> To unsubscribe from this group and stop receiving emails from it, send an email to kernel-team+unsubscribe@android.com.
>
Quentin and I were able to create a setup which reproduces the issue.
Given this, I tried Peter's proposed fix and was still able to reproduce the
issue unfortunately. Current patch is located here -
https://android-review.googlesource.com/c/kernel/common/+/1153487
Our mitigation for this issue on the android-mainline branch has been to
revert 67692435c411 ("sched: Rework pick_next_task() slow-path").
https://android-review.googlesource.com/c/kernel/common/+/1152564
I'll spend some time detailing repro steps next. I should be able to
provide an update on those details early next week.
We appreciate the help so far.
Thanks,
Ram
next prev parent reply other threads:[~2019-10-30 22:50 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-28 17:46 NULL pointer dereference in pick_next_task_fair Quentin Perret
2019-10-28 21:49 ` Peter Zijlstra
2019-10-29 11:34 ` Peter Zijlstra
2019-10-29 11:50 ` Quentin Perret
2019-10-30 22:50 ` Ram Muthiah [this message]
2019-10-31 1:33 ` Valentin Schneider
2019-10-31 10:54 ` Valentin Schneider
2019-10-31 14:24 ` Valentin Schneider
2019-10-31 22:15 ` Valentin Schneider
2019-11-06 12:05 ` Peter Zijlstra
2019-11-06 13:08 ` Peter Zijlstra
2019-11-06 15:04 ` Qais Yousef
2019-11-06 16:57 ` Peter Zijlstra
2019-11-06 17:26 ` Qais Yousef
2019-11-06 15:51 ` Kirill Tkhai
2019-11-06 16:54 ` Peter Zijlstra
2019-11-06 17:27 ` Peter Zijlstra
2019-11-07 8:36 ` Kirill Tkhai
2019-11-07 13:26 ` Peter Zijlstra
2019-11-07 15:12 ` Kirill Tkhai
2019-11-07 15:42 ` Peter Zijlstra
2019-11-07 15:53 ` Kirill Tkhai
2019-11-07 15:38 ` Quentin Perret
2019-11-07 18:43 ` Peter Zijlstra
2019-11-07 19:27 ` Quentin Perret
2019-11-07 19:31 ` Peter Zijlstra
2019-11-07 19:42 ` Quentin Perret
2019-11-07 19:29 ` Peter Zijlstra
2019-11-08 11:02 ` Quentin Perret
2019-11-08 11:47 ` Valentin Schneider
2019-11-08 11:58 ` Quentin Perret
2019-11-08 12:00 ` Peter Zijlstra
2019-11-08 12:15 ` Quentin Perret
2019-11-08 12:35 ` Peter Zijlstra
2019-11-08 12:24 ` Peter Zijlstra
2019-11-08 11:55 ` Peter Zijlstra
2019-11-08 12:52 ` Peter Zijlstra
2019-11-07 16:09 ` Qais Yousef
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CA+CXyWsoW8ann52pcR66ejRmjJ=4QmoaHTRVhb3=ohe0ZDnm-A@mail.gmail.com' \
--to=rammuthiah@google.com \
--cc=aaron.lwe@gmail.com \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=jdesfossez@digitalocean.com \
--cc=john.stultz@linaro.org \
--cc=juri.lelli@redhat.com \
--cc=kernel-team@android.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@kernel.org \
--cc=naravamudan@digitalocean.com \
--cc=pauld@redhat.com \
--cc=peterz@infradead.org \
--cc=qperret@google.com \
--cc=rostedt@goodmis.org \
--cc=valentin.schneider@arm.com \
--cc=vincent.guittot@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).