From: Pavan Kondeti <pkondeti@codeaurora.org>
To: Qais Yousef <qais.yousef@arm.com>
Cc: Ingo Molnar <mingo@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
Steven Rostedt <rostedt@goodmis.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH 3/3] sched/rt: fix pushing unfit tasks to a better CPU
Date: Fri, 21 Feb 2020 13:45:51 +0530 [thread overview]
Message-ID: <20200221081551.GG28029@codeaurora.org> (raw)
In-Reply-To: <20200219140243.wfljmupcrwm2jelo@e107158-lin>
On Wed, Feb 19, 2020 at 02:02:44PM +0000, Qais Yousef wrote:
> On 02/17/20 13:53, Qais Yousef wrote:
> > On 02/17/20 14:53, Pavan Kondeti wrote:
> > > I notice a case where tasks would migrate for no reason (happens without this
> > > patch also). Assuming BIG cores are busy with other RT tasks. Now this RT
> > > task can go to *any* little CPU. There is no bias towards its previous CPU.
> > > I don't know if it makes any difference but I see RT task placement is too
> > > keen on reducing the migrations unless it is absolutely needed.
> >
> > In find_lowest_rq() there's a check if the task_cpu(p) is in the lowest_mask
> > and prefer it if it is.
> >
> > But yeah I see it happening too
> >
> > https://imgur.com/a/FYqLIko
> >
> > Tasks on CPU 0 and 3 swap. Note that my tasks are periodic but the plots don't
> > show that.
> >
> > I shouldn't have changed something to affect this bias. Do you think it's
> > something I introduced?
> >
> > It's something maybe worth digging into though. I'll try to have a look.
>
> FWIW, I dug a bit into this and I found out we have a thundering herd issue.
>
> Since I just have a set of periodic task that all start together,
> select_task_rq_rt() ends up selecting the same fitting CPU for all of them
> (CPU1). The end up all waking up on CPU1, only to get pushed back out
> again with only one surviving.
>
> This reshuffles the task placement ending with some tasks being swapped.
>
> I don't think this problem is specific to my change and could happen without
> it.
>
> The problem is caused by the way find_lowest_rq() selects a cpu in the mask
>
> 1750 best_cpu = cpumask_first_and(lowest_mask,
> 1751 sched_domain_span(sd));
> 1752 if (best_cpu < nr_cpu_ids) {
> 1753 rcu_read_unlock();
> 1754 return best_cpu;
> 1755 }
>
> It always returns the first CPU in the mask. Or the mask could only contain
> a single CPU too. The end result is that we most likely end up herding all the
> tasks that wake up simultaneously to the same CPU.
>
> I'm not sure how to fix this problem yet.
>
Yes, I have seen this problem too. This is not limited to RT even fair class
(find_energy_efficient_cpu path) also have the same issue. There is a window
where we select a CPU for the task and the task being queued there. Because of
this, we may select the same CPU for two successive waking tasks. Turning off
TTWU_QUEUE sched feature addresses this up to some extent. At least it would
solve the cases like multiple tasks getting woken up from an interrupt handler.
Thanks,
Pavan
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project.
next prev parent reply other threads:[~2020-02-21 8:16 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-02-14 16:39 [PATCH 0/3] RT Capacity Awareness Improvements Qais Yousef
2020-02-14 16:39 ` [PATCH 1/3] sched/rt: cpupri_find: implement fallback mechanism for !fit case Qais Yousef
2020-02-17 17:07 ` Valentin Schneider
2020-02-17 23:34 ` Qais Yousef
2020-02-18 10:01 ` Valentin Schneider
2020-02-17 19:09 ` Dietmar Eggemann
2020-02-17 23:45 ` Qais Yousef
2020-02-18 9:53 ` Dietmar Eggemann
2020-02-18 17:28 ` Qais Yousef
2020-02-18 16:46 ` Steven Rostedt
2020-02-18 17:27 ` Qais Yousef
2020-02-18 18:03 ` Steven Rostedt
2020-02-18 18:52 ` Qais Yousef
2020-02-14 16:39 ` [PATCH 2/3] sched/rt: allow pulling unfitting task Qais Yousef
2020-02-17 9:10 ` Pavan Kondeti
2020-02-17 11:20 ` Qais Yousef
2020-02-19 13:43 ` Qais Yousef
2020-02-21 8:07 ` Pavan Kondeti
2020-02-21 11:08 ` Qais Yousef
2020-02-14 16:39 ` [PATCH 3/3] sched/rt: fix pushing unfit tasks to a better CPU Qais Yousef
2020-02-17 9:23 ` Pavan Kondeti
2020-02-17 13:53 ` Qais Yousef
2020-02-18 4:16 ` Pavan Kondeti
2020-02-18 17:47 ` Qais Yousef
2020-02-19 2:46 ` Pavan Kondeti
2020-02-19 10:46 ` Qais Yousef
2020-02-19 14:02 ` Qais Yousef
2020-02-21 8:15 ` Pavan Kondeti [this message]
2020-02-21 11:12 ` Qais Yousef
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200221081551.GG28029@codeaurora.org \
--to=pkondeti@codeaurora.org \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=juri.lelli@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=qais.yousef@arm.com \
--cc=rostedt@goodmis.org \
--cc=vincent.guittot@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).