linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Qais Yousef <qais.yousef@arm.com>
To: Pavan Kondeti <pkondeti@codeaurora.org>
Cc: Ingo Molnar <mingo@redhat.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Steven Rostedt <rostedt@goodmis.org>,
	Dietmar Eggemann <dietmar.eggemann@arm.com>,
	Juri Lelli <juri.lelli@redhat.com>,
	Vincent Guittot <vincent.guittot@linaro.org>,
	Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH 3/3] sched/rt: fix pushing unfit tasks to a better CPU
Date: Wed, 19 Feb 2020 14:02:44 +0000	[thread overview]
Message-ID: <20200219140243.wfljmupcrwm2jelo@e107158-lin> (raw)
In-Reply-To: <20200217135306.cjc2225wdlwqiicu@e107158-lin.cambridge.arm.com>

On 02/17/20 13:53, Qais Yousef wrote:
> On 02/17/20 14:53, Pavan Kondeti wrote:
> > I notice a case where tasks would migrate for no reason (happens without this
> > patch also). Assuming BIG cores are busy with other RT tasks. Now this RT
> > task can go to *any* little CPU. There is no bias towards its previous CPU.
> > I don't know if it makes any difference but I see RT task placement is too
> > keen on reducing the migrations unless it is absolutely needed.
> 
> In find_lowest_rq() there's a check if the task_cpu(p) is in the lowest_mask
> and prefer it if it is.
> 
> But yeah I see it happening too
> 
> https://imgur.com/a/FYqLIko
> 
> Tasks on CPU 0 and 3 swap. Note that my tasks are periodic but the plots don't
> show that.
> 
> I shouldn't have changed something to affect this bias. Do you think it's
> something I introduced?
> 
> It's something maybe worth digging into though. I'll try to have a look.

FWIW, I dug a bit into this and I found out we have a thundering herd issue.

Since I just have a set of periodic task that all start together,
select_task_rq_rt() ends up selecting the same fitting CPU for all of them
(CPU1). The end up all waking up on CPU1, only to get pushed back out
again with only one surviving.

This reshuffles the task placement ending with some tasks being swapped.

I don't think this problem is specific to my change and could happen without
it.

The problem is caused by the way find_lowest_rq() selects a cpu in the mask

1750                         best_cpu = cpumask_first_and(lowest_mask,
1751                                                      sched_domain_span(sd));
1752                         if (best_cpu < nr_cpu_ids) {
1753                                 rcu_read_unlock();
1754                                 return best_cpu;
1755                         }

It always returns the first CPU in the mask. Or the mask could only contain
a single CPU too. The end result is that we most likely end up herding all the
tasks that wake up simultaneously to the same CPU.

I'm not sure how to fix this problem yet.

--
Qais Yousef

  parent reply	other threads:[~2020-02-19 14:02 UTC|newest]

Thread overview: 29+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-02-14 16:39 [PATCH 0/3] RT Capacity Awareness Improvements Qais Yousef
2020-02-14 16:39 ` [PATCH 1/3] sched/rt: cpupri_find: implement fallback mechanism for !fit case Qais Yousef
2020-02-17 17:07   ` Valentin Schneider
2020-02-17 23:34     ` Qais Yousef
2020-02-18 10:01       ` Valentin Schneider
2020-02-17 19:09   ` Dietmar Eggemann
2020-02-17 23:45     ` Qais Yousef
2020-02-18  9:53       ` Dietmar Eggemann
2020-02-18 17:28         ` Qais Yousef
2020-02-18 16:46       ` Steven Rostedt
2020-02-18 17:27         ` Qais Yousef
2020-02-18 18:03           ` Steven Rostedt
2020-02-18 18:52             ` Qais Yousef
2020-02-14 16:39 ` [PATCH 2/3] sched/rt: allow pulling unfitting task Qais Yousef
2020-02-17  9:10   ` Pavan Kondeti
2020-02-17 11:20     ` Qais Yousef
2020-02-19 13:43     ` Qais Yousef
2020-02-21  8:07       ` Pavan Kondeti
2020-02-21 11:08         ` Qais Yousef
2020-02-14 16:39 ` [PATCH 3/3] sched/rt: fix pushing unfit tasks to a better CPU Qais Yousef
2020-02-17  9:23   ` Pavan Kondeti
2020-02-17 13:53     ` Qais Yousef
2020-02-18  4:16       ` Pavan Kondeti
2020-02-18 17:47         ` Qais Yousef
2020-02-19  2:46           ` Pavan Kondeti
2020-02-19 10:46             ` Qais Yousef
2020-02-19 14:02       ` Qais Yousef [this message]
2020-02-21  8:15         ` Pavan Kondeti
2020-02-21 11:12           ` Qais Yousef

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200219140243.wfljmupcrwm2jelo@e107158-lin \
    --to=qais.yousef@arm.com \
    --cc=bsegall@google.com \
    --cc=dietmar.eggemann@arm.com \
    --cc=juri.lelli@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mgorman@suse.de \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=pkondeti@codeaurora.org \
    --cc=rostedt@goodmis.org \
    --cc=vincent.guittot@linaro.org \
    --subject='Re: [PATCH 3/3] sched/rt: fix pushing unfit tasks to a better CPU' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).