From: Yafang Shao <laoar.shao@gmail.com>
To: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Ingo Molnar <mingo@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
Juri Lelli <juri.lelli@redhat.com>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
Daniel Bristot de Oliveira <bristot@redhat.com>,
linux-kernel <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 1/1] sched: do active load balance on the new idle cpu
Date: Mon, 14 Jun 2021 18:23:14 +0800 [thread overview]
Message-ID: <CALOAHbB7Re8ES86bC10vz7KuNjLV1fA7tuLUmJY==sR-fB-+qQ@mail.gmail.com> (raw)
In-Reply-To: <CAKfTPtCS6bVGK1EFUHygj+uZL5N2kEzyyEeoyT4Cuc7r-65yVw@mail.gmail.com>
On Wed, Jun 2, 2021 at 8:37 PM Vincent Guittot
<vincent.guittot@linaro.org> wrote:
>
> On Wed, 2 Jun 2021 at 14:26, Yafang Shao <laoar.shao@gmail.com> wrote:
> >
> > We monitored our latency-sensitive RT tasks are randomly preempted by the
> > kthreads migration/n, which means to migrate tasks on CPUn to other new
> > idle CPU. The logical as follows,
> >
> > new idle CPU CPU n
> > (no task to run) (busy running)
> > wakeup migration/n (busy running)
> > (idle) migraion/n preempts current task
> > run the migrated task (busy running)
>
> migration thread is only used when we want to migrate the currently
> running task of the source cpu.
> This doesn't seem to be your case as it's a RT thread that is
> currently running so the migration thread should not be woken up as we
> don't need it to migrate a runnable but not running cfs thread from
> coin to new idle CPU
>
> Do you have more details about the UC. Could it be a race between new
> idle load balance starting migration thread to pull the cfs running
> thread and the RT thread waking up and preempting cfs task before
> migration threads which then preempt your RT threads
>
Hi Vincent,
When I analyze it on my test server, I find the race really exists. For example,
sensing_node-2511 [007] d... 945.351566: sched_switch:
prev_comm=sensing_node prev_pid=2511 prev_prio=98 prev_state=S ==>
next_comm=cat next_pid=2686 next_prio=120
cat-2686 [007] d... 945.351569: sched_switch: prev_comm=cat
prev_pid=2686 prev_prio=120 prev_state=R+ ==> next_comm=sensing_node
next_pid=2512 next_prio=98
sensing_node-2516 [004] dn.. 945.351571: sched_wakeup:
comm=migration/7 pid=47 prio=0 target_cpu=007
sensing_node-2512 [007] d... 945.351572: sched_switch:
prev_comm=sensing_node prev_pid=2512 prev_prio=98 prev_state=R ==>
next_comm=migration/7 next_pid=47 next_prio=0
sensing_node-2516 [004] d... 945.351572: sched_switch:
prev_comm=sensing_node prev_pid=2516 prev_prio=98 prev_state=S ==>
next_comm=sensing_node next_pid=2502 next_prio=98
migration/7-47 [007] d... 945.351580: sched_switch:
prev_comm=migration/7 prev_pid=47 prev_prio=0 prev_state=S ==>
next_comm=sensing_node next_pid=2512 next_prio=98
sensing_node-2502 [004] d... 945.351605: sched_switch:
prev_comm=sensing_node prev_pid=2502 prev_prio=98 prev_state=S ==>
next_comm=cat next_pid=2686 next_prio=120
When CPU4 is waking migration/7, the CFS thread 'cat' is running on
CPU7, but then 'cat' is preempted by a RT task 'sensing_node', and
then the migration/7 preempts the RT task.
What about below patch to improve the race ? It can't avoid the race,
but it could reduce the race.
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 3248e24a90b0..0e8d31e17dc7 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -9794,6 +9794,20 @@ static int load_balance(int this_cpu, struct rq *this_rq,
goto out_one_pinned;
}
+ /*
+ * There may be a race between new idle load
balance starting
+ * migration thread to pull the cfs running
thread and the RT
+ * thread waking up and preempting cfs task
before migration
+ * threads which then preempt the RT thread.
+ * We'd better do the last minute check before starting
+ * migration thread to avoid preempting
latency-sensitive RT thread.
+ */
+ if (dl_task(busiest->curr) || rt_task(busiest->curr)) {
+ raw_spin_unlock_irqrestore(&busiest->lock,
+ flags);
+ goto out_one_pinned;
+ }
+
/* Record that we found at least one task that
could run on this_cpu */
env.flags &= ~LBF_ALL_PINNED;
>
>
> >
> > As the new idle CPU is going to be idle, we'd better move the migration
> > work on it instead of burdening the busy CPU. After this change, the
> > logic is,
> > new idle CPU CPU n
> > (no task to run) (busy running)
> > migrate task from CPU n (busy running)
> > run the migrated task (busy running)
> >
> > Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
> > ---
> > kernel/sched/fair.c | 17 +++++------------
> > 1 file changed, 5 insertions(+), 12 deletions(-)
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 3248e24a90b0..3e8b98b982ff 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -9807,13 +9807,11 @@ static int load_balance(int this_cpu, struct rq *this_rq,
> > busiest->push_cpu = this_cpu;
> > active_balance = 1;
> > }
> > - raw_spin_unlock_irqrestore(&busiest->lock, flags);
> >
> > - if (active_balance) {
> > - stop_one_cpu_nowait(cpu_of(busiest),
> > - active_load_balance_cpu_stop, busiest,
> > - &busiest->active_balance_work);
> > - }
> > + if (active_balance)
> > + active_load_balance_cpu_stop(busiest);
>
> this doesn't make sense because we reach this point if we want to
> migrate the current running task of the busiest cpu and in order to do
> this we need the preempt this current running thread
>
> > +
> > + raw_spin_unlock_irqrestore(&busiest->lock, flags);
> > }
> > } else {
> > sd->nr_balance_failed = 0;
> > @@ -9923,7 +9921,6 @@ static int active_load_balance_cpu_stop(void *data)
> > struct task_struct *p = NULL;
> > struct rq_flags rf;
> >
> > - rq_lock_irq(busiest_rq, &rf);
> > /*
> > * Between queueing the stop-work and running it is a hole in which
> > * CPUs can become inactive. We should not move tasks from or to
> > @@ -9933,8 +9930,7 @@ static int active_load_balance_cpu_stop(void *data)
> > goto out_unlock;
> >
> > /* Make sure the requested CPU hasn't gone down in the meantime: */
> > - if (unlikely(busiest_cpu != smp_processor_id() ||
> > - !busiest_rq->active_balance))
> > + if (unlikely(!busiest_rq->active_balance))
> > goto out_unlock;
> >
> > /* Is there any task to move? */
> > @@ -9981,13 +9977,10 @@ static int active_load_balance_cpu_stop(void *data)
> > rcu_read_unlock();
> > out_unlock:
> > busiest_rq->active_balance = 0;
> > - rq_unlock(busiest_rq, &rf);
> >
> > if (p)
> > attach_one_task(target_rq, p);
> >
> > - local_irq_enable();
> > -
> > return 0;
> > }
> >
> > --
> > 2.17.1
> >
--
Thanks
Yafang
next prev parent reply other threads:[~2021-06-14 10:23 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-02 12:25 [PATCH 1/1] sched: do active load balance on the new idle cpu Yafang Shao
2021-06-02 12:37 ` Vincent Guittot
2021-06-02 12:38 ` Vincent Guittot
2021-06-02 12:58 ` Yafang Shao
2021-06-02 13:26 ` Vincent Guittot
2021-06-03 1:29 ` Yafang Shao
2021-06-14 10:23 ` Yafang Shao [this message]
2021-06-14 14:20 ` Vincent Guittot
2021-06-14 15:59 ` Yafang Shao
2021-06-04 1:23 ` [sched] 7a86d20411: phoronix-test-suite.x264.0.frames_per_second -3.8% regression kernel test robot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CALOAHbB7Re8ES86bC10vz7KuNjLV1fA7tuLUmJY==sR-fB-+qQ@mail.gmail.com' \
--to=laoar.shao@gmail.com \
--cc=bristot@redhat.com \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=juri.lelli@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=vincent.guittot@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).