From: Joel Fernandes <joel@joelfernandes.org>
To: Vincent Guittot <vincent.guittot@linaro.org>
Cc: linux-kernel <linux-kernel@vger.kernel.org>,
Paul McKenney <paulmck@kernel.org>,
Frederic Weisbecker <fweisbec@gmail.com>,
Dietmar Eggeman <dietmar.eggemann@arm.com>,
Qais Yousef <qais.yousef@arm.com>,
Ben Segall <bsegall@google.com>,
Daniel Bristot de Oliveira <bristot@redhat.com>,
Ingo Molnar <mingo@redhat.com>,
Juri Lelli <juri.lelli@redhat.com>, Mel Gorman <mgorman@suse.de>,
Peter Zijlstra <peterz@infradead.org>,
Steven Rostedt <rostedt@goodmis.org>,
urezki@gmail.com, neeraj.iitr10@gmail.com
Subject: Re: [PATCH] sched/fair: Rate limit calls to update_blocked_averages() for NOHZ
Date: Wed, 27 Jan 2021 13:43:39 -0500 [thread overview]
Message-ID: <YBG0W5PFGtGRCEuB@google.com> (raw)
In-Reply-To: <CAKfTPtBWoRuwwkaqQKNgHTnQBE4fevyYqEoeGc5RpCsBbOS1sQ@mail.gmail.com>
Hi Vincent,
On Mon, Jan 25, 2021 at 03:42:41PM +0100, Vincent Guittot wrote:
> On Fri, 22 Jan 2021 at 20:10, Joel Fernandes <joel@joelfernandes.org> wrote:
> > On Fri, Jan 22, 2021 at 05:56:22PM +0100, Vincent Guittot wrote:
> > > On Fri, 22 Jan 2021 at 16:46, Joel Fernandes (Google)
> > > <joel@joelfernandes.org> wrote:
> > > >
> > > > On an octacore ARM64 device running ChromeOS Linux kernel v5.4, I found
> > > > that there are a lot of calls to update_blocked_averages(). This causes
> > > > the schedule loop to slow down to taking upto 500 micro seconds at
> > > > times (due to newidle load balance). I have also seen this manifest in
> > > > the periodic balancer.
> > > >
> > > > Closer look shows that the problem is caused by the following
> > > > ingredients:
> > > > 1. If the system has a lot of inactive CGroups (thanks Dietmar for
> > > > suggesting to inspect /proc/sched_debug for this), this can make
> > > > __update_blocked_fair() take a long time.
> > >
> > > Inactive cgroups are removed from the list so they should not impact
> > > the duration
> >
> > I meant blocked CGroups. According to this code, a cfs_rq can be partially
> > decayed and not have any tasks running on it but its load needs to be
> > decayed, correct? That's what I meant by 'inactive'. I can reword it to
> > 'blocked'.
>
> How many blocked cgroups have you got ?
I put a counter in for_each_leaf_cfs_rq_safe() { } to count how many times
this loop runs per new idle balance. When the problem happens I see this loop
run 35-40 times (for one single instance of newidle balance). So in total
there are at least these many cfs_rq load updates.
I also see that new idle balance can be called 200-500 times per second.
> >
> > * There can be a lot of idle CPU cgroups. Don't let fully
> > * decayed cfs_rqs linger on the list.
> > */
> > if (cfs_rq_is_decayed(cfs_rq))
> > list_del_leaf_cfs_rq(cfs_rq);
> >
> > > > 2. The device has a lot of CPUs in a cluster which causes schedutil in a
> > > > shared frequency domain configuration to be slower than usual. (the load
> > >
> > > What do you mean exactly by it causes schedutil to be slower than usual ?
> >
> > sugov_next_freq_shared() is order number of CPUs in the a cluster. This
> > system is a 6+2 system with 6 CPUs in a cluster. schedutil shared policy
> > frequency update needs to go through utilization of other CPUs in the
> > cluster. I believe this could be adding to the problem but is not really
> > needed to optimize if we can rate limit the calls to update_blocked_averages
> > to begin with.
>
> Qais mentioned half of the time being used by
> sugov_next_freq_shared(). Are there any frequency changes resulting in
> this call ?
I do not see a frequency update happening at the time of the problem. However
note that sugov_iowait_boost() does run even if frequency is not being
updated. IIRC, this function is also not that light weight and I am not sure
if it is a good idea to call this that often.
> > > > average updates also try to update the frequency in schedutil).
> > > >
> > > > 3. The CPU is running at a low frequency causing the scheduler/schedutil
> > > > code paths to take longer than when running at a high CPU frequency.
> > >
> > > Low frequency usually means low utilization so it should happen that much.
> >
> > It happens a lot as can be seen with schbench. It is super easy to reproduce.
>
> Happening a lot in itself is not a problem if there is nothing else to
> do so it's not a argument in itself
It is a problem - it shows up in the preempt off critical section latency
tracer. Are you saying its Ok for preemption to be disabled on system for 500
micro seconds? That hurts real-time applications (audio etc).
> So why is it a problem for you ? You are mentioning newly idle load
> balance so I assume that your root problem is the scheduling delay
> generated by the newly idle load balance which then calls
> update_blocked_averages
Yes, the new idle balance is when I see it happen quite often. I do see it
happen with other load balance as well, but it not that often as those LB
don't run as often as new idle balance.
>
> rate limiting the call to update_blocked_averages() will only reduce
> the number of time it happens but it will not prevent it to happen.
Sure, but soft real-time issue can tolerate if the issue does not happen very
often. In this case though, it is frequent.
> IIUC, your real problem is that newidle_balance is running whereas a
> task is about to wake up on the cpu and we don't abort quickly during
> this load_balance
Yes.
> so we could also try to abort earlier in case of newly idle load balance
I think interrupts are disabled when the load balance runs, so there's no way
for say an audio interrupt to even run in order to wake up a task. How would
you know to abort the new idle load balance?
Could you elaborate more also on the drawback of the rate limiting patch we
posted? Do you see a side effect?
> > > > sometimes, which seems overkill.
> > > >
> > > > schbench shows a clear improvement with the change:
> > >
> > > Have you got more details about your test setup ?
> > > which platform ?
> > > which kernel ?
> >
> > I mentioned in the commit message it is a v5.4 kernel.
>
> I was not sure if the tests results done with this kernel because we
> usually ask for results against mainline to make sure you are not
> facing a problem that has solved since v5.4 has been released
Ok, yes I have a userspace up and running only on 5.4 kernel unfortunately. I
was hoping that is recent enough for this debug.
thanks,
- Joel
next prev parent reply other threads:[~2021-01-27 18:44 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-22 15:46 [PATCH] sched/fair: Rate limit calls to update_blocked_averages() for NOHZ Joel Fernandes (Google)
2021-01-22 16:56 ` Vincent Guittot
2021-01-22 18:39 ` Qais Yousef
2021-01-22 19:14 ` Joel Fernandes
2021-01-25 13:23 ` Vincent Guittot
2021-01-26 16:36 ` Qais Yousef
2021-01-22 19:10 ` Joel Fernandes
2021-01-25 10:44 ` Dietmar Eggemann
2021-01-25 17:30 ` Vincent Guittot
2021-01-25 17:53 ` Dietmar Eggemann
2021-01-25 14:42 ` Vincent Guittot
2021-01-27 18:43 ` Joel Fernandes [this message]
2021-01-28 13:57 ` Vincent Guittot
2021-01-28 15:09 ` Joel Fernandes
2021-01-28 16:57 ` Qais Yousef
[not found] ` <CAKfTPtBvwm9vZb5C=2oTF6N-Ht6Rvip4Lv18yi7O3G8e-_ZWdg@mail.gmail.com>
2021-01-29 17:27 ` Vincent Guittot
2021-02-03 11:54 ` Dietmar Eggemann
2021-02-03 13:12 ` Vincent Guittot
2021-02-04 9:47 ` Dietmar Eggemann
2021-02-03 17:09 ` Qais Yousef
2021-02-03 17:35 ` Vincent Guittot
2021-02-04 10:45 ` Qais Yousef
2021-02-03 19:56 ` Joel Fernandes
2021-03-23 21:37 ` Tim Chen
2021-03-24 13:44 ` Vincent Guittot
2021-03-24 16:05 ` Tim Chen
2021-04-07 14:02 ` Vincent Guittot
2021-04-07 17:19 ` Tim Chen
2021-04-08 14:51 ` Vincent Guittot
2021-04-08 23:05 ` Tim Chen
2021-04-09 15:26 ` Vincent Guittot
2021-04-09 17:59 ` Tim Chen
2021-05-10 21:59 ` Tim Chen
2021-05-11 15:25 ` Vincent Guittot
2021-05-11 17:25 ` Tim Chen
2021-05-11 17:56 ` Vincent Guittot
2021-05-12 13:59 ` Qais Yousef
2021-05-13 18:45 ` Tim Chen
2021-05-17 16:14 ` Qais Yousef
2021-06-11 20:00 ` Tim Chen
2021-06-18 10:28 ` Vincent Guittot
2021-06-18 16:14 ` Tim Chen
2021-06-25 8:50 ` Vincent Guittot
2021-02-01 15:13 ` Joel Fernandes
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YBG0W5PFGtGRCEuB@google.com \
--to=joel@joelfernandes.org \
--cc=bristot@redhat.com \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=fweisbec@gmail.com \
--cc=juri.lelli@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=neeraj.iitr10@gmail.com \
--cc=paulmck@kernel.org \
--cc=peterz@infradead.org \
--cc=qais.yousef@arm.com \
--cc=rostedt@goodmis.org \
--cc=urezki@gmail.com \
--cc=vincent.guittot@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).