From: Valentin Schneider <valentin.schneider@arm.com>
To: Matt Fleming <matt@codeblueprint.co.uk>,
Peter Zijlstra <peterz@infradead.org>
Cc: linux-kernel@vger.kernel.org, Ingo Molnar <mingo@kernel.org>,
Mike Galbraith <umgwanakikbuti@gmail.com>
Subject: Re: [PATCH] sched/fair: Avoid divide by zero when rebalancing domains
Date: Thu, 5 Jul 2018 11:10:42 +0100 [thread overview]
Message-ID: <55afee27-4143-e08c-b254-0d68a05d5ee6@arm.com> (raw)
In-Reply-To: <20180704142455.16035-1-matt@codeblueprint.co.uk>
Hi,
On 04/07/18 15:24, Matt Fleming wrote:
> It's possible that the CPU doing nohz idle balance hasn't had its own
> load updated for many seconds. This can lead to huge deltas between
> rq->avg_stamp and rq->clock when rebalancing, and has been seen to
> cause the following crash:
>
> divide error: 0000 [#1] SMP
> Call Trace:
> [<ffffffff810bcba8>] update_sd_lb_stats+0xe8/0x560
> [<ffffffff810bd04d>] find_busiest_group+0x2d/0x4b0
> [<ffffffff810bd640>] load_balance+0x170/0x950
> [<ffffffff810be3ff>] rebalance_domains+0x13f/0x290
> [<ffffffff810852bc>] __do_softirq+0xec/0x300
> [<ffffffff8108578a>] irq_exit+0xfa/0x110
> [<ffffffff816167d9>] reschedule_interrupt+0xc9/0xd0
>
Do you have some sort of reproducer for that crash? If not I guess I can cook
something up with a quiet userspace & rt-app, though I've never seen that one
on arm64.
> Make sure we update the rq clock and load before balancing.
>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Mike Galbraith <umgwanakikbuti@gmail.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Signed-off-by: Matt Fleming <matt@codeblueprint.co.uk>
> ---
> kernel/sched/fair.c | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 2f0a0be4d344..2c81662c858a 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -9597,6 +9597,16 @@ static bool _nohz_idle_balance(struct rq *this_rq, unsigned int flags,
> */
> smp_mb();
>
> + /*
> + * Ensure this_rq's clock and load are up-to-date before we
> + * rebalance since it's possible that they haven't been
> + * updated for multiple schedule periods, i.e. many seconds.
> + */
> + raw_spin_lock_irq(&this_rq->lock);
> + update_rq_clock(this_rq);
> + cpu_load_update_idle(this_rq);
> + raw_spin_unlock_irq(&this_rq->lock);
> +
I'm failing to understand why the updates further down below are seemingly
not enough. After we've potentially done
update_rq_clock(rq);
cpu_load_update_idle(rq);
for all nohz cpus != this_cpu, we still end up doing:
if (idle != CPU_NEWLY_IDLE) {
update_blocked_averages(this_cpu);
has_blocked_load |= this_rq->has_blocked_load;
}
which should properly update this_rq's clock and load before we attempt to do
any balancing on it.
> for_each_cpu(balance_cpu, nohz.idle_cpus_mask) {
> if (balance_cpu == this_cpu || !idle_cpu(balance_cpu))
> continue;
>
next prev parent reply other threads:[~2018-07-05 10:10 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-07-04 14:24 [PATCH] sched/fair: Avoid divide by zero when rebalancing domains Matt Fleming
2018-07-05 8:02 ` [lkp-robot] [sched/fair] fbd5188493: WARNING:inconsistent_lock_state kernel test robot
2018-07-05 8:58 ` Dietmar Eggemann
2018-07-05 9:52 ` Dietmar Eggemann
2018-07-05 13:24 ` Matt Fleming
2018-07-05 14:43 ` Matt Fleming
2018-07-05 14:59 ` Dietmar Eggemann
2018-07-05 10:10 ` Valentin Schneider [this message]
2018-07-05 13:27 ` [PATCH] sched/fair: Avoid divide by zero when rebalancing domains Matt Fleming
2018-07-05 16:54 ` Valentin Schneider
2018-08-17 10:27 ` Matt Fleming
2018-08-17 12:58 ` Valentin Schneider
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=55afee27-4143-e08c-b254-0d68a05d5ee6@arm.com \
--to=valentin.schneider@arm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=matt@codeblueprint.co.uk \
--cc=mingo@kernel.org \
--cc=peterz@infradead.org \
--cc=umgwanakikbuti@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).