All of
 help / color / mirror / Atom feed
From: Tianchen Ding <>
To: Ingo Molnar <>,
	Peter Zijlstra <>,
	Juri Lelli <>,
	Vincent Guittot <>,
	Dietmar Eggemann <>,
	Steven Rostedt <>,
	Ben Segall <>, Mel Gorman <>,
	Daniel Bristot de Oliveira <>
Subject: [RFC PATCH] sched: Queue task on wakelist in the same llc if the wakee cpu is idle
Date: Fri, 13 May 2022 14:24:27 +0800	[thread overview]
Message-ID: <> (raw)

We notice the commit 518cd6234178 ("sched: Only queue remote wakeups
when crossing cache boundaries") disabled queuing tasks on wakelist when
the cpus share llc. This is because, at that time, the scheduler must
send IPIs to do ttwu_queue_wakelist. Nowadays, ttwu_queue_wakelist also
supports TIF_POLLING, so this is not a problem now when the wakee cpu is
in idle polling.

  Queuing the task on idle cpu can help improving performance on waker cpu
  and utilization on wakee cpu, and further improve locality because
  the wakee cpu can handle its own rq. This patch helps improving rt on
  our real java workloads where wakeup happens frequently.

Does this patch bring IPI flooding?
  For archs with TIF_POLLING_NRFLAG (e.g., x86), there will be no
  difference if the wakee cpu is idle polling. If the wakee cpu is idle
  but not polling, the later check_preempt_curr() will send IPI too.

  For archs without TIF_POLLING_NRFLAG (e.g., arm64), the IPI is
  unavoidable, since the later check_preempt_curr() will send IPI when
  wakee cpu is idle.

running schbench -m 2 -t 8 on 8269CY:

without patch:
Latency percentiles (usec)
        50.0000th: 10
        75.0000th: 14
        90.0000th: 16
        95.0000th: 16
        *99.0000th: 17
        99.5000th: 20
        99.9000th: 23
        min=0, max=28

with patch:
Latency percentiles (usec)
        50.0000th: 6
        75.0000th: 8
        90.0000th: 9
        95.0000th: 9
        *99.0000th: 10
        99.5000th: 10
        99.9000th: 14
        min=0, max=16

We've also tested unixbench and see about 10% improvement on Pipe-based
Context Switching, and no performance regression on other test cases.

For arm64, we've tested schbench and unixbench on Kunpeng920, the
results show that, the improvement is not as obvious as on x86, and
there's no performance regression.

Signed-off-by: Tianchen Ding <>
 kernel/sched/core.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 51efaabac3e4..cae5011a8b1f 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3820,6 +3820,9 @@ static inline bool ttwu_queue_cond(int cpu, int wake_flags)
 	if (!cpu_active(cpu))
 		return false;
+	if (cpu == smp_processor_id())
+		return false;
 	 * If the CPU does not share cache, then queue the task on the
 	 * remote rqs wakelist to avoid accessing remote data.
@@ -3827,6 +3830,12 @@ static inline bool ttwu_queue_cond(int cpu, int wake_flags)
 	if (!cpus_share_cache(smp_processor_id(), cpu))
 		return true;
+	/*
+	 * If the CPU is idle, let itself do activation to improve utilization.
+	 */
+	if (available_idle_cpu(cpu))
+		return true;
 	 * If the task is descheduling and the only running task on the
 	 * CPU then use the wakelist to offload the task activation to
@@ -3842,9 +3851,6 @@ static inline bool ttwu_queue_cond(int cpu, int wake_flags)
 static bool ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags)
 	if (sched_feat(TTWU_QUEUE) && ttwu_queue_cond(cpu, wake_flags)) {
-		if (WARN_ON_ONCE(cpu == smp_processor_id()))
-			return false;
 		sched_clock_cpu(cpu); /* Sync clocks across CPUs */
 		__ttwu_queue_wakelist(p, cpu, wake_flags);
 		return true;

             reply	other threads:[~2022-05-13  6:24 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-05-13  6:24 Tianchen Ding [this message]
2022-05-13  6:37 ` [RFC PATCH] sched: Queue task on wakelist in the same llc if the wakee cpu is idle Peter Zijlstra
2022-05-13  7:05   ` Tianchen Ding
2022-05-17 13:58 ` Mel Gorman
2022-05-18  8:05   ` Tianchen Ding

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.