Re: Scheduling tasks on idle cpu

From: Vincent Guittot <vincent.guittot@linaro.org>
To: David Laight <David.Laight@aculab.com>
Cc: Qais Yousef <qais.yousef@arm.com>,
	Dietmar Eggemann <dietmar.eggemann@arm.com>,
	"mingo@redhat.com" <mingo@redhat.com>,
	"peterz@infradead.org" <peterz@infradead.org>,
	"juri.lelli@redhat.com" <juri.lelli@redhat.com>,
	"rostedt@goodmis.org" <rostedt@goodmis.org>,
	"bsegall@google.com" <bsegall@google.com>,
	"mgorman@suse.de" <mgorman@suse.de>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"parth@linux.ibm.com" <parth@linux.ibm.com>,
	"chris.hyser@oracle.com" <chris.hyser@oracle.com>,
	"pkondeti@codeaurora.org" <pkondeti@codeaurora.org>,
	"Valentin.Schneider@arm.com" <Valentin.Schneider@arm.com>,
	"patrick.bellasi@matbug.net" <patrick.bellasi@matbug.net>,
	"pjt@google.com" <pjt@google.com>, "pavel@ucw.cz" <pavel@ucw.cz>,
	"tj@kernel.org" <tj@kernel.org>,
	"qperret@google.com" <qperret@google.com>,
	"tim.c.chen@linux.intel.com" <tim.c.chen@linux.intel.com>,
	Wei Wang <wvw@google.com>
Subject: Re: Scheduling tasks on idle cpu
Date: Fri, 15 Apr 2022 09:15:57 +0200	[thread overview]
Message-ID: <CAKfTPtDFZ+Kom5=2Eo8OPaw0v3d0fnxBHgWYLinGp3jHnx0BhA@mail.gmail.com> (raw)
In-Reply-To: <5fc162f60ba84ab3a61992084d76c726@AcuMS.aculab.com>

On Thu, 14 Apr 2022 at 16:14, David Laight <David.Laight@aculab.com> wrote:
>
> From: Vincent Guittot
> > Sent: 14 April 2022 11:17
> ...
> > > > David can rerun is use case after disabling sched_feat(SIS_PROP)
> > >
> > > How would I do that?
> >
> > echo NO_SIS_PROP > /sys/kernel/debug/sched/features
>
> That may not be in the kernel I'm using.
>
> # cat /sys/kernel/debug/sched_features
> GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION ARCH_POWER NO_HRTICK NO_DOUBLE_TICK LB_BIAS NONTASK_POWER TTWU_QUEUE RT_RUNTIME_SHARE NO_LB_MIN NUMA NUMA_FAVOUR_HIGHER NO_NUMA_RESIST_LOWER

SIS_PROP has been normally added in v4.13  so I wonder which kernel
version you are using
Before SIS_PROP, the policy was either to loop all cpu or none
depending of avg_idle and avg_cost

>
> I've been looking at another ftrace output.
> The scheduler does like migrating the process to the current cpu.
> I have seen it migrate from one idle cpu to another idle cpu.
> I've not seen it migrate from an idle cpu to the current cpu.
>   (But I've not looked hard.)
>
> These are all the migrates:
>     TiNG task:12-1005  [026] d... 1111081.796560: sched_migrate_task: comm=RTP sockets pid=990 prio=120 orig_cpu=11 dest_cpu=16
>     TiNG task:31-1026  [005] d... 1111081.836556: sched_migrate_task: comm=RTP sockets pid=990 prio=120 orig_cpu=16 dest_cpu=11
>     TiNG task:28-1023  [033] d... 1111081.856589: sched_migrate_task: comm=RTP sockets pid=990 prio=120 orig_cpu=11 dest_cpu=33
>     TiNG task:11-1004  [013] d... 1111081.856606: sched_migrate_task: comm=RTP sockets pid=990 prio=120 orig_cpu=33 dest_cpu=13
>     TiNG task:19-1012  [002] d... 1111081.896564: sched_migrate_task: comm=RTP sockets pid=990 prio=120 orig_cpu=13 dest_cpu=10
>     TiNG task:26-1019  [008] d... 1111081.956551: sched_migrate_task: comm=RTP sockets pid=990 prio=120 orig_cpu=10 dest_cpu=18
>     TiNG task:34-1029  [001] d... 1111082.016527: sched_migrate_task: comm=RTP sockets pid=990 prio=120 orig_cpu=18 dest_cpu=1
>     TiNG task:20-1013  [021] d... 1111082.016589: sched_migrate_task: comm=RTP sockets pid=990 prio=120 orig_cpu=1 dest_cpu=21
>     TiNG task:32-1027  [000] d... 1111082.036455: sched_migrate_task: comm=RTP sockets pid=990 prio=120 orig_cpu=21 dest_cpu=2
>     TiNG task:15-1008  [006] d... 1111082.056539: sched_migrate_task: comm=RTP sockets pid=990 prio=120 orig_cpu=2 dest_cpu=14
>     TiNG task:34-1029  [001] d... 1111082.076536: sched_migrate_task: comm=RTP sockets pid=990 prio=120 orig_cpu=14 dest_cpu=1
>     TiNG task:21-1014  [004] d... 1111082.076589: sched_migrate_task: comm=RTP sockets pid=990 prio=120 orig_cpu=1 dest_cpu=4
>     TiNG task:11-1004  [013] d... 1111082.096526: sched_migrate_task: comm=RTP sockets pid=990 prio=120 orig_cpu=4 dest_cpu=13
>     TiNG task:28-1023  [033] d... 1111082.096584: sched_migrate_task: comm=RTP sockets pid=990 prio=120 orig_cpu=13 dest_cpu=33
>     TiNG task:25-1018  [029] d... 1111082.116549: sched_migrate_task: comm=RTP sockets pid=990 prio=120 orig_cpu=33 dest_cpu=11
>     TiNG task:27-1020  [032] d... 1111082.176519: sched_migrate_task: comm=RTP sockets pid=990 prio=120 orig_cpu=11 dest_cpu=16
> There are a couple of places where there 2 wakeups before the schedule.
> The scheduler definitely doesn't like waking up a process on an even cpu from on odd one.

odd and even cpus don't belong to the same llc and we only migrate in
the llc at wakeup

> But there are also the 13->33 and 1->21 ones.
>
>         David
>
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
> Registration No: 1397386 (Wales)