Re: [PATCH 00/10] sched/fair: wake_affine improvements

From: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
To: Mel Gorman <mgorman@techsingularity.net>
Cc: Ingo Molnar <mingo@kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Rik van Riel <riel@surriel.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Valentin Schneider <valentin.schneider@arm.com>,
	Vincent Guittot <vincent.guittot@linaro.org>,
	Dietmar Eggemann <dietmar.eggemann@arm.com>,
	Michael Ellerman <mpe@ellerman.id.au>,
	Gautham R Shenoy <ego@linux.vnet.ibm.com>,
	Parth Shah <parth@linux.ibm.com>
Subject: Re: [PATCH 00/10] sched/fair: wake_affine improvements
Date: Fri, 23 Apr 2021 16:01:29 +0530	[thread overview]
Message-ID: <20210423103129.GH2633526@linux.vnet.ibm.com> (raw)
In-Reply-To: <20210423082532.GA4239@techsingularity.net>

* Mel Gorman <mgorman@techsingularity.net> [2021-04-23 09:25:32]:

> On Thu, Apr 22, 2021 at 03:53:16PM +0530, Srikar Dronamraju wrote:
> > Recently we found that some of the benchmark numbers on Power10 were lesser
> > than expected. Some analysis showed that the problem lies in the fact that
> > L2-Cache on Power10 is at core level i.e only 4 threads share the L2-cache.
> > 
> 
> I didn't get the chance to review this properly although I am suspicious
> of tracking idle_core and updating that more frequently. It becomes a very
> hot cache line that bounces. I did experiement with tracking an idle core
> but the data either went stale too quickly or the updates incurred more
> overhead than a reduced search saved.

Thanks a lot for trying this.

> The series also oopses a *lot* and didn't get through a run of basic
> workloads on x86 on any of three machines. An example oops is
> 

Can you pass me your failing config. I am somehow not been seeing this
either on x86 or on Powerpc on multiple systems.
Also if possible cat /proc/schedstat and cat
/proc/sys/kernel/sched_domain/cpu0/domain*/name

> [  137.770968] BUG: unable to handle page fault for address: 000000000001a5c8
> [  137.777836] #PF: supervisor read access in kernel mode
> [  137.782965] #PF: error_code(0x0000) - not-present page
> [  137.788097] PGD 8000004098a42067 P4D 8000004098a42067 PUD 4092e36067 PMD 40883ac067 PTE 0
> [  137.796261] Oops: 0000 [#1] SMP PTI
> [  137.799747] CPU: 0 PID: 14913 Comm: GC Slave Tainted: G            E     5.12.0-rc8-llcfallback-v1r1 #1
> [  137.809123] Hardware name: SGI.COM C2112-4GP3/X10DRT-P-Series, BIOS 2.0a 05/09/2016
> [  137.816765] RIP: 0010:cpus_share_cache+0x22/0x30
> [  137.821396] Code: fc ff 0f 0b eb 80 66 90 0f 1f 44 00 00 48 63 ff 48 63 f6 48 c7 c0 c8 a5 01 00 48 8b 0c fd 00 59 9d 9a 48 8b 14 f5 00 59 9d 9a <8b> 14 02 39 14 01 0f 94 c0 c3 0f 1f 40 00 0f 1f 44 00 00 41 57 41

IP says cpus_share_cache, and it takes 2 ints,
RAX is 000000000001a5c8 but the panic says
"unable to handle page fault for address: 000000000001a5c8"
so it must have failed for "per_cpu(sd_llc_id, xx_cpu)"

> [  137.840133] RSP: 0018:ffff9e0aa26a7c60 EFLAGS: 00010086
> [  137.845360] RAX: 000000000001a5c8 RBX: ffff8bfbc65cf280 RCX: ffff8c3abf440000
> [  137.852483] RDX: 0000000000000000 RSI: ffffffffffffffff RDI: 0000000000000015
> [  137.859605] RBP: ffff8bbc478b9fe0 R08: 0000000000000000 R09: 000000000001a5b8
> [  137.866730] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8bbc47c88a00
> [  137.873855] R13: 0000000000000015 R14: 0000000000000001 R15: ffff8bfde0199fc0
> [  137.880978] FS:  00007fa33c173700(0000) GS:ffff8bfabf400000(0000) knlGS:0000000000000000
> [  137.889055] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  137.894794] CR2: 000000000001a5c8 CR3: 0000004096b22006 CR4: 00000000003706f0
> [  137.901918] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [  137.909041] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

-- 
Thanks and Regards
Srikar Dronamraju