* [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
@ 2021-06-21 6:32 ` Sachin Sant
0 siblings, 0 replies; 26+ messages in thread
From: Sachin Sant @ 2021-06-21 6:32 UTC (permalink / raw)
To: linux-kernel; +Cc: linuxppc-dev, odin, peterz
While running LTP tests (cfs_bandwidth01) against 5.13.0-rc7 kernel on a powerpc box
following warning is seen
[ 6611.331827] ------------[ cut here ]------------
[ 6611.331855] rq->tmp_alone_branch != &rq->leaf_cfs_rq_list
[ 6611.331862] WARNING: CPU: 8 PID: 0 at kernel/sched/fair.c:401 unthrottle_cfs_rq+0x4cc/0x590
[ 6611.331883] Modules linked in: nfsv3 nfs_acl nfs lockd grace fscache netfs tun brd overlay vfat fat btrfs blake2b_generic xor zstd_compress raid6_pq xfs loop sctp ip6_udp_tunnel udp_tunnel libcrc32c dm_mod bonding rfkill sunrpc pseries_rng xts vmx_crypto sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse [last unloaded: init_module]
[ 6611.331957] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G OE 5.13.0-rc6-gcba5e97280f5 #1
[ 6611.331968] NIP: c0000000001b7aac LR: c0000000001b7aa8 CTR: c000000000722d30
[ 6611.331976] REGS: c00000000274f3a0 TRAP: 0700 Tainted: G OE (5.13.0-rc6-gcba5e97280f5)
[ 6611.331985] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 48000224 XER: 00000005
[ 6611.332002] CFAR: c00000000014ca20 IRQMASK: 1
[ 6611.332002] GPR00: c0000000001b7aa8 c00000000274f640 c000000001abaf00 000000000000002d
[ 6611.332002] GPR04: 00000000ffff7fff c00000000274f300 0000000000000027 c000000efdb07e08
[ 6611.332002] GPR08: 0000000000000023 0000000000000001 0000000000000027 c000000001976680
[ 6611.332002] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200
[ 6611.332002] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 6611.332002] GPR20: 0000000000000001 c000000000fa6c08 c000000000fa6030 0000000000000001
[ 6611.332002] GPR24: 0000000000000000 0000000000000000 c000000efde12380 0000000000000001
[ 6611.332002] GPR28: 0000000000000001 0000000000000000 c000000efde12400 0000000000000000
[ 6611.332094] NIP [c0000000001b7aac] unthrottle_cfs_rq+0x4cc/0x590
[ 6611.332104] LR [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590
[ 6611.332113] Call Trace:
[ 6611.332116] [c00000000274f640] [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590 (unreliable)
[ 6611.332128] [c00000000274f6e0] [c0000000001b7e38] distribute_cfs_runtime+0x1d8/0x280
[ 6611.332139] [c00000000274f7b0] [c0000000001b81d0] sched_cfs_period_timer+0x140/0x330
[ 6611.332149] [c00000000274f870] [c00000000022a03c] __hrtimer_run_queues+0x17c/0x380
[ 6611.332158] [c00000000274f8f0] [c00000000022ac68] hrtimer_interrupt+0x128/0x2f0
[ 6611.332168] [c00000000274f9a0] [c00000000002940c] timer_interrupt+0x13c/0x370
[ 6611.332179] [c00000000274fa00] [c000000000009c04] decrementer_common_virt+0x1a4/0x1b0
[ 6611.332189] --- interrupt: 900 at plpar_hcall_norets_notrace+0x18/0x24
[ 6611.332199] NIP: c0000000000f6af8 LR: c000000000a05f68 CTR: 0000000000000000
[ 6611.332206] REGS: c00000000274fa70 TRAP: 0900 Tainted: G OE (5.13.0-rc6-gcba5e97280f5)
[ 6611.332214] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 28000224 XER: 00000000
[ 6611.332234] CFAR: 0000000000000c00 IRQMASK: 0
[ 6611.332234] GPR00: 0000000000000000 c00000000274fd10 c000000001abaf00 0000000000000000
[ 6611.332234] GPR04: 00000000000000c0 0000000000000080 0001a91c68b80fa1 00000000000003dc
[ 6611.332234] GPR08: 000000000001f400 0000000000000001 0000000000000000 0000000000000000
[ 6611.332234] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200
[ 6611.332234] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 6611.332234] GPR20: 0000000000000001 0000000000000002 0000000000000010 c0000000019fe2f8
[ 6611.332234] GPR24: 0000000000000001 00000603517d757e 0000000000000000 0000000000000000
[ 6611.332234] GPR28: 0000000000000001 0000000000000000 c000000001231f90 c000000001231f98
[ 6611.332323] NIP [c0000000000f6af8] plpar_hcall_norets_notrace+0x18/0x24
[ 6611.332332] LR [c000000000a05f68] check_and_cede_processor+0x48/0x60
[ 6611.332340] --- interrupt: 900
[ 6611.332345] [c00000000274fd10] [c000000efdb92380] 0xc000000efdb92380 (unreliable)
[ 6611.332355] [c00000000274fd70] [c000000000a063bc] dedicated_cede_loop+0x9c/0x1b0
[ 6611.332364] [c00000000274fdc0] [c000000000a02b04] cpuidle_enter_state+0x2e4/0x4e0
[ 6611.332375] [c00000000274fe20] [c000000000a02da0] cpuidle_enter+0x50/0x70
[ 6611.332385] [c00000000274fe60] [c0000000001a883c] call_cpuidle+0x4c/0x80
[ 6611.332393] [c00000000274fe80] [c0000000001a8ee0] do_idle+0x380/0x3e0
[ 6611.332402] [c00000000274ff00] [c0000000001a91bc] cpu_startup_entry+0x3c/0x40
[ 6611.332411] [c00000000274ff30] [c000000000063ff8] start_secondary+0x298/0x2b0
[ 6611.332421] [c00000000274ff90] [c00000000000c754] start_secondary_prolog+0x10/0x14
[ 6611.332430] Instruction dump:
[ 6611.332435] 4bfffc44 3d22fff6 8929f328 2f890000 409efea4 39200001 3d42fff6 3c62ff4f
[ 6611.332451] 3863bcd8 992af328 4bf94f15 60000000 <0fe00000> 4bfffe80 7f6407b4 7f43d378
[ 6611.332466] ---[ end trace 1346f865cd1cae91 ]—
5.13.0-rc6 was good. Bisect points to following patch
commit a7b359fc6a37
sched/fair: Correctly insert cfs_rq's to list on unthrottle
The test runs to completion(without this warning) if the patch is reverted.
Thanks
-Sachin
^ permalink raw reply [flat|nested] 26+ messages in thread
* [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
@ 2021-06-21 6:32 ` Sachin Sant
0 siblings, 0 replies; 26+ messages in thread
From: Sachin Sant @ 2021-06-21 6:32 UTC (permalink / raw)
To: linux-kernel; +Cc: peterz, odin, linuxppc-dev
While running LTP tests (cfs_bandwidth01) against 5.13.0-rc7 kernel on a powerpc box
following warning is seen
[ 6611.331827] ------------[ cut here ]------------
[ 6611.331855] rq->tmp_alone_branch != &rq->leaf_cfs_rq_list
[ 6611.331862] WARNING: CPU: 8 PID: 0 at kernel/sched/fair.c:401 unthrottle_cfs_rq+0x4cc/0x590
[ 6611.331883] Modules linked in: nfsv3 nfs_acl nfs lockd grace fscache netfs tun brd overlay vfat fat btrfs blake2b_generic xor zstd_compress raid6_pq xfs loop sctp ip6_udp_tunnel udp_tunnel libcrc32c dm_mod bonding rfkill sunrpc pseries_rng xts vmx_crypto sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse [last unloaded: init_module]
[ 6611.331957] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G OE 5.13.0-rc6-gcba5e97280f5 #1
[ 6611.331968] NIP: c0000000001b7aac LR: c0000000001b7aa8 CTR: c000000000722d30
[ 6611.331976] REGS: c00000000274f3a0 TRAP: 0700 Tainted: G OE (5.13.0-rc6-gcba5e97280f5)
[ 6611.331985] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 48000224 XER: 00000005
[ 6611.332002] CFAR: c00000000014ca20 IRQMASK: 1
[ 6611.332002] GPR00: c0000000001b7aa8 c00000000274f640 c000000001abaf00 000000000000002d
[ 6611.332002] GPR04: 00000000ffff7fff c00000000274f300 0000000000000027 c000000efdb07e08
[ 6611.332002] GPR08: 0000000000000023 0000000000000001 0000000000000027 c000000001976680
[ 6611.332002] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200
[ 6611.332002] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 6611.332002] GPR20: 0000000000000001 c000000000fa6c08 c000000000fa6030 0000000000000001
[ 6611.332002] GPR24: 0000000000000000 0000000000000000 c000000efde12380 0000000000000001
[ 6611.332002] GPR28: 0000000000000001 0000000000000000 c000000efde12400 0000000000000000
[ 6611.332094] NIP [c0000000001b7aac] unthrottle_cfs_rq+0x4cc/0x590
[ 6611.332104] LR [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590
[ 6611.332113] Call Trace:
[ 6611.332116] [c00000000274f640] [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590 (unreliable)
[ 6611.332128] [c00000000274f6e0] [c0000000001b7e38] distribute_cfs_runtime+0x1d8/0x280
[ 6611.332139] [c00000000274f7b0] [c0000000001b81d0] sched_cfs_period_timer+0x140/0x330
[ 6611.332149] [c00000000274f870] [c00000000022a03c] __hrtimer_run_queues+0x17c/0x380
[ 6611.332158] [c00000000274f8f0] [c00000000022ac68] hrtimer_interrupt+0x128/0x2f0
[ 6611.332168] [c00000000274f9a0] [c00000000002940c] timer_interrupt+0x13c/0x370
[ 6611.332179] [c00000000274fa00] [c000000000009c04] decrementer_common_virt+0x1a4/0x1b0
[ 6611.332189] --- interrupt: 900 at plpar_hcall_norets_notrace+0x18/0x24
[ 6611.332199] NIP: c0000000000f6af8 LR: c000000000a05f68 CTR: 0000000000000000
[ 6611.332206] REGS: c00000000274fa70 TRAP: 0900 Tainted: G OE (5.13.0-rc6-gcba5e97280f5)
[ 6611.332214] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 28000224 XER: 00000000
[ 6611.332234] CFAR: 0000000000000c00 IRQMASK: 0
[ 6611.332234] GPR00: 0000000000000000 c00000000274fd10 c000000001abaf00 0000000000000000
[ 6611.332234] GPR04: 00000000000000c0 0000000000000080 0001a91c68b80fa1 00000000000003dc
[ 6611.332234] GPR08: 000000000001f400 0000000000000001 0000000000000000 0000000000000000
[ 6611.332234] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200
[ 6611.332234] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 6611.332234] GPR20: 0000000000000001 0000000000000002 0000000000000010 c0000000019fe2f8
[ 6611.332234] GPR24: 0000000000000001 00000603517d757e 0000000000000000 0000000000000000
[ 6611.332234] GPR28: 0000000000000001 0000000000000000 c000000001231f90 c000000001231f98
[ 6611.332323] NIP [c0000000000f6af8] plpar_hcall_norets_notrace+0x18/0x24
[ 6611.332332] LR [c000000000a05f68] check_and_cede_processor+0x48/0x60
[ 6611.332340] --- interrupt: 900
[ 6611.332345] [c00000000274fd10] [c000000efdb92380] 0xc000000efdb92380 (unreliable)
[ 6611.332355] [c00000000274fd70] [c000000000a063bc] dedicated_cede_loop+0x9c/0x1b0
[ 6611.332364] [c00000000274fdc0] [c000000000a02b04] cpuidle_enter_state+0x2e4/0x4e0
[ 6611.332375] [c00000000274fe20] [c000000000a02da0] cpuidle_enter+0x50/0x70
[ 6611.332385] [c00000000274fe60] [c0000000001a883c] call_cpuidle+0x4c/0x80
[ 6611.332393] [c00000000274fe80] [c0000000001a8ee0] do_idle+0x380/0x3e0
[ 6611.332402] [c00000000274ff00] [c0000000001a91bc] cpu_startup_entry+0x3c/0x40
[ 6611.332411] [c00000000274ff30] [c000000000063ff8] start_secondary+0x298/0x2b0
[ 6611.332421] [c00000000274ff90] [c00000000000c754] start_secondary_prolog+0x10/0x14
[ 6611.332430] Instruction dump:
[ 6611.332435] 4bfffc44 3d22fff6 8929f328 2f890000 409efea4 39200001 3d42fff6 3c62ff4f
[ 6611.332451] 3863bcd8 992af328 4bf94f15 60000000 <0fe00000> 4bfffe80 7f6407b4 7f43d378
[ 6611.332466] ---[ end trace 1346f865cd1cae91 ]—
5.13.0-rc6 was good. Bisect points to following patch
commit a7b359fc6a37
sched/fair: Correctly insert cfs_rq's to list on unthrottle
The test runs to completion(without this warning) if the patch is reverted.
Thanks
-Sachin
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
2021-06-21 6:32 ` Sachin Sant
@ 2021-06-21 9:39 ` Odin Ugedal
-1 siblings, 0 replies; 26+ messages in thread
From: Odin Ugedal @ 2021-06-21 9:39 UTC (permalink / raw)
To: Sachin Sant
Cc: open list, linuxppc-dev, Odin Ugedal, Peter Zijlstra, Vincent Guittot
man. 21. jun. 2021 kl. 08:33 skrev Sachin Sant <sachinp@linux.vnet.ibm.com>:
>
> While running LTP tests (cfs_bandwidth01) against 5.13.0-rc7 kernel on a powerpc box
> following warning is seen
>
> [ 6611.331827] ------------[ cut here ]------------
> [ 6611.331855] rq->tmp_alone_branch != &rq->leaf_cfs_rq_list
> [ 6611.331862] WARNING: CPU: 8 PID: 0 at kernel/sched/fair.c:401 unthrottle_cfs_rq+0x4cc/0x590
> [ 6611.331883] Modules linked in: nfsv3 nfs_acl nfs lockd grace fscache netfs tun brd overlay vfat fat btrfs blake2b_generic xor zstd_compress raid6_pq xfs loop sctp ip6_udp_tunnel udp_tunnel libcrc32c dm_mod bonding rfkill sunrpc pseries_rng xts vmx_crypto sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse [last unloaded: init_module]
> [ 6611.331957] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G OE 5.13.0-rc6-gcba5e97280f5 #1
> [ 6611.331968] NIP: c0000000001b7aac LR: c0000000001b7aa8 CTR: c000000000722d30
> [ 6611.331976] REGS: c00000000274f3a0 TRAP: 0700 Tainted: G OE (5.13.0-rc6-gcba5e97280f5)
> [ 6611.331985] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 48000224 XER: 00000005
> [ 6611.332002] CFAR: c00000000014ca20 IRQMASK: 1
> [ 6611.332002] GPR00: c0000000001b7aa8 c00000000274f640 c000000001abaf00 000000000000002d
> [ 6611.332002] GPR04: 00000000ffff7fff c00000000274f300 0000000000000027 c000000efdb07e08
> [ 6611.332002] GPR08: 0000000000000023 0000000000000001 0000000000000027 c000000001976680
> [ 6611.332002] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200
> [ 6611.332002] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ 6611.332002] GPR20: 0000000000000001 c000000000fa6c08 c000000000fa6030 0000000000000001
> [ 6611.332002] GPR24: 0000000000000000 0000000000000000 c000000efde12380 0000000000000001
> [ 6611.332002] GPR28: 0000000000000001 0000000000000000 c000000efde12400 0000000000000000
> [ 6611.332094] NIP [c0000000001b7aac] unthrottle_cfs_rq+0x4cc/0x590
> [ 6611.332104] LR [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590
> [ 6611.332113] Call Trace:
> [ 6611.332116] [c00000000274f640] [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590 (unreliable)
> [ 6611.332128] [c00000000274f6e0] [c0000000001b7e38] distribute_cfs_runtime+0x1d8/0x280
> [ 6611.332139] [c00000000274f7b0] [c0000000001b81d0] sched_cfs_period_timer+0x140/0x330
> [ 6611.332149] [c00000000274f870] [c00000000022a03c] __hrtimer_run_queues+0x17c/0x380
> [ 6611.332158] [c00000000274f8f0] [c00000000022ac68] hrtimer_interrupt+0x128/0x2f0
> [ 6611.332168] [c00000000274f9a0] [c00000000002940c] timer_interrupt+0x13c/0x370
> [ 6611.332179] [c00000000274fa00] [c000000000009c04] decrementer_common_virt+0x1a4/0x1b0
> [ 6611.332189] --- interrupt: 900 at plpar_hcall_norets_notrace+0x18/0x24
> [ 6611.332199] NIP: c0000000000f6af8 LR: c000000000a05f68 CTR: 0000000000000000
> [ 6611.332206] REGS: c00000000274fa70 TRAP: 0900 Tainted: G OE (5.13.0-rc6-gcba5e97280f5)
> [ 6611.332214] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 28000224 XER: 00000000
> [ 6611.332234] CFAR: 0000000000000c00 IRQMASK: 0
> [ 6611.332234] GPR00: 0000000000000000 c00000000274fd10 c000000001abaf00 0000000000000000
> [ 6611.332234] GPR04: 00000000000000c0 0000000000000080 0001a91c68b80fa1 00000000000003dc
> [ 6611.332234] GPR08: 000000000001f400 0000000000000001 0000000000000000 0000000000000000
> [ 6611.332234] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200
> [ 6611.332234] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ 6611.332234] GPR20: 0000000000000001 0000000000000002 0000000000000010 c0000000019fe2f8
> [ 6611.332234] GPR24: 0000000000000001 00000603517d757e 0000000000000000 0000000000000000
> [ 6611.332234] GPR28: 0000000000000001 0000000000000000 c000000001231f90 c000000001231f98
> [ 6611.332323] NIP [c0000000000f6af8] plpar_hcall_norets_notrace+0x18/0x24
> [ 6611.332332] LR [c000000000a05f68] check_and_cede_processor+0x48/0x60
> [ 6611.332340] --- interrupt: 900
> [ 6611.332345] [c00000000274fd10] [c000000efdb92380] 0xc000000efdb92380 (unreliable)
> [ 6611.332355] [c00000000274fd70] [c000000000a063bc] dedicated_cede_loop+0x9c/0x1b0
> [ 6611.332364] [c00000000274fdc0] [c000000000a02b04] cpuidle_enter_state+0x2e4/0x4e0
> [ 6611.332375] [c00000000274fe20] [c000000000a02da0] cpuidle_enter+0x50/0x70
> [ 6611.332385] [c00000000274fe60] [c0000000001a883c] call_cpuidle+0x4c/0x80
> [ 6611.332393] [c00000000274fe80] [c0000000001a8ee0] do_idle+0x380/0x3e0
> [ 6611.332402] [c00000000274ff00] [c0000000001a91bc] cpu_startup_entry+0x3c/0x40
> [ 6611.332411] [c00000000274ff30] [c000000000063ff8] start_secondary+0x298/0x2b0
> [ 6611.332421] [c00000000274ff90] [c00000000000c754] start_secondary_prolog+0x10/0x14
> [ 6611.332430] Instruction dump:
> [ 6611.332435] 4bfffc44 3d22fff6 8929f328 2f890000 409efea4 39200001 3d42fff6 3c62ff4f
> [ 6611.332451] 3863bcd8 992af328 4bf94f15 60000000 <0fe00000> 4bfffe80 7f6407b4 7f43d378
> [ 6611.332466] ---[ end trace 1346f865cd1cae91 ]—
>
> 5.13.0-rc6 was good. Bisect points to following patch
>
> commit a7b359fc6a37
> sched/fair: Correctly insert cfs_rq's to list on unthrottle
>
> The test runs to completion(without this warning) if the patch is reverted.
>
> Thanks
> -Sachin
>
Hi,
Thanks for the report! I have a theory about what is possibly causing
this, so I will try to reproduce it and see if my assumptions are
correct.
Odin
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
@ 2021-06-21 9:39 ` Odin Ugedal
0 siblings, 0 replies; 26+ messages in thread
From: Odin Ugedal @ 2021-06-21 9:39 UTC (permalink / raw)
To: Sachin Sant
Cc: Peter Zijlstra, Odin Ugedal, linuxppc-dev, open list, Vincent Guittot
man. 21. jun. 2021 kl. 08:33 skrev Sachin Sant <sachinp@linux.vnet.ibm.com>:
>
> While running LTP tests (cfs_bandwidth01) against 5.13.0-rc7 kernel on a powerpc box
> following warning is seen
>
> [ 6611.331827] ------------[ cut here ]------------
> [ 6611.331855] rq->tmp_alone_branch != &rq->leaf_cfs_rq_list
> [ 6611.331862] WARNING: CPU: 8 PID: 0 at kernel/sched/fair.c:401 unthrottle_cfs_rq+0x4cc/0x590
> [ 6611.331883] Modules linked in: nfsv3 nfs_acl nfs lockd grace fscache netfs tun brd overlay vfat fat btrfs blake2b_generic xor zstd_compress raid6_pq xfs loop sctp ip6_udp_tunnel udp_tunnel libcrc32c dm_mod bonding rfkill sunrpc pseries_rng xts vmx_crypto sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse [last unloaded: init_module]
> [ 6611.331957] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G OE 5.13.0-rc6-gcba5e97280f5 #1
> [ 6611.331968] NIP: c0000000001b7aac LR: c0000000001b7aa8 CTR: c000000000722d30
> [ 6611.331976] REGS: c00000000274f3a0 TRAP: 0700 Tainted: G OE (5.13.0-rc6-gcba5e97280f5)
> [ 6611.331985] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 48000224 XER: 00000005
> [ 6611.332002] CFAR: c00000000014ca20 IRQMASK: 1
> [ 6611.332002] GPR00: c0000000001b7aa8 c00000000274f640 c000000001abaf00 000000000000002d
> [ 6611.332002] GPR04: 00000000ffff7fff c00000000274f300 0000000000000027 c000000efdb07e08
> [ 6611.332002] GPR08: 0000000000000023 0000000000000001 0000000000000027 c000000001976680
> [ 6611.332002] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200
> [ 6611.332002] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ 6611.332002] GPR20: 0000000000000001 c000000000fa6c08 c000000000fa6030 0000000000000001
> [ 6611.332002] GPR24: 0000000000000000 0000000000000000 c000000efde12380 0000000000000001
> [ 6611.332002] GPR28: 0000000000000001 0000000000000000 c000000efde12400 0000000000000000
> [ 6611.332094] NIP [c0000000001b7aac] unthrottle_cfs_rq+0x4cc/0x590
> [ 6611.332104] LR [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590
> [ 6611.332113] Call Trace:
> [ 6611.332116] [c00000000274f640] [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590 (unreliable)
> [ 6611.332128] [c00000000274f6e0] [c0000000001b7e38] distribute_cfs_runtime+0x1d8/0x280
> [ 6611.332139] [c00000000274f7b0] [c0000000001b81d0] sched_cfs_period_timer+0x140/0x330
> [ 6611.332149] [c00000000274f870] [c00000000022a03c] __hrtimer_run_queues+0x17c/0x380
> [ 6611.332158] [c00000000274f8f0] [c00000000022ac68] hrtimer_interrupt+0x128/0x2f0
> [ 6611.332168] [c00000000274f9a0] [c00000000002940c] timer_interrupt+0x13c/0x370
> [ 6611.332179] [c00000000274fa00] [c000000000009c04] decrementer_common_virt+0x1a4/0x1b0
> [ 6611.332189] --- interrupt: 900 at plpar_hcall_norets_notrace+0x18/0x24
> [ 6611.332199] NIP: c0000000000f6af8 LR: c000000000a05f68 CTR: 0000000000000000
> [ 6611.332206] REGS: c00000000274fa70 TRAP: 0900 Tainted: G OE (5.13.0-rc6-gcba5e97280f5)
> [ 6611.332214] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 28000224 XER: 00000000
> [ 6611.332234] CFAR: 0000000000000c00 IRQMASK: 0
> [ 6611.332234] GPR00: 0000000000000000 c00000000274fd10 c000000001abaf00 0000000000000000
> [ 6611.332234] GPR04: 00000000000000c0 0000000000000080 0001a91c68b80fa1 00000000000003dc
> [ 6611.332234] GPR08: 000000000001f400 0000000000000001 0000000000000000 0000000000000000
> [ 6611.332234] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200
> [ 6611.332234] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ 6611.332234] GPR20: 0000000000000001 0000000000000002 0000000000000010 c0000000019fe2f8
> [ 6611.332234] GPR24: 0000000000000001 00000603517d757e 0000000000000000 0000000000000000
> [ 6611.332234] GPR28: 0000000000000001 0000000000000000 c000000001231f90 c000000001231f98
> [ 6611.332323] NIP [c0000000000f6af8] plpar_hcall_norets_notrace+0x18/0x24
> [ 6611.332332] LR [c000000000a05f68] check_and_cede_processor+0x48/0x60
> [ 6611.332340] --- interrupt: 900
> [ 6611.332345] [c00000000274fd10] [c000000efdb92380] 0xc000000efdb92380 (unreliable)
> [ 6611.332355] [c00000000274fd70] [c000000000a063bc] dedicated_cede_loop+0x9c/0x1b0
> [ 6611.332364] [c00000000274fdc0] [c000000000a02b04] cpuidle_enter_state+0x2e4/0x4e0
> [ 6611.332375] [c00000000274fe20] [c000000000a02da0] cpuidle_enter+0x50/0x70
> [ 6611.332385] [c00000000274fe60] [c0000000001a883c] call_cpuidle+0x4c/0x80
> [ 6611.332393] [c00000000274fe80] [c0000000001a8ee0] do_idle+0x380/0x3e0
> [ 6611.332402] [c00000000274ff00] [c0000000001a91bc] cpu_startup_entry+0x3c/0x40
> [ 6611.332411] [c00000000274ff30] [c000000000063ff8] start_secondary+0x298/0x2b0
> [ 6611.332421] [c00000000274ff90] [c00000000000c754] start_secondary_prolog+0x10/0x14
> [ 6611.332430] Instruction dump:
> [ 6611.332435] 4bfffc44 3d22fff6 8929f328 2f890000 409efea4 39200001 3d42fff6 3c62ff4f
> [ 6611.332451] 3863bcd8 992af328 4bf94f15 60000000 <0fe00000> 4bfffe80 7f6407b4 7f43d378
> [ 6611.332466] ---[ end trace 1346f865cd1cae91 ]—
>
> 5.13.0-rc6 was good. Bisect points to following patch
>
> commit a7b359fc6a37
> sched/fair: Correctly insert cfs_rq's to list on unthrottle
>
> The test runs to completion(without this warning) if the patch is reverted.
>
> Thanks
> -Sachin
>
Hi,
Thanks for the report! I have a theory about what is possibly causing
this, so I will try to reproduce it and see if my assumptions are
correct.
Odin
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
2021-06-21 9:39 ` Odin Ugedal
@ 2021-06-21 9:50 ` Vincent Guittot
-1 siblings, 0 replies; 26+ messages in thread
From: Vincent Guittot @ 2021-06-21 9:50 UTC (permalink / raw)
To: Odin Ugedal; +Cc: Sachin Sant, open list, linuxppc-dev, Peter Zijlstra
On Mon, 21 Jun 2021 at 11:39, Odin Ugedal <odin@uged.al> wrote:
>
> man. 21. jun. 2021 kl. 08:33 skrev Sachin Sant <sachinp@linux.vnet.ibm.com>:
> >
> > While running LTP tests (cfs_bandwidth01) against 5.13.0-rc7 kernel on a powerpc box
> > following warning is seen
> >
> > [ 6611.331827] ------------[ cut here ]------------
> > [ 6611.331855] rq->tmp_alone_branch != &rq->leaf_cfs_rq_list
> > [ 6611.331862] WARNING: CPU: 8 PID: 0 at kernel/sched/fair.c:401 unthrottle_cfs_rq+0x4cc/0x590
> > [ 6611.331883] Modules linked in: nfsv3 nfs_acl nfs lockd grace fscache netfs tun brd overlay vfat fat btrfs blake2b_generic xor zstd_compress raid6_pq xfs loop sctp ip6_udp_tunnel udp_tunnel libcrc32c dm_mod bonding rfkill sunrpc pseries_rng xts vmx_crypto sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse [last unloaded: init_module]
> > [ 6611.331957] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G OE 5.13.0-rc6-gcba5e97280f5 #1
> > [ 6611.331968] NIP: c0000000001b7aac LR: c0000000001b7aa8 CTR: c000000000722d30
> > [ 6611.331976] REGS: c00000000274f3a0 TRAP: 0700 Tainted: G OE (5.13.0-rc6-gcba5e97280f5)
> > [ 6611.331985] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 48000224 XER: 00000005
> > [ 6611.332002] CFAR: c00000000014ca20 IRQMASK: 1
> > [ 6611.332002] GPR00: c0000000001b7aa8 c00000000274f640 c000000001abaf00 000000000000002d
> > [ 6611.332002] GPR04: 00000000ffff7fff c00000000274f300 0000000000000027 c000000efdb07e08
> > [ 6611.332002] GPR08: 0000000000000023 0000000000000001 0000000000000027 c000000001976680
> > [ 6611.332002] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200
> > [ 6611.332002] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> > [ 6611.332002] GPR20: 0000000000000001 c000000000fa6c08 c000000000fa6030 0000000000000001
> > [ 6611.332002] GPR24: 0000000000000000 0000000000000000 c000000efde12380 0000000000000001
> > [ 6611.332002] GPR28: 0000000000000001 0000000000000000 c000000efde12400 0000000000000000
> > [ 6611.332094] NIP [c0000000001b7aac] unthrottle_cfs_rq+0x4cc/0x590
> > [ 6611.332104] LR [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590
> > [ 6611.332113] Call Trace:
> > [ 6611.332116] [c00000000274f640] [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590 (unreliable)
> > [ 6611.332128] [c00000000274f6e0] [c0000000001b7e38] distribute_cfs_runtime+0x1d8/0x280
> > [ 6611.332139] [c00000000274f7b0] [c0000000001b81d0] sched_cfs_period_timer+0x140/0x330
> > [ 6611.332149] [c00000000274f870] [c00000000022a03c] __hrtimer_run_queues+0x17c/0x380
> > [ 6611.332158] [c00000000274f8f0] [c00000000022ac68] hrtimer_interrupt+0x128/0x2f0
> > [ 6611.332168] [c00000000274f9a0] [c00000000002940c] timer_interrupt+0x13c/0x370
> > [ 6611.332179] [c00000000274fa00] [c000000000009c04] decrementer_common_virt+0x1a4/0x1b0
> > [ 6611.332189] --- interrupt: 900 at plpar_hcall_norets_notrace+0x18/0x24
> > [ 6611.332199] NIP: c0000000000f6af8 LR: c000000000a05f68 CTR: 0000000000000000
> > [ 6611.332206] REGS: c00000000274fa70 TRAP: 0900 Tainted: G OE (5.13.0-rc6-gcba5e97280f5)
> > [ 6611.332214] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 28000224 XER: 00000000
> > [ 6611.332234] CFAR: 0000000000000c00 IRQMASK: 0
> > [ 6611.332234] GPR00: 0000000000000000 c00000000274fd10 c000000001abaf00 0000000000000000
> > [ 6611.332234] GPR04: 00000000000000c0 0000000000000080 0001a91c68b80fa1 00000000000003dc
> > [ 6611.332234] GPR08: 000000000001f400 0000000000000001 0000000000000000 0000000000000000
> > [ 6611.332234] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200
> > [ 6611.332234] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> > [ 6611.332234] GPR20: 0000000000000001 0000000000000002 0000000000000010 c0000000019fe2f8
> > [ 6611.332234] GPR24: 0000000000000001 00000603517d757e 0000000000000000 0000000000000000
> > [ 6611.332234] GPR28: 0000000000000001 0000000000000000 c000000001231f90 c000000001231f98
> > [ 6611.332323] NIP [c0000000000f6af8] plpar_hcall_norets_notrace+0x18/0x24
> > [ 6611.332332] LR [c000000000a05f68] check_and_cede_processor+0x48/0x60
> > [ 6611.332340] --- interrupt: 900
> > [ 6611.332345] [c00000000274fd10] [c000000efdb92380] 0xc000000efdb92380 (unreliable)
> > [ 6611.332355] [c00000000274fd70] [c000000000a063bc] dedicated_cede_loop+0x9c/0x1b0
> > [ 6611.332364] [c00000000274fdc0] [c000000000a02b04] cpuidle_enter_state+0x2e4/0x4e0
> > [ 6611.332375] [c00000000274fe20] [c000000000a02da0] cpuidle_enter+0x50/0x70
> > [ 6611.332385] [c00000000274fe60] [c0000000001a883c] call_cpuidle+0x4c/0x80
> > [ 6611.332393] [c00000000274fe80] [c0000000001a8ee0] do_idle+0x380/0x3e0
> > [ 6611.332402] [c00000000274ff00] [c0000000001a91bc] cpu_startup_entry+0x3c/0x40
> > [ 6611.332411] [c00000000274ff30] [c000000000063ff8] start_secondary+0x298/0x2b0
> > [ 6611.332421] [c00000000274ff90] [c00000000000c754] start_secondary_prolog+0x10/0x14
> > [ 6611.332430] Instruction dump:
> > [ 6611.332435] 4bfffc44 3d22fff6 8929f328 2f890000 409efea4 39200001 3d42fff6 3c62ff4f
> > [ 6611.332451] 3863bcd8 992af328 4bf94f15 60000000 <0fe00000> 4bfffe80 7f6407b4 7f43d378
> > [ 6611.332466] ---[ end trace 1346f865cd1cae91 ]—
> >
> > 5.13.0-rc6 was good. Bisect points to following patch
> >
> > commit a7b359fc6a37
> > sched/fair: Correctly insert cfs_rq's to list on unthrottle
> >
> > The test runs to completion(without this warning) if the patch is reverted.
> >
> > Thanks
> > -Sachin
> >
>
> Hi,
>
> Thanks for the report! I have a theory about what is possibly causing
> this, so I will try to reproduce it and see if my assumptions are
> correct.
This means that a child's load was not null and it was inserted
whereas parent's load was null. This should not happen unless the
propagation failed somewhere
>
>
> Odin
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
@ 2021-06-21 9:50 ` Vincent Guittot
0 siblings, 0 replies; 26+ messages in thread
From: Vincent Guittot @ 2021-06-21 9:50 UTC (permalink / raw)
To: Odin Ugedal; +Cc: Sachin Sant, Peter Zijlstra, linuxppc-dev, open list
On Mon, 21 Jun 2021 at 11:39, Odin Ugedal <odin@uged.al> wrote:
>
> man. 21. jun. 2021 kl. 08:33 skrev Sachin Sant <sachinp@linux.vnet.ibm.com>:
> >
> > While running LTP tests (cfs_bandwidth01) against 5.13.0-rc7 kernel on a powerpc box
> > following warning is seen
> >
> > [ 6611.331827] ------------[ cut here ]------------
> > [ 6611.331855] rq->tmp_alone_branch != &rq->leaf_cfs_rq_list
> > [ 6611.331862] WARNING: CPU: 8 PID: 0 at kernel/sched/fair.c:401 unthrottle_cfs_rq+0x4cc/0x590
> > [ 6611.331883] Modules linked in: nfsv3 nfs_acl nfs lockd grace fscache netfs tun brd overlay vfat fat btrfs blake2b_generic xor zstd_compress raid6_pq xfs loop sctp ip6_udp_tunnel udp_tunnel libcrc32c dm_mod bonding rfkill sunrpc pseries_rng xts vmx_crypto sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse [last unloaded: init_module]
> > [ 6611.331957] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G OE 5.13.0-rc6-gcba5e97280f5 #1
> > [ 6611.331968] NIP: c0000000001b7aac LR: c0000000001b7aa8 CTR: c000000000722d30
> > [ 6611.331976] REGS: c00000000274f3a0 TRAP: 0700 Tainted: G OE (5.13.0-rc6-gcba5e97280f5)
> > [ 6611.331985] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 48000224 XER: 00000005
> > [ 6611.332002] CFAR: c00000000014ca20 IRQMASK: 1
> > [ 6611.332002] GPR00: c0000000001b7aa8 c00000000274f640 c000000001abaf00 000000000000002d
> > [ 6611.332002] GPR04: 00000000ffff7fff c00000000274f300 0000000000000027 c000000efdb07e08
> > [ 6611.332002] GPR08: 0000000000000023 0000000000000001 0000000000000027 c000000001976680
> > [ 6611.332002] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200
> > [ 6611.332002] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> > [ 6611.332002] GPR20: 0000000000000001 c000000000fa6c08 c000000000fa6030 0000000000000001
> > [ 6611.332002] GPR24: 0000000000000000 0000000000000000 c000000efde12380 0000000000000001
> > [ 6611.332002] GPR28: 0000000000000001 0000000000000000 c000000efde12400 0000000000000000
> > [ 6611.332094] NIP [c0000000001b7aac] unthrottle_cfs_rq+0x4cc/0x590
> > [ 6611.332104] LR [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590
> > [ 6611.332113] Call Trace:
> > [ 6611.332116] [c00000000274f640] [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590 (unreliable)
> > [ 6611.332128] [c00000000274f6e0] [c0000000001b7e38] distribute_cfs_runtime+0x1d8/0x280
> > [ 6611.332139] [c00000000274f7b0] [c0000000001b81d0] sched_cfs_period_timer+0x140/0x330
> > [ 6611.332149] [c00000000274f870] [c00000000022a03c] __hrtimer_run_queues+0x17c/0x380
> > [ 6611.332158] [c00000000274f8f0] [c00000000022ac68] hrtimer_interrupt+0x128/0x2f0
> > [ 6611.332168] [c00000000274f9a0] [c00000000002940c] timer_interrupt+0x13c/0x370
> > [ 6611.332179] [c00000000274fa00] [c000000000009c04] decrementer_common_virt+0x1a4/0x1b0
> > [ 6611.332189] --- interrupt: 900 at plpar_hcall_norets_notrace+0x18/0x24
> > [ 6611.332199] NIP: c0000000000f6af8 LR: c000000000a05f68 CTR: 0000000000000000
> > [ 6611.332206] REGS: c00000000274fa70 TRAP: 0900 Tainted: G OE (5.13.0-rc6-gcba5e97280f5)
> > [ 6611.332214] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 28000224 XER: 00000000
> > [ 6611.332234] CFAR: 0000000000000c00 IRQMASK: 0
> > [ 6611.332234] GPR00: 0000000000000000 c00000000274fd10 c000000001abaf00 0000000000000000
> > [ 6611.332234] GPR04: 00000000000000c0 0000000000000080 0001a91c68b80fa1 00000000000003dc
> > [ 6611.332234] GPR08: 000000000001f400 0000000000000001 0000000000000000 0000000000000000
> > [ 6611.332234] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200
> > [ 6611.332234] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> > [ 6611.332234] GPR20: 0000000000000001 0000000000000002 0000000000000010 c0000000019fe2f8
> > [ 6611.332234] GPR24: 0000000000000001 00000603517d757e 0000000000000000 0000000000000000
> > [ 6611.332234] GPR28: 0000000000000001 0000000000000000 c000000001231f90 c000000001231f98
> > [ 6611.332323] NIP [c0000000000f6af8] plpar_hcall_norets_notrace+0x18/0x24
> > [ 6611.332332] LR [c000000000a05f68] check_and_cede_processor+0x48/0x60
> > [ 6611.332340] --- interrupt: 900
> > [ 6611.332345] [c00000000274fd10] [c000000efdb92380] 0xc000000efdb92380 (unreliable)
> > [ 6611.332355] [c00000000274fd70] [c000000000a063bc] dedicated_cede_loop+0x9c/0x1b0
> > [ 6611.332364] [c00000000274fdc0] [c000000000a02b04] cpuidle_enter_state+0x2e4/0x4e0
> > [ 6611.332375] [c00000000274fe20] [c000000000a02da0] cpuidle_enter+0x50/0x70
> > [ 6611.332385] [c00000000274fe60] [c0000000001a883c] call_cpuidle+0x4c/0x80
> > [ 6611.332393] [c00000000274fe80] [c0000000001a8ee0] do_idle+0x380/0x3e0
> > [ 6611.332402] [c00000000274ff00] [c0000000001a91bc] cpu_startup_entry+0x3c/0x40
> > [ 6611.332411] [c00000000274ff30] [c000000000063ff8] start_secondary+0x298/0x2b0
> > [ 6611.332421] [c00000000274ff90] [c00000000000c754] start_secondary_prolog+0x10/0x14
> > [ 6611.332430] Instruction dump:
> > [ 6611.332435] 4bfffc44 3d22fff6 8929f328 2f890000 409efea4 39200001 3d42fff6 3c62ff4f
> > [ 6611.332451] 3863bcd8 992af328 4bf94f15 60000000 <0fe00000> 4bfffe80 7f6407b4 7f43d378
> > [ 6611.332466] ---[ end trace 1346f865cd1cae91 ]—
> >
> > 5.13.0-rc6 was good. Bisect points to following patch
> >
> > commit a7b359fc6a37
> > sched/fair: Correctly insert cfs_rq's to list on unthrottle
> >
> > The test runs to completion(without this warning) if the patch is reverted.
> >
> > Thanks
> > -Sachin
> >
>
> Hi,
>
> Thanks for the report! I have a theory about what is possibly causing
> this, so I will try to reproduce it and see if my assumptions are
> correct.
This means that a child's load was not null and it was inserted
whereas parent's load was null. This should not happen unless the
propagation failed somewhere
>
>
> Odin
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
2021-06-21 9:50 ` Vincent Guittot
@ 2021-06-21 9:54 ` Odin Ugedal
-1 siblings, 0 replies; 26+ messages in thread
From: Odin Ugedal @ 2021-06-21 9:54 UTC (permalink / raw)
To: Vincent Guittot
Cc: Odin Ugedal, Sachin Sant, open list, linuxppc-dev, Peter Zijlstra
man. 21. jun. 2021 kl. 11:50 skrev Vincent Guittot <vincent.guittot@linaro.org>:
> This means that a child's load was not null and it was inserted
> whereas parent's load was null. This should not happen unless the
> propagation failed somewhere
My initial thought is that the patch below will fix it, if that is the
issue (that a leaf is inserted, but the propagation is not "completed"
in unthrottle). Might that be the case? Still working on reproducing
the issue tho.
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index bfaa6e1f6067..015c5a5c1a4d 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4930,12 +4930,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)
if (cfs_rq_throttled(cfs_rq))
goto unthrottle_throttle;
- /*
- * One parent has been throttled and cfs_rq removed from the
- * list. Add it back to not break the leaf list.
- */
- if (throttled_hierarchy(cfs_rq))
- list_add_leaf_cfs_rq(cfs_rq);
+ list_add_leaf_cfs_rq(cfs_rq);
}
/* At this point se is NULL and we are at root level*/
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
@ 2021-06-21 9:54 ` Odin Ugedal
0 siblings, 0 replies; 26+ messages in thread
From: Odin Ugedal @ 2021-06-21 9:54 UTC (permalink / raw)
To: Vincent Guittot
Cc: Sachin Sant, linuxppc-dev, Peter Zijlstra, Odin Ugedal, open list
man. 21. jun. 2021 kl. 11:50 skrev Vincent Guittot <vincent.guittot@linaro.org>:
> This means that a child's load was not null and it was inserted
> whereas parent's load was null. This should not happen unless the
> propagation failed somewhere
My initial thought is that the patch below will fix it, if that is the
issue (that a leaf is inserted, but the propagation is not "completed"
in unthrottle). Might that be the case? Still working on reproducing
the issue tho.
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index bfaa6e1f6067..015c5a5c1a4d 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4930,12 +4930,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)
if (cfs_rq_throttled(cfs_rq))
goto unthrottle_throttle;
- /*
- * One parent has been throttled and cfs_rq removed from the
- * list. Add it back to not break the leaf list.
- */
- if (throttled_hierarchy(cfs_rq))
- list_add_leaf_cfs_rq(cfs_rq);
+ list_add_leaf_cfs_rq(cfs_rq);
}
/* At this point se is NULL and we are at root level*/
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
2021-06-21 9:54 ` Odin Ugedal
@ 2021-06-21 10:57 ` Sachin Sant
-1 siblings, 0 replies; 26+ messages in thread
From: Sachin Sant @ 2021-06-21 10:57 UTC (permalink / raw)
To: Odin Ugedal; +Cc: Vincent Guittot, open list, linuxppc-dev, Peter Zijlstra
[-- Attachment #1: Type: text/plain, Size: 1475 bytes --]
> On 21-Jun-2021, at 3:24 PM, Odin Ugedal <odin@uged.al> wrote:
>
> man. 21. jun. 2021 kl. 11:50 skrev Vincent Guittot <vincent.guittot@linaro.org>:
>> This means that a child's load was not null and it was inserted
>> whereas parent's load was null. This should not happen unless the
>> propagation failed somewhere
>
> My initial thought is that the patch below will fix it, if that is the
> issue (that a leaf is inserted, but the propagation is not "completed"
> in unthrottle). Might that be the case? Still working on reproducing
> the issue tho.
>
Unfortunately this does not help. I can still recreate the failure.
Have attached the o/p from test run.
Thanks
-Sachin
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index bfaa6e1f6067..015c5a5c1a4d 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -4930,12 +4930,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)
> if (cfs_rq_throttled(cfs_rq))
> goto unthrottle_throttle;
>
> - /*
> - * One parent has been throttled and cfs_rq removed from the
> - * list. Add it back to not break the leaf list.
> - */
> - if (throttled_hierarchy(cfs_rq))
> - list_add_leaf_cfs_rq(cfs_rq);
> + list_add_leaf_cfs_rq(cfs_rq);
> }
>
> /* At this point se is NULL and we are at root level*/
[-- Attachment #2: run.txt --]
[-- Type: text/plain, Size: 5966 bytes --]
# cd /opt/ltp/testcases/bin
# ./cfs_bandwidth01 -i 5
tst_test.c:1313: TINFO: Timeout per run is 0h 05m 00s
tst_buffers.c:55: TINFO: Test is using guarded buffers
cfs_bandwidth01.c:49: TINFO: Set 'worker1/cpu.max' = '3000 10000'
cfs_bandwidth01.c:49: TINFO: Set 'worker2/cpu.max' = '2000 10000'
cfs_bandwidth01.c:49: TINFO: Set 'worker3/cpu.max' = '3000 10000'
cfs_bandwidth01.c:113: TPASS: Scheduled bandwidth constrained workers
cfs_bandwidth01.c:49: TINFO: Set 'level2/cpu.max' = '5000 10000'
cfs_bandwidth01.c:125: TPASS: Workers exited
cfs_bandwidth01.c:113: TPASS: Scheduled bandwidth constrained workers
[ 48.343143] ------------[ cut here ]------------
[ 48.343164] rq->tmp_alone_branch != &rq->leaf_cfs_rq_list
[ 48.343172] WARNING: CPU: 24 PID: 4405 at kernel/sched/fair.c:401 unthrottle_cfs_rq+0x49c/0x560
[ 48.343196] Modules linked in: nf_tables nfnetlink tun bridge stp llc rfkill sunrpc pseries_rng xts vmx_crypto uio_pdrv_genirq uio sch_fq_codel ip_tables xfs libcrc32c sr_mod sd_mod cdrom t10_pi sg ibmvscsi ibmveth scsi_transport_srp dm_mirror dm_region_hash dm_log dm_mod fuse
[ 48.343251] CPU: 24 PID: 4405 Comm: cfs_bandwidth01 Not tainted 5.13.0-rc7-dirty #4
[ 48.343261] NIP: c0000000001b88fc LR: c0000000001b88f8 CTR: c000000000723d10
[ 48.343269] REGS: c00000000fb13780 TRAP: 0700 Not tainted (5.13.0-rc7-dirty)
[ 48.343278] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 48044224 XER: 00000005
[ 48.343295] CFAR: c00000000014d8a0 IRQMASK: 1
[ 48.343295] GPR00: c0000000001b88f8 c00000000fb13a20 c0000000029ab400 000000000000002d
[ 48.343295] GPR04: 00000000fffeffff c00000000fb136e0 0000000000000027 c00000154f817e08
[ 48.343295] GPR08: 0000000000000023 0000000000000001 0000000000000027 c00000167f1d7fe8
[ 48.343295] GPR12: 0000000000004000 c00000154ffdc680 0000000000000000 0000000000000000
[ 48.343295] GPR16: c000000000fa6660 0000000000000001 0000000000000000 c0000000024e1cd8
[ 48.343295] GPR20: 0000000000000000 c00000000290a69a 0000000000000000 c0000000024e1cc0
[ 48.343295] GPR24: 0000000000000000 c0000000029f2140 c00000154f762380 0000000000000001
[ 48.343295] GPR28: 0000000000000001 0000000000000000 c00000154f762400 0000000000000000
[ 48.343388] NIP [c0000000001b88fc] unthrottle_cfs_rq+0x49c/0x560
[ 48.343397] LR [c0000000001b88f8] unthrottle_cfs_rq+0x498/0x560
[ 48.343406] Call Trace:
[ 48.343410] [c00000000fb13a20] [c0000000001b88f8] unthrottle_cfs_rq+0x498/0x560 (unreliable)
[ 48.343422] [c00000000fb13ac0] [c00000000019edb8] tg_set_cfs_bandwidth+0x2c8/0x470
[ 48.343433] [c00000000fb13bc0] [c000000000263874] cgroup_file_write+0x164/0x210
[ 48.343444] [c00000000fb13c20] [c00000000058cfac] kernfs_fop_write_iter+0x1cc/0x280
[ 48.343455] [c00000000fb13c70] [c00000000047024c] new_sync_write+0x14c/0x1d0
[ 48.343467] [c00000000fb13d10] [c000000000473844] vfs_write+0x224/0x330
[ 48.343476] [c00000000fb13d60] [c000000000473b2c] ksys_write+0x7c/0x140
[ 48.343485] [c00000000fb13db0] [c000000000030fb0] system_call_exception+0x150/0x2d0
[ 48.343495] [c00000000fb13e10] [c00000000000d45c] system_call_common+0xec/0x278
[ 48.343504] --- interrupt: c00 at 0x7fffaa67bd74
[ 48.343511] NIP: 00007fffaa67bd74 LR: 00007fffaa5f34c4 CTR: 0000000000000000
[ 48.343519] REGS: c00000000fb13e80 TRAP: 0c00 Not tainted (5.13.0-rc7-dirty)
[ 48.343527] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28002282 XER: 00000000
[ 48.343548] IRQMASK: 0
[ 48.343548] GPR00: 0000000000000004 00007fffcb534d60 00007fffaa777100 0000000000000010
[ 48.343548] GPR04: 00000000415623d0 0000000000000005 0000000000000010 00007fffcb534df8
[ 48.343548] GPR08: 0000000010028618 0000000000000000 0000000000000000 0000000000000000
[ 48.343548] GPR12: 0000000000000000 00007fffaa81a310 0000000000000000 0000000000000000
[ 48.343548] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 48.343548] GPR20: 0000000000000000 0000000000000000 0000000000000002 0000000000000000
[ 48.343548] GPR24: 0000000000000000 000000000000002b 0000000000000005 00000000415623d0
[ 48.343548] GPR28: 0000000000000005 00007fffcb534eb0 00000000415623d0 0000000000000005
[ 48.343634] NIP [00007fffaa67bd74] 0x7fffaa67bd74
[ 48.343640] LR [00007fffaa5f34c4] 0x7fffaa5f34c4
[ 48.343646] --- interrupt: c00
[ 48.343651] Instruction dump:
[ 48.343656] 4bfffc74 3d22fff6 8929f2a9 2f890000 409efed4 39200001 3d42fff6 3c62fe60
[ 48.343672] 3863be08 992af2a9 4bf94f45 60000000 <0fe00000> 4bfffeb0 7f6407b4 7f43d378
[ 48.343687] ---[ end trace 61db91af8340603f ]---
cfs_bandwidth01.c:49: TINFO: Set 'level2/cpu.max' = '5000 10000'
cfs_bandwidth01.c:125: TPASS: Workers exited
cfs_bandwidth01.c:113: TPASS: Scheduled bandwidth constrained workers
cfs_bandwidth01.c:49: TINFO: Set 'level2/cpu.max' = '5000 10000'
cfs_bandwidth01.c:125: TPASS: Workers exited
cfs_bandwidth01.c:113: TPASS: Scheduled bandwidth constrained workers
cfs_bandwidth01.c:49: TINFO: Set 'level2/cpu.max' = '5000 10000'
cfs_bandwidth01.c:125: TPASS: Workers exited
cfs_bandwidth01.c:113: TPASS: Scheduled bandwidth constrained workers
cfs_bandwidth01.c:49: TINFO: Set 'level2/cpu.max' = '5000 10000'
cfs_bandwidth01.c:125: TPASS: Workers exited
tst_test.c:1349: TFAIL: Kernel is now tainted.
HINT: You _MAY_ be missing kernel fixes, see:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=39f23ce07b93
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b34cb07dde7c
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fe61468b2cbc
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ab297bab984
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6d4d22468dae
Summary:
passed 10
failed 1
broken 0
skipped 0
warnings 0
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
@ 2021-06-21 10:57 ` Sachin Sant
0 siblings, 0 replies; 26+ messages in thread
From: Sachin Sant @ 2021-06-21 10:57 UTC (permalink / raw)
To: Odin Ugedal; +Cc: Peter Zijlstra, linuxppc-dev, Vincent Guittot, open list
[-- Attachment #1: Type: text/plain, Size: 1475 bytes --]
> On 21-Jun-2021, at 3:24 PM, Odin Ugedal <odin@uged.al> wrote:
>
> man. 21. jun. 2021 kl. 11:50 skrev Vincent Guittot <vincent.guittot@linaro.org>:
>> This means that a child's load was not null and it was inserted
>> whereas parent's load was null. This should not happen unless the
>> propagation failed somewhere
>
> My initial thought is that the patch below will fix it, if that is the
> issue (that a leaf is inserted, but the propagation is not "completed"
> in unthrottle). Might that be the case? Still working on reproducing
> the issue tho.
>
Unfortunately this does not help. I can still recreate the failure.
Have attached the o/p from test run.
Thanks
-Sachin
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index bfaa6e1f6067..015c5a5c1a4d 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -4930,12 +4930,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)
> if (cfs_rq_throttled(cfs_rq))
> goto unthrottle_throttle;
>
> - /*
> - * One parent has been throttled and cfs_rq removed from the
> - * list. Add it back to not break the leaf list.
> - */
> - if (throttled_hierarchy(cfs_rq))
> - list_add_leaf_cfs_rq(cfs_rq);
> + list_add_leaf_cfs_rq(cfs_rq);
> }
>
> /* At this point se is NULL and we are at root level*/
[-- Attachment #2: run.txt --]
[-- Type: text/plain, Size: 5966 bytes --]
# cd /opt/ltp/testcases/bin
# ./cfs_bandwidth01 -i 5
tst_test.c:1313: TINFO: Timeout per run is 0h 05m 00s
tst_buffers.c:55: TINFO: Test is using guarded buffers
cfs_bandwidth01.c:49: TINFO: Set 'worker1/cpu.max' = '3000 10000'
cfs_bandwidth01.c:49: TINFO: Set 'worker2/cpu.max' = '2000 10000'
cfs_bandwidth01.c:49: TINFO: Set 'worker3/cpu.max' = '3000 10000'
cfs_bandwidth01.c:113: TPASS: Scheduled bandwidth constrained workers
cfs_bandwidth01.c:49: TINFO: Set 'level2/cpu.max' = '5000 10000'
cfs_bandwidth01.c:125: TPASS: Workers exited
cfs_bandwidth01.c:113: TPASS: Scheduled bandwidth constrained workers
[ 48.343143] ------------[ cut here ]------------
[ 48.343164] rq->tmp_alone_branch != &rq->leaf_cfs_rq_list
[ 48.343172] WARNING: CPU: 24 PID: 4405 at kernel/sched/fair.c:401 unthrottle_cfs_rq+0x49c/0x560
[ 48.343196] Modules linked in: nf_tables nfnetlink tun bridge stp llc rfkill sunrpc pseries_rng xts vmx_crypto uio_pdrv_genirq uio sch_fq_codel ip_tables xfs libcrc32c sr_mod sd_mod cdrom t10_pi sg ibmvscsi ibmveth scsi_transport_srp dm_mirror dm_region_hash dm_log dm_mod fuse
[ 48.343251] CPU: 24 PID: 4405 Comm: cfs_bandwidth01 Not tainted 5.13.0-rc7-dirty #4
[ 48.343261] NIP: c0000000001b88fc LR: c0000000001b88f8 CTR: c000000000723d10
[ 48.343269] REGS: c00000000fb13780 TRAP: 0700 Not tainted (5.13.0-rc7-dirty)
[ 48.343278] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 48044224 XER: 00000005
[ 48.343295] CFAR: c00000000014d8a0 IRQMASK: 1
[ 48.343295] GPR00: c0000000001b88f8 c00000000fb13a20 c0000000029ab400 000000000000002d
[ 48.343295] GPR04: 00000000fffeffff c00000000fb136e0 0000000000000027 c00000154f817e08
[ 48.343295] GPR08: 0000000000000023 0000000000000001 0000000000000027 c00000167f1d7fe8
[ 48.343295] GPR12: 0000000000004000 c00000154ffdc680 0000000000000000 0000000000000000
[ 48.343295] GPR16: c000000000fa6660 0000000000000001 0000000000000000 c0000000024e1cd8
[ 48.343295] GPR20: 0000000000000000 c00000000290a69a 0000000000000000 c0000000024e1cc0
[ 48.343295] GPR24: 0000000000000000 c0000000029f2140 c00000154f762380 0000000000000001
[ 48.343295] GPR28: 0000000000000001 0000000000000000 c00000154f762400 0000000000000000
[ 48.343388] NIP [c0000000001b88fc] unthrottle_cfs_rq+0x49c/0x560
[ 48.343397] LR [c0000000001b88f8] unthrottle_cfs_rq+0x498/0x560
[ 48.343406] Call Trace:
[ 48.343410] [c00000000fb13a20] [c0000000001b88f8] unthrottle_cfs_rq+0x498/0x560 (unreliable)
[ 48.343422] [c00000000fb13ac0] [c00000000019edb8] tg_set_cfs_bandwidth+0x2c8/0x470
[ 48.343433] [c00000000fb13bc0] [c000000000263874] cgroup_file_write+0x164/0x210
[ 48.343444] [c00000000fb13c20] [c00000000058cfac] kernfs_fop_write_iter+0x1cc/0x280
[ 48.343455] [c00000000fb13c70] [c00000000047024c] new_sync_write+0x14c/0x1d0
[ 48.343467] [c00000000fb13d10] [c000000000473844] vfs_write+0x224/0x330
[ 48.343476] [c00000000fb13d60] [c000000000473b2c] ksys_write+0x7c/0x140
[ 48.343485] [c00000000fb13db0] [c000000000030fb0] system_call_exception+0x150/0x2d0
[ 48.343495] [c00000000fb13e10] [c00000000000d45c] system_call_common+0xec/0x278
[ 48.343504] --- interrupt: c00 at 0x7fffaa67bd74
[ 48.343511] NIP: 00007fffaa67bd74 LR: 00007fffaa5f34c4 CTR: 0000000000000000
[ 48.343519] REGS: c00000000fb13e80 TRAP: 0c00 Not tainted (5.13.0-rc7-dirty)
[ 48.343527] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28002282 XER: 00000000
[ 48.343548] IRQMASK: 0
[ 48.343548] GPR00: 0000000000000004 00007fffcb534d60 00007fffaa777100 0000000000000010
[ 48.343548] GPR04: 00000000415623d0 0000000000000005 0000000000000010 00007fffcb534df8
[ 48.343548] GPR08: 0000000010028618 0000000000000000 0000000000000000 0000000000000000
[ 48.343548] GPR12: 0000000000000000 00007fffaa81a310 0000000000000000 0000000000000000
[ 48.343548] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 48.343548] GPR20: 0000000000000000 0000000000000000 0000000000000002 0000000000000000
[ 48.343548] GPR24: 0000000000000000 000000000000002b 0000000000000005 00000000415623d0
[ 48.343548] GPR28: 0000000000000005 00007fffcb534eb0 00000000415623d0 0000000000000005
[ 48.343634] NIP [00007fffaa67bd74] 0x7fffaa67bd74
[ 48.343640] LR [00007fffaa5f34c4] 0x7fffaa5f34c4
[ 48.343646] --- interrupt: c00
[ 48.343651] Instruction dump:
[ 48.343656] 4bfffc74 3d22fff6 8929f2a9 2f890000 409efed4 39200001 3d42fff6 3c62fe60
[ 48.343672] 3863be08 992af2a9 4bf94f45 60000000 <0fe00000> 4bfffeb0 7f6407b4 7f43d378
[ 48.343687] ---[ end trace 61db91af8340603f ]---
cfs_bandwidth01.c:49: TINFO: Set 'level2/cpu.max' = '5000 10000'
cfs_bandwidth01.c:125: TPASS: Workers exited
cfs_bandwidth01.c:113: TPASS: Scheduled bandwidth constrained workers
cfs_bandwidth01.c:49: TINFO: Set 'level2/cpu.max' = '5000 10000'
cfs_bandwidth01.c:125: TPASS: Workers exited
cfs_bandwidth01.c:113: TPASS: Scheduled bandwidth constrained workers
cfs_bandwidth01.c:49: TINFO: Set 'level2/cpu.max' = '5000 10000'
cfs_bandwidth01.c:125: TPASS: Workers exited
cfs_bandwidth01.c:113: TPASS: Scheduled bandwidth constrained workers
cfs_bandwidth01.c:49: TINFO: Set 'level2/cpu.max' = '5000 10000'
cfs_bandwidth01.c:125: TPASS: Workers exited
tst_test.c:1349: TFAIL: Kernel is now tainted.
HINT: You _MAY_ be missing kernel fixes, see:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=39f23ce07b93
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b34cb07dde7c
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fe61468b2cbc
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ab297bab984
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6d4d22468dae
Summary:
passed 10
failed 1
broken 0
skipped 0
warnings 0
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
2021-06-21 10:57 ` Sachin Sant
@ 2021-06-21 11:04 ` Odin Ugedal
-1 siblings, 0 replies; 26+ messages in thread
From: Odin Ugedal @ 2021-06-21 11:04 UTC (permalink / raw)
To: Sachin Sant
Cc: Odin Ugedal, Vincent Guittot, open list, linuxppc-dev, Peter Zijlstra
man. 21. jun. 2021 kl. 12:57 skrev Sachin Sant <sachinp@linux.vnet.ibm.com>:
>
> Unfortunately this does not help. I can still recreate the failure.
>
> Have attached the o/p from test run.
>
> Thanks
> -Sachin
Yes, thanks!
I am able to reproduce it locally now, so will keep looking to see if
I find the cause. Thanks!
Odin
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
@ 2021-06-21 11:04 ` Odin Ugedal
0 siblings, 0 replies; 26+ messages in thread
From: Odin Ugedal @ 2021-06-21 11:04 UTC (permalink / raw)
To: Sachin Sant
Cc: Peter Zijlstra, Vincent Guittot, linuxppc-dev, Odin Ugedal, open list
man. 21. jun. 2021 kl. 12:57 skrev Sachin Sant <sachinp@linux.vnet.ibm.com>:
>
> Unfortunately this does not help. I can still recreate the failure.
>
> Have attached the o/p from test run.
>
> Thanks
> -Sachin
Yes, thanks!
I am able to reproduce it locally now, so will keep looking to see if
I find the cause. Thanks!
Odin
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
2021-06-21 11:04 ` Odin Ugedal
@ 2021-06-21 12:42 ` Odin Ugedal
-1 siblings, 0 replies; 26+ messages in thread
From: Odin Ugedal @ 2021-06-21 12:42 UTC (permalink / raw)
To: Odin Ugedal
Cc: Sachin Sant, Vincent Guittot, open list, linuxppc-dev, Peter Zijlstra
Hi,
Did some more research, and it looks like this is what happens:
$ tree /sys/fs/cgroup/ltp/ -d --charset=ascii
/sys/fs/cgroup/ltp/
|-- drain
`-- test-6851
`-- level2
|-- level3a
| |-- worker1
| `-- worker2
`-- level3b
`-- worker3
Timeline (ish):
- worker3 gets throttled
- level3b is decayed, since it has no more load
- level2 get throttled
- worker3 get unthrottled
- level2 get unthrottled
- worker3 is added to list
- level3b is not added to list, since nr_running==0 and is decayed
The attached diff (based on
https://lore.kernel.org/lkml/20210518125202.78658-3-odin@uged.al/)
fixes the issue for me. Not the most elegant solution, but the
simplest one as of now, and to show what is wrong.
Any thoughts Vincent?
Thanks
Odin
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index bfaa6e1f6067..aa32e9c29efd 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -376,7 +376,8 @@ static inline bool list_add_leaf_cfs_rq(struct
cfs_rq *cfs_rq)
return false;
}
-static inline void list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq)
+/* Returns 1 if cfs_rq was present in the list and removed */
+static inline bool list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq)
{
if (cfs_rq->on_list) {
struct rq *rq = rq_of(cfs_rq);
@@ -393,7 +394,9 @@ static inline void list_del_leaf_cfs_rq(struct
cfs_rq *cfs_rq)
list_del_rcu(&cfs_rq->leaf_cfs_rq_list);
cfs_rq->on_list = 0;
+ return 1;
}
+ return 0;
}
static inline void assert_list_leaf_cfs_rq(struct rq *rq)
@@ -3298,24 +3301,6 @@ static inline void cfs_rq_util_change(struct
cfs_rq *cfs_rq, int flags)
#ifdef CONFIG_SMP
#ifdef CONFIG_FAIR_GROUP_SCHED
-
-static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
-{
- if (cfs_rq->load.weight)
- return false;
-
- if (cfs_rq->avg.load_sum)
- return false;
-
- if (cfs_rq->avg.util_sum)
- return false;
-
- if (cfs_rq->avg.runnable_sum)
- return false;
-
- return true;
-}
-
/**
* update_tg_load_avg - update the tg's load avg
* @cfs_rq: the cfs_rq whose avg changed
@@ -4109,11 +4094,6 @@ static inline void update_misfit_status(struct
task_struct *p, struct rq *rq)
#else /* CONFIG_SMP */
-static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
-{
- return true;
-}
-
#define UPDATE_TG 0x0
#define SKIP_AGE_LOAD 0x0
#define DO_ATTACH 0x0
@@ -4771,10 +4751,11 @@ static int tg_unthrottle_up(struct task_group
*tg, void *data)
if (!cfs_rq->throttle_count) {
cfs_rq->throttled_clock_task_time += rq_clock_task(rq) -
cfs_rq->throttled_clock_task;
-
- /* Add cfs_rq with load or one or more already running
entities to the list */
- if (!cfs_rq_is_decayed(cfs_rq) || cfs_rq->nr_running)
+ if (cfs_rq->insert_on_unthrottle) {
list_add_leaf_cfs_rq(cfs_rq);
+ if (tg->parent)
+
tg->parent->cfs_rq[cpu_of(rq)]->insert_on_unthrottle = true;
+ }
}
return 0;
@@ -4788,7 +4769,7 @@ static int tg_throttle_down(struct task_group
*tg, void *data)
/* group is entering throttled state, stop time */
if (!cfs_rq->throttle_count) {
cfs_rq->throttled_clock_task = rq_clock_task(rq);
- list_del_leaf_cfs_rq(cfs_rq);
+ cfs_rq->insert_on_unthrottle = list_del_leaf_cfs_rq(cfs_rq);
}
cfs_rq->throttle_count++;
@@ -8019,6 +8000,23 @@ static bool __update_blocked_others(struct rq
*rq, bool *done)
#ifdef CONFIG_FAIR_GROUP_SCHED
+static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
+{
+ if (cfs_rq->load.weight)
+ return false;
+
+ if (cfs_rq->avg.load_sum)
+ return false;
+
+ if (cfs_rq->avg.util_sum)
+ return false;
+
+ if (cfs_rq->avg.runnable_sum)
+ return false;
+
+ return true;
+}
+
static bool __update_blocked_fair(struct rq *rq, bool *done)
{
struct cfs_rq *cfs_rq, *pos;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a189bec13729..12a707d99ee6 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -602,6 +602,7 @@ struct cfs_rq {
u64 throttled_clock_task_time;
int throttled;
int throttle_count;
+ int insert_on_unthrottle;
struct list_head throttled_list;
#endif /* CONFIG_CFS_BANDWIDTH */
#endif /* CONFIG_FAIR_GROUP_SCHED */
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
@ 2021-06-21 12:42 ` Odin Ugedal
0 siblings, 0 replies; 26+ messages in thread
From: Odin Ugedal @ 2021-06-21 12:42 UTC (permalink / raw)
To: Odin Ugedal
Cc: Sachin Sant, linuxppc-dev, Peter Zijlstra, Vincent Guittot, open list
Hi,
Did some more research, and it looks like this is what happens:
$ tree /sys/fs/cgroup/ltp/ -d --charset=ascii
/sys/fs/cgroup/ltp/
|-- drain
`-- test-6851
`-- level2
|-- level3a
| |-- worker1
| `-- worker2
`-- level3b
`-- worker3
Timeline (ish):
- worker3 gets throttled
- level3b is decayed, since it has no more load
- level2 get throttled
- worker3 get unthrottled
- level2 get unthrottled
- worker3 is added to list
- level3b is not added to list, since nr_running==0 and is decayed
The attached diff (based on
https://lore.kernel.org/lkml/20210518125202.78658-3-odin@uged.al/)
fixes the issue for me. Not the most elegant solution, but the
simplest one as of now, and to show what is wrong.
Any thoughts Vincent?
Thanks
Odin
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index bfaa6e1f6067..aa32e9c29efd 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -376,7 +376,8 @@ static inline bool list_add_leaf_cfs_rq(struct
cfs_rq *cfs_rq)
return false;
}
-static inline void list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq)
+/* Returns 1 if cfs_rq was present in the list and removed */
+static inline bool list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq)
{
if (cfs_rq->on_list) {
struct rq *rq = rq_of(cfs_rq);
@@ -393,7 +394,9 @@ static inline void list_del_leaf_cfs_rq(struct
cfs_rq *cfs_rq)
list_del_rcu(&cfs_rq->leaf_cfs_rq_list);
cfs_rq->on_list = 0;
+ return 1;
}
+ return 0;
}
static inline void assert_list_leaf_cfs_rq(struct rq *rq)
@@ -3298,24 +3301,6 @@ static inline void cfs_rq_util_change(struct
cfs_rq *cfs_rq, int flags)
#ifdef CONFIG_SMP
#ifdef CONFIG_FAIR_GROUP_SCHED
-
-static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
-{
- if (cfs_rq->load.weight)
- return false;
-
- if (cfs_rq->avg.load_sum)
- return false;
-
- if (cfs_rq->avg.util_sum)
- return false;
-
- if (cfs_rq->avg.runnable_sum)
- return false;
-
- return true;
-}
-
/**
* update_tg_load_avg - update the tg's load avg
* @cfs_rq: the cfs_rq whose avg changed
@@ -4109,11 +4094,6 @@ static inline void update_misfit_status(struct
task_struct *p, struct rq *rq)
#else /* CONFIG_SMP */
-static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
-{
- return true;
-}
-
#define UPDATE_TG 0x0
#define SKIP_AGE_LOAD 0x0
#define DO_ATTACH 0x0
@@ -4771,10 +4751,11 @@ static int tg_unthrottle_up(struct task_group
*tg, void *data)
if (!cfs_rq->throttle_count) {
cfs_rq->throttled_clock_task_time += rq_clock_task(rq) -
cfs_rq->throttled_clock_task;
-
- /* Add cfs_rq with load or one or more already running
entities to the list */
- if (!cfs_rq_is_decayed(cfs_rq) || cfs_rq->nr_running)
+ if (cfs_rq->insert_on_unthrottle) {
list_add_leaf_cfs_rq(cfs_rq);
+ if (tg->parent)
+
tg->parent->cfs_rq[cpu_of(rq)]->insert_on_unthrottle = true;
+ }
}
return 0;
@@ -4788,7 +4769,7 @@ static int tg_throttle_down(struct task_group
*tg, void *data)
/* group is entering throttled state, stop time */
if (!cfs_rq->throttle_count) {
cfs_rq->throttled_clock_task = rq_clock_task(rq);
- list_del_leaf_cfs_rq(cfs_rq);
+ cfs_rq->insert_on_unthrottle = list_del_leaf_cfs_rq(cfs_rq);
}
cfs_rq->throttle_count++;
@@ -8019,6 +8000,23 @@ static bool __update_blocked_others(struct rq
*rq, bool *done)
#ifdef CONFIG_FAIR_GROUP_SCHED
+static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
+{
+ if (cfs_rq->load.weight)
+ return false;
+
+ if (cfs_rq->avg.load_sum)
+ return false;
+
+ if (cfs_rq->avg.util_sum)
+ return false;
+
+ if (cfs_rq->avg.runnable_sum)
+ return false;
+
+ return true;
+}
+
static bool __update_blocked_fair(struct rq *rq, bool *done)
{
struct cfs_rq *cfs_rq, *pos;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a189bec13729..12a707d99ee6 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -602,6 +602,7 @@ struct cfs_rq {
u64 throttled_clock_task_time;
int throttled;
int throttle_count;
+ int insert_on_unthrottle;
struct list_head throttled_list;
#endif /* CONFIG_CFS_BANDWIDTH */
#endif /* CONFIG_FAIR_GROUP_SCHED */
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
2021-06-21 12:42 ` Odin Ugedal
@ 2021-06-21 16:22 ` Vincent Guittot
-1 siblings, 0 replies; 26+ messages in thread
From: Vincent Guittot @ 2021-06-21 16:22 UTC (permalink / raw)
To: Odin Ugedal; +Cc: Sachin Sant, open list, linuxppc-dev, Peter Zijlstra
Le lundi 21 juin 2021 à 14:42:23 (+0200), Odin Ugedal a écrit :
> Hi,
>
> Did some more research, and it looks like this is what happens:
>
> $ tree /sys/fs/cgroup/ltp/ -d --charset=ascii
> /sys/fs/cgroup/ltp/
> |-- drain
> `-- test-6851
> `-- level2
> |-- level3a
> | |-- worker1
> | `-- worker2
> `-- level3b
> `-- worker3
>
> Timeline (ish):
> - worker3 gets throttled
> - level3b is decayed, since it has no more load
> - level2 get throttled
> - worker3 get unthrottled
> - level2 get unthrottled
> - worker3 is added to list
> - level3b is not added to list, since nr_running==0 and is decayed
>
>
> The attached diff (based on
> https://lore.kernel.org/lkml/20210518125202.78658-3-odin@uged.al/)
> fixes the issue for me. Not the most elegant solution, but the
> simplest one as of now, and to show what is wrong.
>
> Any thoughts Vincent?
I would prefer that we use the reason of adding the cfs in the list instead.
Something like the below should also fixed the problem. It is based on a
proposal I made to Rik sometimes ago when he tried to flatten the rq:
https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/
This will ensure that a cfs is added in the list whenever one of its child
is still in the list.
---
kernel/sched/fair.c | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index ea7de54cb022..e751061a9449 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -3272,6 +3272,31 @@ static inline void cfs_rq_util_change(struct cfs_rq *cfs_rq, int flags)
#ifdef CONFIG_SMP
#ifdef CONFIG_FAIR_GROUP_SCHED
+/*
+ * Because list_add_leaf_cfs_rq always places a child cfs_rq on the list
+ * immediately before a parent cfs_rq, and cfs_rqs are removed from the list
+ * bottom-up, we only have to test whether the cfs_rq before us on the list
+ * is our child.
+ * If cfs_rq is not on the list, test wether a child needs its to be added to
+ * connect a branch to the tree * (see list_add_leaf_cfs_rq() for details).
+ */
+static inline bool child_cfs_rq_on_list(struct cfs_rq *cfs_rq)
+{
+ struct cfs_rq *prev_cfs_rq;
+ struct list_head *prev;
+
+ if (cfs_rq->on_list) {
+ prev = cfs_rq->leaf_cfs_rq_list.prev;
+ } else {
+ struct rq *rq = rq_of(cfs_rq);
+
+ prev = rq->tmp_alone_branch;
+ }
+
+ prev_cfs_rq = container_of(prev, struct cfs_rq, leaf_cfs_rq_list);
+
+ return (prev_cfs_rq->tg->parent == cfs_rq->tg);
+}
static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
{
@@ -3287,6 +3312,9 @@ static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
if (cfs_rq->avg.runnable_sum)
return false;
+ if (child_cfs_rq_on_list(cfs_rq))
+ return false;
+
return true;
}
--
2.17.1
>
> Thanks
> Odin
>
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index bfaa6e1f6067..aa32e9c29efd 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -376,7 +376,8 @@ static inline bool list_add_leaf_cfs_rq(struct
> cfs_rq *cfs_rq)
> return false;
> }
>
> -static inline void list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq)
> +/* Returns 1 if cfs_rq was present in the list and removed */
> +static inline bool list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq)
> {
> if (cfs_rq->on_list) {
> struct rq *rq = rq_of(cfs_rq);
> @@ -393,7 +394,9 @@ static inline void list_del_leaf_cfs_rq(struct
> cfs_rq *cfs_rq)
>
> list_del_rcu(&cfs_rq->leaf_cfs_rq_list);
> cfs_rq->on_list = 0;
> + return 1;
> }
> + return 0;
> }
>
> static inline void assert_list_leaf_cfs_rq(struct rq *rq)
> @@ -3298,24 +3301,6 @@ static inline void cfs_rq_util_change(struct
> cfs_rq *cfs_rq, int flags)
>
> #ifdef CONFIG_SMP
> #ifdef CONFIG_FAIR_GROUP_SCHED
> -
> -static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
> -{
> - if (cfs_rq->load.weight)
> - return false;
> -
> - if (cfs_rq->avg.load_sum)
> - return false;
> -
> - if (cfs_rq->avg.util_sum)
> - return false;
> -
> - if (cfs_rq->avg.runnable_sum)
> - return false;
> -
> - return true;
> -}
> -
> /**
> * update_tg_load_avg - update the tg's load avg
> * @cfs_rq: the cfs_rq whose avg changed
> @@ -4109,11 +4094,6 @@ static inline void update_misfit_status(struct
> task_struct *p, struct rq *rq)
>
> #else /* CONFIG_SMP */
>
> -static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
> -{
> - return true;
> -}
> -
> #define UPDATE_TG 0x0
> #define SKIP_AGE_LOAD 0x0
> #define DO_ATTACH 0x0
> @@ -4771,10 +4751,11 @@ static int tg_unthrottle_up(struct task_group
> *tg, void *data)
> if (!cfs_rq->throttle_count) {
> cfs_rq->throttled_clock_task_time += rq_clock_task(rq) -
> cfs_rq->throttled_clock_task;
> -
> - /* Add cfs_rq with load or one or more already running
> entities to the list */
> - if (!cfs_rq_is_decayed(cfs_rq) || cfs_rq->nr_running)
> + if (cfs_rq->insert_on_unthrottle) {
> list_add_leaf_cfs_rq(cfs_rq);
> + if (tg->parent)
> +
> tg->parent->cfs_rq[cpu_of(rq)]->insert_on_unthrottle = true;
> + }
> }
>
> return 0;
> @@ -4788,7 +4769,7 @@ static int tg_throttle_down(struct task_group
> *tg, void *data)
> /* group is entering throttled state, stop time */
> if (!cfs_rq->throttle_count) {
> cfs_rq->throttled_clock_task = rq_clock_task(rq);
> - list_del_leaf_cfs_rq(cfs_rq);
> + cfs_rq->insert_on_unthrottle = list_del_leaf_cfs_rq(cfs_rq);
> }
> cfs_rq->throttle_count++;
>
> @@ -8019,6 +8000,23 @@ static bool __update_blocked_others(struct rq
> *rq, bool *done)
>
> #ifdef CONFIG_FAIR_GROUP_SCHED
>
> +static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
> +{
> + if (cfs_rq->load.weight)
> + return false;
> +
> + if (cfs_rq->avg.load_sum)
> + return false;
> +
> + if (cfs_rq->avg.util_sum)
> + return false;
> +
> + if (cfs_rq->avg.runnable_sum)
> + return false;
> +
> + return true;
> +}
> +
> static bool __update_blocked_fair(struct rq *rq, bool *done)
> {
> struct cfs_rq *cfs_rq, *pos;
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index a189bec13729..12a707d99ee6 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -602,6 +602,7 @@ struct cfs_rq {
> u64 throttled_clock_task_time;
> int throttled;
> int throttle_count;
> + int insert_on_unthrottle;
> struct list_head throttled_list;
> #endif /* CONFIG_CFS_BANDWIDTH */
> #endif /* CONFIG_FAIR_GROUP_SCHED */
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
@ 2021-06-21 16:22 ` Vincent Guittot
0 siblings, 0 replies; 26+ messages in thread
From: Vincent Guittot @ 2021-06-21 16:22 UTC (permalink / raw)
To: Odin Ugedal; +Cc: Sachin Sant, Peter Zijlstra, linuxppc-dev, open list
Le lundi 21 juin 2021 à 14:42:23 (+0200), Odin Ugedal a écrit :
> Hi,
>
> Did some more research, and it looks like this is what happens:
>
> $ tree /sys/fs/cgroup/ltp/ -d --charset=ascii
> /sys/fs/cgroup/ltp/
> |-- drain
> `-- test-6851
> `-- level2
> |-- level3a
> | |-- worker1
> | `-- worker2
> `-- level3b
> `-- worker3
>
> Timeline (ish):
> - worker3 gets throttled
> - level3b is decayed, since it has no more load
> - level2 get throttled
> - worker3 get unthrottled
> - level2 get unthrottled
> - worker3 is added to list
> - level3b is not added to list, since nr_running==0 and is decayed
>
>
> The attached diff (based on
> https://lore.kernel.org/lkml/20210518125202.78658-3-odin@uged.al/)
> fixes the issue for me. Not the most elegant solution, but the
> simplest one as of now, and to show what is wrong.
>
> Any thoughts Vincent?
I would prefer that we use the reason of adding the cfs in the list instead.
Something like the below should also fixed the problem. It is based on a
proposal I made to Rik sometimes ago when he tried to flatten the rq:
https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/
This will ensure that a cfs is added in the list whenever one of its child
is still in the list.
---
kernel/sched/fair.c | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index ea7de54cb022..e751061a9449 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -3272,6 +3272,31 @@ static inline void cfs_rq_util_change(struct cfs_rq *cfs_rq, int flags)
#ifdef CONFIG_SMP
#ifdef CONFIG_FAIR_GROUP_SCHED
+/*
+ * Because list_add_leaf_cfs_rq always places a child cfs_rq on the list
+ * immediately before a parent cfs_rq, and cfs_rqs are removed from the list
+ * bottom-up, we only have to test whether the cfs_rq before us on the list
+ * is our child.
+ * If cfs_rq is not on the list, test wether a child needs its to be added to
+ * connect a branch to the tree * (see list_add_leaf_cfs_rq() for details).
+ */
+static inline bool child_cfs_rq_on_list(struct cfs_rq *cfs_rq)
+{
+ struct cfs_rq *prev_cfs_rq;
+ struct list_head *prev;
+
+ if (cfs_rq->on_list) {
+ prev = cfs_rq->leaf_cfs_rq_list.prev;
+ } else {
+ struct rq *rq = rq_of(cfs_rq);
+
+ prev = rq->tmp_alone_branch;
+ }
+
+ prev_cfs_rq = container_of(prev, struct cfs_rq, leaf_cfs_rq_list);
+
+ return (prev_cfs_rq->tg->parent == cfs_rq->tg);
+}
static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
{
@@ -3287,6 +3312,9 @@ static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
if (cfs_rq->avg.runnable_sum)
return false;
+ if (child_cfs_rq_on_list(cfs_rq))
+ return false;
+
return true;
}
--
2.17.1
>
> Thanks
> Odin
>
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index bfaa6e1f6067..aa32e9c29efd 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -376,7 +376,8 @@ static inline bool list_add_leaf_cfs_rq(struct
> cfs_rq *cfs_rq)
> return false;
> }
>
> -static inline void list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq)
> +/* Returns 1 if cfs_rq was present in the list and removed */
> +static inline bool list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq)
> {
> if (cfs_rq->on_list) {
> struct rq *rq = rq_of(cfs_rq);
> @@ -393,7 +394,9 @@ static inline void list_del_leaf_cfs_rq(struct
> cfs_rq *cfs_rq)
>
> list_del_rcu(&cfs_rq->leaf_cfs_rq_list);
> cfs_rq->on_list = 0;
> + return 1;
> }
> + return 0;
> }
>
> static inline void assert_list_leaf_cfs_rq(struct rq *rq)
> @@ -3298,24 +3301,6 @@ static inline void cfs_rq_util_change(struct
> cfs_rq *cfs_rq, int flags)
>
> #ifdef CONFIG_SMP
> #ifdef CONFIG_FAIR_GROUP_SCHED
> -
> -static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
> -{
> - if (cfs_rq->load.weight)
> - return false;
> -
> - if (cfs_rq->avg.load_sum)
> - return false;
> -
> - if (cfs_rq->avg.util_sum)
> - return false;
> -
> - if (cfs_rq->avg.runnable_sum)
> - return false;
> -
> - return true;
> -}
> -
> /**
> * update_tg_load_avg - update the tg's load avg
> * @cfs_rq: the cfs_rq whose avg changed
> @@ -4109,11 +4094,6 @@ static inline void update_misfit_status(struct
> task_struct *p, struct rq *rq)
>
> #else /* CONFIG_SMP */
>
> -static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
> -{
> - return true;
> -}
> -
> #define UPDATE_TG 0x0
> #define SKIP_AGE_LOAD 0x0
> #define DO_ATTACH 0x0
> @@ -4771,10 +4751,11 @@ static int tg_unthrottle_up(struct task_group
> *tg, void *data)
> if (!cfs_rq->throttle_count) {
> cfs_rq->throttled_clock_task_time += rq_clock_task(rq) -
> cfs_rq->throttled_clock_task;
> -
> - /* Add cfs_rq with load or one or more already running
> entities to the list */
> - if (!cfs_rq_is_decayed(cfs_rq) || cfs_rq->nr_running)
> + if (cfs_rq->insert_on_unthrottle) {
> list_add_leaf_cfs_rq(cfs_rq);
> + if (tg->parent)
> +
> tg->parent->cfs_rq[cpu_of(rq)]->insert_on_unthrottle = true;
> + }
> }
>
> return 0;
> @@ -4788,7 +4769,7 @@ static int tg_throttle_down(struct task_group
> *tg, void *data)
> /* group is entering throttled state, stop time */
> if (!cfs_rq->throttle_count) {
> cfs_rq->throttled_clock_task = rq_clock_task(rq);
> - list_del_leaf_cfs_rq(cfs_rq);
> + cfs_rq->insert_on_unthrottle = list_del_leaf_cfs_rq(cfs_rq);
> }
> cfs_rq->throttle_count++;
>
> @@ -8019,6 +8000,23 @@ static bool __update_blocked_others(struct rq
> *rq, bool *done)
>
> #ifdef CONFIG_FAIR_GROUP_SCHED
>
> +static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
> +{
> + if (cfs_rq->load.weight)
> + return false;
> +
> + if (cfs_rq->avg.load_sum)
> + return false;
> +
> + if (cfs_rq->avg.util_sum)
> + return false;
> +
> + if (cfs_rq->avg.runnable_sum)
> + return false;
> +
> + return true;
> +}
> +
> static bool __update_blocked_fair(struct rq *rq, bool *done)
> {
> struct cfs_rq *cfs_rq, *pos;
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index a189bec13729..12a707d99ee6 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -602,6 +602,7 @@ struct cfs_rq {
> u64 throttled_clock_task_time;
> int throttled;
> int throttle_count;
> + int insert_on_unthrottle;
> struct list_head throttled_list;
> #endif /* CONFIG_CFS_BANDWIDTH */
> #endif /* CONFIG_FAIR_GROUP_SCHED */
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
2021-06-21 16:22 ` Vincent Guittot
@ 2021-06-21 16:45 ` Odin Ugedal
-1 siblings, 0 replies; 26+ messages in thread
From: Odin Ugedal @ 2021-06-21 16:45 UTC (permalink / raw)
To: Vincent Guittot
Cc: Odin Ugedal, Sachin Sant, open list, linuxppc-dev, Peter Zijlstra
man. 21. jun. 2021 kl. 18:22 skrev Vincent Guittot <vincent.guittot@linaro.org>:
> I would prefer that we use the reason of adding the cfs in the list instead.
>
> Something like the below should also fixed the problem. It is based on a
> proposal I made to Rik sometimes ago when he tried to flatten the rq:
> https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/
>
> This will ensure that a cfs is added in the list whenever one of its child
> is still in the list.
Oh, yeah, that is a much more elegant solution! It fixes the issue as well!
Feel free to add this when/if you submit it as a patch:
Acked-by: Odin Ugedal <odin@uged.al>
Odin
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
@ 2021-06-21 16:45 ` Odin Ugedal
0 siblings, 0 replies; 26+ messages in thread
From: Odin Ugedal @ 2021-06-21 16:45 UTC (permalink / raw)
To: Vincent Guittot
Cc: Sachin Sant, linuxppc-dev, Peter Zijlstra, Odin Ugedal, open list
man. 21. jun. 2021 kl. 18:22 skrev Vincent Guittot <vincent.guittot@linaro.org>:
> I would prefer that we use the reason of adding the cfs in the list instead.
>
> Something like the below should also fixed the problem. It is based on a
> proposal I made to Rik sometimes ago when he tried to flatten the rq:
> https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/
>
> This will ensure that a cfs is added in the list whenever one of its child
> is still in the list.
Oh, yeah, that is a much more elegant solution! It fixes the issue as well!
Feel free to add this when/if you submit it as a patch:
Acked-by: Odin Ugedal <odin@uged.al>
Odin
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
2021-06-21 16:45 ` Odin Ugedal
@ 2021-06-21 17:07 ` Vincent Guittot
-1 siblings, 0 replies; 26+ messages in thread
From: Vincent Guittot @ 2021-06-21 17:07 UTC (permalink / raw)
To: Odin Ugedal; +Cc: Sachin Sant, open list, linuxppc-dev, Peter Zijlstra
On Mon, 21 Jun 2021 at 18:45, Odin Ugedal <odin@uged.al> wrote:
>
> man. 21. jun. 2021 kl. 18:22 skrev Vincent Guittot <vincent.guittot@linaro.org>:
> > I would prefer that we use the reason of adding the cfs in the list instead.
> >
> > Something like the below should also fixed the problem. It is based on a
> > proposal I made to Rik sometimes ago when he tried to flatten the rq:
> > https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/
> >
> > This will ensure that a cfs is added in the list whenever one of its child
> > is still in the list.
>
> Oh, yeah, that is a much more elegant solution! It fixes the issue as well!
>
> Feel free to add this when/if you submit it as a patch:
> Acked-by: Odin Ugedal <odin@uged.al>
Thanks
>
> Odin
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
@ 2021-06-21 17:07 ` Vincent Guittot
0 siblings, 0 replies; 26+ messages in thread
From: Vincent Guittot @ 2021-06-21 17:07 UTC (permalink / raw)
To: Odin Ugedal; +Cc: Sachin Sant, Peter Zijlstra, linuxppc-dev, open list
On Mon, 21 Jun 2021 at 18:45, Odin Ugedal <odin@uged.al> wrote:
>
> man. 21. jun. 2021 kl. 18:22 skrev Vincent Guittot <vincent.guittot@linaro.org>:
> > I would prefer that we use the reason of adding the cfs in the list instead.
> >
> > Something like the below should also fixed the problem. It is based on a
> > proposal I made to Rik sometimes ago when he tried to flatten the rq:
> > https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/
> >
> > This will ensure that a cfs is added in the list whenever one of its child
> > is still in the list.
>
> Oh, yeah, that is a much more elegant solution! It fixes the issue as well!
>
> Feel free to add this when/if you submit it as a patch:
> Acked-by: Odin Ugedal <odin@uged.al>
Thanks
>
> Odin
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
2021-06-21 16:22 ` Vincent Guittot
@ 2021-06-21 17:09 ` Vincent Guittot
-1 siblings, 0 replies; 26+ messages in thread
From: Vincent Guittot @ 2021-06-21 17:09 UTC (permalink / raw)
To: Sachin Sant; +Cc: open list, linuxppc-dev, Peter Zijlstra, Odin Ugedal
Hi Sacha
On Mon, 21 Jun 2021 at 18:22, Vincent Guittot
<vincent.guittot@linaro.org> wrote:
>
> Le lundi 21 juin 2021 à 14:42:23 (+0200), Odin Ugedal a écrit :
> > Hi,
> >
> > Did some more research, and it looks like this is what happens:
> >
> > $ tree /sys/fs/cgroup/ltp/ -d --charset=ascii
> > /sys/fs/cgroup/ltp/
> > |-- drain
> > `-- test-6851
> > `-- level2
> > |-- level3a
> > | |-- worker1
> > | `-- worker2
> > `-- level3b
> > `-- worker3
> >
> > Timeline (ish):
> > - worker3 gets throttled
> > - level3b is decayed, since it has no more load
> > - level2 get throttled
> > - worker3 get unthrottled
> > - level2 get unthrottled
> > - worker3 is added to list
> > - level3b is not added to list, since nr_running==0 and is decayed
> >
> >
> > The attached diff (based on
> > https://lore.kernel.org/lkml/20210518125202.78658-3-odin@uged.al/)
> > fixes the issue for me. Not the most elegant solution, but the
> > simplest one as of now, and to show what is wrong.
> >
> > Any thoughts Vincent?
>
>
> I would prefer that we use the reason of adding the cfs in the list instead.
>
> Something like the below should also fixed the problem. It is based on a
> proposal I made to Rik sometimes ago when he tried to flatten the rq:
> https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/
>
> This will ensure that a cfs is added in the list whenever one of its child
> is still in the list.
Could you confirm that this patch fixes the problem for you too ?
>
> ---
> kernel/sched/fair.c | 28 ++++++++++++++++++++++++++++
> 1 file changed, 28 insertions(+)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index ea7de54cb022..e751061a9449 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -3272,6 +3272,31 @@ static inline void cfs_rq_util_change(struct cfs_rq *cfs_rq, int flags)
>
> #ifdef CONFIG_SMP
> #ifdef CONFIG_FAIR_GROUP_SCHED
> +/*
> + * Because list_add_leaf_cfs_rq always places a child cfs_rq on the list
> + * immediately before a parent cfs_rq, and cfs_rqs are removed from the list
> + * bottom-up, we only have to test whether the cfs_rq before us on the list
> + * is our child.
> + * If cfs_rq is not on the list, test wether a child needs its to be added to
> + * connect a branch to the tree * (see list_add_leaf_cfs_rq() for details).
> + */
> +static inline bool child_cfs_rq_on_list(struct cfs_rq *cfs_rq)
> +{
> + struct cfs_rq *prev_cfs_rq;
> + struct list_head *prev;
> +
> + if (cfs_rq->on_list) {
> + prev = cfs_rq->leaf_cfs_rq_list.prev;
> + } else {
> + struct rq *rq = rq_of(cfs_rq);
> +
> + prev = rq->tmp_alone_branch;
> + }
> +
> + prev_cfs_rq = container_of(prev, struct cfs_rq, leaf_cfs_rq_list);
> +
> + return (prev_cfs_rq->tg->parent == cfs_rq->tg);
> +}
>
> static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
> {
> @@ -3287,6 +3312,9 @@ static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
> if (cfs_rq->avg.runnable_sum)
> return false;
>
> + if (child_cfs_rq_on_list(cfs_rq))
> + return false;
> +
> return true;
> }
>
> --
> 2.17.1
>
>
>
> >
> > Thanks
> > Odin
> >
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index bfaa6e1f6067..aa32e9c29efd 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -376,7 +376,8 @@ static inline bool list_add_leaf_cfs_rq(struct
> > cfs_rq *cfs_rq)
> > return false;
> > }
> >
> > -static inline void list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq)
> > +/* Returns 1 if cfs_rq was present in the list and removed */
> > +static inline bool list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq)
> > {
> > if (cfs_rq->on_list) {
> > struct rq *rq = rq_of(cfs_rq);
> > @@ -393,7 +394,9 @@ static inline void list_del_leaf_cfs_rq(struct
> > cfs_rq *cfs_rq)
> >
> > list_del_rcu(&cfs_rq->leaf_cfs_rq_list);
> > cfs_rq->on_list = 0;
> > + return 1;
> > }
> > + return 0;
> > }
> >
> > static inline void assert_list_leaf_cfs_rq(struct rq *rq)
> > @@ -3298,24 +3301,6 @@ static inline void cfs_rq_util_change(struct
> > cfs_rq *cfs_rq, int flags)
> >
> > #ifdef CONFIG_SMP
> > #ifdef CONFIG_FAIR_GROUP_SCHED
> > -
> > -static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
> > -{
> > - if (cfs_rq->load.weight)
> > - return false;
> > -
> > - if (cfs_rq->avg.load_sum)
> > - return false;
> > -
> > - if (cfs_rq->avg.util_sum)
> > - return false;
> > -
> > - if (cfs_rq->avg.runnable_sum)
> > - return false;
> > -
> > - return true;
> > -}
> > -
> > /**
> > * update_tg_load_avg - update the tg's load avg
> > * @cfs_rq: the cfs_rq whose avg changed
> > @@ -4109,11 +4094,6 @@ static inline void update_misfit_status(struct
> > task_struct *p, struct rq *rq)
> >
> > #else /* CONFIG_SMP */
> >
> > -static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
> > -{
> > - return true;
> > -}
> > -
> > #define UPDATE_TG 0x0
> > #define SKIP_AGE_LOAD 0x0
> > #define DO_ATTACH 0x0
> > @@ -4771,10 +4751,11 @@ static int tg_unthrottle_up(struct task_group
> > *tg, void *data)
> > if (!cfs_rq->throttle_count) {
> > cfs_rq->throttled_clock_task_time += rq_clock_task(rq) -
> > cfs_rq->throttled_clock_task;
> > -
> > - /* Add cfs_rq with load or one or more already running
> > entities to the list */
> > - if (!cfs_rq_is_decayed(cfs_rq) || cfs_rq->nr_running)
> > + if (cfs_rq->insert_on_unthrottle) {
> > list_add_leaf_cfs_rq(cfs_rq);
> > + if (tg->parent)
> > +
> > tg->parent->cfs_rq[cpu_of(rq)]->insert_on_unthrottle = true;
> > + }
> > }
> >
> > return 0;
> > @@ -4788,7 +4769,7 @@ static int tg_throttle_down(struct task_group
> > *tg, void *data)
> > /* group is entering throttled state, stop time */
> > if (!cfs_rq->throttle_count) {
> > cfs_rq->throttled_clock_task = rq_clock_task(rq);
> > - list_del_leaf_cfs_rq(cfs_rq);
> > + cfs_rq->insert_on_unthrottle = list_del_leaf_cfs_rq(cfs_rq);
> > }
> > cfs_rq->throttle_count++;
> >
> > @@ -8019,6 +8000,23 @@ static bool __update_blocked_others(struct rq
> > *rq, bool *done)
> >
> > #ifdef CONFIG_FAIR_GROUP_SCHED
> >
> > +static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
> > +{
> > + if (cfs_rq->load.weight)
> > + return false;
> > +
> > + if (cfs_rq->avg.load_sum)
> > + return false;
> > +
> > + if (cfs_rq->avg.util_sum)
> > + return false;
> > +
> > + if (cfs_rq->avg.runnable_sum)
> > + return false;
> > +
> > + return true;
> > +}
> > +
> > static bool __update_blocked_fair(struct rq *rq, bool *done)
> > {
> > struct cfs_rq *cfs_rq, *pos;
> > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > index a189bec13729..12a707d99ee6 100644
> > --- a/kernel/sched/sched.h
> > +++ b/kernel/sched/sched.h
> > @@ -602,6 +602,7 @@ struct cfs_rq {
> > u64 throttled_clock_task_time;
> > int throttled;
> > int throttle_count;
> > + int insert_on_unthrottle;
> > struct list_head throttled_list;
> > #endif /* CONFIG_CFS_BANDWIDTH */
> > #endif /* CONFIG_FAIR_GROUP_SCHED */
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
@ 2021-06-21 17:09 ` Vincent Guittot
0 siblings, 0 replies; 26+ messages in thread
From: Vincent Guittot @ 2021-06-21 17:09 UTC (permalink / raw)
To: Sachin Sant; +Cc: Peter Zijlstra, Odin Ugedal, linuxppc-dev, open list
Hi Sacha
On Mon, 21 Jun 2021 at 18:22, Vincent Guittot
<vincent.guittot@linaro.org> wrote:
>
> Le lundi 21 juin 2021 à 14:42:23 (+0200), Odin Ugedal a écrit :
> > Hi,
> >
> > Did some more research, and it looks like this is what happens:
> >
> > $ tree /sys/fs/cgroup/ltp/ -d --charset=ascii
> > /sys/fs/cgroup/ltp/
> > |-- drain
> > `-- test-6851
> > `-- level2
> > |-- level3a
> > | |-- worker1
> > | `-- worker2
> > `-- level3b
> > `-- worker3
> >
> > Timeline (ish):
> > - worker3 gets throttled
> > - level3b is decayed, since it has no more load
> > - level2 get throttled
> > - worker3 get unthrottled
> > - level2 get unthrottled
> > - worker3 is added to list
> > - level3b is not added to list, since nr_running==0 and is decayed
> >
> >
> > The attached diff (based on
> > https://lore.kernel.org/lkml/20210518125202.78658-3-odin@uged.al/)
> > fixes the issue for me. Not the most elegant solution, but the
> > simplest one as of now, and to show what is wrong.
> >
> > Any thoughts Vincent?
>
>
> I would prefer that we use the reason of adding the cfs in the list instead.
>
> Something like the below should also fixed the problem. It is based on a
> proposal I made to Rik sometimes ago when he tried to flatten the rq:
> https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/
>
> This will ensure that a cfs is added in the list whenever one of its child
> is still in the list.
Could you confirm that this patch fixes the problem for you too ?
>
> ---
> kernel/sched/fair.c | 28 ++++++++++++++++++++++++++++
> 1 file changed, 28 insertions(+)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index ea7de54cb022..e751061a9449 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -3272,6 +3272,31 @@ static inline void cfs_rq_util_change(struct cfs_rq *cfs_rq, int flags)
>
> #ifdef CONFIG_SMP
> #ifdef CONFIG_FAIR_GROUP_SCHED
> +/*
> + * Because list_add_leaf_cfs_rq always places a child cfs_rq on the list
> + * immediately before a parent cfs_rq, and cfs_rqs are removed from the list
> + * bottom-up, we only have to test whether the cfs_rq before us on the list
> + * is our child.
> + * If cfs_rq is not on the list, test wether a child needs its to be added to
> + * connect a branch to the tree * (see list_add_leaf_cfs_rq() for details).
> + */
> +static inline bool child_cfs_rq_on_list(struct cfs_rq *cfs_rq)
> +{
> + struct cfs_rq *prev_cfs_rq;
> + struct list_head *prev;
> +
> + if (cfs_rq->on_list) {
> + prev = cfs_rq->leaf_cfs_rq_list.prev;
> + } else {
> + struct rq *rq = rq_of(cfs_rq);
> +
> + prev = rq->tmp_alone_branch;
> + }
> +
> + prev_cfs_rq = container_of(prev, struct cfs_rq, leaf_cfs_rq_list);
> +
> + return (prev_cfs_rq->tg->parent == cfs_rq->tg);
> +}
>
> static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
> {
> @@ -3287,6 +3312,9 @@ static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
> if (cfs_rq->avg.runnable_sum)
> return false;
>
> + if (child_cfs_rq_on_list(cfs_rq))
> + return false;
> +
> return true;
> }
>
> --
> 2.17.1
>
>
>
> >
> > Thanks
> > Odin
> >
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index bfaa6e1f6067..aa32e9c29efd 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -376,7 +376,8 @@ static inline bool list_add_leaf_cfs_rq(struct
> > cfs_rq *cfs_rq)
> > return false;
> > }
> >
> > -static inline void list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq)
> > +/* Returns 1 if cfs_rq was present in the list and removed */
> > +static inline bool list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq)
> > {
> > if (cfs_rq->on_list) {
> > struct rq *rq = rq_of(cfs_rq);
> > @@ -393,7 +394,9 @@ static inline void list_del_leaf_cfs_rq(struct
> > cfs_rq *cfs_rq)
> >
> > list_del_rcu(&cfs_rq->leaf_cfs_rq_list);
> > cfs_rq->on_list = 0;
> > + return 1;
> > }
> > + return 0;
> > }
> >
> > static inline void assert_list_leaf_cfs_rq(struct rq *rq)
> > @@ -3298,24 +3301,6 @@ static inline void cfs_rq_util_change(struct
> > cfs_rq *cfs_rq, int flags)
> >
> > #ifdef CONFIG_SMP
> > #ifdef CONFIG_FAIR_GROUP_SCHED
> > -
> > -static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
> > -{
> > - if (cfs_rq->load.weight)
> > - return false;
> > -
> > - if (cfs_rq->avg.load_sum)
> > - return false;
> > -
> > - if (cfs_rq->avg.util_sum)
> > - return false;
> > -
> > - if (cfs_rq->avg.runnable_sum)
> > - return false;
> > -
> > - return true;
> > -}
> > -
> > /**
> > * update_tg_load_avg - update the tg's load avg
> > * @cfs_rq: the cfs_rq whose avg changed
> > @@ -4109,11 +4094,6 @@ static inline void update_misfit_status(struct
> > task_struct *p, struct rq *rq)
> >
> > #else /* CONFIG_SMP */
> >
> > -static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
> > -{
> > - return true;
> > -}
> > -
> > #define UPDATE_TG 0x0
> > #define SKIP_AGE_LOAD 0x0
> > #define DO_ATTACH 0x0
> > @@ -4771,10 +4751,11 @@ static int tg_unthrottle_up(struct task_group
> > *tg, void *data)
> > if (!cfs_rq->throttle_count) {
> > cfs_rq->throttled_clock_task_time += rq_clock_task(rq) -
> > cfs_rq->throttled_clock_task;
> > -
> > - /* Add cfs_rq with load or one or more already running
> > entities to the list */
> > - if (!cfs_rq_is_decayed(cfs_rq) || cfs_rq->nr_running)
> > + if (cfs_rq->insert_on_unthrottle) {
> > list_add_leaf_cfs_rq(cfs_rq);
> > + if (tg->parent)
> > +
> > tg->parent->cfs_rq[cpu_of(rq)]->insert_on_unthrottle = true;
> > + }
> > }
> >
> > return 0;
> > @@ -4788,7 +4769,7 @@ static int tg_throttle_down(struct task_group
> > *tg, void *data)
> > /* group is entering throttled state, stop time */
> > if (!cfs_rq->throttle_count) {
> > cfs_rq->throttled_clock_task = rq_clock_task(rq);
> > - list_del_leaf_cfs_rq(cfs_rq);
> > + cfs_rq->insert_on_unthrottle = list_del_leaf_cfs_rq(cfs_rq);
> > }
> > cfs_rq->throttle_count++;
> >
> > @@ -8019,6 +8000,23 @@ static bool __update_blocked_others(struct rq
> > *rq, bool *done)
> >
> > #ifdef CONFIG_FAIR_GROUP_SCHED
> >
> > +static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
> > +{
> > + if (cfs_rq->load.weight)
> > + return false;
> > +
> > + if (cfs_rq->avg.load_sum)
> > + return false;
> > +
> > + if (cfs_rq->avg.util_sum)
> > + return false;
> > +
> > + if (cfs_rq->avg.runnable_sum)
> > + return false;
> > +
> > + return true;
> > +}
> > +
> > static bool __update_blocked_fair(struct rq *rq, bool *done)
> > {
> > struct cfs_rq *cfs_rq, *pos;
> > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > index a189bec13729..12a707d99ee6 100644
> > --- a/kernel/sched/sched.h
> > +++ b/kernel/sched/sched.h
> > @@ -602,6 +602,7 @@ struct cfs_rq {
> > u64 throttled_clock_task_time;
> > int throttled;
> > int throttle_count;
> > + int insert_on_unthrottle;
> > struct list_head throttled_list;
> > #endif /* CONFIG_CFS_BANDWIDTH */
> > #endif /* CONFIG_FAIR_GROUP_SCHED */
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
2021-06-21 17:09 ` Vincent Guittot
@ 2021-06-21 17:31 ` Sachin Sant
-1 siblings, 0 replies; 26+ messages in thread
From: Sachin Sant @ 2021-06-21 17:31 UTC (permalink / raw)
To: Vincent Guittot; +Cc: Peter Zijlstra, Odin Ugedal, linuxppc-dev, open list
>>> Any thoughts Vincent?
>>
>>
>> I would prefer that we use the reason of adding the cfs in the list instead.
>>
>> Something like the below should also fixed the problem. It is based on a
>> proposal I made to Rik sometimes ago when he tried to flatten the rq:
>> https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/
>>
>> This will ensure that a cfs is added in the list whenever one of its child
>> is still in the list.
>
> Could you confirm that this patch fixes the problem for you too ?
>
Thanks for the fix.
The patch fixes the reported problem. The test ran to completion without
any failure.
Reported-by: Sachin Sant <sachinp@linux.vnet.ibm.com>
Tested-by: Sachin Sant <sachinp@linux.vnet.ibm.com>
-Sachin
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
@ 2021-06-21 17:31 ` Sachin Sant
0 siblings, 0 replies; 26+ messages in thread
From: Sachin Sant @ 2021-06-21 17:31 UTC (permalink / raw)
To: Vincent Guittot; +Cc: Peter Zijlstra, linuxppc-dev, Odin Ugedal, open list
>>> Any thoughts Vincent?
>>
>>
>> I would prefer that we use the reason of adding the cfs in the list instead.
>>
>> Something like the below should also fixed the problem. It is based on a
>> proposal I made to Rik sometimes ago when he tried to flatten the rq:
>> https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/
>>
>> This will ensure that a cfs is added in the list whenever one of its child
>> is still in the list.
>
> Could you confirm that this patch fixes the problem for you too ?
>
Thanks for the fix.
The patch fixes the reported problem. The test ran to completion without
any failure.
Reported-by: Sachin Sant <sachinp@linux.vnet.ibm.com>
Tested-by: Sachin Sant <sachinp@linux.vnet.ibm.com>
-Sachin
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
2021-06-21 17:31 ` Sachin Sant
@ 2021-06-21 17:44 ` Vincent Guittot
-1 siblings, 0 replies; 26+ messages in thread
From: Vincent Guittot @ 2021-06-21 17:44 UTC (permalink / raw)
To: Sachin Sant; +Cc: Peter Zijlstra, Odin Ugedal, linuxppc-dev, open list
On Mon, 21 Jun 2021 at 19:32, Sachin Sant <sachinp@linux.vnet.ibm.com> wrote:
>
> >>> Any thoughts Vincent?
> >>
> >>
> >> I would prefer that we use the reason of adding the cfs in the list instead.
> >>
> >> Something like the below should also fixed the problem. It is based on a
> >> proposal I made to Rik sometimes ago when he tried to flatten the rq:
> >> https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/
> >>
> >> This will ensure that a cfs is added in the list whenever one of its child
> >> is still in the list.
> >
> > Could you confirm that this patch fixes the problem for you too ?
> >
> Thanks for the fix.
>
> The patch fixes the reported problem. The test ran to completion without
> any failure.
>
> Reported-by: Sachin Sant <sachinp@linux.vnet.ibm.com>
> Tested-by: Sachin Sant <sachinp@linux.vnet.ibm.com>
Thanks
>
> -Sachin
>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests
@ 2021-06-21 17:44 ` Vincent Guittot
0 siblings, 0 replies; 26+ messages in thread
From: Vincent Guittot @ 2021-06-21 17:44 UTC (permalink / raw)
To: Sachin Sant; +Cc: Peter Zijlstra, linuxppc-dev, Odin Ugedal, open list
On Mon, 21 Jun 2021 at 19:32, Sachin Sant <sachinp@linux.vnet.ibm.com> wrote:
>
> >>> Any thoughts Vincent?
> >>
> >>
> >> I would prefer that we use the reason of adding the cfs in the list instead.
> >>
> >> Something like the below should also fixed the problem. It is based on a
> >> proposal I made to Rik sometimes ago when he tried to flatten the rq:
> >> https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/
> >>
> >> This will ensure that a cfs is added in the list whenever one of its child
> >> is still in the list.
> >
> > Could you confirm that this patch fixes the problem for you too ?
> >
> Thanks for the fix.
>
> The patch fixes the reported problem. The test ran to completion without
> any failure.
>
> Reported-by: Sachin Sant <sachinp@linux.vnet.ibm.com>
> Tested-by: Sachin Sant <sachinp@linux.vnet.ibm.com>
Thanks
>
> -Sachin
>
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2021-06-21 17:44 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-21 6:32 [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests Sachin Sant
2021-06-21 6:32 ` Sachin Sant
2021-06-21 9:39 ` Odin Ugedal
2021-06-21 9:39 ` Odin Ugedal
2021-06-21 9:50 ` Vincent Guittot
2021-06-21 9:50 ` Vincent Guittot
2021-06-21 9:54 ` Odin Ugedal
2021-06-21 9:54 ` Odin Ugedal
2021-06-21 10:57 ` Sachin Sant
2021-06-21 10:57 ` Sachin Sant
2021-06-21 11:04 ` Odin Ugedal
2021-06-21 11:04 ` Odin Ugedal
2021-06-21 12:42 ` Odin Ugedal
2021-06-21 12:42 ` Odin Ugedal
2021-06-21 16:22 ` Vincent Guittot
2021-06-21 16:22 ` Vincent Guittot
2021-06-21 16:45 ` Odin Ugedal
2021-06-21 16:45 ` Odin Ugedal
2021-06-21 17:07 ` Vincent Guittot
2021-06-21 17:07 ` Vincent Guittot
2021-06-21 17:09 ` Vincent Guittot
2021-06-21 17:09 ` Vincent Guittot
2021-06-21 17:31 ` Sachin Sant
2021-06-21 17:31 ` Sachin Sant
2021-06-21 17:44 ` Vincent Guittot
2021-06-21 17:44 ` Vincent Guittot
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.