Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue

From: Mike Galbraith <efault@gmx.de>
To: K Prateek Nayak <kprateek.nayak@amd.com>,
	Peter Zijlstra <peterz@infradead.org>,
	mingo@redhat.com, juri.lelli@redhat.com,
	 vincent.guittot@linaro.org, dietmar.eggemann@arm.com,
	rostedt@goodmis.org,  bsegall@google.com, mgorman@suse.de,
	bristot@redhat.com, vschneid@redhat.com,
	 linux-kernel@vger.kernel.org
Cc: wuyun.abel@bytedance.com, tglx@linutronix.de,
	Chen Yu <yu.c.chen@intel.com>,
	 Oliver Sang <oliver.sang@intel.com>
Subject: Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Date: Thu, 18 Apr 2024 18:24:59 +0200	[thread overview]
Message-ID: <2b9f7617f2b2130bb6270504ec3858f15d463f1d.camel@gmx.de> (raw)
In-Reply-To: <747627a1414f1f33d0c237f555494149d6937800.camel@gmx.de>

Greetings,

I tossed a couple rocks at it today, and seem to have hit the little
bugger.  The root cause seems to be doing the delay dequeue business on
exiting tasks.  Hunk #1 of hacklet below seems to quell the explosions.

crash> bt -sx
PID: 21722    TASK: ffff88815710ee40  CPU: 6    COMMAND: "hackbench"
 #0 [ffff88822773bb90] machine_kexec+0x156 at ffffffff810642d6
 #1 [ffff88822773bbe0] __crash_kexec+0xd7 at ffffffff81152a07
 #2 [ffff88822773bc98] crash_kexec+0x23 at ffffffff81152eb3
 #3 [ffff88822773bca8] oops_end+0xbe at ffffffff810329be
 #4 [ffff88822773bcc8] page_fault_oops+0x81 at ffffffff81071951
 #5 [ffff88822773bd28] exc_page_fault+0x62 at ffffffff8194e9e2
 #6 [ffff88822773bd50] asm_exc_page_fault+0x22 at ffffffff81a00ba2
    [exception RIP: pick_next_task_fair+178]
    RIP: ffffffff810d8d12  RSP: ffff88822773be00  RFLAGS: 00010006
    RAX: ffff88813cb780b8  RBX: ffff88840edb0e80  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: ffff88813cb78080  RDI: ffff88840ec30f00
    RBP: ffff88813cb78000   R8: ffff88815710eec0   R9: 0000000000000001
    R10: ffff88822773bdc8  R11: 0000000000000013  R12: 0000000000030e80
    R13: ffff88815710ee40  R14: ffff88813cb78080  R15: ffff88815710ee40
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffff88822773be28] __schedule+0x10d at ffffffff8195865d
 #8 [ffff88822773be98] do_task_dead+0x3e at ffffffff810cc00e
 #9 [ffff88822773beb0] do_exit+0x770 at ffffffff8108f0e0
#10 [ffff88822773bf00] do_group_exit+0x2c at ffffffff8108f64c
#11 [ffff88822773bf28] __x64_sys_exit_group+0x14 at ffffffff8108f6f4
#12 [ffff88822773bf30] do_syscall_64+0x57 at ffffffff8194ac37
#13 [ffff88822773bf50] entry_SYSCALL_64_after_hwframe+0x76 at ffffffff81a0012b
    RIP: 00007f4f2aa76136  RSP: 00007ffcbba84748  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: 0000000000000000  RCX: 00007f4f2aa76136
    RDX: 0000000000000000  RSI: 000000000000003c  RDI: 0000000000000000
    RBP: 00007f4f2ab86970   R8: 00000000000000e7   R9: ffffffffffffff80
    R10: 0000000000000004  R11: 0000000000000246  R12: 00007f4f2ab86970
    R13: 0000000000000001  R14: 00007f4f2ab8a328  R15: 0000000000000000
    ORIG_RAX: 00000000000000e7  CS: 0033  SS: 002b
crash> task_struct ffff88815710ee40 | grep sched_delayed
    sched_delayed = 1 '\001',
crash>

---
 kernel/sched/fair.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5374,6 +5374,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
 		update_curr(cfs_rq);

 		if (sched_feat(DELAY_DEQUEUE) && sleep &&
+		    !(entity_is_task(se) && (task_of(se)->flags & PF_EXITING)) &&
 		    !entity_eligible(cfs_rq, se)) {
 			if (cfs_rq->next == se)
 				cfs_rq->next = NULL;
@@ -5495,14 +5496,14 @@ pick_next_entity(struct rq *rq, struct c
 	}

 	struct sched_entity *se = pick_eevdf(cfs_rq);
-	if (se->sched_delayed) {
+	while (se && se->sched_delayed) {
 		dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
 		SCHED_WARN_ON(se->sched_delayed);
 		SCHED_WARN_ON(se->on_rq);
 		if (sched_feat(DELAY_ZERO) && se->vlag > 0)
 			se->vlag = 0;

-		return NULL;
+		se = pick_eevdf(cfs_rq);
 	}
 	return se;
 }

