* Re: BUG: HANG_DETECT waiting for migration_cpu_stop() complete
[not found] <88b2910181bda955ac46011b695c53f7da39ac47.camel@mediatek.com>
@ 2022-09-07 0:07 ` Hillf Danton
2022-09-22 5:40 ` Jing-Ting Wu
0 siblings, 1 reply; 3+ messages in thread
From: Hillf Danton @ 2022-09-07 0:07 UTC (permalink / raw)
To: Jing-Ting Wu; +Cc: Peter Zijlstra, linux-kernel, linux-mm, Waiman Long
On 5 Sep 2022 10:47:36 +0800 Jing-Ting Wu <jing-ting.wu@mediatek.com> wrote
>
> We meet the HANG_DETECT happened in T SW version with kernel-5.15.
> Many tasks have been blocked for a long time.
>
> Root cause:
> migration_cpu_stop() is not complete due to is_migration_disabled(p) is
> true, complete is false and complete_all() never get executed.
> It let other task wait the rwsem.
See if handing task over to stopper again in case of migration disabled
could survive your tests.
Hillf
--- linux-5.15/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2322,9 +2322,7 @@ static int migration_cpu_stop(void *data
* holding rq->lock, if p->on_rq == 0 it cannot get enqueued because
* we're holding p->pi_lock.
*/
- if (task_rq(p) == rq) {
- if (is_migration_disabled(p))
- goto out;
+ if (task_rq(p) == rq && !is_migration_disabled(p)) {
if (pending) {
p->migration_pending = NULL;
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: BUG: HANG_DETECT waiting for migration_cpu_stop() complete
2022-09-07 0:07 ` BUG: HANG_DETECT waiting for migration_cpu_stop() complete Hillf Danton
@ 2022-09-22 5:40 ` Jing-Ting Wu
2022-09-22 12:02 ` Hillf Danton
0 siblings, 1 reply; 3+ messages in thread
From: Jing-Ting Wu @ 2022-09-22 5:40 UTC (permalink / raw)
To: Hillf Danton
Cc: Peter Zijlstra, linux-kernel, linux-mm, Waiman Long,
ValentinSchneider, TejunHeo, wsd_upstream, linux-arm-kernel,
linux-mediatek, Jonathan.JMChen, chris.redpath, Dietmar Eggemann,
Vincent Donnefort, Ingo Molnar, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Christian Brauner, cgroups, lixiong.liu, wenju.xu
On Wed, 2022-09-07 at 08:07 +0800, Hillf Danton wrote:
> On 5 Sep 2022 10:47:36 +0800 Jing-Ting Wu <jing-ting.wu@mediatek.com>
> wrote
> >
> > We meet the HANG_DETECT happened in T SW version with kernel-5.15.
> > Many tasks have been blocked for a long time.
> >
> > Root cause:
> > migration_cpu_stop() is not complete due to
> > is_migration_disabled(p) is
> > true, complete is false and complete_all() never get executed.
> > It let other task wait the rwsem.
>
> See if handing task over to stopper again in case of migration
> disabled
> could survive your tests.
>
> Hillf
>
> --- linux-5.15/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -2322,9 +2322,7 @@ static int migration_cpu_stop(void *data
> * holding rq->lock, if p->on_rq == 0 it cannot get enqueued
> because
> * we're holding p->pi_lock.
> */
> - if (task_rq(p) == rq) {
> - if (is_migration_disabled(p))
> - goto out;
> + if (task_rq(p) == rq && !is_migration_disabled(p)) {
>
> if (pending) {
> p->migration_pending = NULL;
Because Peter have some concern for patch by Waiman.
We add Hillf's patch to our stability test.
But there are side effects after patched.
The warning appear once < two weeks.
Backtrace as follows:
[name:panic&]WARNING: CPU: 6 PID: 32583 at affine_move_task
pc : affine_move_task
lr : __set_cpus_allowed_ptr_locked
Call trace:
affine_move_task
__set_cpus_allowed_ptr_locked
migrate_enable
__cgroup_bpf_run_filter_skb
ip_finish_output
ip_output
The root cause is when is_migration_disabled(p) is true,the patched
version will set p->migration_pending to NULL by migration_cpu_stop.
And in affine_move_task will raise a WARN_ON_ONCE(!pending).
Kernel-5.15/kernel/sched/core.c:
static int affine_move_task(struct rq *rq, struct task_struct *p,
struct rq_flags *rf, int dest_cpu, unsigned int flags) {
...
If (WARN_ON_ONCE(!pending)) {
Task_rq_unlock(rq,p,fr);
return -EINVAL;
}
...
}
But the tasks have not been migrated to the new affinity CPU, so there
should be pending tasks to be processed, so p->migration_pending should
not be NULL.
Without patch:
When is_migration_disabled is true, then goto out and not set p-
>migration_pending to NULL.
static int migration_cpu_stop(void *data) {
...
If (task_rq(p) == rq) {
if (is_migration_disabled(p))
goto out;
}
...
}
With patch:
When is_migration_disabled is true and pending is true, goto else if
flow. Because p->cpus_ptr not updated when migrate_disable, so this
condition is always true and p->migration_pending will set to NULL.
static int migration_cpu_stop(void *data) {
...
If (task_rq(p) == rq && !is_migration_disabled(p) ) {
...
} else if (pending) {
...
If (cpumask_test_cpu(task_cpu(p), p-> cpus_ ptr)) {
p->migration_pending = NULL;
complete = true;
goto out;
}
...
}
Best regards,
Jing-Ting Wu
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: BUG: HANG_DETECT waiting for migration_cpu_stop() complete
2022-09-22 5:40 ` Jing-Ting Wu
@ 2022-09-22 12:02 ` Hillf Danton
0 siblings, 0 replies; 3+ messages in thread
From: Hillf Danton @ 2022-09-22 12:02 UTC (permalink / raw)
To: Jing-Ting Wu
Cc: Peter Zijlstra, linux-kernel, linux-mm, Waiman Long,
Vincent Guittot, Mel Gorman, wsd_upstream
On 22 Sep 2022 13:40:47 +0800 Jing-Ting Wu <jing-ting.wu@mediatek.com> wrote
>
> Because Peter have some concern for patch by Waiman.
> We add Hillf's patch to our stability test.
> But there are side effects after patched.
> The warning appear once < two weeks.
Thanks for your test. Any other effects observed?
>
> Backtrace as follows:
> [name:panic&]WARNING: CPU: 6 PID: 32583 at affine_move_task
> pc : affine_move_task
> lr : __set_cpus_allowed_ptr_locked
> Call trace:
> affine_move_task
> __set_cpus_allowed_ptr_locked
> migrate_enable
> __cgroup_bpf_run_filter_skb
> ip_finish_output
> ip_output
>
>
> The root cause is when is_migration_disabled(p) is true,the patched
> version will set p->migration_pending to NULL by migration_cpu_stop.
> And in affine_move_task will raise a WARN_ON_ONCE(!pending).
>
> Kernel-5.15/kernel/sched/core.c:
> static int affine_move_task(struct rq *rq, struct task_struct *p,
> struct rq_flags *rf, int dest_cpu, unsigned int flags) {
> ...
> If (WARN_ON_ONCE(!pending)) {
> Task_rq_unlock(rq,p,fr);
> return -EINVAL;
> }
> ...
> }
>
> But the tasks have not been migrated to the new affinity CPU, so there
> should be pending tasks to be processed, so p->migration_pending should
> not be NULL.
>
>
>
> Without patch:
> When is_migration_disabled is true, then goto out and not set p-
> >migration_pending to NULL.
>
> static int migration_cpu_stop(void *data) {
> ...
> If (task_rq(p) == rq) {
> if (is_migration_disabled(p))
> goto out;
> }
> ...
> }
>
>
> With patch:
> When is_migration_disabled is true and pending is true, goto else if
> flow. Because p->cpus_ptr not updated when migrate_disable, so this
> condition is always true and p->migration_pending will set to NULL.
>
> static int migration_cpu_stop(void *data) {
> ...
> If (task_rq(p) == rq && !is_migration_disabled(p) ) {
> ...
> } else if (pending) {
> ...
/*
* The task moved before the stopper got to run. We're holding
* ->pi_lock, so the allowed mask is stable - if it got
* somewhere allowed, we're done.
*/
> If (cpumask_test_cpu(task_cpu(p), p-> cpus_ ptr)) {
> p->migration_pending = NULL;
> complete = true;
> goto out;
> }
> ...
> }
Given p->migration_pending reset in case of job done, the warning you saw
is benign without other negative effects observed.
It should be fixed (example by simply cutting it) if you have a reproducer
on top of the mainline tree.
Hillf
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2022-09-22 12:02 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <88b2910181bda955ac46011b695c53f7da39ac47.camel@mediatek.com>
2022-09-07 0:07 ` BUG: HANG_DETECT waiting for migration_cpu_stop() complete Hillf Danton
2022-09-22 5:40 ` Jing-Ting Wu
2022-09-22 12:02 ` Hillf Danton
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).