[PATCH] sched/debug: Reset watchdog on all CPUs while processing sysrq-t

[PATCH] sched/debug: Reset watchdog on all CPUs while processing sysrq-t

[Wei Li]

Lengthy output of sysrq-t may take a lot of time on slow serial console
with lots of processes and CPUs.

So we need to reset NMI-watchdog to avoid spurious lockup messages, and
we also reset softlockup watchdogs on all other CPUs since another CPU
might be blocked waiting for us to process an IPI or stop_machine.

Add to sysrq_sched_debug_show() as what we did in show_state_filter().

Signed-off-by: Wei Li <liwei391@huawei.com>
---
 kernel/sched/debug.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index f7e4579e746c..879d3ccf3806 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -751,9 +751,16 @@ void sysrq_sched_debug_show(void)
 	int cpu;
 
 	sched_debug_header(NULL);
-	for_each_online_cpu(cpu)
+	for_each_online_cpu(cpu) {
+		/*
+		 * Need to reset softlockup watchdogs on all CPUs, because
+		 * another CPU might be blocked waiting for us to process
+		 * an IPI or stop_machine.
+		 */
+		touch_nmi_watchdog();
+		touch_all_softlockup_watchdogs();
 		print_cpu(NULL, cpu);
-
+	}
 }
 
 /*
-- 
2.17.1

Re: [PATCH] sched/debug: Reset watchdog on all CPUs while processing sysrq-t

[Steven Rostedt]

On Thu, 26 Dec 2019 16:52:24 +0800
Wei Li <liwei391@huawei.com> wrote:

> Lengthy output of sysrq-t may take a lot of time on slow serial console
> with lots of processes and CPUs.
> 
> So we need to reset NMI-watchdog to avoid spurious lockup messages, and
> we also reset softlockup watchdogs on all other CPUs since another CPU
> might be blocked waiting for us to process an IPI or stop_machine.

Have you had this triggered?

> 
> Add to sysrq_sched_debug_show() as what we did in show_state_filter().
> 
> Signed-off-by: Wei Li <liwei391@huawei.com>
> ---
>  kernel/sched/debug.c | 11 +++++++++--
>  1 file changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
> index f7e4579e746c..879d3ccf3806 100644
> --- a/kernel/sched/debug.c
> +++ b/kernel/sched/debug.c
> @@ -751,9 +751,16 @@ void sysrq_sched_debug_show(void)
>  	int cpu;
>  
>  	sched_debug_header(NULL);
> -	for_each_online_cpu(cpu)
> +	for_each_online_cpu(cpu) {
> +		/*
> +		 * Need to reset softlockup watchdogs on all CPUs, because
> +		 * another CPU might be blocked waiting for us to process
> +		 * an IPI or stop_machine.
> +		 */
> +		touch_nmi_watchdog();
> +		touch_all_softlockup_watchdogs();

This doesn't seem to hurt to add, thus.

Reviewed-by: Steven Rostedt (VMware) <rostedt@goodmis.org>

-- Steve

>  		print_cpu(NULL, cpu);
> -
> +	}
>  }
>  
>  /*

Re: [PATCH] sched/debug: Reset watchdog on all CPUs while processing sysrq-t

[liwei (GF)]

Hi Steven,
Yes, it can be triggered on the Hi1620 system (128 cores) as follows:
stress-ng -c 50 &
stress-ng -m 50 &
stress-ng -i 20 &
echo 7 > /proc/sys/kernel/printk
echo t > /proc/sysrq-trigger

Then a soft lockup will be reported at migration thread
[39636.303531] watchdog: BUG: soft lockup - CPU#67 stuck for 23s! [migration/67:348]
which is waiting for the CPU handling sysrq-t to process stop_two_cpus.

Thanks,
Wei

On 2020/1/3 3:45, Steven Rostedt wrote:
> On Thu, 26 Dec 2019 16:52:24 +0800
> Wei Li <liwei391@huawei.com> wrote:
> 
>> Lengthy output of sysrq-t may take a lot of time on slow serial console
>> with lots of processes and CPUs.
>>
>> So we need to reset NMI-watchdog to avoid spurious lockup messages, and
>> we also reset softlockup watchdogs on all other CPUs since another CPU
>> might be blocked waiting for us to process an IPI or stop_machine.
> 
> Have you had this triggered?
> 
>>
>> Add to sysrq_sched_debug_show() as what we did in show_state_filter().
>>
>> Signed-off-by: Wei Li <liwei391@huawei.com>
>> ---
>>  kernel/sched/debug.c | 11 +++++++++--
>>  1 file changed, 9 insertions(+), 2 deletions(-)
>>
>> diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
>> index f7e4579e746c..879d3ccf3806 100644
>> --- a/kernel/sched/debug.c
>> +++ b/kernel/sched/debug.c
>> @@ -751,9 +751,16 @@ void sysrq_sched_debug_show(void)
>>  	int cpu;
>>  
>>  	sched_debug_header(NULL);
>> -	for_each_online_cpu(cpu)
>> +	for_each_online_cpu(cpu) {
>> +		/*
>> +		 * Need to reset softlockup watchdogs on all CPUs, because
>> +		 * another CPU might be blocked waiting for us to process
>> +		 * an IPI or stop_machine.
>> +		 */
>> +		touch_nmi_watchdog();
>> +		touch_all_softlockup_watchdogs();
> 
> This doesn't seem to hurt to add, thus.
> 
> Reviewed-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
> 
> -- Steve
> 
>>  		print_cpu(NULL, cpu);
>> -
>> +	}
>>  }
>>  
>>  /*
> 
> 
> .
> 

Re: [PATCH] sched/debug: Reset watchdog on all CPUs while processing sysrq-t

[Peter Zijlstra]

On Thu, Jan 02, 2020 at 02:45:14PM -0500, Steven Rostedt wrote:
> On Thu, 26 Dec 2019 16:52:24 +0800
> Wei Li <liwei391@huawei.com> wrote:
> 
> > Lengthy output of sysrq-t may take a lot of time on slow serial console
> > with lots of processes and CPUs.
> > 
> > So we need to reset NMI-watchdog to avoid spurious lockup messages, and
> > we also reset softlockup watchdogs on all other CPUs since another CPU
> > might be blocked waiting for us to process an IPI or stop_machine.
> 
> Have you had this triggered?
> 
> > 
> > Add to sysrq_sched_debug_show() as what we did in show_state_filter().
> > 
> > Signed-off-by: Wei Li <liwei391@huawei.com>
> > ---
> >  kernel/sched/debug.c | 11 +++++++++--
> >  1 file changed, 9 insertions(+), 2 deletions(-)
> > 
> > diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
> > index f7e4579e746c..879d3ccf3806 100644
> > --- a/kernel/sched/debug.c
> > +++ b/kernel/sched/debug.c
> > @@ -751,9 +751,16 @@ void sysrq_sched_debug_show(void)
> >  	int cpu;
> >  
> >  	sched_debug_header(NULL);
> > -	for_each_online_cpu(cpu)
> > +	for_each_online_cpu(cpu) {
> > +		/*
> > +		 * Need to reset softlockup watchdogs on all CPUs, because
> > +		 * another CPU might be blocked waiting for us to process
> > +		 * an IPI or stop_machine.
> > +		 */
> > +		touch_nmi_watchdog();
> > +		touch_all_softlockup_watchdogs();
> 
> This doesn't seem to hurt to add, thus.
> 
> Reviewed-by: Steven Rostedt (VMware) <rostedt@goodmis.org>

Thanks!

[tip: sched/core] sched/debug: Reset watchdog on all CPUs while processing sysrq-t

[tip-bot2 for Wei Li]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     02d4ac5885a18d326b500b94808f0956dcce2832
Gitweb:        https://git.kernel.org/tip/02d4ac5885a18d326b500b94808f0956dcce2832
Author:        Wei Li <liwei391@huawei.com>
AuthorDate:    Thu, 26 Dec 2019 16:52:24 +08:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Fri, 17 Jan 2020 10:19:20 +01:00

sched/debug: Reset watchdog on all CPUs while processing sysrq-t

Lengthy output of sysrq-t may take a lot of time on slow serial console
with lots of processes and CPUs.

So we need to reset NMI-watchdog to avoid spurious lockup messages, and
we also reset softlockup watchdogs on all other CPUs since another CPU
might be blocked waiting for us to process an IPI or stop_machine.

Add to sysrq_sched_debug_show() as what we did in show_state_filter().

Signed-off-by: Wei Li <liwei391@huawei.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
Link: https://lkml.kernel.org/r/20191226085224.48942-1-liwei391@huawei.com
---
 kernel/sched/debug.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index f7e4579..879d3cc 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -751,9 +751,16 @@ void sysrq_sched_debug_show(void)
 	int cpu;
 
 	sched_debug_header(NULL);
-	for_each_online_cpu(cpu)
+	for_each_online_cpu(cpu) {
+		/*
+		 * Need to reset softlockup watchdogs on all CPUs, because
+		 * another CPU might be blocked waiting for us to process
+		 * an IPI or stop_machine.
+		 */
+		touch_nmi_watchdog();
+		touch_all_softlockup_watchdogs();
 		print_cpu(NULL, cpu);
-
+	}
 }
 
 /*