PATCH: softlockup: Fix hung_task_check_count sysctl

PATCH: softlockup: Fix hung_task_check_count sysctl

[Anton Blanchard]

I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce
this source of jitter I tried setting hung_task_check_count to 0:

# echo 0 > /proc/sys/kernel/hung_task_check_count

which didn't have the intended response. Change to a post increment of
max_count, so a value of 0 means check 0 tasks.

Signed-off-by: Anton Blanchard <anton@samba.org>
---

Index: linux.trees.git/kernel/hung_task.c
===================================================================
--- linux.trees.git.orig/kernel/hung_task.c	2009-11-27 13:11:46.000000000 +1100
+++ linux.trees.git/kernel/hung_task.c	2009-11-27 13:11:57.000000000 +1100
@@ -144,7 +144,7 @@ static void check_hung_uninterruptible_t
 
 	rcu_read_lock();
 	do_each_thread(g, t) {
-		if (!--max_count)
+		if (!max_count--)
 			goto unlock;
 		if (!--batch_count) {
 			batch_count = HUNG_TASK_BATCHING;

Re: PATCH: softlockup: Fix hung_task_check_count sysctl

[Frederic Weisbecker]

On Fri, Nov 27, 2009 at 01:28:20PM +1100, Anton Blanchard wrote:
> 
> I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce
> this source of jitter I tried setting hung_task_check_count to 0:
> 
> # echo 0 > /proc/sys/kernel/hung_task_check_count
> 
> which didn't have the intended response. Change to a post increment of
> max_count, so a value of 0 means check 0 tasks.
> 
> Signed-off-by: Anton Blanchard <anton@samba.org>


Acked-by: Frederic Weisbecker <fweisbec@gmail.com>


> ---
> 
> Index: linux.trees.git/kernel/hung_task.c
> ===================================================================
> --- linux.trees.git.orig/kernel/hung_task.c	2009-11-27 13:11:46.000000000 +1100
> +++ linux.trees.git/kernel/hung_task.c	2009-11-27 13:11:57.000000000 +1100
> @@ -144,7 +144,7 @@ static void check_hung_uninterruptible_t
>  
>  	rcu_read_lock();
>  	do_each_thread(g, t) {
> -		if (!--max_count)
> +		if (!max_count--)
>  			goto unlock;
>  		if (!--batch_count) {
>  			batch_count = HUNG_TASK_BATCHING;

Re: PATCH: softlockup: Fix hung_task_check_count sysctl

[Américo Wang]

On Fri, Nov 27, 2009 at 10:28 AM, Anton Blanchard <anton@samba.org> wrote:
>
> I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce
> this source of jitter I tried setting hung_task_check_count to 0:
>
> # echo 0 > /proc/sys/kernel/hung_task_check_count
>
> which didn't have the intended response. Change to a post increment of
> max_count, so a value of 0 means check 0 tasks.
>
> Signed-off-by: Anton Blanchard <anton@samba.org>


Ack.

I would also suggest to make 'max_count' as unsigned long,
since sysctl_hung_task_check_count is.

Thanks.

> ---
>
> Index: linux.trees.git/kernel/hung_task.c
> ===================================================================
> --- linux.trees.git.orig/kernel/hung_task.c     2009-11-27 13:11:46.000000000 +1100
> +++ linux.trees.git/kernel/hung_task.c  2009-11-27 13:11:57.000000000 +1100
> @@ -144,7 +144,7 @@ static void check_hung_uninterruptible_t
>
>        rcu_read_lock();
>        do_each_thread(g, t) {
> -               if (!--max_count)
> +               if (!max_count--)
>                        goto unlock;
>                if (!--batch_count) {
>                        batch_count = HUNG_TASK_BATCHING;

Re: PATCH: softlockup: Fix hung_task_check_count sysctl

[Frederic Weisbecker]

On Fri, Nov 27, 2009 at 10:46:53AM +0800, Américo Wang wrote:
> On Fri, Nov 27, 2009 at 10:28 AM, Anton Blanchard <anton@samba.org> wrote:
> >
> > I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce
> > this source of jitter I tried setting hung_task_check_count to 0:
> >
> > # echo 0 > /proc/sys/kernel/hung_task_check_count
> >
> > which didn't have the intended response. Change to a post increment of
> > max_count, so a value of 0 means check 0 tasks.
> >
> > Signed-off-by: Anton Blanchard <anton@samba.org>
> 
> 
> Ack.
> 
> I would also suggest to make 'max_count' as unsigned long,
> since sysctl_hung_task_check_count is.
> 
> Thanks.


Also, the batch_count thing should be dropped I think.
This is a hardcoded, not overridable pause after 1024
threads checks to avoid latencies caused by rcu_read_lock.
But now we have PREEMPT_RCU so people can enable it if
they care about latency. We should remove it as it adds
unnecessary complexity.

I'm preparing a patch for that, on top of Anton patch.

Re: PATCH: softlockup: Fix hung_task_check_count sysctl

[Américo Wang]

On Fri, Nov 27, 2009 at 10:55 AM, Frederic Weisbecker
<fweisbec@gmail.com> wrote:
> On Fri, Nov 27, 2009 at 10:46:53AM +0800, Américo Wang wrote:
>> On Fri, Nov 27, 2009 at 10:28 AM, Anton Blanchard <anton@samba.org> wrote:
>> >
>> > I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce
>> > this source of jitter I tried setting hung_task_check_count to 0:
>> >
>> > # echo 0 > /proc/sys/kernel/hung_task_check_count
>> >
>> > which didn't have the intended response. Change to a post increment of
>> > max_count, so a value of 0 means check 0 tasks.
>> >
>> > Signed-off-by: Anton Blanchard <anton@samba.org>
>>
>>
>> Ack.
>>
>> I would also suggest to make 'max_count' as unsigned long,
>> since sysctl_hung_task_check_count is.
>>
>> Thanks.
>
>
> Also, the batch_count thing should be dropped I think.
> This is a hardcoded, not overridable pause after 1024
> threads checks to avoid latencies caused by rcu_read_lock.
> But now we have PREEMPT_RCU so people can enable it if
> they care about latency. We should remove it as it adds
> unnecessary complexity.

This sounds OK for me.

>
> I'm preparing a patch for that, on top of Anton patch.
>

Great!

Thanks.

Re: PATCH: softlockup: Fix hung_task_check_count sysctl

[Frederic Weisbecker]

On Fri, Nov 27, 2009 at 11:02:13AM +0800, Américo Wang wrote:
> On Fri, Nov 27, 2009 at 10:55 AM, Frederic Weisbecker
> <fweisbec@gmail.com> wrote:
> > On Fri, Nov 27, 2009 at 10:46:53AM +0800, Américo Wang wrote:
> >> On Fri, Nov 27, 2009 at 10:28 AM, Anton Blanchard <anton@samba.org> wrote:
> >> >
> >> > I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce
> >> > this source of jitter I tried setting hung_task_check_count to 0:
> >> >
> >> > # echo 0 > /proc/sys/kernel/hung_task_check_count
> >> >
> >> > which didn't have the intended response. Change to a post increment of
> >> > max_count, so a value of 0 means check 0 tasks.
> >> >
> >> > Signed-off-by: Anton Blanchard <anton@samba.org>
> >>
> >>
> >> Ack.
> >>
> >> I would also suggest to make 'max_count' as unsigned long,
> >> since sysctl_hung_task_check_count is.
> >>
> >> Thanks.
> >
> >
> > Also, the batch_count thing should be dropped I think.
> > This is a hardcoded, not overridable pause after 1024
> > threads checks to avoid latencies caused by rcu_read_lock.
> > But now we have PREEMPT_RCU so people can enable it if
> > they care about latency. We should remove it as it adds
> > unnecessary complexity.
> 
> This sounds OK for me.
> 
> >
> > I'm preparing a patch for that, on top of Anton patch.
> >
> 
> Great!
> 
> Thanks.


Actually it looks like this check is not only there to take
care of latency but also to avoid too much waiting before
a grace period.

Hm, well, I'm not sure what to do.

[tip:core/softlockup] softlockup: Fix hung_task_check_count sysctl

[tip-bot for Anton Blanchard]

Commit-ID:  e5af02261668350b43eb7381648930bde8e872f7
Gitweb:     http://git.kernel.org/tip/e5af02261668350b43eb7381648930bde8e872f7
Author:     Anton Blanchard <anton@samba.org>
AuthorDate: Fri, 27 Nov 2009 13:28:20 +1100
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Fri, 27 Nov 2009 06:21:57 +0100

softlockup: Fix hung_task_check_count sysctl

I'm seeing spikes of up to 0.5ms in khungtaskd on a large
machine. To reduce this source of jitter I tried setting
hung_task_check_count to 0:

 # echo 0 > /proc/sys/kernel/hung_task_check_count

which didn't have the intended response. Change to a post
increment of max_count, so a value of 0 means check 0 tasks.

Signed-off-by: Anton Blanchard <anton@samba.org>
Acked-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: msb@google.com
LKML-Reference: <20091127022820.GU32182@kryten>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 kernel/hung_task.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index d4e8417..0c642d5 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -144,7 +144,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 
 	rcu_read_lock();
 	do_each_thread(g, t) {
-		if (!--max_count)
+		if (!max_count--)
 			goto unlock;
 		if (!--batch_count) {
 			batch_count = HUNG_TASK_BATCHING;

Re: PATCH: softlockup: Fix hung_task_check_count sysctl

[Mandeep Singh Baines]

To disable hung_task at runtime:

# echo 0 > /proc/sys/kernel/hung_task_check_count

33 /*
34  * Zero means infinite timeout - no checking done:
35  */
36 unsigned long __read_mostly sysctl_hung_task_timeout_secs = 120;

For hung_task_check_count, 0 means there is no maximum so all tasks
are checked.

Anton Blanchard (anton@samba.org) wrote:
> 
> I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce
> this source of jitter I tried setting hung_task_check_count to 0:
> 

Hmm, maybe khungtaskd should be run using SCHED_IDLE.

> # echo 0 > /proc/sys/kernel/hung_task_check_count
> 
> which didn't have the intended response. Change to a post increment of
> max_count, so a value of 0 means check 0 tasks.
> 
> Signed-off-by: Anton Blanchard <anton@samba.org>
> ---
> 
> Index: linux.trees.git/kernel/hung_task.c
> ===================================================================
> --- linux.trees.git.orig/kernel/hung_task.c	2009-11-27 13:11:46.000000000 +1100
> +++ linux.trees.git/kernel/hung_task.c	2009-11-27 13:11:57.000000000 +1100
> @@ -144,7 +144,7 @@ static void check_hung_uninterruptible_t
>  
>  	rcu_read_lock();
>  	do_each_thread(g, t) {
> -		if (!--max_count)
> +		if (!max_count--)
>  			goto unlock;
>  		if (!--batch_count) {
>  			batch_count = HUNG_TASK_BATCHING;

Re: PATCH: softlockup: Fix hung_task_check_count sysctl

[Mandeep Baines]

On Mon, Nov 30, 2009 at 11:13 AM, Mandeep Singh Baines <msb@google.com> wrote:
> To disable hung_task at runtime:
>
> # echo 0 > /proc/sys/kernel/hung_task_check_count
>

D'oh. Copy and paste error. I meant

# echo 0 > /proc/sys/kernel/hung_task_timeout_secs

The nice thing about hung_task_timeout_secs=0 is that hungtaskd will
no longer run. Implementing
no checking done with hung_task_check_count=0 would result in the task
getting scheduled,
running, and then doing nothing.

> 33 /*
> 34  * Zero means infinite timeout - no checking done:
> 35  */
> 36 unsigned long __read_mostly sysctl_hung_task_timeout_secs = 120;
>
> For hung_task_check_count, 0 means there is no maximum so all tasks
> are checked.
>
> Anton Blanchard (anton@samba.org) wrote:
>>
>> I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce
>> this source of jitter I tried setting hung_task_check_count to 0:
>>
>
> Hmm, maybe khungtaskd should be run using SCHED_IDLE.
>
>> # echo 0 > /proc/sys/kernel/hung_task_check_count
>>
>> which didn't have the intended response. Change to a post increment of
>> max_count, so a value of 0 means check 0 tasks.
>>
>> Signed-off-by: Anton Blanchard <anton@samba.org>
>> ---
>>
>> Index: linux.trees.git/kernel/hung_task.c
>> ===================================================================
>> --- linux.trees.git.orig/kernel/hung_task.c   2009-11-27 13:11:46.000000000 +1100
>> +++ linux.trees.git/kernel/hung_task.c        2009-11-27 13:11:57.000000000 +1100
>> @@ -144,7 +144,7 @@ static void check_hung_uninterruptible_t
>>
>>       rcu_read_lock();
>>       do_each_thread(g, t) {
>> -             if (!--max_count)
>> +             if (!max_count--)
>>                       goto unlock;
>>               if (!--batch_count) {
>>                       batch_count = HUNG_TASK_BATCHING;
>