* Re: PATCH: softlockup: Fix hung_task_check_count sysctl
2009-11-27 2:28 PATCH: softlockup: Fix hung_task_check_count sysctl Anton Blanchard
@ 2009-11-27 2:40 ` Frederic Weisbecker
2009-11-27 2:46 ` Américo Wang
` (2 subsequent siblings)
3 siblings, 0 replies; 9+ messages in thread
From: Frederic Weisbecker @ 2009-11-27 2:40 UTC (permalink / raw)
To: Anton Blanchard; +Cc: mingo, linux-kernel, msb
On Fri, Nov 27, 2009 at 01:28:20PM +1100, Anton Blanchard wrote:
>
> I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce
> this source of jitter I tried setting hung_task_check_count to 0:
>
> # echo 0 > /proc/sys/kernel/hung_task_check_count
>
> which didn't have the intended response. Change to a post increment of
> max_count, so a value of 0 means check 0 tasks.
>
> Signed-off-by: Anton Blanchard <anton@samba.org>
Acked-by: Frederic Weisbecker <fweisbec@gmail.com>
> ---
>
> Index: linux.trees.git/kernel/hung_task.c
> ===================================================================
> --- linux.trees.git.orig/kernel/hung_task.c 2009-11-27 13:11:46.000000000 +1100
> +++ linux.trees.git/kernel/hung_task.c 2009-11-27 13:11:57.000000000 +1100
> @@ -144,7 +144,7 @@ static void check_hung_uninterruptible_t
>
> rcu_read_lock();
> do_each_thread(g, t) {
> - if (!--max_count)
> + if (!max_count--)
> goto unlock;
> if (!--batch_count) {
> batch_count = HUNG_TASK_BATCHING;
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: PATCH: softlockup: Fix hung_task_check_count sysctl
2009-11-27 2:28 PATCH: softlockup: Fix hung_task_check_count sysctl Anton Blanchard
2009-11-27 2:40 ` Frederic Weisbecker
@ 2009-11-27 2:46 ` Américo Wang
2009-11-27 2:55 ` Frederic Weisbecker
2009-11-27 5:48 ` [tip:core/softlockup] " tip-bot for Anton Blanchard
2009-11-30 19:13 ` PATCH: " Mandeep Singh Baines
3 siblings, 1 reply; 9+ messages in thread
From: Américo Wang @ 2009-11-27 2:46 UTC (permalink / raw)
To: Anton Blanchard; +Cc: mingo, linux-kernel, msb, fweisbec
On Fri, Nov 27, 2009 at 10:28 AM, Anton Blanchard <anton@samba.org> wrote:
>
> I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce
> this source of jitter I tried setting hung_task_check_count to 0:
>
> # echo 0 > /proc/sys/kernel/hung_task_check_count
>
> which didn't have the intended response. Change to a post increment of
> max_count, so a value of 0 means check 0 tasks.
>
> Signed-off-by: Anton Blanchard <anton@samba.org>
Ack.
I would also suggest to make 'max_count' as unsigned long,
since sysctl_hung_task_check_count is.
Thanks.
> ---
>
> Index: linux.trees.git/kernel/hung_task.c
> ===================================================================
> --- linux.trees.git.orig/kernel/hung_task.c 2009-11-27 13:11:46.000000000 +1100
> +++ linux.trees.git/kernel/hung_task.c 2009-11-27 13:11:57.000000000 +1100
> @@ -144,7 +144,7 @@ static void check_hung_uninterruptible_t
>
> rcu_read_lock();
> do_each_thread(g, t) {
> - if (!--max_count)
> + if (!max_count--)
> goto unlock;
> if (!--batch_count) {
> batch_count = HUNG_TASK_BATCHING;
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: PATCH: softlockup: Fix hung_task_check_count sysctl
2009-11-27 2:46 ` Américo Wang
@ 2009-11-27 2:55 ` Frederic Weisbecker
2009-11-27 3:02 ` Américo Wang
0 siblings, 1 reply; 9+ messages in thread
From: Frederic Weisbecker @ 2009-11-27 2:55 UTC (permalink / raw)
To: Américo Wang; +Cc: Anton Blanchard, mingo, linux-kernel, msb
On Fri, Nov 27, 2009 at 10:46:53AM +0800, Américo Wang wrote:
> On Fri, Nov 27, 2009 at 10:28 AM, Anton Blanchard <anton@samba.org> wrote:
> >
> > I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce
> > this source of jitter I tried setting hung_task_check_count to 0:
> >
> > # echo 0 > /proc/sys/kernel/hung_task_check_count
> >
> > which didn't have the intended response. Change to a post increment of
> > max_count, so a value of 0 means check 0 tasks.
> >
> > Signed-off-by: Anton Blanchard <anton@samba.org>
>
>
> Ack.
>
> I would also suggest to make 'max_count' as unsigned long,
> since sysctl_hung_task_check_count is.
>
> Thanks.
Also, the batch_count thing should be dropped I think.
This is a hardcoded, not overridable pause after 1024
threads checks to avoid latencies caused by rcu_read_lock.
But now we have PREEMPT_RCU so people can enable it if
they care about latency. We should remove it as it adds
unnecessary complexity.
I'm preparing a patch for that, on top of Anton patch.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: PATCH: softlockup: Fix hung_task_check_count sysctl
2009-11-27 2:55 ` Frederic Weisbecker
@ 2009-11-27 3:02 ` Américo Wang
2009-11-27 4:22 ` Frederic Weisbecker
0 siblings, 1 reply; 9+ messages in thread
From: Américo Wang @ 2009-11-27 3:02 UTC (permalink / raw)
To: Frederic Weisbecker; +Cc: Anton Blanchard, mingo, linux-kernel, msb
On Fri, Nov 27, 2009 at 10:55 AM, Frederic Weisbecker
<fweisbec@gmail.com> wrote:
> On Fri, Nov 27, 2009 at 10:46:53AM +0800, Américo Wang wrote:
>> On Fri, Nov 27, 2009 at 10:28 AM, Anton Blanchard <anton@samba.org> wrote:
>> >
>> > I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce
>> > this source of jitter I tried setting hung_task_check_count to 0:
>> >
>> > # echo 0 > /proc/sys/kernel/hung_task_check_count
>> >
>> > which didn't have the intended response. Change to a post increment of
>> > max_count, so a value of 0 means check 0 tasks.
>> >
>> > Signed-off-by: Anton Blanchard <anton@samba.org>
>>
>>
>> Ack.
>>
>> I would also suggest to make 'max_count' as unsigned long,
>> since sysctl_hung_task_check_count is.
>>
>> Thanks.
>
>
> Also, the batch_count thing should be dropped I think.
> This is a hardcoded, not overridable pause after 1024
> threads checks to avoid latencies caused by rcu_read_lock.
> But now we have PREEMPT_RCU so people can enable it if
> they care about latency. We should remove it as it adds
> unnecessary complexity.
This sounds OK for me.
>
> I'm preparing a patch for that, on top of Anton patch.
>
Great!
Thanks.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: PATCH: softlockup: Fix hung_task_check_count sysctl
2009-11-27 3:02 ` Américo Wang
@ 2009-11-27 4:22 ` Frederic Weisbecker
0 siblings, 0 replies; 9+ messages in thread
From: Frederic Weisbecker @ 2009-11-27 4:22 UTC (permalink / raw)
To: Américo Wang; +Cc: Anton Blanchard, mingo, linux-kernel, msb
On Fri, Nov 27, 2009 at 11:02:13AM +0800, Américo Wang wrote:
> On Fri, Nov 27, 2009 at 10:55 AM, Frederic Weisbecker
> <fweisbec@gmail.com> wrote:
> > On Fri, Nov 27, 2009 at 10:46:53AM +0800, Américo Wang wrote:
> >> On Fri, Nov 27, 2009 at 10:28 AM, Anton Blanchard <anton@samba.org> wrote:
> >> >
> >> > I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce
> >> > this source of jitter I tried setting hung_task_check_count to 0:
> >> >
> >> > # echo 0 > /proc/sys/kernel/hung_task_check_count
> >> >
> >> > which didn't have the intended response. Change to a post increment of
> >> > max_count, so a value of 0 means check 0 tasks.
> >> >
> >> > Signed-off-by: Anton Blanchard <anton@samba.org>
> >>
> >>
> >> Ack.
> >>
> >> I would also suggest to make 'max_count' as unsigned long,
> >> since sysctl_hung_task_check_count is.
> >>
> >> Thanks.
> >
> >
> > Also, the batch_count thing should be dropped I think.
> > This is a hardcoded, not overridable pause after 1024
> > threads checks to avoid latencies caused by rcu_read_lock.
> > But now we have PREEMPT_RCU so people can enable it if
> > they care about latency. We should remove it as it adds
> > unnecessary complexity.
>
> This sounds OK for me.
>
> >
> > I'm preparing a patch for that, on top of Anton patch.
> >
>
> Great!
>
> Thanks.
Actually it looks like this check is not only there to take
care of latency but also to avoid too much waiting before
a grace period.
Hm, well, I'm not sure what to do.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [tip:core/softlockup] softlockup: Fix hung_task_check_count sysctl
2009-11-27 2:28 PATCH: softlockup: Fix hung_task_check_count sysctl Anton Blanchard
2009-11-27 2:40 ` Frederic Weisbecker
2009-11-27 2:46 ` Américo Wang
@ 2009-11-27 5:48 ` tip-bot for Anton Blanchard
2009-11-30 19:13 ` PATCH: " Mandeep Singh Baines
3 siblings, 0 replies; 9+ messages in thread
From: tip-bot for Anton Blanchard @ 2009-11-27 5:48 UTC (permalink / raw)
To: linux-tip-commits; +Cc: linux-kernel, anton, hpa, mingo, fweisbec, tglx, mingo
Commit-ID: e5af02261668350b43eb7381648930bde8e872f7
Gitweb: http://git.kernel.org/tip/e5af02261668350b43eb7381648930bde8e872f7
Author: Anton Blanchard <anton@samba.org>
AuthorDate: Fri, 27 Nov 2009 13:28:20 +1100
Committer: Ingo Molnar <mingo@elte.hu>
CommitDate: Fri, 27 Nov 2009 06:21:57 +0100
softlockup: Fix hung_task_check_count sysctl
I'm seeing spikes of up to 0.5ms in khungtaskd on a large
machine. To reduce this source of jitter I tried setting
hung_task_check_count to 0:
# echo 0 > /proc/sys/kernel/hung_task_check_count
which didn't have the intended response. Change to a post
increment of max_count, so a value of 0 means check 0 tasks.
Signed-off-by: Anton Blanchard <anton@samba.org>
Acked-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: msb@google.com
LKML-Reference: <20091127022820.GU32182@kryten>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
kernel/hung_task.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index d4e8417..0c642d5 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -144,7 +144,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
rcu_read_lock();
do_each_thread(g, t) {
- if (!--max_count)
+ if (!max_count--)
goto unlock;
if (!--batch_count) {
batch_count = HUNG_TASK_BATCHING;
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: PATCH: softlockup: Fix hung_task_check_count sysctl
2009-11-27 2:28 PATCH: softlockup: Fix hung_task_check_count sysctl Anton Blanchard
` (2 preceding siblings ...)
2009-11-27 5:48 ` [tip:core/softlockup] " tip-bot for Anton Blanchard
@ 2009-11-30 19:13 ` Mandeep Singh Baines
2009-11-30 20:21 ` Mandeep Baines
3 siblings, 1 reply; 9+ messages in thread
From: Mandeep Singh Baines @ 2009-11-30 19:13 UTC (permalink / raw)
To: Anton Blanchard; +Cc: mingo, linux-kernel, fweisbec
To disable hung_task at runtime:
# echo 0 > /proc/sys/kernel/hung_task_check_count
33 /*
34 * Zero means infinite timeout - no checking done:
35 */
36 unsigned long __read_mostly sysctl_hung_task_timeout_secs = 120;
For hung_task_check_count, 0 means there is no maximum so all tasks
are checked.
Anton Blanchard (anton@samba.org) wrote:
>
> I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce
> this source of jitter I tried setting hung_task_check_count to 0:
>
Hmm, maybe khungtaskd should be run using SCHED_IDLE.
> # echo 0 > /proc/sys/kernel/hung_task_check_count
>
> which didn't have the intended response. Change to a post increment of
> max_count, so a value of 0 means check 0 tasks.
>
> Signed-off-by: Anton Blanchard <anton@samba.org>
> ---
>
> Index: linux.trees.git/kernel/hung_task.c
> ===================================================================
> --- linux.trees.git.orig/kernel/hung_task.c 2009-11-27 13:11:46.000000000 +1100
> +++ linux.trees.git/kernel/hung_task.c 2009-11-27 13:11:57.000000000 +1100
> @@ -144,7 +144,7 @@ static void check_hung_uninterruptible_t
>
> rcu_read_lock();
> do_each_thread(g, t) {
> - if (!--max_count)
> + if (!max_count--)
> goto unlock;
> if (!--batch_count) {
> batch_count = HUNG_TASK_BATCHING;
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: PATCH: softlockup: Fix hung_task_check_count sysctl
2009-11-30 19:13 ` PATCH: " Mandeep Singh Baines
@ 2009-11-30 20:21 ` Mandeep Baines
0 siblings, 0 replies; 9+ messages in thread
From: Mandeep Baines @ 2009-11-30 20:21 UTC (permalink / raw)
To: Anton Blanchard; +Cc: mingo, linux-kernel, fweisbec
On Mon, Nov 30, 2009 at 11:13 AM, Mandeep Singh Baines <msb@google.com> wrote:
> To disable hung_task at runtime:
>
> # echo 0 > /proc/sys/kernel/hung_task_check_count
>
D'oh. Copy and paste error. I meant
# echo 0 > /proc/sys/kernel/hung_task_timeout_secs
The nice thing about hung_task_timeout_secs=0 is that hungtaskd will
no longer run. Implementing
no checking done with hung_task_check_count=0 would result in the task
getting scheduled,
running, and then doing nothing.
> 33 /*
> 34 * Zero means infinite timeout - no checking done:
> 35 */
> 36 unsigned long __read_mostly sysctl_hung_task_timeout_secs = 120;
>
> For hung_task_check_count, 0 means there is no maximum so all tasks
> are checked.
>
> Anton Blanchard (anton@samba.org) wrote:
>>
>> I'm seeing spikes of up to 0.5ms in khungtaskd on a large machine. To reduce
>> this source of jitter I tried setting hung_task_check_count to 0:
>>
>
> Hmm, maybe khungtaskd should be run using SCHED_IDLE.
>
>> # echo 0 > /proc/sys/kernel/hung_task_check_count
>>
>> which didn't have the intended response. Change to a post increment of
>> max_count, so a value of 0 means check 0 tasks.
>>
>> Signed-off-by: Anton Blanchard <anton@samba.org>
>> ---
>>
>> Index: linux.trees.git/kernel/hung_task.c
>> ===================================================================
>> --- linux.trees.git.orig/kernel/hung_task.c 2009-11-27 13:11:46.000000000 +1100
>> +++ linux.trees.git/kernel/hung_task.c 2009-11-27 13:11:57.000000000 +1100
>> @@ -144,7 +144,7 @@ static void check_hung_uninterruptible_t
>>
>> rcu_read_lock();
>> do_each_thread(g, t) {
>> - if (!--max_count)
>> + if (!max_count--)
>> goto unlock;
>> if (!--batch_count) {
>> batch_count = HUNG_TASK_BATCHING;
>
^ permalink raw reply [flat|nested] 9+ messages in thread