* [PATCH] softlockup: Make detector be aware of task switch of processes hogging cpu
@ 2014-08-28 4:52 Don Zickus
2014-08-28 23:07 ` Andrew Morton
0 siblings, 1 reply; 9+ messages in thread
From: Don Zickus @ 2014-08-28 4:52 UTC (permalink / raw)
To: Andrew Morton; +Cc: LKML, Ingo Molnar, chai wen, Don Zickus
From: chai wen <chaiw.fnst@cn.fujitsu.com>
For now, soft lockup detector warns once for each case of process softlockup.
But the thread 'watchdog/n' may not always get the cpu at the time slot between
the task switch of two processes hogging that cpu to reset soft_watchdog_warn.
An example would be two processes hogging the cpu. Process A causes the
softlockup warning and is killed manually by a user. Process B immediately
becomes the new process hogging the cpu preventing the softlockup code from
resetting the soft_watchdog_warn variable.
This case is a false negative of "warn only once for a process", as there may
be a different process that is going to hog the cpu. Resolve this by
saving/checking the task pointer of the hogging process and use that to reset
soft_watchdog_warn too.
Signed-off-by: chai wen <chaiw.fnst@cn.fujitsu.com>
Signed-off-by: Don Zickus <dzickus@redhat.com>
---
kernel/watchdog.c | 16 +++++++++++++++-
1 files changed, 15 insertions(+), 1 deletions(-)
diff --git a/kernel/watchdog.c b/kernel/watchdog.c
index c3319bd..499f65f 100644
--- a/kernel/watchdog.c
+++ b/kernel/watchdog.c
@@ -47,6 +47,7 @@ static DEFINE_PER_CPU(bool, softlockup_touch_sync);
static DEFINE_PER_CPU(bool, soft_watchdog_warn);
static DEFINE_PER_CPU(unsigned long, hrtimer_interrupts);
static DEFINE_PER_CPU(unsigned long, soft_lockup_hrtimer_cnt);
+static DEFINE_PER_CPU(struct task_struct *, softlockup_task_ptr_saved);
#ifdef CONFIG_HARDLOCKUP_DETECTOR
static DEFINE_PER_CPU(bool, hard_watchdog_warn);
static DEFINE_PER_CPU(bool, watchdog_nmi_touch);
@@ -331,8 +332,20 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
return HRTIMER_RESTART;
/* only warn once */
- if (__this_cpu_read(soft_watchdog_warn) == true)
+ if (__this_cpu_read(soft_watchdog_warn) == true) {
+ /*
+ * Handle the case where multiple processes are
+ * causing softlockups but the duration is small
+ * enough, the softlockup detector can not reset
+ * itself in time. Use task pointers to detect this.
+ */
+ if (__this_cpu_read(softlockup_task_ptr_saved) !=
+ current) {
+ __this_cpu_write(soft_watchdog_warn, false);
+ __touch_watchdog();
+ }
return HRTIMER_RESTART;
+ }
if (softlockup_all_cpu_backtrace) {
/* Prevent multiple soft-lockup reports if one cpu is already
@@ -348,6 +361,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
printk(KERN_EMERG "BUG: soft lockup - CPU#%d stuck for %us! [%s:%d]\n",
smp_processor_id(), duration,
current->comm, task_pid_nr(current));
+ __this_cpu_write(softlockup_task_ptr_saved, current);
print_modules();
print_irqtrace_events(current);
if (regs)
--
1.7.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH] softlockup: Make detector be aware of task switch of processes hogging cpu
2014-08-28 4:52 [PATCH] softlockup: Make detector be aware of task switch of processes hogging cpu Don Zickus
@ 2014-08-28 23:07 ` Andrew Morton
2014-08-29 1:27 ` Don Zickus
0 siblings, 1 reply; 9+ messages in thread
From: Andrew Morton @ 2014-08-28 23:07 UTC (permalink / raw)
To: Don Zickus; +Cc: LKML, Ingo Molnar, chai wen
On Thu, 28 Aug 2014 00:52:24 -0400 Don Zickus <dzickus@redhat.com> wrote:
> From: chai wen <chaiw.fnst@cn.fujitsu.com>
>
> For now, soft lockup detector warns once for each case of process softlockup.
> But the thread 'watchdog/n' may not always get the cpu at the time slot between
> the task switch of two processes hogging that cpu to reset soft_watchdog_warn.
>
> An example would be two processes hogging the cpu. Process A causes the
> softlockup warning and is killed manually by a user. Process B immediately
> becomes the new process hogging the cpu preventing the softlockup code from
> resetting the soft_watchdog_warn variable.
>
> This case is a false negative of "warn only once for a process", as there may
> be a different process that is going to hog the cpu. Resolve this by
> saving/checking the task pointer of the hogging process and use that to reset
> soft_watchdog_warn too.
>
OK, this should address the PID uniqueness issue which Ingo identified.
> --- a/kernel/watchdog.c
> +++ b/kernel/watchdog.c
> @@ -47,6 +47,7 @@ static DEFINE_PER_CPU(bool, softlockup_touch_sync);
> static DEFINE_PER_CPU(bool, soft_watchdog_warn);
> static DEFINE_PER_CPU(unsigned long, hrtimer_interrupts);
> static DEFINE_PER_CPU(unsigned long, soft_lockup_hrtimer_cnt);
> +static DEFINE_PER_CPU(struct task_struct *, softlockup_task_ptr_saved);
> #ifdef CONFIG_HARDLOCKUP_DETECTOR
> static DEFINE_PER_CPU(bool, hard_watchdog_warn);
> static DEFINE_PER_CPU(bool, watchdog_nmi_touch);
> @@ -331,8 +332,20 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
> return HRTIMER_RESTART;
>
> /* only warn once */
> - if (__this_cpu_read(soft_watchdog_warn) == true)
> + if (__this_cpu_read(soft_watchdog_warn) == true) {
> + /*
> + * Handle the case where multiple processes are
> + * causing softlockups but the duration is small
> + * enough, the softlockup detector can not reset
> + * itself in time. Use task pointers to detect this.
> + */
This comment is rather hard to follow ("the duration" of what?). Can
you think of some words which are a bit more complete/clear?
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] softlockup: Make detector be aware of task switch of processes hogging cpu
2014-08-28 23:07 ` Andrew Morton
@ 2014-08-29 1:27 ` Don Zickus
0 siblings, 0 replies; 9+ messages in thread
From: Don Zickus @ 2014-08-29 1:27 UTC (permalink / raw)
To: Andrew Morton; +Cc: LKML, Ingo Molnar, chai wen
On Thu, Aug 28, 2014 at 04:07:23PM -0700, Andrew Morton wrote:
> On Thu, 28 Aug 2014 00:52:24 -0400 Don Zickus <dzickus@redhat.com> wrote:
>
> > From: chai wen <chaiw.fnst@cn.fujitsu.com>
> >
> > For now, soft lockup detector warns once for each case of process softlockup.
> > But the thread 'watchdog/n' may not always get the cpu at the time slot between
> > the task switch of two processes hogging that cpu to reset soft_watchdog_warn.
> >
> > An example would be two processes hogging the cpu. Process A causes the
> > softlockup warning and is killed manually by a user. Process B immediately
> > becomes the new process hogging the cpu preventing the softlockup code from
> > resetting the soft_watchdog_warn variable.
> >
> > This case is a false negative of "warn only once for a process", as there may
> > be a different process that is going to hog the cpu. Resolve this by
> > saving/checking the task pointer of the hogging process and use that to reset
> > soft_watchdog_warn too.
> >
>
> OK, this should address the PID uniqueness issue which Ingo identified.
>
> > --- a/kernel/watchdog.c
> > +++ b/kernel/watchdog.c
> > @@ -47,6 +47,7 @@ static DEFINE_PER_CPU(bool, softlockup_touch_sync);
> > static DEFINE_PER_CPU(bool, soft_watchdog_warn);
> > static DEFINE_PER_CPU(unsigned long, hrtimer_interrupts);
> > static DEFINE_PER_CPU(unsigned long, soft_lockup_hrtimer_cnt);
> > +static DEFINE_PER_CPU(struct task_struct *, softlockup_task_ptr_saved);
> > #ifdef CONFIG_HARDLOCKUP_DETECTOR
> > static DEFINE_PER_CPU(bool, hard_watchdog_warn);
> > static DEFINE_PER_CPU(bool, watchdog_nmi_touch);
> > @@ -331,8 +332,20 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
> > return HRTIMER_RESTART;
> >
> > /* only warn once */
> > - if (__this_cpu_read(soft_watchdog_warn) == true)
> > + if (__this_cpu_read(soft_watchdog_warn) == true) {
> > + /*
> > + * Handle the case where multiple processes are
> > + * causing softlockups but the duration is small
> > + * enough, the softlockup detector can not reset
> > + * itself in time. Use task pointers to detect this.
> > + */
>
> This comment is rather hard to follow ("the duration" of what?). Can
> you think of some words which are a bit more complete/clear?
Agreed. Does this work better?
"
/*
* When multiple processes are causing softlockups
* the softlockup detector only warns on the first
* one because the code relies on a full quiet cycle
* to re-arm. The second process prevents the
* quiet cycle and never gets reported. Use task
* pointers to detect this.
*/
Cheers,
Don
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] softlockup: make detector be aware of task switch of processes hogging cpu
2014-08-26 14:22 ` Don Zickus
@ 2014-08-27 1:33 ` Chai Wen
0 siblings, 0 replies; 9+ messages in thread
From: Chai Wen @ 2014-08-27 1:33 UTC (permalink / raw)
To: Don Zickus; +Cc: akpm, mingo, linux-kernel
On 08/26/2014 10:22 PM, Don Zickus wrote:
> On Tue, Aug 26, 2014 at 08:51:30PM +0800, Chai Wen wrote:
>> On 08/22/2014 09:58 AM, Don Zickus wrote:
>>
>>> On Thu, Aug 21, 2014 at 01:42:22PM +0800, chai wen wrote:
>>>> For now, soft lockup detector warns once for each case of process softlockup.
>>>> But the thread 'watchdog/n' may not always get the cpu at the time slot between
>>>> the task switch of two processes hogging that cpu to reset soft_watchdog_warn.
>>>>
>>>> An example would be two processes hogging the cpu. Process A causes the
>>>> softlockup warning and is killed manually by a user. Process B immediately
>>>> becomes the new process hogging the cpu preventing the softlockup code from
>>>> resetting the soft_watchdog_warn variable.
>>>>
>>>> This case is a false negative of "warn only once for a process", as there may
>>>> be a different process that is going to hog the cpu. Resolve this by
>>>> saving/checking the task pointer of the hogging process and use that to reset
>>>> soft_watchdog_warn too.
>>>>
>>>> Signed-off-by: chai wen <chaiw.fnst@cn.fujitsu.com>
>>>> Signed-off-by: Don Zickus <dzickus@redhat.com>
>>>
>>> Acked-by: Don Zickus <dzickus@redhat.com>
>>>
>>
>>
>> Hi Andrew
>>
>> Sorry for some disturbing.
>> Could you help to check and pick up this little improvement patch ?
>>
>> I am not sure which MAINTAINER I should talk to, but the original version of
>> this patch is queued to -mm tree by you, so I assume that they are in the charge of you.
>>
>>
>> thanks
>> chai wen
>
> Hi Chai,
>
> Sorry about that. Ingo asked me privately to pick this up and re-post
> with my signoff. I was converting to a new test env and was going to use this
> patch as an excuse to exercise it. That is the delay. Let me get this
> out today.
>
OK， It is kind of you to do that, thanks for your work. :)
thanks
chai wen
> Cheers,
> Don
>
>>
>>>> ---
>>>> kernel/watchdog.c | 16 +++++++++++++++-
>>>> 1 files changed, 15 insertions(+), 1 deletions(-)
>>>>
>>>> diff --git a/kernel/watchdog.c b/kernel/watchdog.c
>>>> index 0037db6..2e55620 100644
>>>> --- a/kernel/watchdog.c
>>>> +++ b/kernel/watchdog.c
>>>> @@ -42,6 +42,7 @@ static DEFINE_PER_CPU(bool, softlockup_touch_sync);
>>>> static DEFINE_PER_CPU(bool, soft_watchdog_warn);
>>>> static DEFINE_PER_CPU(unsigned long, hrtimer_interrupts);
>>>> static DEFINE_PER_CPU(unsigned long, soft_lockup_hrtimer_cnt);
>>>> +static DEFINE_PER_CPU(struct task_struct *, softlockup_task_ptr_saved);
>>>> #ifdef CONFIG_HARDLOCKUP_DETECTOR
>>>> static DEFINE_PER_CPU(bool, hard_watchdog_warn);
>>>> static DEFINE_PER_CPU(bool, watchdog_nmi_touch);
>>>> @@ -328,8 +329,20 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
>>>> return HRTIMER_RESTART;
>>>>
>>>> /* only warn once */
>>>> - if (__this_cpu_read(soft_watchdog_warn) == true)
>>>> + if (__this_cpu_read(soft_watchdog_warn) == true) {
>>>> + /*
>>>> + * Handle the case where multiple processes are
>>>> + * causing softlockups but the duration is small
>>>> + * enough, the softlockup detector can not reset
>>>> + * itself in time. Use task pointers to detect this.
>>>> + */
>>>> + if (__this_cpu_read(softlockup_task_ptr_saved) !=
>>>> + current) {
>>>> + __this_cpu_write(soft_watchdog_warn, false);
>>>> + __touch_watchdog();
>>>> + }
>>>> return HRTIMER_RESTART;
>>>> + }
>>>>
>>>> if (softlockup_all_cpu_backtrace) {
>>>> /* Prevent multiple soft-lockup reports if one cpu is already
>>>> @@ -345,6 +358,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
>>>> pr_emerg("BUG: soft lockup - CPU#%d stuck for %us! [%s:%d]\n",
>>>> smp_processor_id(), duration,
>>>> current->comm, task_pid_nr(current));
>>>> + __this_cpu_write(softlockup_task_ptr_saved, current);
>>>> print_modules();
>>>> print_irqtrace_events(current);
>>>> if (regs)
>>>> --
>>>> 1.7.1
>>>>
>>> .
>>>
>>
>>
>>
>> --
>> Regards
>>
>> Chai Wen
> .
>
--
Regards
Chai Wen
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] softlockup: make detector be aware of task switch of processes hogging cpu
2014-08-26 12:51 ` Chai Wen
@ 2014-08-26 14:22 ` Don Zickus
2014-08-27 1:33 ` Chai Wen
0 siblings, 1 reply; 9+ messages in thread
From: Don Zickus @ 2014-08-26 14:22 UTC (permalink / raw)
To: Chai Wen; +Cc: akpm, mingo, linux-kernel
On Tue, Aug 26, 2014 at 08:51:30PM +0800, Chai Wen wrote:
> On 08/22/2014 09:58 AM, Don Zickus wrote:
>
> > On Thu, Aug 21, 2014 at 01:42:22PM +0800, chai wen wrote:
> >> For now, soft lockup detector warns once for each case of process softlockup.
> >> But the thread 'watchdog/n' may not always get the cpu at the time slot between
> >> the task switch of two processes hogging that cpu to reset soft_watchdog_warn.
> >>
> >> An example would be two processes hogging the cpu. Process A causes the
> >> softlockup warning and is killed manually by a user. Process B immediately
> >> becomes the new process hogging the cpu preventing the softlockup code from
> >> resetting the soft_watchdog_warn variable.
> >>
> >> This case is a false negative of "warn only once for a process", as there may
> >> be a different process that is going to hog the cpu. Resolve this by
> >> saving/checking the task pointer of the hogging process and use that to reset
> >> soft_watchdog_warn too.
> >>
> >> Signed-off-by: chai wen <chaiw.fnst@cn.fujitsu.com>
> >> Signed-off-by: Don Zickus <dzickus@redhat.com>
> >
> > Acked-by: Don Zickus <dzickus@redhat.com>
> >
>
>
> Hi Andrew
>
> Sorry for some disturbing.
> Could you help to check and pick up this little improvement patch ?
>
> I am not sure which MAINTAINER I should talk to, but the original version of
> this patch is queued to -mm tree by you, so I assume that they are in the charge of you.
>
>
> thanks
> chai wen
Hi Chai,
Sorry about that. Ingo asked me privately to pick this up and re-post
with my signoff. I was converting to a new test env and was going to use this
patch as an excuse to exercise it. That is the delay. Let me get this
out today.
Cheers,
Don
>
> >> ---
> >> kernel/watchdog.c | 16 +++++++++++++++-
> >> 1 files changed, 15 insertions(+), 1 deletions(-)
> >>
> >> diff --git a/kernel/watchdog.c b/kernel/watchdog.c
> >> index 0037db6..2e55620 100644
> >> --- a/kernel/watchdog.c
> >> +++ b/kernel/watchdog.c
> >> @@ -42,6 +42,7 @@ static DEFINE_PER_CPU(bool, softlockup_touch_sync);
> >> static DEFINE_PER_CPU(bool, soft_watchdog_warn);
> >> static DEFINE_PER_CPU(unsigned long, hrtimer_interrupts);
> >> static DEFINE_PER_CPU(unsigned long, soft_lockup_hrtimer_cnt);
> >> +static DEFINE_PER_CPU(struct task_struct *, softlockup_task_ptr_saved);
> >> #ifdef CONFIG_HARDLOCKUP_DETECTOR
> >> static DEFINE_PER_CPU(bool, hard_watchdog_warn);
> >> static DEFINE_PER_CPU(bool, watchdog_nmi_touch);
> >> @@ -328,8 +329,20 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
> >> return HRTIMER_RESTART;
> >>
> >> /* only warn once */
> >> - if (__this_cpu_read(soft_watchdog_warn) == true)
> >> + if (__this_cpu_read(soft_watchdog_warn) == true) {
> >> + /*
> >> + * Handle the case where multiple processes are
> >> + * causing softlockups but the duration is small
> >> + * enough, the softlockup detector can not reset
> >> + * itself in time. Use task pointers to detect this.
> >> + */
> >> + if (__this_cpu_read(softlockup_task_ptr_saved) !=
> >> + current) {
> >> + __this_cpu_write(soft_watchdog_warn, false);
> >> + __touch_watchdog();
> >> + }
> >> return HRTIMER_RESTART;
> >> + }
> >>
> >> if (softlockup_all_cpu_backtrace) {
> >> /* Prevent multiple soft-lockup reports if one cpu is already
> >> @@ -345,6 +358,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
> >> pr_emerg("BUG: soft lockup - CPU#%d stuck for %us! [%s:%d]\n",
> >> smp_processor_id(), duration,
> >> current->comm, task_pid_nr(current));
> >> + __this_cpu_write(softlockup_task_ptr_saved, current);
> >> print_modules();
> >> print_irqtrace_events(current);
> >> if (regs)
> >> --
> >> 1.7.1
> >>
> > .
> >
>
>
>
> --
> Regards
>
> Chai Wen
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] softlockup: make detector be aware of task switch of processes hogging cpu
2014-08-22 1:58 ` Don Zickus
@ 2014-08-26 12:51 ` Chai Wen
2014-08-26 14:22 ` Don Zickus
0 siblings, 1 reply; 9+ messages in thread
From: Chai Wen @ 2014-08-26 12:51 UTC (permalink / raw)
To: akpm; +Cc: Don Zickus, mingo, linux-kernel
On 08/22/2014 09:58 AM, Don Zickus wrote:
> On Thu, Aug 21, 2014 at 01:42:22PM +0800, chai wen wrote:
>> For now, soft lockup detector warns once for each case of process softlockup.
>> But the thread 'watchdog/n' may not always get the cpu at the time slot between
>> the task switch of two processes hogging that cpu to reset soft_watchdog_warn.
>>
>> An example would be two processes hogging the cpu. Process A causes the
>> softlockup warning and is killed manually by a user. Process B immediately
>> becomes the new process hogging the cpu preventing the softlockup code from
>> resetting the soft_watchdog_warn variable.
>>
>> This case is a false negative of "warn only once for a process", as there may
>> be a different process that is going to hog the cpu. Resolve this by
>> saving/checking the task pointer of the hogging process and use that to reset
>> soft_watchdog_warn too.
>>
>> Signed-off-by: chai wen <chaiw.fnst@cn.fujitsu.com>
>> Signed-off-by: Don Zickus <dzickus@redhat.com>
>
> Acked-by: Don Zickus <dzickus@redhat.com>
>
Hi Andrew
Sorry for some disturbing.
Could you help to check and pick up this little improvement patch ?
I am not sure which MAINTAINER I should talk to, but the original version of
this patch is queued to -mm tree by you, so I assume that they are in the charge of you.
thanks
chai wen
>> ---
>> kernel/watchdog.c | 16 +++++++++++++++-
>> 1 files changed, 15 insertions(+), 1 deletions(-)
>>
>> diff --git a/kernel/watchdog.c b/kernel/watchdog.c
>> index 0037db6..2e55620 100644
>> --- a/kernel/watchdog.c
>> +++ b/kernel/watchdog.c
>> @@ -42,6 +42,7 @@ static DEFINE_PER_CPU(bool, softlockup_touch_sync);
>> static DEFINE_PER_CPU(bool, soft_watchdog_warn);
>> static DEFINE_PER_CPU(unsigned long, hrtimer_interrupts);
>> static DEFINE_PER_CPU(unsigned long, soft_lockup_hrtimer_cnt);
>> +static DEFINE_PER_CPU(struct task_struct *, softlockup_task_ptr_saved);
>> #ifdef CONFIG_HARDLOCKUP_DETECTOR
>> static DEFINE_PER_CPU(bool, hard_watchdog_warn);
>> static DEFINE_PER_CPU(bool, watchdog_nmi_touch);
>> @@ -328,8 +329,20 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
>> return HRTIMER_RESTART;
>>
>> /* only warn once */
>> - if (__this_cpu_read(soft_watchdog_warn) == true)
>> + if (__this_cpu_read(soft_watchdog_warn) == true) {
>> + /*
>> + * Handle the case where multiple processes are
>> + * causing softlockups but the duration is small
>> + * enough, the softlockup detector can not reset
>> + * itself in time. Use task pointers to detect this.
>> + */
>> + if (__this_cpu_read(softlockup_task_ptr_saved) !=
>> + current) {
>> + __this_cpu_write(soft_watchdog_warn, false);
>> + __touch_watchdog();
>> + }
>> return HRTIMER_RESTART;
>> + }
>>
>> if (softlockup_all_cpu_backtrace) {
>> /* Prevent multiple soft-lockup reports if one cpu is already
>> @@ -345,6 +358,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
>> pr_emerg("BUG: soft lockup - CPU#%d stuck for %us! [%s:%d]\n",
>> smp_processor_id(), duration,
>> current->comm, task_pid_nr(current));
>> + __this_cpu_write(softlockup_task_ptr_saved, current);
>> print_modules();
>> print_irqtrace_events(current);
>> if (regs)
>> --
>> 1.7.1
>>
> .
>
--
Regards
Chai Wen
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] softlockup: make detector be aware of task switch of processes hogging cpu
2014-08-21 5:42 ` [PATCH] " chai wen
2014-08-22 1:12 ` Chai Wen
@ 2014-08-22 1:58 ` Don Zickus
2014-08-26 12:51 ` Chai Wen
1 sibling, 1 reply; 9+ messages in thread
From: Don Zickus @ 2014-08-22 1:58 UTC (permalink / raw)
To: chai wen; +Cc: mingo, linux-kernel
On Thu, Aug 21, 2014 at 01:42:22PM +0800, chai wen wrote:
> For now, soft lockup detector warns once for each case of process softlockup.
> But the thread 'watchdog/n' may not always get the cpu at the time slot between
> the task switch of two processes hogging that cpu to reset soft_watchdog_warn.
>
> An example would be two processes hogging the cpu. Process A causes the
> softlockup warning and is killed manually by a user. Process B immediately
> becomes the new process hogging the cpu preventing the softlockup code from
> resetting the soft_watchdog_warn variable.
>
> This case is a false negative of "warn only once for a process", as there may
> be a different process that is going to hog the cpu. Resolve this by
> saving/checking the task pointer of the hogging process and use that to reset
> soft_watchdog_warn too.
>
> Signed-off-by: chai wen <chaiw.fnst@cn.fujitsu.com>
> Signed-off-by: Don Zickus <dzickus@redhat.com>
Acked-by: Don Zickus <dzickus@redhat.com>
> ---
> kernel/watchdog.c | 16 +++++++++++++++-
> 1 files changed, 15 insertions(+), 1 deletions(-)
>
> diff --git a/kernel/watchdog.c b/kernel/watchdog.c
> index 0037db6..2e55620 100644
> --- a/kernel/watchdog.c
> +++ b/kernel/watchdog.c
> @@ -42,6 +42,7 @@ static DEFINE_PER_CPU(bool, softlockup_touch_sync);
> static DEFINE_PER_CPU(bool, soft_watchdog_warn);
> static DEFINE_PER_CPU(unsigned long, hrtimer_interrupts);
> static DEFINE_PER_CPU(unsigned long, soft_lockup_hrtimer_cnt);
> +static DEFINE_PER_CPU(struct task_struct *, softlockup_task_ptr_saved);
> #ifdef CONFIG_HARDLOCKUP_DETECTOR
> static DEFINE_PER_CPU(bool, hard_watchdog_warn);
> static DEFINE_PER_CPU(bool, watchdog_nmi_touch);
> @@ -328,8 +329,20 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
> return HRTIMER_RESTART;
>
> /* only warn once */
> - if (__this_cpu_read(soft_watchdog_warn) == true)
> + if (__this_cpu_read(soft_watchdog_warn) == true) {
> + /*
> + * Handle the case where multiple processes are
> + * causing softlockups but the duration is small
> + * enough, the softlockup detector can not reset
> + * itself in time. Use task pointers to detect this.
> + */
> + if (__this_cpu_read(softlockup_task_ptr_saved) !=
> + current) {
> + __this_cpu_write(soft_watchdog_warn, false);
> + __touch_watchdog();
> + }
> return HRTIMER_RESTART;
> + }
>
> if (softlockup_all_cpu_backtrace) {
> /* Prevent multiple soft-lockup reports if one cpu is already
> @@ -345,6 +358,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
> pr_emerg("BUG: soft lockup - CPU#%d stuck for %us! [%s:%d]\n",
> smp_processor_id(), duration,
> current->comm, task_pid_nr(current));
> + __this_cpu_write(softlockup_task_ptr_saved, current);
> print_modules();
> print_irqtrace_events(current);
> if (regs)
> --
> 1.7.1
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] softlockup: make detector be aware of task switch of processes hogging cpu
2014-08-21 5:42 ` [PATCH] " chai wen
@ 2014-08-22 1:12 ` Chai Wen
2014-08-22 1:58 ` Don Zickus
1 sibling, 0 replies; 9+ messages in thread
From: Chai Wen @ 2014-08-22 1:12 UTC (permalink / raw)
To: mingo, Don Zickus; +Cc: chai wen, linux-kernel
On 08/21/2014 01:42 PM, chai wen wrote:
> For now, soft lockup detector warns once for each case of process softlockup.
> But the thread 'watchdog/n' may not always get the cpu at the time slot between
> the task switch of two processes hogging that cpu to reset soft_watchdog_warn.
>
> An example would be two processes hogging the cpu. Process A causes the
> softlockup warning and is killed manually by a user. Process B immediately
> becomes the new process hogging the cpu preventing the softlockup code from
> resetting the soft_watchdog_warn variable.
>
> This case is a false negative of "warn only once for a process", as there may
> be a different process that is going to hog the cpu. Resolve this by
> saving/checking the task pointer of the hogging process and use that to reset
> soft_watchdog_warn too.
>
> Signed-off-by: chai wen <chaiw.fnst@cn.fujitsu.com>
> Signed-off-by: Don Zickus <dzickus@redhat.com>
Hi Ingo & Don
Ping...
This patch is using the task pointer to check cases that softlockup can
not reset itself, and has been tested.
thanks
chai wen
> ---
> kernel/watchdog.c | 16 +++++++++++++++-
> 1 files changed, 15 insertions(+), 1 deletions(-)
>
> diff --git a/kernel/watchdog.c b/kernel/watchdog.c
> index 0037db6..2e55620 100644
> --- a/kernel/watchdog.c
> +++ b/kernel/watchdog.c
> @@ -42,6 +42,7 @@ static DEFINE_PER_CPU(bool, softlockup_touch_sync);
> static DEFINE_PER_CPU(bool, soft_watchdog_warn);
> static DEFINE_PER_CPU(unsigned long, hrtimer_interrupts);
> static DEFINE_PER_CPU(unsigned long, soft_lockup_hrtimer_cnt);
> +static DEFINE_PER_CPU(struct task_struct *, softlockup_task_ptr_saved);
> #ifdef CONFIG_HARDLOCKUP_DETECTOR
> static DEFINE_PER_CPU(bool, hard_watchdog_warn);
> static DEFINE_PER_CPU(bool, watchdog_nmi_touch);
> @@ -328,8 +329,20 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
> return HRTIMER_RESTART;
>
> /* only warn once */
> - if (__this_cpu_read(soft_watchdog_warn) == true)
> + if (__this_cpu_read(soft_watchdog_warn) == true) {
> + /*
> + * Handle the case where multiple processes are
> + * causing softlockups but the duration is small
> + * enough, the softlockup detector can not reset
> + * itself in time. Use task pointers to detect this.
> + */
> + if (__this_cpu_read(softlockup_task_ptr_saved) !=
> + current) {
> + __this_cpu_write(soft_watchdog_warn, false);
> + __touch_watchdog();
> + }
> return HRTIMER_RESTART;
> + }
>
> if (softlockup_all_cpu_backtrace) {
> /* Prevent multiple soft-lockup reports if one cpu is already
> @@ -345,6 +358,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
> pr_emerg("BUG: soft lockup - CPU#%d stuck for %us! [%s:%d]\n",
> smp_processor_id(), duration,
> current->comm, task_pid_nr(current));
> + __this_cpu_write(softlockup_task_ptr_saved, current);
> print_modules();
> print_irqtrace_events(current);
> if (regs)
--
Regards
Chai Wen
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH] softlockup: make detector be aware of task switch of processes hogging cpu
2014-08-21 2:30 [PATCH 2/5] softlockup: make " Don Zickus
@ 2014-08-21 5:42 ` chai wen
2014-08-22 1:12 ` Chai Wen
2014-08-22 1:58 ` Don Zickus
0 siblings, 2 replies; 9+ messages in thread
From: chai wen @ 2014-08-21 5:42 UTC (permalink / raw)
Cc: mingo, linux-kernel, chai wen, Don Zickus
For now, soft lockup detector warns once for each case of process softlockup.
But the thread 'watchdog/n' may not always get the cpu at the time slot between
the task switch of two processes hogging that cpu to reset soft_watchdog_warn.
An example would be two processes hogging the cpu. Process A causes the
softlockup warning and is killed manually by a user. Process B immediately
becomes the new process hogging the cpu preventing the softlockup code from
resetting the soft_watchdog_warn variable.
This case is a false negative of "warn only once for a process", as there may
be a different process that is going to hog the cpu. Resolve this by
saving/checking the task pointer of the hogging process and use that to reset
soft_watchdog_warn too.
Signed-off-by: chai wen <chaiw.fnst@cn.fujitsu.com>
Signed-off-by: Don Zickus <dzickus@redhat.com>
---
kernel/watchdog.c | 16 +++++++++++++++-
1 files changed, 15 insertions(+), 1 deletions(-)
diff --git a/kernel/watchdog.c b/kernel/watchdog.c
index 0037db6..2e55620 100644
--- a/kernel/watchdog.c
+++ b/kernel/watchdog.c
@@ -42,6 +42,7 @@ static DEFINE_PER_CPU(bool, softlockup_touch_sync);
static DEFINE_PER_CPU(bool, soft_watchdog_warn);
static DEFINE_PER_CPU(unsigned long, hrtimer_interrupts);
static DEFINE_PER_CPU(unsigned long, soft_lockup_hrtimer_cnt);
+static DEFINE_PER_CPU(struct task_struct *, softlockup_task_ptr_saved);
#ifdef CONFIG_HARDLOCKUP_DETECTOR
static DEFINE_PER_CPU(bool, hard_watchdog_warn);
static DEFINE_PER_CPU(bool, watchdog_nmi_touch);
@@ -328,8 +329,20 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
return HRTIMER_RESTART;
/* only warn once */
- if (__this_cpu_read(soft_watchdog_warn) == true)
+ if (__this_cpu_read(soft_watchdog_warn) == true) {
+ /*
+ * Handle the case where multiple processes are
+ * causing softlockups but the duration is small
+ * enough, the softlockup detector can not reset
+ * itself in time. Use task pointers to detect this.
+ */
+ if (__this_cpu_read(softlockup_task_ptr_saved) !=
+ current) {
+ __this_cpu_write(soft_watchdog_warn, false);
+ __touch_watchdog();
+ }
return HRTIMER_RESTART;
+ }
if (softlockup_all_cpu_backtrace) {
/* Prevent multiple soft-lockup reports if one cpu is already
@@ -345,6 +358,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
pr_emerg("BUG: soft lockup - CPU#%d stuck for %us! [%s:%d]\n",
smp_processor_id(), duration,
current->comm, task_pid_nr(current));
+ __this_cpu_write(softlockup_task_ptr_saved, current);
print_modules();
print_irqtrace_events(current);
if (regs)
--
1.7.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2014-08-29 1:27 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-08-28 4:52 [PATCH] softlockup: Make detector be aware of task switch of processes hogging cpu Don Zickus
2014-08-28 23:07 ` Andrew Morton
2014-08-29 1:27 ` Don Zickus
-- strict thread matches above, loose matches on Subject: below --
2014-08-21 2:30 [PATCH 2/5] softlockup: make " Don Zickus
2014-08-21 5:42 ` [PATCH] " chai wen
2014-08-22 1:12 ` Chai Wen
2014-08-22 1:58 ` Don Zickus
2014-08-26 12:51 ` Chai Wen
2014-08-26 14:22 ` Don Zickus
2014-08-27 1:33 ` Chai Wen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).