[BUG] perf_events: NMI watchdog event cannot be throttled

[BUG] perf_events: NMI watchdog event cannot be throttled

[Stephane Eranian]

Hi,

I ran into some issue  with the NMI watchdog not firing in a deadlock
situation. After some debugging I found the source of the problem.

The NMI watchdog is currently subject, like any other events, to interrupt
throttling. The heart of the problem is that if you are deadlocked on a CPU
with interrupts masked, the timer interrupt won't fire, therefore the
hwc->interrupts
field won't be reset. Then, depending on the max sampling rate, you
could eventually
fail the max interrupt rate test in __pfm_overflow_handler() and
perf_events would
throttle, i.e., stop, the NMI watchdog event before the 5s delay to panic.
Thus, you would never get the panic. I ran into this problem myself.

This is a serious issue because perf_events must ensure the watchdog can
always fire, regardless of the interrupt masking situation.

Look like one way of solving the problem would be to mark the NMI watchdog
event as immune to throttling. The event being internal to the kernel we could
trust the event setup from perf_event_create_kernel_counter().

Re: [BUG] perf_events: NMI watchdog event cannot be throttled

[Peter Zijlstra]

On Wed, 2010-08-18 at 22:26 +0200, Stephane Eranian wrote:
> Hi,
> 
> I ran into some issue  with the NMI watchdog not firing in a deadlock
> situation. After some debugging I found the source of the problem.
> 
> The NMI watchdog is currently subject, like any other events, to interrupt
> throttling. The heart of the problem is that if you are deadlocked on a CPU
> with interrupts masked, the timer interrupt won't fire, therefore the
> hwc->interrupts
> field won't be reset. Then, depending on the max sampling rate, you
> could eventually
> fail the max interrupt rate test in __pfm_overflow_handler() and
> perf_events would
> throttle, i.e., stop, the NMI watchdog event before the 5s delay to panic.
> Thus, you would never get the panic. I ran into this problem myself.
> 
> This is a serious issue because perf_events must ensure the watchdog can
> always fire, regardless of the interrupt masking situation.
> 
> Look like one way of solving the problem would be to mark the NMI watchdog
> event as immune to throttling. The event being internal to the kernel we could
> trust the event setup from perf_event_create_kernel_counter().

Something like so?

---
 kernel/watchdog.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/kernel/watchdog.c b/kernel/watchdog.c
index 613bc1f..e0fe6e4 100644
--- a/kernel/watchdog.c
+++ b/kernel/watchdog.c
@@ -206,6 +206,9 @@ void watchdog_overflow_callback(struct perf_event *event, int nmi,
 		 struct perf_sample_data *data,
 		 struct pt_regs *regs)
 {
+	/* Ensure the watchdog never gets throttled. */
+	event->hw.interrupts = 0;
+
 	if (__get_cpu_var(watchdog_nmi_touch) == true) {
 		__get_cpu_var(watchdog_nmi_touch) = false;
 		return;

Re: [BUG] perf_events: NMI watchdog event cannot be throttled

[Stephane Eranian]

Yeah, that should probably fix it. Let me try it out.


On Thu, Aug 19, 2010 at 1:05 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Wed, 2010-08-18 at 22:26 +0200, Stephane Eranian wrote:
>> Hi,
>>
>> I ran into some issue  with the NMI watchdog not firing in a deadlock
>> situation. After some debugging I found the source of the problem.
>>
>> The NMI watchdog is currently subject, like any other events, to interrupt
>> throttling. The heart of the problem is that if you are deadlocked on a CPU
>> with interrupts masked, the timer interrupt won't fire, therefore the
>> hwc->interrupts
>> field won't be reset. Then, depending on the max sampling rate, you
>> could eventually
>> fail the max interrupt rate test in __pfm_overflow_handler() and
>> perf_events would
>> throttle, i.e., stop, the NMI watchdog event before the 5s delay to panic.
>> Thus, you would never get the panic. I ran into this problem myself.
>>
>> This is a serious issue because perf_events must ensure the watchdog can
>> always fire, regardless of the interrupt masking situation.
>>
>> Look like one way of solving the problem would be to mark the NMI watchdog
>> event as immune to throttling. The event being internal to the kernel we could
>> trust the event setup from perf_event_create_kernel_counter().
>
> Something like so?
>
> ---
>  kernel/watchdog.c |    3 +++
>  1 files changed, 3 insertions(+), 0 deletions(-)
>
> diff --git a/kernel/watchdog.c b/kernel/watchdog.c
> index 613bc1f..e0fe6e4 100644
> --- a/kernel/watchdog.c
> +++ b/kernel/watchdog.c
> @@ -206,6 +206,9 @@ void watchdog_overflow_callback(struct perf_event *event, int nmi,
>                 struct perf_sample_data *data,
>                 struct pt_regs *regs)
>  {
> +       /* Ensure the watchdog never gets throttled. */
> +       event->hw.interrupts = 0;
> +
>        if (__get_cpu_var(watchdog_nmi_touch) == true) {
>                __get_cpu_var(watchdog_nmi_touch) = false;
>                return;
>
>

Re: [BUG] perf_events: NMI watchdog event cannot be throttled

[Stephane Eranian]

On Thu, Aug 19, 2010 at 1:24 PM, Stephane Eranian <eranian@google.com> wrote:
> Yeah, that should probably fix it. Let me try it out.
>
Works for me.
Thanks.

>
> On Thu, Aug 19, 2010 at 1:05 PM, Peter Zijlstra <peterz@infradead.org> wrote:
>> On Wed, 2010-08-18 at 22:26 +0200, Stephane Eranian wrote:
>>> Hi,
>>>
>>> I ran into some issue  with the NMI watchdog not firing in a deadlock
>>> situation. After some debugging I found the source of the problem.
>>>
>>> The NMI watchdog is currently subject, like any other events, to interrupt
>>> throttling. The heart of the problem is that if you are deadlocked on a CPU
>>> with interrupts masked, the timer interrupt won't fire, therefore the
>>> hwc->interrupts
>>> field won't be reset. Then, depending on the max sampling rate, you
>>> could eventually
>>> fail the max interrupt rate test in __pfm_overflow_handler() and
>>> perf_events would
>>> throttle, i.e., stop, the NMI watchdog event before the 5s delay to panic.
>>> Thus, you would never get the panic. I ran into this problem myself.
>>>
>>> This is a serious issue because perf_events must ensure the watchdog can
>>> always fire, regardless of the interrupt masking situation.
>>>
>>> Look like one way of solving the problem would be to mark the NMI watchdog
>>> event as immune to throttling. The event being internal to the kernel we could
>>> trust the event setup from perf_event_create_kernel_counter().
>>
>> Something like so?
>>
>> ---
>>  kernel/watchdog.c |    3 +++
>>  1 files changed, 3 insertions(+), 0 deletions(-)
>>
>> diff --git a/kernel/watchdog.c b/kernel/watchdog.c
>> index 613bc1f..e0fe6e4 100644
>> --- a/kernel/watchdog.c
>> +++ b/kernel/watchdog.c
>> @@ -206,6 +206,9 @@ void watchdog_overflow_callback(struct perf_event *event, int nmi,
>>                 struct perf_sample_data *data,
>>                 struct pt_regs *regs)
>>  {
>> +       /* Ensure the watchdog never gets throttled. */
>> +       event->hw.interrupts = 0;
>> +
>>        if (__get_cpu_var(watchdog_nmi_touch) == true) {
>>                __get_cpu_var(watchdog_nmi_touch) = false;
>>                return;
>>
>>
>

[tip:perf/urgent] watchdog: Don't throttle the watchdog

[tip-bot for Peter Zijlstra]

Commit-ID:  b847b94fe2caff0f28a99af6353e9a27282e771a
Gitweb:     http://git.kernel.org/tip/b847b94fe2caff0f28a99af6353e9a27282e771a
Author:     Peter Zijlstra <a.p.zijlstra@chello.nl>
AuthorDate: Fri, 20 Aug 2010 11:49:15 +0200
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Fri, 20 Aug 2010 15:00:06 +0200

watchdog: Don't throttle the watchdog

Stephane reported that when the machine locks up, the regular ticks,
which are responsible to resetting the throttle count, stop too.

Hence the NMI watchdog can end up being throttled before it reports on
the locked up state, and we end up being sad..

Cure this by having the watchdog overflow reset its own throttle count.

Reported-by: Stephane Eranian <eranian@google.com>
Tested-by: Stephane Eranian <eranian@google.com>
Cc: Don Zickus <dzickus@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <1282215916.1926.4696.camel@laptop>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 kernel/watchdog.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/kernel/watchdog.c b/kernel/watchdog.c
index 613bc1f..0d53c8e 100644
--- a/kernel/watchdog.c
+++ b/kernel/watchdog.c
@@ -206,6 +206,9 @@ void watchdog_overflow_callback(struct perf_event *event, int nmi,
 		 struct perf_sample_data *data,
 		 struct pt_regs *regs)
 {
+	/* Ensure the watchdog never gets throttled */
+	event->hw.interrupts = 0;
+
 	if (__get_cpu_var(watchdog_nmi_touch) == true) {
 		__get_cpu_var(watchdog_nmi_touch) = false;
 		return;

[tip:perf/urgent] watchdog: Don't throttle the watchdog

[tip-bot for Peter Zijlstra]

Commit-ID:  c6db67cda735d8ace5f19c3831240e1408679790
Gitweb:     http://git.kernel.org/tip/c6db67cda735d8ace5f19c3831240e1408679790
Author:     Peter Zijlstra <a.p.zijlstra@chello.nl>
AuthorDate: Fri, 20 Aug 2010 11:49:15 +0200
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Mon, 23 Aug 2010 10:48:05 +0200

watchdog: Don't throttle the watchdog

Stephane reported that when the machine locks up, the regular ticks,
which are responsible to resetting the throttle count, stop too.

Hence the NMI watchdog can end up being throttled before it reports on
the locked up state, and we end up being sad..

Cure this by having the watchdog overflow reset its own throttle count.

Reported-by: Stephane Eranian <eranian@google.com>
Tested-by: Stephane Eranian <eranian@google.com>
Cc: Don Zickus <dzickus@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <1282215916.1926.4696.camel@laptop>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 kernel/watchdog.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/kernel/watchdog.c b/kernel/watchdog.c
index 613bc1f..0d53c8e 100644
--- a/kernel/watchdog.c
+++ b/kernel/watchdog.c
@@ -206,6 +206,9 @@ void watchdog_overflow_callback(struct perf_event *event, int nmi,
 		 struct perf_sample_data *data,
 		 struct pt_regs *regs)
 {
+	/* Ensure the watchdog never gets throttled */
+	event->hw.interrupts = 0;
+
 	if (__get_cpu_var(watchdog_nmi_touch) == true) {
 		__get_cpu_var(watchdog_nmi_touch) = false;
 		return;