* [PATCH] tracing: Have osnoise_main() add a quiescent state for task rcu
@ 2021-06-28 15:49 Steven Rostedt
2021-06-28 15:58 ` Paul E. McKenney
2021-06-28 16:25 ` Daniel Bristot de Oliveira
0 siblings, 2 replies; 3+ messages in thread
From: Steven Rostedt @ 2021-06-28 15:49 UTC (permalink / raw)
To: LKML; +Cc: Daniel Bristot de Oliveira, Paul E. McKenney, Ingo Molnar
From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>
ftracetest triggered:
INFO: rcu_tasks detected stalls on tasks:
00000000b92b832d: .. nvcsw: 1/1 holdout: 1 idle_cpu: -1/7
task:osnoise/7 state:R running task stack: 0 pid: 2133 ppid: 2 flags:0x00004000
Call Trace:
? asm_sysvec_apic_timer_interrupt+0x12/0x20
? asm_sysvec_apic_timer_interrupt+0x12/0x20
? trace_hardirqs_on+0x2b/0xe0
? asm_sysvec_apic_timer_interrupt+0x12/0x20
? trace_clock_local+0xc/0x20
? osnoise_main+0x10e/0x450
? trace_softirq_entry_callback+0x50/0x50
? kthread+0x153/0x170
? __kthread_bind_mask+0x60/0x60
? ret_from_fork+0x22/0x30
While running osnoise tracer with other tracers that rely on
synchronize_rcu_tasks(), where that just hung.
The reason is that osnoise_main() never schedules out if the interval
is less than 1, and this will cause synchronize_rcu_tasks() to never
return.
Fixes: bce29ac9ce0bb ("trace: Add osnoise tracer")
Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
kernel/trace/trace_osnoise.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c
index 38aa5e208ffd..556d530af805 100644
--- a/kernel/trace/trace_osnoise.c
+++ b/kernel/trace/trace_osnoise.c
@@ -1216,8 +1216,11 @@ static int osnoise_main(void *data)
* differently from hwlat_detector, the osnoise tracer can run
* without a pause because preemption is on.
*/
- if (interval < 1)
+ if (interval < 1) {
+ /* Let synchronize_rcu_tasks() make progress */
+ cond_resched_tasks_rcu_qs();
continue;
+ }
if (msleep_interruptible(interval))
break;
--
2.29.2
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH] tracing: Have osnoise_main() add a quiescent state for task rcu
2021-06-28 15:49 [PATCH] tracing: Have osnoise_main() add a quiescent state for task rcu Steven Rostedt
@ 2021-06-28 15:58 ` Paul E. McKenney
2021-06-28 16:25 ` Daniel Bristot de Oliveira
1 sibling, 0 replies; 3+ messages in thread
From: Paul E. McKenney @ 2021-06-28 15:58 UTC (permalink / raw)
To: Steven Rostedt; +Cc: LKML, Daniel Bristot de Oliveira, Ingo Molnar
On Mon, Jun 28, 2021 at 11:49:53AM -0400, Steven Rostedt wrote:
> From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>
>
> ftracetest triggered:
>
> INFO: rcu_tasks detected stalls on tasks:
> 00000000b92b832d: .. nvcsw: 1/1 holdout: 1 idle_cpu: -1/7
> task:osnoise/7 state:R running task stack: 0 pid: 2133 ppid: 2 flags:0x00004000
> Call Trace:
> ? asm_sysvec_apic_timer_interrupt+0x12/0x20
> ? asm_sysvec_apic_timer_interrupt+0x12/0x20
> ? trace_hardirqs_on+0x2b/0xe0
> ? asm_sysvec_apic_timer_interrupt+0x12/0x20
> ? trace_clock_local+0xc/0x20
> ? osnoise_main+0x10e/0x450
> ? trace_softirq_entry_callback+0x50/0x50
> ? kthread+0x153/0x170
> ? __kthread_bind_mask+0x60/0x60
> ? ret_from_fork+0x22/0x30
>
> While running osnoise tracer with other tracers that rely on
> synchronize_rcu_tasks(), where that just hung.
>
> The reason is that osnoise_main() never schedules out if the interval
> is less than 1, and this will cause synchronize_rcu_tasks() to never
> return.
>
> Fixes: bce29ac9ce0bb ("trace: Add osnoise tracer")
> Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
That looks like something I would do!
Acked-by: Paul E. McKenney <paulmck@kernel.org>
> ---
> kernel/trace/trace_osnoise.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c
> index 38aa5e208ffd..556d530af805 100644
> --- a/kernel/trace/trace_osnoise.c
> +++ b/kernel/trace/trace_osnoise.c
> @@ -1216,8 +1216,11 @@ static int osnoise_main(void *data)
> * differently from hwlat_detector, the osnoise tracer can run
> * without a pause because preemption is on.
> */
> - if (interval < 1)
> + if (interval < 1) {
> + /* Let synchronize_rcu_tasks() make progress */
> + cond_resched_tasks_rcu_qs();
> continue;
> + }
>
> if (msleep_interruptible(interval))
> break;
> --
> 2.29.2
>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH] tracing: Have osnoise_main() add a quiescent state for task rcu
2021-06-28 15:49 [PATCH] tracing: Have osnoise_main() add a quiescent state for task rcu Steven Rostedt
2021-06-28 15:58 ` Paul E. McKenney
@ 2021-06-28 16:25 ` Daniel Bristot de Oliveira
1 sibling, 0 replies; 3+ messages in thread
From: Daniel Bristot de Oliveira @ 2021-06-28 16:25 UTC (permalink / raw)
To: Steven Rostedt, LKML; +Cc: Paul E. McKenney, Ingo Molnar
On 6/28/21 5:49 PM, Steven Rostedt wrote:
> From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>
>
> ftracetest triggered:
>
> INFO: rcu_tasks detected stalls on tasks:
> 00000000b92b832d: .. nvcsw: 1/1 holdout: 1 idle_cpu: -1/7
> task:osnoise/7 state:R running task stack: 0 pid: 2133 ppid: 2 flags:0x00004000
> Call Trace:
> ? asm_sysvec_apic_timer_interrupt+0x12/0x20
> ? asm_sysvec_apic_timer_interrupt+0x12/0x20
> ? trace_hardirqs_on+0x2b/0xe0
> ? asm_sysvec_apic_timer_interrupt+0x12/0x20
> ? trace_clock_local+0xc/0x20
> ? osnoise_main+0x10e/0x450
> ? trace_softirq_entry_callback+0x50/0x50
> ? kthread+0x153/0x170
> ? __kthread_bind_mask+0x60/0x60
> ? ret_from_fork+0x22/0x30
>
> While running osnoise tracer with other tracers that rely on
> synchronize_rcu_tasks(), where that just hung.
>
> The reason is that osnoise_main() never schedules out if the interval
> is less than 1,
and there is no other task setting need resched (it has a regular cond_resched()
on the run_osnoise() loop).
and this will cause synchronize_rcu_tasks() to never
> return.
>
> Fixes: bce29ac9ce0bb ("trace: Add osnoise tracer")
> Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
Reviewed-by: Daniel Bristot de Oliveira <bristot@redhat.com>
Thanks!
-- Daniel
> ---
> kernel/trace/trace_osnoise.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c
> index 38aa5e208ffd..556d530af805 100644
> --- a/kernel/trace/trace_osnoise.c
> +++ b/kernel/trace/trace_osnoise.c
> @@ -1216,8 +1216,11 @@ static int osnoise_main(void *data)
> * differently from hwlat_detector, the osnoise tracer can run
> * without a pause because preemption is on.
> */
> - if (interval < 1)
> + if (interval < 1) {
> + /* Let synchronize_rcu_tasks() make progress */
> + cond_resched_tasks_rcu_qs();
> continue;
> + }
>
> if (msleep_interruptible(interval))
> break;
>
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-06-28 16:25 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-28 15:49 [PATCH] tracing: Have osnoise_main() add a quiescent state for task rcu Steven Rostedt
2021-06-28 15:58 ` Paul E. McKenney
2021-06-28 16:25 ` Daniel Bristot de Oliveira
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).