LKML Archive on lore.kernel.org
 help / Atom feed
* [PATCH] ftrace: Add missing check for existing hwlat thread
@ 2018-08-01 10:45 Erica Bugden
  2018-08-01 19:40 ` Steven Rostedt
  0 siblings, 1 reply; 8+ messages in thread
From: Erica Bugden @ 2018-08-01 10:45 UTC (permalink / raw)
  To: linux-kernel; +Cc: rostedt, peterz, tglx, anna-maria, bigeasy, Erica Bugden

The hwlat tracer uses a kernel thread to measure latencies. The function
that creates this kernel thread, start_kthread(), can be called when the
tracer is initialized and when the tracer is explicitly enabled.
start_kthread() does not check if there is an existing hwlat kernel
thread and will create a new one each time it is called.

This causes the reference to the previous thread to be lost. Without the
thread reference, the old kernel thread becomes unstoppable and
continues to use CPU time even after the hwlat tracer has been disabled.
This problem can be observed when a system is booted with tracing
enabled and the hwlat tracer is configured like this:

	echo hwlat > current_tracer; echo 1 > tracing_on

Add the missing check for an existing kernel thread in start_kthread()
to prevent this problem. This function and the rest of the hwlat kernel
thread setup and teardown are already serialized because they are called
through the tracer core code with trace_type_lock held.

Signed-off-by: Erica Bugden <erica.bugden@linutronix.de>
---
 kernel/trace/trace_hwlat.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/kernel/trace/trace_hwlat.c b/kernel/trace/trace_hwlat.c
index d7c8e4e..2d9d36d 100644
--- a/kernel/trace/trace_hwlat.c
+++ b/kernel/trace/trace_hwlat.c
@@ -354,6 +354,9 @@ static int start_kthread(struct trace_array *tr)
 	struct task_struct *kthread;
 	int next_cpu;
 
+	if (hwlat_kthread)
+		return 0;
+
 	/* Just pick the first CPU on first iteration */
 	current_mask = &save_cpumask;
 	get_online_cpus();
-- 
2.1.4


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] ftrace: Add missing check for existing hwlat thread
  2018-08-01 10:45 [PATCH] ftrace: Add missing check for existing hwlat thread Erica Bugden
@ 2018-08-01 19:40 ` Steven Rostedt
  2018-08-01 19:59   ` Thomas Gleixner
  2018-08-03  8:46   ` Erica Bugden
  0 siblings, 2 replies; 8+ messages in thread
From: Steven Rostedt @ 2018-08-01 19:40 UTC (permalink / raw)
  To: Erica Bugden; +Cc: linux-kernel, peterz, tglx, anna-maria, bigeasy

On Wed,  1 Aug 2018 12:45:54 +0200
Erica Bugden <erica.bugden@linutronix.de> wrote:

> The hwlat tracer uses a kernel thread to measure latencies. The function
> that creates this kernel thread, start_kthread(), can be called when the
> tracer is initialized and when the tracer is explicitly enabled.
> start_kthread() does not check if there is an existing hwlat kernel
> thread and will create a new one each time it is called.
> 
> This causes the reference to the previous thread to be lost. Without the
> thread reference, the old kernel thread becomes unstoppable and
> continues to use CPU time even after the hwlat tracer has been disabled.
> This problem can be observed when a system is booted with tracing
> enabled and the hwlat tracer is configured like this:
> 
> 	echo hwlat > current_tracer; echo 1 > tracing_on
> 
> Add the missing check for an existing kernel thread in start_kthread()
> to prevent this problem. This function and the rest of the hwlat kernel
> thread setup and teardown are already serialized because they are called
> through the tracer core code with trace_type_lock held.
> 
> Signed-off-by: Erica Bugden <erica.bugden@linutronix.de>
> ---
>  kernel/trace/trace_hwlat.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/kernel/trace/trace_hwlat.c b/kernel/trace/trace_hwlat.c
> index d7c8e4e..2d9d36d 100644
> --- a/kernel/trace/trace_hwlat.c
> +++ b/kernel/trace/trace_hwlat.c
> @@ -354,6 +354,9 @@ static int start_kthread(struct trace_array *tr)
>  	struct task_struct *kthread;
>  	int next_cpu;
>  
> +	if (hwlat_kthread)
> +		return 0;
> +

This looks like it is treating the symptom and not the disease.

>  	/* Just pick the first CPU on first iteration */
>  	current_mask = &save_cpumask;
>  	get_online_cpus();

Can you try this patch?

-- Steve

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 823687997b01..15862044db05 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -7628,7 +7628,9 @@ rb_simple_write(struct file *filp, const char __user *ubuf,
 
 	if (buffer) {
 		mutex_lock(&trace_types_lock);
-		if (val) {
+		if (!!val == tracer_tracing_is_on(tr)) {
+			val = 0; /* do nothing */
+		} else if (val) {
 			tracer_tracing_on(tr);
 			if (tr->current_trace->start)
 				tr->current_trace->start(tr);

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] ftrace: Add missing check for existing hwlat thread
  2018-08-01 19:40 ` Steven Rostedt
@ 2018-08-01 19:59   ` Thomas Gleixner
  2018-08-01 20:03     ` Steven Rostedt
  2018-08-03  8:46   ` Erica Bugden
  1 sibling, 1 reply; 8+ messages in thread
From: Thomas Gleixner @ 2018-08-01 19:59 UTC (permalink / raw)
  To: Steven Rostedt; +Cc: Erica Bugden, linux-kernel, peterz, anna-maria, bigeasy

On Wed, 1 Aug 2018, Steven Rostedt wrote:

> On Wed,  1 Aug 2018 12:45:54 +0200
> Erica Bugden <erica.bugden@linutronix.de> wrote:
> 
> > The hwlat tracer uses a kernel thread to measure latencies. The function
> > that creates this kernel thread, start_kthread(), can be called when the
> > tracer is initialized and when the tracer is explicitly enabled.
> > start_kthread() does not check if there is an existing hwlat kernel
> > thread and will create a new one each time it is called.
> > 
> > This causes the reference to the previous thread to be lost. Without the
> > thread reference, the old kernel thread becomes unstoppable and
> > continues to use CPU time even after the hwlat tracer has been disabled.
> > This problem can be observed when a system is booted with tracing
> > enabled and the hwlat tracer is configured like this:
> > 
> > 	echo hwlat > current_tracer; echo 1 > tracing_on
> > 
> > Add the missing check for an existing kernel thread in start_kthread()
> > to prevent this problem. This function and the rest of the hwlat kernel
> > thread setup and teardown are already serialized because they are called
> > through the tracer core code with trace_type_lock held.
> > 
> > Signed-off-by: Erica Bugden <erica.bugden@linutronix.de>
> > ---
> >  kernel/trace/trace_hwlat.c | 3 +++
> >  1 file changed, 3 insertions(+)
> > 
> > diff --git a/kernel/trace/trace_hwlat.c b/kernel/trace/trace_hwlat.c
> > index d7c8e4e..2d9d36d 100644
> > --- a/kernel/trace/trace_hwlat.c
> > +++ b/kernel/trace/trace_hwlat.c
> > @@ -354,6 +354,9 @@ static int start_kthread(struct trace_array *tr)
> >  	struct task_struct *kthread;
> >  	int next_cpu;
> >  
> > +	if (hwlat_kthread)
> > +		return 0;
> > +
> 
> This looks like it is treating the symptom and not the disease.

My bad. We looked at the other instances of tracers and they all have
protection against this kind of call sequence ...


> >  	/* Just pick the first CPU on first iteration */
> >  	current_mask = &save_cpumask;
> >  	get_online_cpus();
> 
> Can you try this patch?
> 
> -- Steve
> 
> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> index 823687997b01..15862044db05 100644
> --- a/kernel/trace/trace.c
> +++ b/kernel/trace/trace.c
> @@ -7628,7 +7628,9 @@ rb_simple_write(struct file *filp, const char __user *ubuf,
>  
>  	if (buffer) {
>  		mutex_lock(&trace_types_lock);
> -		if (val) {
> +		if (!!val == tracer_tracing_is_on(tr)) {
> +			val = 0; /* do nothing */
> +		} else if (val) {
>  			tracer_tracing_on(tr);
>  			if (tr->current_trace->start)
>  				tr->current_trace->start(tr);
> 

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] ftrace: Add missing check for existing hwlat thread
  2018-08-01 19:59   ` Thomas Gleixner
@ 2018-08-01 20:03     ` Steven Rostedt
  2018-08-01 20:07       ` Thomas Gleixner
  0 siblings, 1 reply; 8+ messages in thread
From: Steven Rostedt @ 2018-08-01 20:03 UTC (permalink / raw)
  To: Thomas Gleixner; +Cc: Erica Bugden, linux-kernel, peterz, anna-maria, bigeasy

On Wed, 1 Aug 2018 21:59:38 +0200 (CEST)
Thomas Gleixner <tglx@linutronix.de> wrote:

> > This looks like it is treating the symptom and not the disease.  
> 
> My bad. We looked at the other instances of tracers and they all have
> protection against this kind of call sequence ...

Probably because they were all fixing the symptom ;-)

I'm going to keep Erica's patch and add a WARN_ON() to it just in case.
But can you still check if the patch I gave fixes it too?

-- Steve

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] ftrace: Add missing check for existing hwlat thread
  2018-08-01 20:03     ` Steven Rostedt
@ 2018-08-01 20:07       ` Thomas Gleixner
  0 siblings, 0 replies; 8+ messages in thread
From: Thomas Gleixner @ 2018-08-01 20:07 UTC (permalink / raw)
  To: Steven Rostedt; +Cc: Erica Bugden, linux-kernel, peterz, anna-maria, bigeasy

On Wed, 1 Aug 2018, Steven Rostedt wrote:

> On Wed, 1 Aug 2018 21:59:38 +0200 (CEST)
> Thomas Gleixner <tglx@linutronix.de> wrote:
> 
> > > This looks like it is treating the symptom and not the disease.  
> > 
> > My bad. We looked at the other instances of tracers and they all have
> > protection against this kind of call sequence ...
> 
> Probably because they were all fixing the symptom ;-)

:)

> I'm going to keep Erica's patch and add a WARN_ON() to it just in case.
> But can you still check if the patch I gave fixes it too?

Sure

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] ftrace: Add missing check for existing hwlat thread
  2018-08-01 19:40 ` Steven Rostedt
  2018-08-01 19:59   ` Thomas Gleixner
@ 2018-08-03  8:46   ` Erica Bugden
  2018-08-03 13:37     ` Steven Rostedt
  1 sibling, 1 reply; 8+ messages in thread
From: Erica Bugden @ 2018-08-03  8:46 UTC (permalink / raw)
  To: Steven Rostedt; +Cc: linux-kernel, peterz, tglx, anna-maria, bigeasy

On Wed, 2018-08-01 at 15:40 -0400, Steven Rostedt wrote:
> On Wed,  1 Aug 2018 12:45:54 +0200
> > Erica Bugden <erica.bugden@linutronix.de> wrote:
> 
> > The hwlat tracer uses a kernel thread to measure latencies. The function
> > that creates this kernel thread, start_kthread(), can be called when the
> > tracer is initialized and when the tracer is explicitly enabled.
> > start_kthread() does not check if there is an existing hwlat kernel
> > thread and will create a new one each time it is called.
> > 
> > This causes the reference to the previous thread to be lost. Without the
> > thread reference, the old kernel thread becomes unstoppable and
> > continues to use CPU time even after the hwlat tracer has been disabled.
> > This problem can be observed when a system is booted with tracing
> > enabled and the hwlat tracer is configured like this:
> > 
> > 	echo hwlat > current_tracer; echo 1 > tracing_on
> > 
> > Add the missing check for an existing kernel thread in start_kthread()
> > to prevent this problem. This function and the rest of the hwlat kernel
> > thread setup and teardown are already serialized because they are called
> > through the tracer core code with trace_type_lock held.
> > 
> > > > Signed-off-by: Erica Bugden <erica.bugden@linutronix.de>
> > ---
> >  kernel/trace/trace_hwlat.c | 3 +++
> >  1 file changed, 3 insertions(+)
> > 
> > diff --git a/kernel/trace/trace_hwlat.c b/kernel/trace/trace_hwlat.c
> > index d7c8e4e..2d9d36d 100644
> > --- a/kernel/trace/trace_hwlat.c
> > +++ b/kernel/trace/trace_hwlat.c
> > @@ -354,6 +354,9 @@ static int start_kthread(struct trace_array *tr)
> > > >  	struct task_struct *kthread;
> > > >  	int next_cpu;
> >  
> > > > +	if (hwlat_kthread)
> > > > +		return 0;
> > +
> 
> This looks like it is treating the symptom and not the disease.
> 
> > > >  	/* Just pick the first CPU on first iteration */
> > > >  	current_mask = &save_cpumask;
> > > >  	get_online_cpus();
> 
> Can you try this patch?

I tested the patch below and it also fixes the problem.

> 
> -- Steve
> 
> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> index 823687997b01..15862044db05 100644
> --- a/kernel/trace/trace.c
> +++ b/kernel/trace/trace.c
> @@ -7628,7 +7628,9 @@ rb_simple_write(struct file *filp, const char __user *ubuf,
>  
>  	if (buffer) {
>  		mutex_lock(&trace_types_lock);
> -		if (val) {
> +		if (!!val == tracer_tracing_is_on(tr)) {
> +			val = 0; /* do nothing */
> +		} else if (val) {
>  			tracer_tracing_on(tr);
>  			if (tr->current_trace->start)
>  				tr->current_trace->start(tr);

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] ftrace: Add missing check for existing hwlat thread
  2018-08-03  8:46   ` Erica Bugden
@ 2018-08-03 13:37     ` Steven Rostedt
  2018-08-03 14:57       ` Erica Bugden
  0 siblings, 1 reply; 8+ messages in thread
From: Steven Rostedt @ 2018-08-03 13:37 UTC (permalink / raw)
  To: Erica Bugden; +Cc: linux-kernel, peterz, tglx, anna-maria, bigeasy

On Fri, 03 Aug 2018 10:46:14 +0200
Erica Bugden <erica.bugden@linutronix.de> wrote:


> > Can you try this patch?  
> 
> I tested the patch below and it also fixes the problem.

Thanks! May I add:

Tested-by: Erica Bugden <erica.bugden@linutronix.de>

?

-- Steve

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] ftrace: Add missing check for existing hwlat thread
  2018-08-03 13:37     ` Steven Rostedt
@ 2018-08-03 14:57       ` Erica Bugden
  0 siblings, 0 replies; 8+ messages in thread
From: Erica Bugden @ 2018-08-03 14:57 UTC (permalink / raw)
  To: Steven Rostedt; +Cc: linux-kernel, peterz, tglx, anna-maria, bigeasy

On Fri, 2018-08-03 at 09:37 -0400, Steven Rostedt wrote:
> On Fri, 03 Aug 2018 10:46:14 +0200
> Erica Bugden <erica.bugden@linutronix.de> wrote:
> 
> 
> > > Can you try this patch?  
> > 
> > I tested the patch below and it also fixes the problem.
> 
> Thanks! May I add:
> 
> Tested-by: Erica Bugden <erica.bugden@linutronix.de>
> 
> ?

Certainly! One must be held responsible when one makes a claim :)

  Erica

> 
> -- Steve

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, back to index

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-08-01 10:45 [PATCH] ftrace: Add missing check for existing hwlat thread Erica Bugden
2018-08-01 19:40 ` Steven Rostedt
2018-08-01 19:59   ` Thomas Gleixner
2018-08-01 20:03     ` Steven Rostedt
2018-08-01 20:07       ` Thomas Gleixner
2018-08-03  8:46   ` Erica Bugden
2018-08-03 13:37     ` Steven Rostedt
2018-08-03 14:57       ` Erica Bugden

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org linux-kernel@archiver.kernel.org
	public-inbox-index lkml


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/ public-inbox