[PATCH 6/9] livepatch/ftrace: Add recursion protection to the ftrace callback

[PATCH 6/9] livepatch/ftrace: Add recursion protection to the ftrace callback

[Steven Rostedt]

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

If a ftrace callback does not supply its own recursion protection and
does not set the RECURSION_SAFE flag in its ftrace_ops, then ftrace will
make a helper trampoline to do so before calling the callback instead of
just calling the callback directly.

The default for ftrace_ops is going to assume recursion protection unless
otherwise specified.

Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Jiri Kosina <jikos@kernel.org>
Cc: Miroslav Benes <mbenes@suse.cz>
Cc: Petr Mladek <pmladek@suse.com>
Cc: Joe Lawrence <joe.lawrence@redhat.com>
Cc: live-patching@vger.kernel.org
Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 kernel/livepatch/patch.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/kernel/livepatch/patch.c b/kernel/livepatch/patch.c
index b552cf2d85f8..6c0164d24bbd 100644
--- a/kernel/livepatch/patch.c
+++ b/kernel/livepatch/patch.c
@@ -45,9 +45,13 @@ static void notrace klp_ftrace_handler(unsigned long ip,
 	struct klp_ops *ops;
 	struct klp_func *func;
 	int patch_state;
+	int bit;
 
 	ops = container_of(fops, struct klp_ops, fops);
 
+	bit = ftrace_test_recursion_trylock();
+	if (bit < 0)
+		return;
 	/*
 	 * A variant of synchronize_rcu() is used to allow patching functions
 	 * where RCU is not watching, see klp_synchronize_transition().
@@ -117,6 +121,7 @@ static void notrace klp_ftrace_handler(unsigned long ip,
 
 unlock:
 	preempt_enable_notrace();
+	ftrace_test_recursion_unlock(bit);
 }
 
 /*
-- 
2.28.0

Re: [PATCH 6/9] livepatch/ftrace: Add recursion protection to the ftrace callback

[Miroslav Benes]

On Wed, 28 Oct 2020, Steven Rostedt wrote:

> From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>
> 
> If a ftrace callback does not supply its own recursion protection and
> does not set the RECURSION_SAFE flag in its ftrace_ops, then ftrace will
> make a helper trampoline to do so before calling the callback instead of
> just calling the callback directly.
> 
> The default for ftrace_ops is going to assume recursion protection unless
> otherwise specified.

Hm, I've always thought that we did not need any kind of recursion 
protection for our callback. It is marked as notrace and it does not call 
anything traceable. In fact, it does not call anything. I even have a note 
in my todo list to mark the callback as RECURSION_SAFE :)

At the same time, it probably does not hurt and the patch is still better 
than what we have now without RECURSION_SAFE if I understand the patch set 
correctly.
 
> Cc: Josh Poimboeuf <jpoimboe@redhat.com>
> Cc: Jiri Kosina <jikos@kernel.org>
> Cc: Miroslav Benes <mbenes@suse.cz>
> Cc: Petr Mladek <pmladek@suse.com>
> Cc: Joe Lawrence <joe.lawrence@redhat.com>
> Cc: live-patching@vger.kernel.org
> Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
> ---
>  kernel/livepatch/patch.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/kernel/livepatch/patch.c b/kernel/livepatch/patch.c
> index b552cf2d85f8..6c0164d24bbd 100644
> --- a/kernel/livepatch/patch.c
> +++ b/kernel/livepatch/patch.c
> @@ -45,9 +45,13 @@ static void notrace klp_ftrace_handler(unsigned long ip,
>  	struct klp_ops *ops;
>  	struct klp_func *func;
>  	int patch_state;
> +	int bit;
>  
>  	ops = container_of(fops, struct klp_ops, fops);
>  
> +	bit = ftrace_test_recursion_trylock();
> +	if (bit < 0)
> +		return;

This means that the original function will be called in case of recursion. 
That's probably fair, but I'm wondering if we should at least WARN about 
it.

Thanks
Miroslav

Re: [PATCH 6/9] livepatch/ftrace: Add recursion protection to the ftrace callback

[Steven Rostedt]

On Thu, 29 Oct 2020 14:51:06 +0100 (CET)
Miroslav Benes <mbenes@suse.cz> wrote:

> > index b552cf2d85f8..6c0164d24bbd 100644
> > --- a/kernel/livepatch/patch.c
> > +++ b/kernel/livepatch/patch.c
> > @@ -45,9 +45,13 @@ static void notrace klp_ftrace_handler(unsigned long ip,
> >  	struct klp_ops *ops;
> >  	struct klp_func *func;
> >  	int patch_state;
> > +	int bit;
> >  
> >  	ops = container_of(fops, struct klp_ops, fops);
> >  
> > +	bit = ftrace_test_recursion_trylock();
> > +	if (bit < 0)
> > +		return;  
> 
> This means that the original function will be called in case of recursion. 
> That's probably fair, but I'm wondering if we should at least WARN about 
> it.

It's probably what happens today. But if you add a WARN_ON_ONCE() it may
not hurt.

I also plan on adding code that reports when recursion has happened,
because even if it's not a problem, recursion adds extra overhead.

-- Steve

Re: [PATCH 6/9] livepatch/ftrace: Add recursion protection to the ftrace callback

[Petr Mladek]

On Thu 2020-10-29 14:51:06, Miroslav Benes wrote:
> On Wed, 28 Oct 2020, Steven Rostedt wrote:
> 
> > From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>
> > 
> > If a ftrace callback does not supply its own recursion protection and
> > does not set the RECURSION_SAFE flag in its ftrace_ops, then ftrace will
> > make a helper trampoline to do so before calling the callback instead of
> > just calling the callback directly.
> > 
> > The default for ftrace_ops is going to assume recursion protection unless
> > otherwise specified.

It might be my lack skills to read English. But the above sentence
sounds ambiguous to me. It is not clear to me who provides the
recursion protection by default. Could you please make it more
explicit, for example by:

"The default for ftrace_ops is going to change. It will expect that
handlers provide their own recursion protection."


> Hm, I've always thought that we did not need any kind of recursion 
> protection for our callback. It is marked as notrace and it does not call 
> anything traceable. In fact, it does not call anything. I even have a note 
> in my todo list to mark the callback as RECURSION_SAFE :)

Well, it calls WARN_ON_ONCE() ;-)

> At the same time, it probably does not hurt and the patch is still better 
> than what we have now without RECURSION_SAFE if I understand the patch set 
> correctly.

And better be on the safe side.


> > Cc: Josh Poimboeuf <jpoimboe@redhat.com>
> > Cc: Jiri Kosina <jikos@kernel.org>
> > Cc: Miroslav Benes <mbenes@suse.cz>
> > Cc: Petr Mladek <pmladek@suse.com>
> > Cc: Joe Lawrence <joe.lawrence@redhat.com>
> > Cc: live-patching@vger.kernel.org
> > Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
> > ---
> >  kernel/livepatch/patch.c | 5 +++++
> >  1 file changed, 5 insertions(+)
> > 
> > diff --git a/kernel/livepatch/patch.c b/kernel/livepatch/patch.c
> > index b552cf2d85f8..6c0164d24bbd 100644
> > --- a/kernel/livepatch/patch.c
> > +++ b/kernel/livepatch/patch.c
> > @@ -45,9 +45,13 @@ static void notrace klp_ftrace_handler(unsigned long ip,
> >  	struct klp_ops *ops;
> >  	struct klp_func *func;
> >  	int patch_state;
> > +	int bit;
> >  
> >  	ops = container_of(fops, struct klp_ops, fops);
> >  
> > +	bit = ftrace_test_recursion_trylock();
> > +	if (bit < 0)
> > +		return;
> 
> This means that the original function will be called in case of recursion. 
> That's probably fair, but I'm wondering if we should at least WARN about 
> it.

Yeah, the early return might break the consistency model and
unexpected things might happen. We should be aware of it.
Please use:

	if (WARN_ON_ONCE(bit < 0))
		return;

WARN_ON_ONCE() might be part of the recursion. But it should happen
only once. IMHO, it is worth the risk.

Otherwise it looks good.

Best Regards,
Petr

Re: [PATCH 6/9] livepatch/ftrace: Add recursion protection to the ftrace callback

[Miroslav Benes]

On Thu, 29 Oct 2020, Petr Mladek wrote:

> On Thu 2020-10-29 14:51:06, Miroslav Benes wrote:
> > On Wed, 28 Oct 2020, Steven Rostedt wrote:
> 
> > Hm, I've always thought that we did not need any kind of recursion 
> > protection for our callback. It is marked as notrace and it does not call 
> > anything traceable. In fact, it does not call anything. I even have a note 
> > in my todo list to mark the callback as RECURSION_SAFE :)
> 
> Well, it calls WARN_ON_ONCE() ;-)

Oh my, I learned to ignore these. Of course there is printk hidden 
everywhere.

> > At the same time, it probably does not hurt and the patch is still better 
> > than what we have now without RECURSION_SAFE if I understand the patch set 
> > correctly.
> 
> And better be on the safe side.

Agreed. 
 
> > > Cc: Josh Poimboeuf <jpoimboe@redhat.com>
> > > Cc: Jiri Kosina <jikos@kernel.org>
> > > Cc: Miroslav Benes <mbenes@suse.cz>
> > > Cc: Petr Mladek <pmladek@suse.com>
> > > Cc: Joe Lawrence <joe.lawrence@redhat.com>
> > > Cc: live-patching@vger.kernel.org
> > > Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
> > > ---
> > >  kernel/livepatch/patch.c | 5 +++++
> > >  1 file changed, 5 insertions(+)
> > > 
> > > diff --git a/kernel/livepatch/patch.c b/kernel/livepatch/patch.c
> > > index b552cf2d85f8..6c0164d24bbd 100644
> > > --- a/kernel/livepatch/patch.c
> > > +++ b/kernel/livepatch/patch.c
> > > @@ -45,9 +45,13 @@ static void notrace klp_ftrace_handler(unsigned long ip,
> > >  	struct klp_ops *ops;
> > >  	struct klp_func *func;
> > >  	int patch_state;
> > > +	int bit;
> > >  
> > >  	ops = container_of(fops, struct klp_ops, fops);
> > >  
> > > +	bit = ftrace_test_recursion_trylock();
> > > +	if (bit < 0)
> > > +		return;
> > 
> > This means that the original function will be called in case of recursion. 
> > That's probably fair, but I'm wondering if we should at least WARN about 
> > it.
> 
> Yeah, the early return might break the consistency model and
> unexpected things might happen. We should be aware of it.
> Please use:
> 
> 	if (WARN_ON_ONCE(bit < 0))
> 		return;
> 
> WARN_ON_ONCE() might be part of the recursion. But it should happen
> only once. IMHO, it is worth the risk.

Agreed.

Miroslav

Re: [PATCH 6/9] livepatch/ftrace: Add recursion protection to the ftrace callback

[Steven Rostedt]

On Thu, 29 Oct 2020 15:57:09 +0100
Petr Mladek <pmladek@suse.com> wrote:

> On Thu 2020-10-29 14:51:06, Miroslav Benes wrote:
> > On Wed, 28 Oct 2020, Steven Rostedt wrote:
> >   
> > > From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>
> > > 
> > > If a ftrace callback does not supply its own recursion protection and
> > > does not set the RECURSION_SAFE flag in its ftrace_ops, then ftrace will
> > > make a helper trampoline to do so before calling the callback instead of
> > > just calling the callback directly.
> > > 
> > > The default for ftrace_ops is going to assume recursion protection unless
> > > otherwise specified.  
> 
> It might be my lack skills to read English. But the above sentence
> sounds ambiguous to me. It is not clear to me who provides the
> recursion protection by default. Could you please make it more
> explicit, for example by:
> 
> "The default for ftrace_ops is going to change. It will expect that
> handlers provide their own recursion protection."

It was originally written as something else, as my first series (that I
didn't post) added the recursion flag, and then I needed one big nasty
patch to remove them. Then I realized it would be fine to just keep the
double recursion testing and remove the flag when it was no longer used. I
then went back and wrote up that sentence, and yeah, it wasn't the best
explanation.

Your sentence is better, I'll update it.

> 
> 
> > Hm, I've always thought that we did not need any kind of recursion 
> > protection for our callback. It is marked as notrace and it does not call 
> > anything traceable. In fact, it does not call anything. I even have a note 
> > in my todo list to mark the callback as RECURSION_SAFE :)  
> 
> Well, it calls WARN_ON_ONCE() ;-)
> 
> > At the same time, it probably does not hurt and the patch is still better 
> > than what we have now without RECURSION_SAFE if I understand the patch set 
> > correctly.  
> 
> And better be on the safe side.

And the WARN_ON_ONCE() use to cause a problem, until I fixed it:

  dfbf2897d0049 ("bug: set warn variable before calling WARN()")

> 
> 
> > > Cc: Josh Poimboeuf <jpoimboe@redhat.com>
> > > Cc: Jiri Kosina <jikos@kernel.org>
> > > Cc: Miroslav Benes <mbenes@suse.cz>
> > > Cc: Petr Mladek <pmladek@suse.com>
> > > Cc: Joe Lawrence <joe.lawrence@redhat.com>
> > > Cc: live-patching@vger.kernel.org
> > > Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
> > > ---
> > >  kernel/livepatch/patch.c | 5 +++++
> > >  1 file changed, 5 insertions(+)
> > > 
> > > diff --git a/kernel/livepatch/patch.c b/kernel/livepatch/patch.c
> > > index b552cf2d85f8..6c0164d24bbd 100644
> > > --- a/kernel/livepatch/patch.c
> > > +++ b/kernel/livepatch/patch.c
> > > @@ -45,9 +45,13 @@ static void notrace klp_ftrace_handler(unsigned long ip,
> > >  	struct klp_ops *ops;
> > >  	struct klp_func *func;
> > >  	int patch_state;
> > > +	int bit;
> > >  
> > >  	ops = container_of(fops, struct klp_ops, fops);
> > >  
> > > +	bit = ftrace_test_recursion_trylock();
> > > +	if (bit < 0)
> > > +		return;  
> > 
> > This means that the original function will be called in case of recursion. 
> > That's probably fair, but I'm wondering if we should at least WARN about 
> > it.  
> 
> Yeah, the early return might break the consistency model and
> unexpected things might happen. We should be aware of it.
> Please use:
> 
> 	if (WARN_ON_ONCE(bit < 0))
> 		return;
> 
> WARN_ON_ONCE() might be part of the recursion. But it should happen
> only once. IMHO, it is worth the risk.
> 
> Otherwise it looks good.

Perhaps we can add that as a separate patch, because this patch doesn't add
any real functionality change. It only moves the recursion testing from the
helper function (which ftrace wraps all callbacks that do not have the
RECURSION flags set, including this one) down to your callback.

In keeping with one patch to do one thing principle, the added of
WARN_ON_ONCE() should be a separate patch, as that will change the
functionality.

If that WARN_ON_ONCE() breaks things, I'd like it to be bisected to another
patch other than this one.

-- Steve

Re: [PATCH 6/9] livepatch/ftrace: Add recursion protection to the ftrace callback

[Miroslav Benes]

> > > > +	bit = ftrace_test_recursion_trylock();
> > > > +	if (bit < 0)
> > > > +		return;  
> > > 
> > > This means that the original function will be called in case of recursion. 
> > > That's probably fair, but I'm wondering if we should at least WARN about 
> > > it.  
> > 
> > Yeah, the early return might break the consistency model and
> > unexpected things might happen. We should be aware of it.
> > Please use:
> > 
> > 	if (WARN_ON_ONCE(bit < 0))
> > 		return;
> > 
> > WARN_ON_ONCE() might be part of the recursion. But it should happen
> > only once. IMHO, it is worth the risk.
> > 
> > Otherwise it looks good.
> 
> Perhaps we can add that as a separate patch, because this patch doesn't add
> any real functionality change. It only moves the recursion testing from the
> helper function (which ftrace wraps all callbacks that do not have the
> RECURSION flags set, including this one) down to your callback.
> 
> In keeping with one patch to do one thing principle, the added of
> WARN_ON_ONCE() should be a separate patch, as that will change the
> functionality.
> 
> If that WARN_ON_ONCE() breaks things, I'd like it to be bisected to another
> patch other than this one.

Works for me.

Miroslav

Re: [PATCH 6/9] livepatch/ftrace: Add recursion protection to the ftrace callback

[Petr Mladek]

On Fri 2020-10-30 10:48:58, Miroslav Benes wrote:
> > > > > +	bit = ftrace_test_recursion_trylock();
> > > > > +	if (bit < 0)
> > > > > +		return;  
> > > > 
> > > > This means that the original function will be called in case of recursion. 
> > > > That's probably fair, but I'm wondering if we should at least WARN about 
> > > > it.  
> > > 
> > > Yeah, the early return might break the consistency model and
> > > unexpected things might happen. We should be aware of it.
> > > Please use:
> > > 
> > > 	if (WARN_ON_ONCE(bit < 0))
> > > 		return;
> > > 
> > > WARN_ON_ONCE() might be part of the recursion. But it should happen
> > > only once. IMHO, it is worth the risk.
> > > 
> > > Otherwise it looks good.
> > 
> > Perhaps we can add that as a separate patch, because this patch doesn't add
> > any real functionality change. It only moves the recursion testing from the
> > helper function (which ftrace wraps all callbacks that do not have the
> > RECURSION flags set, including this one) down to your callback.
> > 
> > In keeping with one patch to do one thing principle, the added of
> > WARN_ON_ONCE() should be a separate patch, as that will change the
> > functionality.
> > 
> > If that WARN_ON_ONCE() breaks things, I'd like it to be bisected to another
> > patch other than this one.
> 
> Works for me.

+1

So, with the updated commit message:

Reviewed-by: Petr Mladek <pmladek@suse.com>

Best Regards,
Petr

Re: [PATCH 6/9] livepatch/ftrace: Add recursion protection to the ftrace callback

[Steven Rostedt]

On Thu, 29 Oct 2020 10:37:44 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> I also plan on adding code that reports when recursion has happened,
> because even if it's not a problem, recursion adds extra overhead.

I did the above (will be posting that later, maybe next week), and
found two bugs with the recursion code. :-/

One was in the nmi handling, where it never cleared the nmi bit
(because it was zero, and thus ignored), and that caused all functions
in NMI handlers to not be traced (because it thought it was a
recursion).
(see https://lore.kernel.org/r/20201030002722.766a22df@oasis.local.home)

The second was the recursion algorithm depends on the preempt_count()
being accurate, but when it transitions between context, and there's
tracing in that transition, it could falsely record it as a recursion.

I have a fix for both of these bugs and will be sending them up marked
for stable after I finish testing them.

This goes to show that the recursion reported should be implemented
(but that will be for the next merge window).

-- Steve