From: Steven Rostedt <rostedt@goodmis.org>
To: Frederic Weisbecker <fweisbec@gmail.com>
Cc: linux-kernel@vger.kernel.org, Ingo Molnar <mingo@kernel.org>,
Andrew Morton <akpm@linux-foundation.org>,
Peter Zijlstra <peterz@infradead.org>,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
stable@vger.kernel.org, Petr Mladek <pmladek@suse.cz>
Subject: Re: [RFA][PATCH 2/5] ftrace/x86: One more missing sync after fixup of function modification failure
Date: Thu, 27 Feb 2014 12:00:14 -0500 [thread overview]
Message-ID: <20140227120014.7ba8b484@gandalf.local.home> (raw)
In-Reply-To: <20140227163731.GC19580@localhost.localdomain>
On Thu, 27 Feb 2014 17:37:32 +0100
Frederic Weisbecker <fweisbec@gmail.com> wrote:
> On Thu, Feb 27, 2014 at 10:46:18AM -0500, Steven Rostedt wrote:
> > [Request for Ack]
> >
> > From: Petr Mladek <pmladek@suse.cz>
> >
> > If a failure occurs while modifying ftrace function, it bails out and will
> > remove the tracepoints to be back to what the code originally was.
> >
> > There is missing the final sync run across the CPUs after the fix up is done
> > and before the ftrace int3 handler flag is reset.
>
> So IIUC the risk is that other CPUs may spuriously ignore non-ftrace traps if we don't sync the
> other cores after reverting the int3 before decrementing the modifying_ftrace_code counter?
Actually, the bug is that they will not ignore the ftrace traps after
we decrement modifying_ftrace_code counter. Here's the race:
CPU0 CPU1
---- ----
remove_breakpoint();
modifying_ftrace_code = 0;
[still sees breakpoint]
<takes trap>
[sees modifying_ftrace_code as zero]
[no breakpoint handler]
[goto failed case]
[trap exception - kernel breakpoint, no
handler]
BUG()
Even if we had a smp_wmb() after removing the breakpoint and clearing
the modifying_ftrace_code, we still need the smp_rmb() on the other
CPUS. The run_sync() does a IPI on all CPUs doing the smp_rmb().
>
> >
> > Link: http://lkml.kernel.org/r/1393258342-29978-2-git-send-email-pmladek@suse.cz
> >
> > Fixes: 8a4d0a687a5 "ftrace: Use breakpoint method to update ftrace caller"
> > Cc: stable@vger.kernel.org # 3.5+
> > Signed-off-by: Petr Mladek <pmladek@suse.cz>
> > Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
> > ---
> > arch/x86/kernel/ftrace.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
> > index 6b566c8..69885e2 100644
> > --- a/arch/x86/kernel/ftrace.c
> > +++ b/arch/x86/kernel/ftrace.c
> > @@ -660,8 +660,8 @@ ftrace_modify_code(unsigned long ip, unsigned const char *old_code,
> > ret = -EPERM;
> > goto out;
> > }
> > - run_sync();
> > out:
> > + run_sync();
> > return ret;
> >
> > fail_update:
>
> This could be further optimized by rather calling run_sync() in the end of the
> fail_update block (after the probe_kernel_write revert) otherwise even failure on
> setting the break will result in run_sync(), which doesn't appear to be needed. But
> that's really just nitpicking as it's a rare failure codepath and shouldn't hurt.
No, the run_sync() must be done after removing the breakpoint. Again,
we don't want one of these breakpoints to be called on another CPU and
then see modifying_ftrace_code as zero. That is bad. The final
run_sync() is required.
I think I'll update the change log to include my race flow graph from
above.
-- Steve
>
> In any case, the fix looks correct.
>
> Acked-by: Frederic Weisbecker <fweisbec@gmail.com>
>
> > --
> > 1.8.5.3
> >
> >
next prev parent reply other threads:[~2014-02-27 17:00 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-02-27 15:46 [RFA][PATCH 0/5] tracing: Ftrace error sync fix and tracepoint warn on failure Steven Rostedt
2014-02-27 15:46 ` [RFA][PATCH 1/5] ftrace/x86: Run a sync after fixup " Steven Rostedt
2014-02-27 15:58 ` Petr Mládek
2014-02-27 16:02 ` Steven Rostedt
2014-03-03 22:12 ` H. Peter Anvin
2014-03-03 22:18 ` Steven Rostedt
2014-03-03 22:27 ` H. Peter Anvin
2014-03-03 22:31 ` Steven Rostedt
2014-02-27 15:46 ` [RFA][PATCH 2/5] ftrace/x86: One more missing sync after fixup of function modification failure Steven Rostedt
2014-02-27 16:05 ` Steven Rostedt
2014-02-27 16:37 ` Frederic Weisbecker
2014-02-27 17:00 ` Steven Rostedt [this message]
2014-02-27 17:19 ` Frederic Weisbecker
2014-02-27 17:35 ` Steven Rostedt
2014-02-27 17:52 ` Frederic Weisbecker
2014-02-27 15:46 ` [RFA][PATCH 3/5] tracing: Do not add event files for modules that fail tracepoints Steven Rostedt
2014-02-27 16:31 ` Mathieu Desnoyers
2014-02-27 15:46 ` [RFA][PATCH 4/5] tracepoint: Do not waste memory on mods with no tracepoints Steven Rostedt
2014-02-27 15:46 ` [RFA][PATCH 5/5] tracepoint: Warn and notify if tracepoints are not loaded due to module taint Steven Rostedt
2014-02-27 16:33 ` Mathieu Desnoyers
2014-02-27 17:06 ` Steven Rostedt
2014-02-27 17:09 ` Steven Rostedt
2014-02-27 17:16 ` Mathieu Desnoyers
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140227120014.7ba8b484@gandalf.local.home \
--to=rostedt@goodmis.org \
--cc=akpm@linux-foundation.org \
--cc=fweisbec@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=mingo@kernel.org \
--cc=peterz@infradead.org \
--cc=pmladek@suse.cz \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).