linux-kselftest.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: torvalds at linux-foundation.org (Linus Torvalds)
Subject: [RFC][PATCH 1/2] x86: Allow breakpoints to emulate call functions
Date: Mon, 6 May 2019 19:22:06 -0700	[thread overview]
Message-ID: <CAHk-=wjLXmOn=Cp=uOfO4gE01eN_-UcOUyrMTTw5-f_OfPO48Q@mail.gmail.com> (raw)
In-Reply-To: <20190506215353.14a8ef78@oasis.local.home>

On Mon, May 6, 2019 at 6:53 PM Steven Rostedt <rostedt at goodmis.org> wrote:
>
> Also, I figured just calling ftrace_regs_caller() was simpler then
> having that int3 handler do the hash look ups to determine what handler
> it needs to call.

So what got me looking at this - and the races (that didn't turn out
to be races) - and why I really hate it, is because of the whole
notion of "atomic state".

Running an "int3" handler (when the int3 is in the kernel) is in some
sense "atomic". There is the state in the caller, of course, and
there's the state that the int3 handler has, but you can *mostly*
think of the two as independent.

In particular, if the int3 handler doesn't ever enable interrupts, and
if it doesn't need to do any stack switches, the int3 handler part
looks "obvious". It can be interrupted by NMI, but it can't be
interrupted (for example) by the cross-cpu IPI.

That was at least the mental model I was going for.

Similarly, the "caller" side mostly looks obvious too. If we don't
take an int3, none of this matter, and if we *do* take an int3, if we
can at least make it "atomic" wrt the rewriter (before or after
state), it should be easy to think about.

One of the things I was thinking of doing, for example, was to simply
make the call emulation do a "load four bytes from the instruction
stream, and just use that as the emulation target offset".

Why would that matter?

We do *not* have very strict guarantees for D$-vs-I$ coherency on x86,
but we *do* have very strict guarantees for D$-vs-D$ coherency. And so
we could use the D$ coherency to give us atomicity guarantees for
loading and storing the instruction offset for instruction emulation,
in ways we can *not* use the D$-to-I$ guarantees and just executing it
directly.

So while we still need those nasty IPI's to guarantee the D$-vs-I$
coherency in the "big picture" model and to get the serialization with
the actual 'int3' exception right, we *could* just do all the other
parts of the instruction emulation using the D$ coherency.

So we could do the actual "call offset" write with a single atomic
4-byte locked cycle (just use "xchg" to write - it's always locked).
And similarly we could do the call offset *read* with a single locked
cycle (cmpxchg with a 0 value, for example). It would be atomic even
if it crosses a cacheline boundary.

And now we'd be guaranteed that on a D$ side, we'd get either the old
value _or_ the new value for the emulation, and never a mix of them.
Which is very much unlike the I$ side guarantees that aren't
documented and people have been worrying about.

Notice? We'd not even have to look up any values.  We'd literally just
do something like

        int offset = locked_atomic_read(ip+1);
        return int3_emulate_call(ip, ip+5+offset);

and it would be *atomic* with respect to whatever other user that
updates the instruction, as long as they update the offset with a
"xchg" instruction.

Wouldn't that be lovely? We'd literally use the emulation as a way to
get the kinds of coherency guarantees that we don't get natively with
the I$.

Except that was _so_ not the case with the whole ftrace_regs_caller()
thing. That whole hack made me go "this doesn't emulate either the old
_or_ the new call". And see above why that would matter. Because a
"let's just make the four bytes after the 'int3' instruction be
atomic" would very much be all about "emulate either the old or the
new call".

So this was why I was then looking at what the heck that code was
doing, and it was making zero sense to me. Emulating something that is
different from what you're actually rewriting means that tricks like
the above don't work.

                  Linus

WARNING: multiple messages have this Message-ID (diff)
From: torvalds@linux-foundation.org (Linus Torvalds)
Subject: [RFC][PATCH 1/2] x86: Allow breakpoints to emulate call functions
Date: Mon, 6 May 2019 19:22:06 -0700	[thread overview]
Message-ID: <CAHk-=wjLXmOn=Cp=uOfO4gE01eN_-UcOUyrMTTw5-f_OfPO48Q@mail.gmail.com> (raw)
Message-ID: <20190507022206.9SKyi3MNJjf6qpQoZ3dGV7_0E5pOU-rQXQ5Nmwi28qs@z> (raw)
In-Reply-To: <20190506215353.14a8ef78@oasis.local.home>

On Mon, May 6, 2019@6:53 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> Also, I figured just calling ftrace_regs_caller() was simpler then
> having that int3 handler do the hash look ups to determine what handler
> it needs to call.

So what got me looking at this - and the races (that didn't turn out
to be races) - and why I really hate it, is because of the whole
notion of "atomic state".

Running an "int3" handler (when the int3 is in the kernel) is in some
sense "atomic". There is the state in the caller, of course, and
there's the state that the int3 handler has, but you can *mostly*
think of the two as independent.

In particular, if the int3 handler doesn't ever enable interrupts, and
if it doesn't need to do any stack switches, the int3 handler part
looks "obvious". It can be interrupted by NMI, but it can't be
interrupted (for example) by the cross-cpu IPI.

That was at least the mental model I was going for.

Similarly, the "caller" side mostly looks obvious too. If we don't
take an int3, none of this matter, and if we *do* take an int3, if we
can at least make it "atomic" wrt the rewriter (before or after
state), it should be easy to think about.

One of the things I was thinking of doing, for example, was to simply
make the call emulation do a "load four bytes from the instruction
stream, and just use that as the emulation target offset".

Why would that matter?

We do *not* have very strict guarantees for D$-vs-I$ coherency on x86,
but we *do* have very strict guarantees for D$-vs-D$ coherency. And so
we could use the D$ coherency to give us atomicity guarantees for
loading and storing the instruction offset for instruction emulation,
in ways we can *not* use the D$-to-I$ guarantees and just executing it
directly.

So while we still need those nasty IPI's to guarantee the D$-vs-I$
coherency in the "big picture" model and to get the serialization with
the actual 'int3' exception right, we *could* just do all the other
parts of the instruction emulation using the D$ coherency.

So we could do the actual "call offset" write with a single atomic
4-byte locked cycle (just use "xchg" to write - it's always locked).
And similarly we could do the call offset *read* with a single locked
cycle (cmpxchg with a 0 value, for example). It would be atomic even
if it crosses a cacheline boundary.

And now we'd be guaranteed that on a D$ side, we'd get either the old
value _or_ the new value for the emulation, and never a mix of them.
Which is very much unlike the I$ side guarantees that aren't
documented and people have been worrying about.

Notice? We'd not even have to look up any values.  We'd literally just
do something like

        int offset = locked_atomic_read(ip+1);
        return int3_emulate_call(ip, ip+5+offset);

and it would be *atomic* with respect to whatever other user that
updates the instruction, as long as they update the offset with a
"xchg" instruction.

Wouldn't that be lovely? We'd literally use the emulation as a way to
get the kinds of coherency guarantees that we don't get natively with
the I$.

Except that was _so_ not the case with the whole ftrace_regs_caller()
thing. That whole hack made me go "this doesn't emulate either the old
_or_ the new call". And see above why that would matter. Because a
"let's just make the four bytes after the 'int3' instruction be
atomic" would very much be all about "emulate either the old or the
new call".

So this was why I was then looking at what the heck that code was
doing, and it was making zero sense to me. Emulating something that is
different from what you're actually rewriting means that tricks like
the above don't work.

                  Linus

  parent reply	other threads:[~2019-05-07  2:22 UTC|newest]

Thread overview: 204+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20190501202830.347656894@goodmis.org>
2019-05-01 20:28 ` [RFC][PATCH 1/2] x86: Allow breakpoints to emulate call functions rostedt
2019-05-01 20:28   ` Steven Rostedt
2019-05-02  3:24   ` rostedt
2019-05-02  3:24     ` Steven Rostedt
2019-05-02 16:21     ` peterz
2019-05-02 16:21       ` Peter Zijlstra
2019-05-02 16:29       ` peterz
2019-05-02 16:29         ` Peter Zijlstra
2019-05-02 18:02       ` torvalds
2019-05-02 18:02         ` Linus Torvalds
2019-05-02 18:18         ` peterz
2019-05-02 18:18           ` Peter Zijlstra
2019-05-02 18:30           ` peterz
2019-05-02 18:30             ` Peter Zijlstra
2019-05-02 18:43           ` torvalds
2019-05-02 18:43             ` Linus Torvalds
2019-05-02 19:28             ` jikos
2019-05-02 19:28               ` Jiri Kosina
2019-05-02 20:25               ` luto
2019-05-02 20:25                 ` Andy Lutomirski
2019-05-02 20:21             ` peterz
2019-05-02 20:21               ` Peter Zijlstra
2019-05-02 20:49               ` torvalds
2019-05-02 20:49                 ` Linus Torvalds
2019-05-02 21:32                 ` peterz
2019-05-02 21:32                   ` Peter Zijlstra
2019-05-03 19:24                 ` rostedt
2019-05-03 19:24                   ` Steven Rostedt
2019-05-03 21:46                   ` torvalds
2019-05-03 21:46                     ` Linus Torvalds
2019-05-03 22:49                     ` rostedt
2019-05-03 22:49                       ` Steven Rostedt
2019-05-03 23:07                       ` torvalds
2019-05-03 23:07                         ` Linus Torvalds
2019-05-04  4:17                         ` rostedt
2019-05-04  4:17                           ` Steven Rostedt
     [not found]                           ` <CAHk-=wiuSFbv_rELND-BLWcP0GSZ0yF=xOAEcf61GE3bU9d=yg@mail.gmail.com>
2019-05-04 18:59                             ` torvalds
2019-05-04 18:59                               ` Linus Torvalds
2019-05-04 20:12                               ` luto
2019-05-04 20:12                                 ` Andy Lutomirski
2019-05-04 20:28                                 ` torvalds
2019-05-04 20:28                                   ` Linus Torvalds
2019-05-04 20:36                                 ` torvalds
2019-05-04 20:36                                   ` Linus Torvalds
2019-05-03 22:55                     ` luto
2019-05-03 22:55                       ` Andy Lutomirski
2019-05-03 23:16                       ` torvalds
2019-05-03 23:16                         ` Linus Torvalds
2019-05-03 23:32                         ` luto
2019-05-03 23:32                           ` Andy Lutomirski
2019-05-02 22:52               ` rostedt
2019-05-02 22:52                 ` Steven Rostedt
2019-05-02 23:31                 ` rostedt
2019-05-02 23:31                   ` Steven Rostedt
2019-05-02 23:50                   ` rostedt
2019-05-02 23:50                     ` Steven Rostedt
2019-05-03  1:51                     ` [RFC][PATCH 1/2 v2] " rostedt
2019-05-03  1:51                       ` Steven Rostedt
2019-05-03  9:29                     ` [RFC][PATCH 1/2] " peterz
2019-05-03  9:29                       ` Peter Zijlstra
2019-05-03 13:22                       ` rostedt
2019-05-03 13:22                         ` Steven Rostedt
2019-05-03 16:20                         ` luto
2019-05-03 16:20                           ` Andy Lutomirski
2019-05-03 16:31                           ` rostedt
2019-05-03 16:31                             ` Steven Rostedt
2019-05-03 16:35                             ` peterz
2019-05-03 16:35                               ` Peter Zijlstra
2019-05-03 16:44                               ` luto
2019-05-03 16:44                                 ` Andy Lutomirski
2019-05-03 16:49                                 ` rostedt
2019-05-03 16:49                                   ` Steven Rostedt
2019-05-03 16:32                           ` peterz
2019-05-03 16:32                             ` Peter Zijlstra
2019-05-03 18:57                           ` torvalds
2019-05-03 18:57                             ` Linus Torvalds
2019-05-06  8:19                             ` peterz
2019-05-06  8:19                               ` Peter Zijlstra
2019-05-06 13:56                               ` rostedt
2019-05-06 13:56                                 ` Steven Rostedt
2019-05-06 16:17                                 ` torvalds
2019-05-06 16:17                                   ` Linus Torvalds
2019-05-06 16:19                                   ` torvalds
2019-05-06 16:19                                     ` Linus Torvalds
2019-05-06 17:06                                   ` rostedt
2019-05-06 17:06                                     ` Steven Rostedt
2019-05-06 18:06                                     ` torvalds
2019-05-06 18:06                                       ` Linus Torvalds
2019-05-06 18:57                                       ` rostedt
2019-05-06 18:57                                         ` Steven Rostedt
2019-05-06 19:46                                         ` torvalds
2019-05-06 19:46                                           ` Linus Torvalds
2019-05-06 20:29                                           ` rostedt
2019-05-06 20:29                                             ` Steven Rostedt
2019-05-06 20:42                                             ` torvalds
2019-05-06 20:42                                               ` Linus Torvalds
2019-05-06 20:44                                               ` torvalds
2019-05-06 20:44                                                 ` Linus Torvalds
2019-05-06 21:45                                               ` rostedt
2019-05-06 21:45                                                 ` Steven Rostedt
2019-05-06 22:06                                                 ` torvalds
2019-05-06 22:06                                                   ` Linus Torvalds
2019-05-06 22:31                                                   ` torvalds
2019-05-06 22:31                                                     ` Linus Torvalds
2019-05-07  0:10                                                     ` rostedt
2019-05-07  0:10                                                       ` Steven Rostedt
2019-05-07  1:06                                                       ` torvalds
2019-05-07  1:06                                                         ` Linus Torvalds
2019-05-07  1:04                                                   ` rostedt
2019-05-07  1:04                                                     ` Steven Rostedt
2019-05-07  1:34                                                     ` rostedt
2019-05-07  1:34                                                       ` Steven Rostedt
2019-05-07  1:34                                                     ` torvalds
2019-05-07  1:34                                                       ` Linus Torvalds
2019-05-07  1:53                                                       ` rostedt
2019-05-07  1:53                                                         ` Steven Rostedt
2019-05-07  2:22                                                         ` torvalds [this message]
2019-05-07  2:22                                                           ` Linus Torvalds
2019-05-07  2:58                                                           ` rostedt
2019-05-07  2:58                                                             ` Steven Rostedt
2019-05-07  3:05                                                             ` torvalds
2019-05-07  3:05                                                               ` Linus Torvalds
2019-05-07  3:21                                                               ` rostedt
2019-05-07  3:21                                                                 ` Steven Rostedt
2019-05-07  3:28                                                                 ` torvalds
2019-05-07  3:28                                                                   ` Linus Torvalds
2019-05-07 14:54                                                                   ` torvalds
2019-05-07 14:54                                                                     ` Linus Torvalds
2019-05-07 15:12                                                                     ` rostedt
2019-05-07 15:12                                                                       ` Steven Rostedt
2019-05-07 15:25                                                                       ` rostedt
2019-05-07 15:25                                                                         ` Steven Rostedt
2019-05-07 16:25                                                                         ` rostedt
2019-05-07 16:25                                                                           ` Steven Rostedt
2019-05-07 15:31                                                                       ` torvalds
2019-05-07 15:31                                                                         ` Linus Torvalds
2019-05-07 15:45                                                                         ` rostedt
2019-05-07 15:45                                                                           ` Steven Rostedt
2019-05-07 16:34                                                                         ` peterz
2019-05-07 16:34                                                                           ` Peter Zijlstra
2019-05-07 17:08                                                                           ` torvalds
2019-05-07 17:08                                                                             ` Linus Torvalds
2019-05-07 17:21                                                                             ` jpoimboe
2019-05-07 17:21                                                                               ` Josh Poimboeuf
2019-05-07 21:24                                                                               ` rostedt
2019-05-07 21:24                                                                                 ` Steven Rostedt
2019-05-08  4:50                                                                                 ` torvalds
2019-05-08  4:50                                                                                   ` Linus Torvalds
2019-05-08 16:37                                                                                   ` rostedt
2019-05-08 16:37                                                                                     ` Steven Rostedt
2019-05-07 17:38                                                                             ` peterz
2019-05-07 17:38                                                                               ` Peter Zijlstra
2019-05-07  9:51                                                           ` peterz
2019-05-07  9:51                                                             ` Peter Zijlstra
2019-05-07 14:48                                                           ` luto
2019-05-07 14:48                                                             ` Andy Lutomirski
2019-05-07 14:57                                                             ` torvalds
2019-05-07 14:57                                                               ` Linus Torvalds
2019-05-07 14:13                                                 ` mhiramat
2019-05-07 14:13                                                   ` Masami Hiramatsu
2019-05-07 17:15                                                   ` mhiramat
2019-05-07 17:15                                                     ` Masami Hiramatsu
2019-05-06 14:22                               ` peterz
2019-05-06 14:22                                 ` Peter Zijlstra
2019-05-07  8:57                               ` peterz
2019-05-07  8:57                                 ` Peter Zijlstra
2019-05-07  9:18                                 ` David.Laight
2019-05-07  9:18                                   ` David Laight
2019-05-07 11:30                                   ` peterz
2019-05-07 11:30                                     ` Peter Zijlstra
2019-05-07 12:57                                     ` David.Laight
2019-05-07 12:57                                       ` David Laight
2019-05-07 13:14                                       ` rostedt
2019-05-07 13:14                                         ` Steven Rostedt
2019-05-07 14:50                                         ` David.Laight
2019-05-07 14:50                                           ` David Laight
2019-05-07 14:57                                           ` rostedt
2019-05-07 14:57                                             ` Steven Rostedt
2019-05-07 15:46                                             ` David.Laight
2019-05-07 15:46                                               ` David Laight
2019-05-07 13:32                                       ` peterz
2019-05-07 13:32                                         ` Peter Zijlstra
2019-05-07  9:27                                 ` peterz
2019-05-07  9:27                                   ` Peter Zijlstra
2019-05-07 12:27                                   ` rostedt
2019-05-07 12:27                                     ` Steven Rostedt
2019-05-07 12:41                                     ` peterz
2019-05-07 12:41                                       ` Peter Zijlstra
2019-05-07 12:54                                       ` rostedt
2019-05-07 12:54                                         ` Steven Rostedt
2019-05-07 17:22                                         ` masami.hiramatsu
2019-05-07 17:22                                           ` Masami Hiramatsu
2019-05-07 14:28                                 ` peterz
2019-05-07 14:28                                   ` Peter Zijlstra
2019-05-02 20:48         ` rostedt
2019-05-02 20:48           ` Steven Rostedt
2019-05-06 15:14         ` jpoimboe
2019-05-06 15:14           ` Josh Poimboeuf
2019-05-01 20:28 ` [RFC][PATCH 2/2] ftrace/x86: Emulate call function while updating in breakpoint handler rostedt
2019-05-01 20:28   ` Steven Rostedt
2019-05-03 10:22 ` [RFC][PATCH 1.5/2] x86: Add int3_emulate_call() selftest peterz
2019-05-03 10:22   ` Peter Zijlstra
2019-05-03 18:46   ` rostedt
2019-05-03 18:46     ` Steven Rostedt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAHk-=wjLXmOn=Cp=uOfO4gE01eN_-UcOUyrMTTw5-f_OfPO48Q@mail.gmail.com' \
    --to=linux-kselftest@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).