From: Josh Poimboeuf <jpoimboe@redhat.com>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: Aleksa Sarai <cyphar@cyphar.com>,
"Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com>,
Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>,
"David S. Miller" <davem@davemloft.net>,
Masami Hiramatsu <mhiramat@kernel.org>,
Jonathan Corbet <corbet@lwn.net>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>,
Arnaldo Carvalho de Melo <acme@kernel.org>,
Alexander Shishkin <alexander.shishkin@linux.intel.com>,
Jiri Olsa <jolsa@redhat.com>, Namhyung Kim <namhyung@kernel.org>,
Shuah Khan <shuah@kernel.org>,
Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Brendan Gregg <bgregg@netflix.com>,
Christian Brauner <christian@brauner.io>,
Aleksa Sarai <asarai@suse.de>,
netdev@vger.kernel.org, linux-doc@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org
Subject: Re: [PATCH v3 1/2] kretprobe: produce sane stack traces
Date: Fri, 2 Nov 2018 10:43:26 -0500 [thread overview]
Message-ID: <20181102154325.bt6xoysl4xdl33wd@treble> (raw)
In-Reply-To: <20181102091658.1bc979a4@gandalf.local.home>
On Fri, Nov 02, 2018 at 09:16:58AM -0400, Steven Rostedt wrote:
> On Fri, 2 Nov 2018 17:59:32 +1100
> Aleksa Sarai <cyphar@cyphar.com> wrote:
>
> > As an aside, I just tested with the frame unwinder and it isn't thrown
> > off-course by kretprobe_trampoline (though obviously the stack is still
> > wrong). So I think we just need to hook into the ORC unwinder to get it
> > to continue skipping up the stack, as well as add the rewriting code for
> > the stack traces (for all unwinders I guess -- though ideally we should
>
> I agree that this is the right solution.
Sounds good to me.
However, it would be *really* nice if function graph and kretprobes
shared the same infrastructure, like they do for function entry.
There's a lot of duplicated effort there.
> > do this without having to add the same code to every architecture).
>
> True, and there's an art to consolidating the code between
> architectures.
>
> I'm currently looking at function graph and seeing if I can consolidate
> it too. And I'm also trying to get multiple uses to hook into its
> infrastructure. I think I finally figured out a way to do so.
>
> The reason it is difficult, is that you need to maintain state between
> the entry of a function and the exit for each task and callback that is
> registered. Hence, it's a 3x tuple (function stack, task, callbacks).
> And this must be maintained with preemption. A task may sleep for
> minutes, and the state needs to be retained.
>
> The only state that must be retained is the function stack with the
> task, because if that gets out of sync, the system crashes. But the
> callback state can be removed.
>
> Here's what is there now:
>
> When something is registered with the function graph tracer, every
> task gets a shadowed stack. A hook is added to fork to add shadow
> stacks to new tasks. Once a shadow stack is added to a task, that
> shadow stack is never removed until the task exits.
>
> When the function is entered, the real return code is stored in the
> shadow stack and the trampoline address is put in its place.
>
> On return, the trampoline is called, and it will pop off the return
> code from the shadow stack and return to that.
>
> The issue with multiple users, is that different users may want to
> trace different functions. On entry, the user could say it doesn't want
> to trace the current function, and the return part must not be called
> on exit. Keeping track of which user needs the return called is the
> tricky part.
>
> Here's what I plan on implementing:
>
> Along with a shadow stack, I was going to add a 4096 byte (one page)
> array that holds 64 8 byte masks to every task as well. This will allow
> 64 simultaneous users (which is rather extreme). If we need to support
> more, we could allocate another page for all tasks. The 8 byte mask
> will represent each depth (allowing to do this for 64 function call
> stack depth, which should also be enough).
>
> Each user will be assigned one of the masks. Each bit in the mask
> represents the depth of the shadow stack. When a function is called,
> each user registered with the function graph tracer will get called
> (if they asked to be called for this function, via the ftrace_ops
> hashes) and if they want to trace the function, then the bit is set in
> the mask for that stack depth.
>
> When the function exits the function and we pop off the return code
> from the shadow stack, we then look at all the bits set for the
> corresponding users, and call their return callbacks, and ignore
> anything that is not set.
>
>
> When a user is unregistered, it the corresponding bits that represent
> it are cleared, and it the return callback will not be called. But the
> tasks being traced will still have their shadow stack to allow it to
> get back to normal.
>
> I'll hopefully have a prototype ready by plumbers.
Why do we need multiple users? It would be a lot simpler if we could
just enforce a single user per fgraphed/kretprobed function (and return
-EBUSY if it's already being traced/probed).
> And this too will require each architecture to probably change. As a
> side project to this, I'm going to try to consolidate the function
> graph code among all the architectures as well. Not an easy task.
Do you mean implementing HAVE_FUNCTION_GRAPH_RET_ADDR_PTR for all the
arches? If so, I think have an old crusty patch which attempted to
that. I could try to dig it up if you're interested.
--
Josh
next prev parent reply other threads:[~2018-11-02 15:43 UTC|newest]
Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-01 8:35 [PATCH v3 0/2] kretprobe: produce sane stack traces Aleksa Sarai
2018-11-01 8:35 ` [PATCH v3 1/2] " Aleksa Sarai
2018-11-01 15:20 ` Masami Hiramatsu
2018-11-01 21:13 ` Aleksa Sarai
2018-11-02 3:04 ` Masami Hiramatsu
2018-11-02 4:37 ` Aleksa Sarai
2018-11-03 12:47 ` Masami Hiramatsu
2018-11-02 0:47 ` Steven Rostedt
2018-11-02 5:05 ` Aleksa Sarai
2018-11-02 6:59 ` Aleksa Sarai
2018-11-02 13:16 ` Steven Rostedt
2018-11-02 15:43 ` Josh Poimboeuf [this message]
2018-11-02 16:13 ` Steven Rostedt
2018-11-03 13:00 ` Masami Hiramatsu
2018-11-03 13:13 ` Steven Rostedt
2018-11-03 16:34 ` Masami Hiramatsu
2018-11-03 17:30 ` Steven Rostedt
2018-11-03 17:33 ` Steven Rostedt
2018-11-04 2:25 ` Masami Hiramatsu
2018-11-03 7:02 ` Aleksa Sarai
2018-11-04 11:59 ` Aleksa Sarai
2018-11-06 22:15 ` Steven Rostedt
2018-11-08 7:46 ` Aleksa Sarai
2018-11-08 8:04 ` Aleksa Sarai
2018-11-08 14:44 ` Josh Poimboeuf
2018-11-09 7:26 ` Masami Hiramatsu
2018-11-09 15:10 ` Aleksa Sarai
2018-11-09 7:15 ` Masami Hiramatsu
2018-11-09 15:06 ` Aleksa Sarai
2018-11-10 15:31 ` Masami Hiramatsu
2018-11-12 10:38 ` Aleksa Sarai
2018-11-03 13:23 ` Masami Hiramatsu
2018-11-02 7:58 ` Aleksa Sarai
2018-11-02 4:01 ` kbuild test robot
2018-11-01 8:35 ` [PATCH v3 2/2] trace: remove kretprobed checks Aleksa Sarai
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181102154325.bt6xoysl4xdl33wd@treble \
--to=jpoimboe@redhat.com \
--cc=acme@kernel.org \
--cc=alexander.shishkin@linux.intel.com \
--cc=anil.s.keshavamurthy@intel.com \
--cc=asarai@suse.de \
--cc=ast@kernel.org \
--cc=bgregg@netflix.com \
--cc=christian@brauner.io \
--cc=corbet@lwn.net \
--cc=cyphar@cyphar.com \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=jolsa@redhat.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=mhiramat@kernel.org \
--cc=mingo@redhat.com \
--cc=namhyung@kernel.org \
--cc=naveen.n.rao@linux.vnet.ibm.com \
--cc=netdev@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=shuah@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).