Re: [POC 5/5] x86_64: invoke SFrame based stack tracer for user space

Re: [POC 5/5] x86_64: invoke SFrame based stack tracer for user space

[Andrii Nakryiko]

adding bpf@

On Tue, May 2, 2023 at 8:27 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Tue, 2 May 2023 12:53:53 +0200
> Peter Zijlstra <peterz@infradead.org> wrote:
>
> > >     while (entry->nr < entry->max_stack) {
> > >             if (!valid_user_frame(fp, sizeof(frame)))
> > >                     break;
> >
> > This is broken, the sframe stuff is not NMI safe. First you need to
> > change perf to emit a forward reference to a 'next' user trace and then
> > emit the user trace on return-to-user.
> >
> > As is, perf does everything in-place from NMI context.
>
> Exactly. What I would like to see with the new sframe infrastructure is
> just a way to tell it "I want a user stack trace before going back to user
> space". Then all the work can be done in the ptrace path, where it's safe
> to memory map in the sframe section. It's not like the user space stack
> frame will change before going back to user space.
>
> This will allow for asking for the user space stack trace at any context,
> because the work will not be done until later. Of course, this may also
> require user space tooling to be updated to handle this case. Perf may
> already do that.
>
> It would also mean that even if you take multiple profiling hits in one
> system call, you will only get one user space stack trace. But that's a
> good thing. As the stack trace doesn't change, and you don't waste ring
> buffer space with unneeded duplicate stacks.

As discussed in the halls of LSF/MM2023, such lazily mapped .sframe in
approach won't work with BPF's bpf_get_stack() approach, which expects
stack trace to be grabbed and returned synchronously from NMI context.
But we can probably retrofit it into bpf_get_stackid()+STACK_TRACE BPF
map API.

Indu, please cc bpf@vger.kernel.org for future revisions so we can
track and plan accordingly. Thank you!

>
> -- Steve

Re: [POC 5/5] x86_64: invoke SFrame based stack tracer for user space

[Steven Rostedt]

On Tue, 16 May 2023 10:25:52 -0700
Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:

> As discussed in the halls of LSF/MM2023, such lazily mapped .sframe in
> approach won't work with BPF's bpf_get_stack() approach, which expects
> stack trace to be grabbed and returned synchronously from NMI context.
> But we can probably retrofit it into bpf_get_stackid()+STACK_TRACE BPF
> map API.

Note that we will likely not be able to use sframe in NMI context, and if
that's a requirement, then BPF will need to continue using the method it is
currently using.

The best way to access sframe reliable is in normal context. NMI is
special, and really should never had been used to access user space
addresses. That was just a simple solution but not a good one. There's a
lot of hacks just to allow a page fault in NMI context.
See https://lwn.net/Articles/484932/

> 
> Indu, please cc bpf@vger.kernel.org for future revisions so we can
> track and plan accordingly. Thank you!

I'll likely be taking over the kernel side of sframes. I'll be happy to
Cc that work to the bpf mailing list.

-- Steve

Re: [POC 5/5] x86_64: invoke SFrame based stack tracer for user space

[Andrii Nakryiko]

On Tue, May 16, 2023 at 10:38 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Tue, 16 May 2023 10:25:52 -0700
> Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
>
> > As discussed in the halls of LSF/MM2023, such lazily mapped .sframe in
> > approach won't work with BPF's bpf_get_stack() approach, which expects
> > stack trace to be grabbed and returned synchronously from NMI context.
> > But we can probably retrofit it into bpf_get_stackid()+STACK_TRACE BPF
> > map API.
>
> Note that we will likely not be able to use sframe in NMI context, and if
> that's a requirement, then BPF will need to continue using the method it is
> currently using.

Yes, unfortunately. We should give users a way to use sframes, but
transition might be slow.

>
> The best way to access sframe reliable is in normal context. NMI is
> special, and really should never had been used to access user space
> addresses. That was just a simple solution but not a good one. There's a
> lot of hacks just to allow a page fault in NMI context.
> See https://lwn.net/Articles/484932/
>
> >
> > Indu, please cc bpf@vger.kernel.org for future revisions so we can
> > track and plan accordingly. Thank you!
>
> I'll likely be taking over the kernel side of sframes. I'll be happy to
> Cc that work to the bpf mailing list.

Thanks!


>
> -- Steve