[GIT PULL] tracing: Updates for 5.18

[GIT PULL] tracing: Updates for 5.18

[Steven Rostedt]

Linus,


** NOTE **

This has major conflicts with two other trees.

A restructure of include/trace caused a conflict with:

 The nfsd tree for commit:

  tracing: Introduce helpers to safely handle dynamic-sized sockaddrs

(currently it is commit d07c9ad622474616e94572e59e725c2c4a494fb4 in linux-next)

And the custom event sample relies on matching the prototype of
the sched_switch event that was changed by:

 The tip sched/core tree:

  sched/tracing: Don't re-read p->state when emitting sched_switch event

(currently is commit fa2c3254d7cfff5f7a916ab928a562d1165f17bb in linux-next)

Which does not conflict, but will cause the build to fail. Merging the two
trees requires the sample in samples/trace_events/trace_custom_sched.h to
match the prototype of the sched_switch event in include/trace/events/sched.h.

My tree has a branch called:

  trace-merge-next-20220315

That resolved both of the above issues.

For more details see the report of the issues mentioned above:

 https://lore.kernel.org/all/20220315145828.413e9301@canb.auug.org.au/



Tracing updates for 5.18:

- New user_events interface. User space can register an event with the kernel
  describing the format of the event. Then it will receive a byte in a page
  mapping that it can check against. A privileged task can then enable that
  event like any other event, which will change the mapped byte to true,
  telling the user space application to start writing the event to the
  tracing buffer.

- Add new "ftrace_boot_snapshot" kernel command line parameter. When set,
  the tracing buffer will be saved in the snapshot buffer at boot up when
  the kernel hands things over to user space. This will keep the traces that
  happened at boot up available even if user space boot up has tracing as
  well.

- Have TRACE_EVENT_ENUM() also update trace event field type descriptions.
  Thus if a static array defines its size with an enum, the user space trace
  event parsers can still know how to parse that array.

- Add new TRACE_CUSTOM_EVENT() macro. This acts the same as the
  TRACE_EVENT() macro, but will attach to an existing tracepoint. This will
  make one tracepoint be able to trace different content and not be stuck at
  only what the original TRACE_EVENT() macro exports.

- Fixes to tracing error logging.

- Better saving of cmdlines to PIDs when tracing (use the wakeup events for
  mapping).

Please pull the latest trace-v5.18 tree, which can be found at:


  git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git
trace-v5.18

Tag SHA1: 61be4324105e5feff3de2eff421cbea80a8d490f
Head SHA1: 795301d3c28996219d555023ac6863401b6076bc


Beau Belgrave (15):
      user_events: Add minimal support for trace_event into ftrace
      user_events: Add print_fmt generation support for basic types
      user_events: Handle matching arguments from dyn_events
      user_events: Add basic perf and eBPF support
      user_events: Optimize writing events by only copying data once
      user_events: Validate user payloads for size and null termination
      user_events: Add self-test for ftrace integration
      user_events: Add self-test for dynamic_events integration
      user_events: Add self-test for perf_event integration
      user_events: Add self-test for validator boundaries
      user_events: Add sample code for typical usage
      user_events: Add documentation file
      user_events: Fix potential uninitialized pointer while parsing field
      user_events: Prevent dyn_event delete racing with ioctl add/delete
      user_events: Add trace event call as root for low permission cases

Steven Rostedt (Google) (11):
      tracing: Save both wakee and current on wakeup events
      tracing: Fix allocation of last_cmd in last_cmd_set()
      tracing: Fix last_cmd_set() string management in histogram code
      tracing: Allow custom events to be added to the tracefs directory
      tracing: Add sample code for custom trace events
      tracing: Move the defines to create TRACE_EVENTS into their own files
      tracing: Add TRACE_CUSTOM_EVENT() macro
      tracing: Have TRACE_DEFINE_ENUM affect trace event types as well
      tracing: Add snapshot at end of kernel boot up
      tracing/user_events: Use alloc_pages instead of kzalloc() for register pages
      tracing: Have type enum modifications copy the strings

Tom Zanussi (4):
      tracing: Remove size restriction on tracing_log_err cmd strings
      tracing: Remove size restriction on hist trigger cmd error logging
      tracing: Remove size restriction on synthetic event cmd error logging
      tracing: Fix strncpy warning in trace_events_synth.c

----
 Documentation/admin-guide/kernel-parameters.txt   |    8 +
 Documentation/trace/index.rst                     |    1 +
 Documentation/trace/user_events.rst               |  216 +++
 include/linux/ftrace.h                            |   11 +-
 include/linux/trace_events.h                      |   24 +-
 include/trace/define_custom_trace.h               |   77 +
 include/trace/stages/init.h                       |   37 +
 include/trace/stages/stage1_defines.h             |   45 +
 include/trace/stages/stage2_defines.h             |   48 +
 include/trace/stages/stage3_defines.h             |  129 ++
 include/trace/stages/stage4_defines.h             |   57 +
 include/trace/stages/stage5_defines.h             |   83 +
 include/trace/stages/stage6_defines.h             |   86 ++
 include/trace/stages/stage7_defines.h             |   34 +
 include/trace/trace_custom_events.h               |  221 +++
 include/trace/trace_events.h                      |  499 +-----
 include/uapi/linux/user_events.h                  |  116 ++
 kernel/trace/Kconfig                              |   14 +
 kernel/trace/Makefile                             |    1 +
 kernel/trace/ftrace.c                             |    2 +
 kernel/trace/trace.c                              |   73 +-
 kernel/trace/trace.h                              |    2 +-
 kernel/trace/trace_events.c                       |   90 ++
 kernel/trace/trace_events_hist.c                  |   33 +-
 kernel/trace/trace_events_synth.c                 |   14 +-
 kernel/trace/trace_events_user.c                  | 1690 +++++++++++++++++++++
 kernel/trace/trace_sched_switch.c                 |    2 +-
 samples/Kconfig                                   |    8 +-
 samples/Makefile                                  |    1 +
 samples/trace_events/Makefile                     |    2 +
 samples/trace_events/trace_custom_sched.c         |   60 +
 samples/trace_events/trace_custom_sched.h         |   95 ++
 samples/user_events/Makefile                      |    5 +
 samples/user_events/example.c                     |   91 ++
 tools/testing/selftests/user_events/Makefile      |    9 +
 tools/testing/selftests/user_events/dyn_test.c    |  130 ++
 tools/testing/selftests/user_events/ftrace_test.c |  452 ++++++
 tools/testing/selftests/user_events/perf_test.c   |  168 ++
 tools/testing/selftests/user_events/settings      |    1 +
 39 files changed, 4116 insertions(+), 519 deletions(-)
 create mode 100644 Documentation/trace/user_events.rst
 create mode 100644 include/trace/define_custom_trace.h
 create mode 100644 include/trace/stages/init.h
 create mode 100644 include/trace/stages/stage1_defines.h
 create mode 100644 include/trace/stages/stage2_defines.h
 create mode 100644 include/trace/stages/stage3_defines.h
 create mode 100644 include/trace/stages/stage4_defines.h
 create mode 100644 include/trace/stages/stage5_defines.h
 create mode 100644 include/trace/stages/stage6_defines.h
 create mode 100644 include/trace/stages/stage7_defines.h
 create mode 100644 include/trace/trace_custom_events.h
 create mode 100644 include/uapi/linux/user_events.h
 create mode 100644 kernel/trace/trace_events_user.c
 create mode 100644 samples/trace_events/trace_custom_sched.c
 create mode 100644 samples/trace_events/trace_custom_sched.h
 create mode 100644 samples/user_events/Makefile
 create mode 100644 samples/user_events/example.c
 create mode 100644 tools/testing/selftests/user_events/Makefile
 create mode 100644 tools/testing/selftests/user_events/dyn_test.c
 create mode 100644 tools/testing/selftests/user_events/ftrace_test.c
 create mode 100644 tools/testing/selftests/user_events/perf_test.c
 create mode 100644 tools/testing/selftests/user_events/settings
---------------------------

Re: [GIT PULL] tracing: Updates for 5.18

[Linus Torvalds]

On Mon, Mar 21, 2022 at 7:56 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> A restructure of include/trace caused a conflict [..]

Ugh. Disgusting. And in the very same pull request it shows why that
TRACE_CUSTOM_EVENT() thing that caused this all was actually a
horribly bad idea, since it also then DIDN'T WORK due to the kernel
interfaces changing.

So this restructuring seems to have been triggered by something that
was a bad idea to begin with.

But the real problem is here:

> Tracing updates for 5.18:
>
> - New user_events interface. User space can register an event with the kernel
>   describing the format of the event. Then it will receive a byte in a page
>   mapping that it can check against. A privileged task can then enable that
>   event like any other event, which will change the mapped byte to true,
>   telling the user space application to start writing the event to the
>   tracing buffer.

That explanation makes no sense, because it doesn't actually explain *why*.

It explains *what*, but the big issue for new interfaces shoudl always
be why the heck a new interface was needed in the first place.

I've pulled this, but under protest.

              Linus

Re: [GIT PULL] tracing: Updates for 5.18

[Linus Torvalds]

On Mon, Mar 21, 2022 at 7:56 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> My tree has a branch called:
>
>   trace-merge-next-20220315
>
> That resolved both of the above issues.

Oh, and that was completely pointless.

That isn't actually a tree that shows the resolution, that's literally
the linux-next tree.

So I did my normal merge, and then after that decided "ok, let's see
how Steven did it" and did my normal

     git checkout -b test-merge HEAD^
     git pull ..yourtree.. trace-merge-next-20220315

but that doesn't work at all because your trace-merge-next-20220315
has not just the tracing tree, but EVERYTHING. So it just conflicts
EVEN MORE.

So that "trace-merge" branch is only pointless noise. For it to be
useful, it should have contained the conflicting stuff, but *only* the
conflicting stuff. Not everything else too, that just causes even more
conflicts.

Anyway, when I actually compare my resolution to what is in
linux-next, it turns out they are fairly different. But I think my
resolution is the correct one in that it actually makes the sockaddr
cases match the other cases, and the original patch that added them.

I'm not particularly happy with my resolution either, but that's
because I'm not super-happy with that code, but whatever.

               Linus

Re: [GIT PULL] tracing: Updates for 5.18

[Linus Torvalds]

On Wed, Mar 23, 2022 at 11:56 AM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> Anyway, when I actually compare my resolution to what is in
> linux-next, it turns out they are fairly different.

Actually, my resolution is fairly similar - but not identical - to
linux-next, but quite different from the one I find in your branch. I
just got confused by having looked at both.

                Linus

Re: [GIT PULL] tracing: Updates for 5.18

[pr-tracker-bot]

The pull request you sent on Mon, 21 Mar 2022 10:56:21 -0400:

> git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git trace-v5.18

has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/1bc191051dca28fa6d20fd1dc34a1903e7d4fb62

Thank you!

-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/prtracker.html

Re: [GIT PULL] tracing: Updates for 5.18

[Steven Rostedt]

On Wed, 23 Mar 2022 11:47:07 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Mon, Mar 21, 2022 at 7:56 AM Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > A restructure of include/trace caused a conflict [..]  
> 
> Ugh. Disgusting. And in the very same pull request it shows why that
> TRACE_CUSTOM_EVENT() thing that caused this all was actually a
> horribly bad idea, since it also then DIDN'T WORK due to the kernel
> interfaces changing.

I thought I explained the reasoning behind that in my pull request.

CUSTOM events have to match the function prototype to the tracepoint that
they are on top of (which is created by the TRACE_EVENT), and if they do
not, bad things can happen at run time. Thus, the added check to make sure
it will fail at build time.

> 
> So this restructuring seems to have been triggered by something that
> was a bad idea to begin with.
> 
> But the real problem is here:
> 
> > Tracing updates for 5.18:
> >
> > - New user_events interface. User space can register an event with the kernel
> >   describing the format of the event. Then it will receive a byte in a page
> >   mapping that it can check against. A privileged task can then enable that
> >   event like any other event, which will change the mapped byte to true,
> >   telling the user space application to start writing the event to the
> >   tracing buffer.  
> 
> That explanation makes no sense, because it doesn't actually explain *why*.

Sorry, sometimes when you are deep into the work, things that are obvious
to yourself may not be obvious to others. Being able to know those things
is an art onto itself.

> 
> It explains *what*, but the big issue for new interfaces shoudl always
> be why the heck a new interface was needed in the first place.

The why is that we have applications that run in containers that want
events to be traced. But we do not want to always trace those events. These
applications will not have an interface to talk to, to say "start tracing
X", as they are inside a container and various other reasons.

Thus, the applications can add their own trace events, and then the admin
that is outside the container can enable the events inside the container,
by enabling them from the tracefs file system, which will set a byte in the
mapping that happened when the application registered the event, where it
can now know to start tracing the event.

It does require that the application has a conditional jmp on that byte,
and not be optimized like the kernel is with static_branches.

I hope that explains things a bit better.

> 
> I've pulled this, but under protest.

Thank you.

-- Steve

Re: [GIT PULL] tracing: Updates for 5.18

[Steven Rostedt]

On Wed, 23 Mar 2022 12:20:33 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Wed, Mar 23, 2022 at 11:56 AM Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > Anyway, when I actually compare my resolution to what is in
> > linux-next, it turns out they are fairly different.  
> 
> Actually, my resolution is fairly similar - but not identical - to
> linux-next, but quite different from the one I find in your branch. I
> just got confused by having looked at both.

I'll take a look at yours. I gave that branch to Stephen to help him resolve
it, but didn't look at what he had done to see if it was different.

I didn't merge against your branch because when I did the pull request you
didn't have the nfsd tree in yet.

-- Steve

Re: [GIT PULL] tracing: Updates for 5.18

[Linus Torvalds]

On Wed, Mar 23, 2022 at 12:43 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> I'll take a look at yours. I gave that branch to Stephen to help him resolve
> it, but didn't look at what he had done to see if it was different.

I may actually have messed up my comparison with your tree, because I
was doing some of that while in my "test-merge" branch (which had that
dirty state due to the merge not working). Maybe I screwed up twice.

I think the only difference I have against linux-next is because I added the

  #undef __assign_sockaddr
...
  #undef __assign_rel_sockaddr

lines to stage6 to match the other pre-existing non-sockaddr cases.

              Linus

Re: [GIT PULL] tracing: Updates for 5.18

[Steven Rostedt]

On Wed, 23 Mar 2022 12:49:08 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Wed, Mar 23, 2022 at 12:43 PM Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > I'll take a look at yours. I gave that branch to Stephen to help him resolve
> > it, but didn't look at what he had done to see if it was different.  
> 
> I may actually have messed up my comparison with your tree, because I
> was doing some of that while in my "test-merge" branch (which had that
> dirty state due to the merge not working). Maybe I screwed up twice.
> 
> I think the only difference I have against linux-next is because I added the
> 
>   #undef __assign_sockaddr
> ...
>   #undef __assign_rel_sockaddr
> 
> lines to stage6 to match the other pre-existing non-sockaddr cases.

Yeah, that looks correct. Looking at my merge branch, it appears I messed
it up :-p

But what is in your tree looks like what it should be.

I also should rename the files to what they are doing. The "stage1, stage2,
etc" is meaningless :-p

I think I'll go write up a patch to do that.

-- Steve