Linux-Doc Archive on lore.kernel.org
 help / color / Atom feed
From: Masami Hiramatsu <mhiramat@kernel.org>
To: Steven Rostedt <rostedt@goodmis.org>,
	Frank Rowand <frowand.list@gmail.com>
Cc: Ingo Molnar <mingo@redhat.com>,
	Namhyung Kim <namhyung@kernel.org>, Tim Bird <Tim.Bird@sony.com>,
	Jiri Olsa <jolsa@redhat.com>,
	Arnaldo Carvalho de Melo <acme@kernel.org>,
	Tom Zanussi <tom.zanussi@linux.intel.com>,
	Rob Herring <robh+dt@kernel.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	Thomas Gleixner <tglx@linutronix.de>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Alexey Dobriyan <adobriyan@gmail.com>,
	Jonathan Corbet <corbet@lwn.net>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org,
	linux-kernel@vger.kernel.org
Subject: [RFC PATCH v4 00/22] tracing: bootconfig: Boot-time tracing and Extra boot config
Date: Mon,  2 Dec 2019 19:13:18 +0900
Message-ID: <157528159833.22451.14878731055438721716.stgit@devnote2> (raw)

Hello,

This is the 4th version of the series for the boot-time tracing.

Previous version is here.

https://lkml.kernel.org/r/156678933823.21459.4100380582025186209.stgit@devnote2

After submitted the previous one, I had a talk on Plumbers
tracing track, and got positive feedbacks (Thanks everyone!)
and some concerns about bootloader modification to load
an SKC file.

I got many ideas from the discussion instead of passing
configuration file via bootloader. It seems most feasible way
is embedding the SKC in vmlinux or initrd.

In this version, I renamed the SKC to "extra boot config" so that
user can easier understand what it does. And I changed it to load
with the initrd image.

I also tried to integrate a part of configs into current
kernel command line. With this change, we can write long
options for subsystems, module and systemd into the boot config.
(we maybe possible to set sysctl default value via boot config,
 but it's another story.)

This series can be applied on Steve's tracing tree (ftrace/core)
or directly available at;

https://github.com/mhiramat/linux.git ftrace-boottrace-v4


Extra Boot Config
=================

Extra boot config allows admin to pass a tree-structured key-value
list when booting up the kernel. This expands the kernel command
line in an efficient way.

Each key is described as a dot-jointed-words. And user can write
the key-words in tree stlye. (In this version, the tailing ';'
becomes optional. See Documentation/admin-guide/bootconfig.rst)

For example,

 feature.option.foo = 1
 feature.option.bar = 2

can be also written in

 feature.option {
    foo = 1
    bar = 2
 }

or more compact,

 feature.option{foo=1;bar=2}

(Note that in both style, the same words are merged automatically
 and make a single tree)
All values are treated as a string, or array of strings, e.g.

 feature.options = "foo", "bar"

User can see the loaded key-value list via /proc/bootconfig.
The size is limited upto 32KB and 1024 key-words and values
in total.

Boot with a Boot Config
=======================

This version doesn't require to modify boot loaders anymore.
The boot config is loaded with initrd, and there is new "bootconfig"
command under tools/bootconfig.
To add (append) a bootconfig file to an initrd, you can use the
bootconfig command like:

 # tools/bootconfig/bootconfig -a your-config /boot/initrd.img-X.Y.Z

This verifies the configuration file too. 


Boot-time Tracing
=================

Boot-time tracing supports following boot configs. Please read
Documentation/trace/boottime-trace.rst for details.

     - kernel.dump_on_oops [= MODE]
     - kernel.traceoff_on_warning
     - kernel.tp_printk
     - kernel.fgraph_filters = FILTER[, FILTER2...]
     - kernel.fgraph_notraces = FILTER[, FILTER2...]
     - kernel.fgraph_max_depth = MAX_DEPTH
     - ftrace.[instance.INSTANCE.]options = OPT1[,OPT2...]
     - ftrace.[instance.INSTANCE.]trace_clock = CLOCK
     - ftrace.[instance.INSTANCE.]buffer_size = SIZE
     - ftrace.[instance.INSTANCE.]alloc_snapshot
     - ftrace.[instance.INSTANCE.]cpumask = CPUMASK
     - ftrace.[instance.INSTANCE.]events = EVENT[, EVENT2...]
     - ftrace.[instance.INSTANCE.]tracer = TRACER
     - ftrace.[instance.INSTANCE.]ftrace.filters
     - ftrace.[instance.INSTANCE.]ftrace.notraces
     - ftrace.[instance.INSTANCE.]event.GROUP.EVENT.filter = FILTER
     - ftrace.[instance.INSTANCE.]event.GROUP.EVENT.actions = ACTION[, ACTION2...]
     - ftrace.[instance.INSTANCE.]event.GROUP.EVENT.enable
     - ftrace.[instance.INSTANCE.]event.kprobes.EVENT.probes = PROBE[, PROBE2...]
     - ftrace.[instance.INSTANCE.]event.synthetic.EVENT.fields = FIELD[, FIELD2...]

Kernel and Init Command Line
============================

Boot config also supports kernel and init command line parameters
except for early kernel parameters.

In boot config, all key-values start with "kernel." are automatically
merged into user passed boot command line, and key-values which
start with "init." are also passed to init. These options are visible
on /proc/cmdline.

For example,

kernel {
  audit = on
  audit_backlog_limit = 256
}
init.systemd.unified_cgroup_hierarchy = 1


Usage
=====

With this series, we can setup new kprobe and synthetic events, more
complicated event filters and trigger actions including histogram
via supplemental kernel cmdline.

We can add filter and actions for each event, define kprobe events,
and synthetic events with histogram like below.

ftrace.event {
	task.task_newtask {
		filter = "pid < 128"
		enable
	}
	kprobes.vfs_read {
		probes = "vfs_read $arg1 $arg2"
		filter = "common_pid < 200"
		enable
	}
	synthetic.initcall_latency {
		fields = "unsigned long func", "u64 lat"
		actions = "hist:keys=func.sym,lat:vals=lat:sort=lat"
	}
	initcall.initcall_start {
		actions = "hist:keys=func:ts0=common_timestamp.usecs"
	}
	initcall.initcall_finish {
		actions = "hist:keys=func:lat=common_timestamp.usecs-$ts0:onmatch(initcall.initcall_start).initcall_latency(func,$lat)"
	}
}

Also, this supports "instance" node, which allows us to run several
tracers for different purpose at once. For example, one tracer is for
tracing functions start with "user_", and others tracing "kernel_",
you can write boot config as:

ftrace.instance {
	foo {
		tracer = "function"
		ftrace-filters = "user_*"
	}
	bar {
		tracer = "function"
		ftrace-filters = "function_*"
	}
}

The instance node also accepts event nodes so that each instance
can customize its event tracing.

This boot-time trace also supports ftrace kernel parameters.
For example, following kernel parameters

trace_options=sym-addr trace_event=initcall:* tp_printk trace_buf_size=1M ftrace=function ftrace_filter="vfs*"

it can be written in boot config like below.

ftrace {
	options = sym-addr;
	events = "initcall:*";
	tp-printk;
	buffer-size = 1MB;
	ftrace-filters = "vfs*";
}

However, since the initialization timing is different, if you need
to trace very early boot, please use normal kernel parameters.

Some Notes
==========

- Since it is not easy to write boot-time tracing without any bug
  in bootconfig, a user-helper command will be needed.
  That command will generate a boot config file from current ftrace
  settings, or try to apply given boot config setting to the ftrace.

- It is also possible to embed boot config file into the kernel
  image for the environment which doesn't use initrd. But in that
  case, user can not update the boot config.

- As you can see, the EVENT.actions value is a bit ugly since histogram
  interface settings are bit complicated. Maybe we can introduce more
  abstractive syntax for that. e.g.

ftrace.event.synthetic.initcall_latency {
	fields = "unsigned long func", "u64 lat"
	hist {
		from {
			event = initcall.initcall_start
			key = func
			assigns = "ts0=common_timestamp.usecs"
		}
		to {
			event = initcall.initcall_finish
			key = func
			assigns = "lat=common_timestamp.usecs-$ts0"
			onmatch = func, $lat
		}
		keys = func.sym, lat
		vals = lat
		sort = lat
	}
}

Any suggestions, thoughts?

Thank you,

---

Masami Hiramatsu (22):
      bootconfig: Add Extra Boot Config support
      bootconfig: Load boot config from the tail of initrd
      tools: bootconfig: Add bootconfig command
      tools: bootconfig: Add bootconfig test script
      proc: bootconfig: Add /proc/bootconfig to show boot config list
      init/main.c: Alloc initcall_command_line in do_initcall() and free it
      bootconfig: init: Allow admin to use bootconfig for kernel command line
      bootconfig: init: Allow admin to use bootconfig for init command line
      Documentation: bootconfig: Add a doc for extended boot config
      tracing: Apply soft-disabled and filter to tracepoints printk
      tracing: kprobes: Output kprobe event to printk buffer
      tracing: kprobes: Register to dynevent earlier stage
      tracing: Accept different type for synthetic event fields
      tracing: Add NULL trace-array check in print_synth_event()
      tracing/boot: Add boot-time tracing
      tracing/boot: Add per-event settings
      tracing/boot Add kprobe event support
      tracing/boot: Add synthetic event support
      tracing/boot: Add instance node support
      tracing/boot: Add cpu_mask option support
      tracing/boot: Add function tracer filter options
      Documentation: tracing: Add boot-time tracing document


 Documentation/admin-guide/bootconfig.rst      |  176 ++++++
 Documentation/admin-guide/index.rst           |    1 
 Documentation/trace/boottime-trace.rst        |  184 ++++++
 Documentation/trace/index.rst                 |    1 
 MAINTAINERS                                   |    9 
 fs/proc/Makefile                              |    1 
 fs/proc/bootconfig.c                          |   89 +++
 include/linux/bootconfig.h                    |  221 +++++++
 include/linux/trace_events.h                  |    1 
 init/Kconfig                                  |   12 
 init/main.c                                   |  213 ++++++-
 kernel/trace/Kconfig                          |    9 
 kernel/trace/Makefile                         |    1 
 kernel/trace/trace.c                          |   63 +-
 kernel/trace/trace_boot.c                     |  353 +++++++++++
 kernel/trace/trace_events.c                   |    1 
 kernel/trace/trace_events_hist.c              |   14 
 kernel/trace/trace_events_trigger.c           |    2 
 kernel/trace/trace_kprobe.c                   |   81 ++-
 lib/Kconfig                                   |    3 
 lib/Makefile                                  |    2 
 lib/bootconfig.c                              |  783 +++++++++++++++++++++++++
 tools/Makefile                                |   11 
 tools/bootconfig/.gitignore                   |    1 
 tools/bootconfig/Makefile                     |   25 +
 tools/bootconfig/include/linux/bootconfig.h   |    7 
 tools/bootconfig/include/linux/bug.h          |   12 
 tools/bootconfig/include/linux/ctype.h        |    7 
 tools/bootconfig/include/linux/errno.h        |    7 
 tools/bootconfig/include/linux/kernel.h       |   18 +
 tools/bootconfig/include/linux/printk.h       |   13 
 tools/bootconfig/include/linux/string.h       |   32 +
 tools/bootconfig/main.c                       |  337 +++++++++++
 tools/bootconfig/samples/bad-dotword.bconf    |    4 
 tools/bootconfig/samples/bad-empty.bconf      |    1 
 tools/bootconfig/samples/bad-keyerror.bconf   |    2 
 tools/bootconfig/samples/bad-longkey.bconf    |    1 
 tools/bootconfig/samples/bad-manywords.bconf  |    1 
 tools/bootconfig/samples/bad-no-keyword.bconf |    2 
 tools/bootconfig/samples/bad-spaceword.bconf  |    2 
 tools/bootconfig/samples/bad-tree.bconf       |    5 
 tools/bootconfig/samples/bad-value.bconf      |    3 
 tools/bootconfig/samples/good-simple.bconf    |   11 
 tools/bootconfig/samples/good-single.bconf    |    4 
 tools/bootconfig/samples/good-tree.bconf      |   15 
 tools/bootconfig/test-bootconfig.sh           |   80 +++
 46 files changed, 2742 insertions(+), 79 deletions(-)
 create mode 100644 Documentation/admin-guide/bootconfig.rst
 create mode 100644 Documentation/trace/boottime-trace.rst
 create mode 100644 fs/proc/bootconfig.c
 create mode 100644 include/linux/bootconfig.h
 create mode 100644 kernel/trace/trace_boot.c
 create mode 100644 lib/bootconfig.c
 create mode 100644 tools/bootconfig/.gitignore
 create mode 100644 tools/bootconfig/Makefile
 create mode 100644 tools/bootconfig/include/linux/bootconfig.h
 create mode 100644 tools/bootconfig/include/linux/bug.h
 create mode 100644 tools/bootconfig/include/linux/ctype.h
 create mode 100644 tools/bootconfig/include/linux/errno.h
 create mode 100644 tools/bootconfig/include/linux/kernel.h
 create mode 100644 tools/bootconfig/include/linux/printk.h
 create mode 100644 tools/bootconfig/include/linux/string.h
 create mode 100644 tools/bootconfig/main.c
 create mode 100644 tools/bootconfig/samples/bad-dotword.bconf
 create mode 100644 tools/bootconfig/samples/bad-empty.bconf
 create mode 100644 tools/bootconfig/samples/bad-keyerror.bconf
 create mode 100644 tools/bootconfig/samples/bad-longkey.bconf
 create mode 100644 tools/bootconfig/samples/bad-manywords.bconf
 create mode 100644 tools/bootconfig/samples/bad-no-keyword.bconf
 create mode 100644 tools/bootconfig/samples/bad-spaceword.bconf
 create mode 100644 tools/bootconfig/samples/bad-tree.bconf
 create mode 100644 tools/bootconfig/samples/bad-value.bconf
 create mode 100644 tools/bootconfig/samples/good-simple.bconf
 create mode 100644 tools/bootconfig/samples/good-single.bconf
 create mode 100644 tools/bootconfig/samples/good-tree.bconf
 create mode 100755 tools/bootconfig/test-bootconfig.sh

--
Masami Hiramatsu (Linaro) <mhiramat@kernel.org>

             reply index

Thread overview: 29+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-02 10:13 Masami Hiramatsu [this message]
2019-12-02 10:13 ` [RFC PATCH v4 01/22] bootconfig: Add Extra Boot Config support Masami Hiramatsu
2019-12-08 19:34   ` Randy Dunlap
2019-12-09  5:50     ` Masami Hiramatsu
2019-12-09 15:54       ` Steven Rostedt
2019-12-10 14:15         ` Masami Hiramatsu
2020-02-06  9:08         ` Geert Uytterhoeven
2020-02-06  9:41           ` Masami Hiramatsu
2019-12-02 10:13 ` [RFC PATCH v4 02/22] bootconfig: Load boot config from the tail of initrd Masami Hiramatsu
2019-12-02 10:13 ` [RFC PATCH v4 03/22] tools: bootconfig: Add bootconfig command Masami Hiramatsu
2019-12-02 10:14 ` [RFC PATCH v4 04/22] tools: bootconfig: Add bootconfig test script Masami Hiramatsu
2019-12-02 10:14 ` [RFC PATCH v4 05/22] proc: bootconfig: Add /proc/bootconfig to show boot config list Masami Hiramatsu
2019-12-02 10:14 ` [RFC PATCH v4 06/22] init/main.c: Alloc initcall_command_line in do_initcall() and free it Masami Hiramatsu
2019-12-02 10:14 ` [RFC PATCH v4 07/22] bootconfig: init: Allow admin to use bootconfig for kernel command line Masami Hiramatsu
2019-12-02 10:14 ` [RFC PATCH v4 08/22] bootconfig: init: Allow admin to use bootconfig for init " Masami Hiramatsu
2019-12-02 10:14 ` [RFC PATCH v4 09/22] Documentation: bootconfig: Add a doc for extended boot config Masami Hiramatsu
2019-12-02 10:15 ` [RFC PATCH v4 10/22] tracing: Apply soft-disabled and filter to tracepoints printk Masami Hiramatsu
2019-12-02 10:15 ` [RFC PATCH v4 11/22] tracing: kprobes: Output kprobe event to printk buffer Masami Hiramatsu
2019-12-02 10:15 ` [RFC PATCH v4 12/22] tracing: kprobes: Register to dynevent earlier stage Masami Hiramatsu
2019-12-02 10:15 ` [RFC PATCH v4 13/22] tracing: Accept different type for synthetic event fields Masami Hiramatsu
2019-12-02 10:15 ` [RFC PATCH v4 14/22] tracing: Add NULL trace-array check in print_synth_event() Masami Hiramatsu
2019-12-02 10:16 ` [RFC PATCH v4 15/22] tracing/boot: Add boot-time tracing Masami Hiramatsu
2019-12-02 10:16 ` [RFC PATCH v4 16/22] tracing/boot: Add per-event settings Masami Hiramatsu
2019-12-02 10:16 ` [RFC PATCH v4 17/22] tracing/boot Add kprobe event support Masami Hiramatsu
2019-12-02 10:16 ` [RFC PATCH v4 18/22] tracing/boot: Add synthetic " Masami Hiramatsu
2019-12-02 10:16 ` [RFC PATCH v4 19/22] tracing/boot: Add instance node support Masami Hiramatsu
2019-12-02 10:17 ` [RFC PATCH v4 20/22] tracing/boot: Add cpu_mask option support Masami Hiramatsu
2019-12-02 10:17 ` [RFC PATCH v4 21/22] tracing/boot: Add function tracer filter options Masami Hiramatsu
2019-12-02 10:17 ` [RFC PATCH v4 22/22] Documentation: tracing: Add boot-time tracing document Masami Hiramatsu

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=157528159833.22451.14878731055438721716.stgit@devnote2 \
    --to=mhiramat@kernel.org \
    --cc=Tim.Bird@sony.com \
    --cc=acme@kernel.org \
    --cc=adobriyan@gmail.com \
    --cc=akpm@linux-foundation.org \
    --cc=corbet@lwn.net \
    --cc=frowand.list@gmail.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=jolsa@redhat.com \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@redhat.com \
    --cc=namhyung@kernel.org \
    --cc=robh+dt@kernel.org \
    --cc=rostedt@goodmis.org \
    --cc=tglx@linutronix.de \
    --cc=tom.zanussi@linux.intel.com \
    --cc=torvalds@linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Doc Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-doc/0 linux-doc/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-doc linux-doc/ https://lore.kernel.org/linux-doc \
		linux-doc@vger.kernel.org
	public-inbox-index linux-doc

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-doc


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git