linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line
@ 2021-08-03 17:05 Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 01/22] libtracefs: Added new API tracefs_sql() Steven Rostedt
                   ` (21 more replies)
  0 siblings, 22 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:05 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

Special thanks to:

  Lukas Bulwahn for first introducing the idea at RT Summit on the Summit
    Spring of 2019.
    
  Daniel Black for sorting out the syntax of mapping the synthetic
    event creations into SQL statements at Linux Plumbers, Fall of 2019.

This patch set depends on:

   https://patchwork.kernel.org/project/linux-trace-devel/list/?series=525783

Lots of bug fixes. Full diff of changes since v2 at bottom of this email.

Below is from the man page, which has a fully functional parser as its
example (in man page, not here. But the usage of that parser is described in
the FUNCTIONAL EXAMPLE section below).

Also, see patch 18, which extracts the program from the man page and builds
it via:

    make sqlhist

For easy testing of this feature.

struct tracefs_synth *tracefs_sql(struct tep_handle *tep, const char *name,
				  const char *sql_buffer, char **err);

struct tracefs_synth *tracefs_sql(struct tep_handle *tep, const char *name,
                                  const char *sql_buffer, char **err);

Synthetic events are dynamically created events that attach two existing
events together via one or more matching fields between the two events. It
can be used to find the latency between the events, or to simply pass fields
of the first event on to the second event to display as one event.

The Linux kernel interface to create synthetic events is complex, and there
needs to be a better way to create synthetic events that is easy and can be
understood via existing technology.

If you think of each event as a table, where the fields are the column of
the table and each instance of the event as a row, you can understand how
SQL can be used to attach two events together and for another event (table).
Utilizing the SQL SELECT FROM JOIN ON [ WHERE ] syntax, a synthetic event
can easily be created from two different events.

For simple SQL queries to make a histogram instead of a synthetic event, see
HISTOGRAMS below.

tracefs_sql() takes in a tep handler (See tep_local_events(3)) that is used
to verify the events within the sql_buffer expression. The name is the name
of the synthetic event to create. If err points to an address of a string,
it will be filled with a detailed message on any type of parsing error,
including fields that do not belong to an event, or if the events or fields
are not properly compared.

The example program below is a fully functional parser where it will create
a synthetic event from a SQL syntax passed in via the command line or a
file.

The SQL format is as follows:

    SELECT <fields> FROM <start-event> JOIN <end-event> ON <matching-fields> WHERE <filter>

Note, although the examples show the SQL commands in uppercase, they are not
required to be so. That is, you can use "SELECT" or "select" or "sElEct".

For example:

    SELECT syscalls.sys_enter_read.fd, syscalls.sys_exit_read.ret FROM syscalls.sys_enter_read
       JOIN syscalls.sys_exit_read
       ON syscalls.sys_enter_read.common_pid = syscalls.sys_exit_write.common_pid

Will create a synthetic event that with the fields:

    u64 fd; s64 ret;

Because the function takes a tep handle, and usually all event names are
unique, you can leave off the system (group) name of the event, and
tracefs_sql() will discover the system for you.

That is, the above statement would work with:

    SELECT sys_enter_read.fd, sys_exit_read.ret FROM sys_enter_read JOIN sys_exit_read
       ON sys_enter_read.common_pid = sys_exit_write.common_pid

The AS keyword can be used to name the fields as well as to give an alias to
the events, such that the above can be simplified even more as;

    SELECT start.fd, end.ret FROM sys_enter_read AS start JOIN sys_exit_read AS end ON start.common_pid = end.common_pid

The above aliases sys_enter_read as start and sys_exit_read as end and uses
those aliases to reference the event throughout the statement.

Using the AS keyword in the selection portion of the SQL statement will
define what those fields will be called in the synthetic event.

    SELECT start.fd AS filed, end.ret AS return FROM sys_enter_read AS start JOIN sys_exit_read AS end
       ON start.common_pid = end.common_pid

The above labels the fd of start as filed and the ret of end as return where
the synthetic event that is created will now have the fields:

    u64 filed; s64 return;

The fields can also be calculated with results passed to the synthetic event:

    select start.truesize, end.len, (start.truesize - end.len) as diff from napi_gro_receive_entry as start
       JOIN netif_receive_skb as end ON start.skbaddr = end.skbaddr

Which would show the truesize" of the napi_gro_receive_entry event, the
actual len of the content, shown by the netif_receive_skb, and alse the
delta between the two and expressed by the field *diff.

The code also supports recording the timestamps at either event, and
performing calculations on them. For wakeup latency, you have:

    select start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) as lat from sched_waking as start
       JOIN sched_switch as end ON start.pid = end.next_pid

The above will create a synthetic event that records the pid of the task
being woken up, and the time difference between the sched_waking event and
the sched_switch event. The TIMESTAMP_USECS will truncate the time down to
microseconds as the timestamp usually recorded in the tracing buffer has
nanosecond resolution. If you do not want that truncation, use TIMESTAMP
instead of TIMESTAMP_USECS.

Finally, the WHERE clause can be added, that will let you add filters on
either or both events.

    select start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) as lat from sched_waking as start
       JOIN sched_switch as end ON start.pid = end.next_pid
       WHERE start.prio < 100 && (!(end.prev_pid < 1 || end.prev_prio > 100) || end.prev_pid == 0)

NOTE

Although both events can be used together in the WHERE clause, they must not
be mixed outside the top most "&&" statements. You can not OR (||) the
events together, where a filter of one event is OR’d to a filter of the
other event. This does not make sense, as the synthetic event requires both
events to take place to be recorded. If one is filtered out, then the
synthetic event does not execute.

    select start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) as lat from sched_waking as start
       JOIN sched_switch as end ON start.pid = end.next_pid
       WHERE start.prio < 100 && end.prev_prio < 100

The above is valid.

Where as the below is not.

    select start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) as lat from sched_waking as start
       JOIN sched_switch as end ON start.pid = end.next_pid
       WHERE start.prio < 100 || end.prev_prio < 100

KEYWORDS AS EVENT FIELDS

In some cases, an event may have a keyword. For example,
regcache_drop_region has "from" as a field and the following will not work

      select from from regcache_drop_region

In such cases, add a backslash to the conflicting field, and this will tell
the parser that the "from" is a field and not a keyword:

      select \from from regcache_drop_region

HISTOGRAMS

Simple SQL statements without the JOIN ON may also be used, which will
create a histogram instead. When doing this, the struct tracefs_hist
descriptor can be retrieved from the returned synthetic event descriptor via
the tracefs_synth_get_start_hist(3).

In order to utilize the histogram types (see xxx) the CAST command of SQL
can be used.

That is:

      select CAST(common_pid AS comm), CAST(id AS syscall) FROM sys_enter

Which produces:

     # echo 'hist:keys=common_pid.execname,id.syscall' > events/raw_syscalls/sys_enter/trigger

     # cat events/raw_syscalls/sys_enter/hist

    { common_pid: bash            [     18248], id: sys_setpgid                   [109] } hitcount:          1
    { common_pid: sendmail        [      1812], id: sys_read                      [  0] } hitcount:          1
    { common_pid: bash            [     18247], id: sys_getpid                    [ 39] } hitcount:          1
    { common_pid: bash            [     18247], id: sys_dup2                      [ 33] } hitcount:          1
    { common_pid: gmain           [     13684], id: sys_inotify_add_watch         [254] } hitcount:          1
    { common_pid: cat             [     18247], id: sys_access                    [ 21] } hitcount:          1
    { common_pid: bash            [     18248], id: sys_getpid                    [ 39] } hitcount:          1
    { common_pid: cat             [     18247], id: sys_fadvise64                 [221] } hitcount:          1
    { common_pid: sendmail        [      1812], id: sys_openat                    [257] } hitcount:          1
    { common_pid: less            [     18248], id: sys_munmap                    [ 11] } hitcount:          1
    { common_pid: sendmail        [      1812], id: sys_close                     [  3] } hitcount:          1
    { common_pid: gmain           [      1534], id: sys_poll                      [  7] } hitcount:          1
    { common_pid: bash            [     18247], id: sys_execve                    [ 59] } hitcount:          1

Note, string fields may not be cast.

The possible types to cast to are:

HEX - convert the value to use hex and not decimal

SYM - convert a pointer to symbolic (kallsyms values)

SYM-OFFSET - convert a pointer to symbolic and include the offset.

SYSCALL - convert the number to the mapped system call name

EXECNAME or COMM - can only be used with the common_pid field. Will show the
task name of the process.

LOG or LOG2 - bucket the key values in a log 2 values (1, 2, 3-4, 5-8, 9-16, 17-32, ...)

The above fields are not case sensitive, and "LOG2" works as good as "log".

A special CAST to COUNTER or COUNTER will make the field a value and not a
key. For example:

      SELECT common_pid, CAST(bytes_req AS _COUNTER_) FROM kmalloc

Which will create

      echo 'hist:keys=common_pid:vals=bytes_req' > events/kmem/kmalloc/trigger

      cat events/kmem/kmalloc/hist

    { common_pid:       1812 } hitcount:          1  bytes_req:         32
    { common_pid:       9111 } hitcount:          2  bytes_req:        272
    { common_pid:       1768 } hitcount:          3  bytes_req:       1112
    { common_pid:          0 } hitcount:          4  bytes_req:        512
    { common_pid:      18297 } hitcount:         11  bytes_req:       2004

RETURN VALUE

Returns 0 on success and -1 on failure. On failure, if err is defined, it
will be allocated to hold a detailed description of what went wrong if it
the error was caused by a parsing error, or that an event, field does not
exist or is not compatible with what it was combined with.

FUNCTIONAL EXAMPLE:
-------------------

After applying this patch, and installing it. If you compile the example from the man
page (calling it sqlhist.c):

 >$ gcc -o sqlhist sqlhist.c `pkg-config --cflags --libs libtracefs`
 >$ su
 ># ./sqlhist -n syscall_wait -e 'select start.id, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) as lat  
    from sys_enter as start join sys_exit as end on start.common_pid = end.common_pid
    where start.id != 23 && start.id != 7 && start.id != 61 && start.id != 230 &&
          start.id != 232 && start.id != 270 && start.id != 271 && start.id != 202'


(All the start.id filtering is hiding the syscalls that block for a long time)

 ># echo 'hist:keys=id.syscall,lat.buckets=10:sort=lat' > /sys/kernel/tracing/events/synthetic/syscall_wait/trigger  

<wait a while>

 ># cat /sys/kernel/tracing/events/synthetic/syscall_wait/hist  
 # event histogram
 #
 # trigger info: hist:keys=id.syscall,lat.buckets=10:vals=hitcount:sort=lat.buckets=10:size=2048 [active]
 #

{ id: sys_fadvise64                 [221], lat: ~ 0-9 } hitcount:          1
{ id: sys_fcntl                     [ 72], lat: ~ 0-9 } hitcount:          5
{ id: sys_read                      [  0], lat: ~ 0-9 } hitcount:         51
{ id: sys_dup2                      [ 33], lat: ~ 0-9 } hitcount:          1
{ id: sys_newfstat                  [  5], lat: ~ 0-9 } hitcount:          7
{ id: sys_getpid                    [ 39], lat: ~ 0-9 } hitcount:         18
{ id: sys_openat                    [257], lat: ~ 0-9 } hitcount:          3
{ id: sys_newstat                   [  4], lat: ~ 0-9 } hitcount:         22
{ id: sys_newlstat                  [  6], lat: ~ 0-9 } hitcount:         44
{ id: sys_write                     [  1], lat: ~ 0-9 } hitcount:         23
{ id: sys_brk                       [ 12], lat: ~ 0-9 } hitcount:          4
{ id: sys_readlink                  [ 89], lat: ~ 0-9 } hitcount:         11
{ id: sys_close                     [  3], lat: ~ 0-9 } hitcount:         21
{ id: sys_mprotect                  [ 10], lat: ~ 0-9 } hitcount:          4
{ id: sys_arch_prctl                [158], lat: ~ 0-9 } hitcount:          2
{ id: sys_getuid                    [102], lat: ~ 0-9 } hitcount:         12
{ id: sys_rt_sigaction              [ 13], lat: ~ 0-9 } hitcount:        105
{ id: sys_statfs                    [137], lat: ~ 0-9 } hitcount:          8
{ id: sys_rt_sigprocmask            [ 14], lat: ~ 0-9 } hitcount:        121
{ id: sys_inotify_add_watch         [254], lat: ~ 0-9 } hitcount:          7
{ id: sys_mmap                      [  9], lat: ~ 0-9 } hitcount:         10
{ id: sys_ioctl                     [ 16], lat: ~ 0-9 } hitcount:         32
{ id: sys_pread64                   [ 17], lat: ~ 0-9 } hitcount:          4
{ id: sys_setpgid                   [109], lat: ~ 0-9 } hitcount:          2
{ id: sys_lseek                     [  8], lat: ~ 0-9 } hitcount:         11
{ id: sys_access                    [ 21], lat: ~ 0-9 } hitcount:          1
{ id: sys_rt_sigprocmask            [ 14], lat: ~ 10-19 } hitcount:          9
{ id: sys_munmap                    [ 11], lat: ~ 10-19 } hitcount:          1
{ id: sys_newstat                   [  4], lat: ~ 10-19 } hitcount:          1
{ id: sys_rt_sigaction              [ 13], lat: ~ 10-19 } hitcount:          1
{ id: sys_read                      [  0], lat: ~ 10-19 } hitcount:         11
{ id: sys_openat                    [257], lat: ~ 10-19 } hitcount:         13
{ id: sys_getpid                    [ 39], lat: ~ 10-19 } hitcount:          1
{ id: sys_inotify_add_watch         [254], lat: ~ 10-19 } hitcount:          2
{ id: sys_write                     [  1], lat: ~ 10-19 } hitcount:         10
{ id: sys_read                      [  0], lat: ~ 20-29 } hitcount:          4
{ id: sys_pipe                      [ 22], lat: ~ 20-29 } hitcount:          1
{ id: sys_write                     [  1], lat: ~ 20-29 } hitcount:         11
{ id: sys_write                     [  1], lat: ~ 30-39 } hitcount:          2
{ id: sys_inotify_add_watch         [254], lat: ~ 30-39 } hitcount:          1
{ id: sys_write                     [  1], lat: ~ 40-49 } hitcount:          1
{ id: sys_inotify_add_watch         [254], lat: ~ 40-49 } hitcount:          1
{ id: sys_openat                    [257], lat: ~ 40-49 } hitcount:          1
{ id: sys_read                      [  0], lat: ~ 70-79 } hitcount:          8
{ id: sys_read                      [  0], lat: ~ 80-89 } hitcount:          1
{ id: sys_read                      [  0], lat: ~ 110-119 } hitcount:          2
{ id: sys_clone                     [ 56], lat: ~ 240-249 } hitcount:          1
{ id: sys_execve                    [ 59], lat: ~ 350-359 } hitcount:          1
{ id: sys_write                     [  1], lat: ~ 1960-1969 } hitcount:          1

Totals:
    Hits: 615
    Entries: 49
    Dropped: 0

Steven Rostedt (VMware) (22):
  libtracefs: Added new API tracefs_sql()
  tracefs: Add unit tests for tracefs_sql()
  libtracefs: Add comparing start and end fields in tracefs_sql()
  libtracefs: Add unit test to test tracefs_sql() compare
  libtracefs: Add filtering for start and end events in tracefs_sql()
  libtracefs: Add unit test to test tracefs_sql() where clause
  libtracefs: Make sqlhist parser reentrant
  libtracefs: Make parser unique to libtracefs
  libtracefs: Add line number and index to expr structure
  libtracefs: Add error message when match fields are not FROM and JOIN
    events
  libtracefs: Add error message when match or init fails from bad events
  libtracefs; Add error message for bad selections to SQL sequence
  libtracefs: Add error message when compare fields fail
  libtracefs: Add error message for grouping events in SQL filter
  libtracefs: Add error message for bad filters in SQL statement
  libtracefs: Add error message when calculation has no label
  libtracefs: Add man page for tracefs_sql()
  libtracefs: Add Makefile rule to create sqlhist
  libtracefs: Allow for simple SQL statements to create a histogram
  libtracefs: Allow trace_sql() to take keywords for fields with
    backslash
  libtracefs: Add CAST() syntax to SQL parsing for histogram types
  libtracefs: Add CAST(x AS _COUNTER_) syntax to create values in
    histograms

 Documentation/libtracefs-sql.txt |  504 ++++++++++
 Makefile                         |   10 +
 include/tracefs-local.h          |    9 +
 include/tracefs.h                |    6 +
 src/Makefile                     |   13 +
 src/sqlhist-parse.h              |   76 ++
 src/sqlhist.l                    |   98 ++
 src/sqlhist.y                    |  250 +++++
 src/tracefs-hist.c               |  163 ++-
 src/tracefs-sqlhist.c            | 1622 ++++++++++++++++++++++++++++++
 utest/tracefs-utest.c            |   78 ++
 11 files changed, 2790 insertions(+), 39 deletions(-)
 create mode 100644 Documentation/libtracefs-sql.txt
 create mode 100644 src/sqlhist-parse.h
 create mode 100644 src/sqlhist.l
 create mode 100644 src/sqlhist.y
 create mode 100644 src/tracefs-sqlhist.c

-- 
2.30.2



diff --git a/Documentation/libtracefs-sql.txt b/Documentation/libtracefs-sql.txt
index 7b616fa..91c99d7 100644
--- a/Documentation/libtracefs-sql.txt
+++ b/Documentation/libtracefs-sql.txt
@@ -42,7 +42,7 @@ with a detailed message on any type of parsing error, including fields that do n
 to an event, or if the events or fields are not properly compared.
 
 The example program below is a fully functional parser where it will create a synthetic
-event from a SQL syntax passed in via the command line or a file. 
+event from a SQL syntax passed in via the command line or a file.
 
 The SQL format is as follows:
 
diff --git a/Makefile b/Makefile
index 96d7f55..7f382c3 100644
--- a/Makefile
+++ b/Makefile
@@ -364,11 +364,21 @@ $(bdir)/libtracefs.so.$(TRACEFS_VERSION): force
 	$(Q)mkdir -p $(bdir)
 	$(Q)$(MAKE) -C $(src)/src libtracefs.so
 
+$(bdir)/sqlhist.c: Documentation/libtracefs-sql.txt
+	cat $< | sed -ne '/^EXAMPLE/,/FILES/ { /EXAMPLE/,+2d ; /^FILES/d ;  /^--/d ; p}' > $@
+
+$(bdir)/sqlhist.o: $(bdir)/sqlhist.c
+	$(CC) -g -Wall -c -o $@ $^ -Iinclude/ $(LIBTRACEEVENT_INCLUDES)
+
+sqlhist: $(bdir)/sqlhist.o $(LIBTRACEFS_STATIC)
+	$(CC) -o $@ $^ $(LIBTRACEEVENT_LIBS)
+
 clean:
 	$(MAKE) -C $(src)/utest clean
 	$(MAKE) -C $(src)/src clean
 	$(RM) $(TARGETS) $(bdir)/*.a $(bdir)/*.so $(bdir)/*.so.* $(bdir)/*.o $(bdir)/.*.d
 	$(RM) $(PKG_CONFIG_FILE)
 	$(RM) $(VERSION_FILE)
+	$(RM) $(bdir)/sqlhist.o $(bdir)/sqlhist.c sqlhist
 
 .PHONY: clean
diff --git a/src/tracefs-hist.c b/src/tracefs-hist.c
index c3aecf9..7f9cf38 100644
--- a/src/tracefs-hist.c
+++ b/src/tracefs-hist.c
@@ -1468,7 +1468,7 @@ tracefs_synth_get_start_hist(struct tracefs_synth *synth)
 			hist = tracefs_hist_alloc(tep, system, event,
 						  key, type);
 			if (!hist)
-				break;
+				return NULL;
 		}
 	}
 
diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
index 1a3cf37..6224677 100644
--- a/src/tracefs-sqlhist.c
+++ b/src/tracefs-sqlhist.c
@@ -36,7 +36,8 @@ enum alias_type {
 struct field {
 	struct expr		*next;	/* private link list */
 	const char		*system;
-	const char		*event;
+	const char		*event_name;
+	struct tep_event	*event;
 	const char		*raw;
 	const char		*label;
 	const char		*field;
@@ -515,89 +516,175 @@ __hidden int table_start(struct sqlhist_bison *sb)
 	return 0;
 }
 
-static int update_vars(struct sql_table *table, struct field *event)
+static int test_event_exists(struct tep_handle *tep,
+			     struct sqlhist_bison *sb,
+			     struct expr *expr, struct tep_event **pevent)
+{
+	struct field *field = &expr->field;
+	const char *system = field->system;
+	const char *event = field->event_name;
+
+	if (!field->event)
+		field->event = tep_find_event_by_name(tep, system, event);
+	if (pevent)
+		*pevent = field->event;
+
+	if (field->event)
+		return 0;
+
+	sb->line_no = expr->line;
+	sb->line_idx = expr->idx;
+
+	parse_error(sb, field->raw, "event not found\n");
+	return -1;
+}
+
+static int test_field_exists(struct tep_handle *tep,
+			     struct sqlhist_bison *sb,
+			     struct expr *expr)
+{
+	struct field *field = &expr->field;
+	struct tep_format_field *tfield;
+	char *field_name;
+	const char *p;
+
+	if (!field->event) {
+		if (test_event_exists(tep, sb, expr, NULL))
+			return -1;
+	}
+
+	/* The field could have a conversion */
+	p = strchr(field->field, '.');
+	if (p)
+		field_name = strndup(field->field, p - field->field);
+	else
+		field_name = strdup(field->field);
+
+	if (!field_name)
+		return -1;
+
+	if (!strcmp(field_name, TRACEFS_TIMESTAMP) ||
+	    !strcmp(field->field, TRACEFS_TIMESTAMP_USECS))
+		tfield = (void *)1L;
+	else
+		tfield = tep_find_any_field(field->event, field_name);
+	free(field_name);
+
+	if (tfield)
+		return 0;
+
+	sb->line_no = expr->line;
+	sb->line_idx = expr->idx;
+
+	parse_error(sb, field->raw,
+		    "Field '%s' not part of event %s\n",
+		    field->field, field->event_name);
+	return -1;
+}
+
+static int update_vars(struct tep_handle *tep,
+		       struct sql_table *table,
+		       struct expr *expr)
 {
 	struct sqlhist_bison *sb = table->sb;
-	struct expr *expr;
+	struct field *event_field = &expr->field;
+	struct tep_event *event;
 	struct field *field;
 	const char *label;
-	const char *p, *r;
-	char *system;
-	int len;
+	const char *raw = event_field->raw;
+	const char *event_name;
+	const char *system;
+	const char *p;
+	int label_len = 0, event_len, system_len;
 
-	p = strchr(event->raw, '.');
+	p = strchr(raw, '.');
 	if (p) {
-		system = strndup(event->raw, p - event->raw);
-		if (!system)
+		char *str;
+
+		str = strndup(raw, p - raw);
+		if (!str)
 			return -1;
-		event->system = store_str(sb, system);
-		free(system);
-		if (!event->system)
+		event_field->system = store_str(sb, str);
+		free(str);
+		if (!event_field->system)
 			return -1;
 		p++;
 	} else {
-		p = event->raw;
+		p = raw;
 	}
 
-	event->event = store_str(sb, p);
-	if (!event->event)
+	event_field->event_name = store_str(sb, p);
+	if (!event_field->event_name)
+		return -1;
+
+	if (test_event_exists(tep, sb, expr, &event))
+		return -1;
+
+	if (!event_field->system)
+		event_field->system = store_str(sb, event->system);
+
+	if (!event_field->system)
 		return -1;
 
-	if (!event->label)
-		event->label = event->event;
+	label = event_field->label;
+	if (label)
+		label_len = strlen(label);
+
+	system = event_field->system;
+	system_len = strlen(system);
 
-	label = event->label;
-	len = strlen(label);
+	event_name = event_field->event_name;
+	event_len = strlen(event_name);
 
 	for_each_field(expr, field, table) {
+		int len;
+
 		field = &expr->field;
 
 		if (field->event)
 			continue;
 
-		p = strchr(field->raw, '.');
-		if (p) {
-			/* Does this field have a system */
-			r = strchr(p + 1, '.');
-			if (r) {
-				/* This has a system, and is not a alias */
-				system = strndup(field->raw, p - field->raw);
-				if (!system)
-					return -1;
-				field->system = store_str(sb, system);
-				free(system);
-				if (!field->system)
-					return -1;
-
-				/* save the event as well */
-				p++;
-				system = strndup(p, r - p);
-				if (!system)
-					return -1;
-				field->event = store_str(sb, system);
-				free(system);
-				if (!field->event)
-					return -1;
-				r++;
-				field->field = store_str(sb, r);
-				goto check_timestamps;
-			}
+		raw = field->raw;
+
+		/*
+		 * The field could be:
+		 *     system.event.field...
+		 *     event.field...
+		 *     label.field...
+		 * We check label first.
+		 */
+
+		len = label_len;
+		if (label && !strncmp(raw, label, len) &&
+		    raw[len] == '.') {
+			/* Label matches and takes precedence */
+			goto found;
 		}
 
-		if (strncmp(field->raw, label, len))
-			continue;
+		if (!strncmp(raw, system, system_len) &&
+		    raw[system_len] == '.') {
+			raw += system_len + 1;
+			/* Check the event portion next */
+		}
 
-		if (field->raw[len] != '.')
+		len = event_len;
+		if (strncmp(raw, event_name, len) ||
+		    raw[len] != '.') {
+			/* Does not match */
 			continue;
+		}
+ found:
+		field->system = system;
+		field->event_name = event_name;
+		field->event = event;
+		field->field = raw + len + 1;
 
-		field->system = event->system;
-		field->event = event->event;
-		field->field = field->raw + len + 1;
- check_timestamps:
 		if (!strcmp(field->field, "TIMESTAMP"))
 			field->field = store_str(sb, TRACEFS_TIMESTAMP);
-		else if (!strcmp(field->field, "TIMESTAMP_USECS"))
+		if (!strcmp(field->field, "TIMESTAMP_USECS"))
 			field->field = store_str(sb, TRACEFS_TIMESTAMP_USECS);
+		if (test_field_exists(tep, sb, expr))
+			return -1;
 	}
 
 	return 0;
@@ -608,29 +695,29 @@ static int update_vars(struct sql_table *table, struct field *event)
  * selections can be fields and not mention the event itself.
  */
 static int update_fields(struct tep_handle *tep,
-			 struct sql_table *table, struct field *event_field)
+			 struct sql_table *table,
+			 struct expr *expr)
 {
+	struct field *event_field = &expr->field;
 	struct sqlhist_bison *sb = table->sb;
 	struct tep_format_field *tfield;
 	struct tep_event *event;
-	struct expr *expr;
 	struct field *field;
 	const char *p;
 	int len;
 
-	/* First update fields with aliases an such */
-	update_vars(table, event_field);
+	/* First update fields with aliases an such and add event */
+	update_vars(tep, table, expr);
 
-	/* The update_vars already updated event->system and event->event */
-	event = tep_find_event_by_name(tep, event_field->system,
-				       event_field->event);
 	/*
 	 * If event is not found, the creation of the synth will
 	 * add a proper error, so return "success".
 	*/
-	if (!event)
+	if (!event_field->event)
 		return 0;
 
+	event = event_field->event;
+
 	for_each_field(expr, field, table) {
 		const char *field_name;
 
@@ -659,7 +746,8 @@ static int update_fields(struct tep_handle *tep,
 			continue;
 
 		field->system = event_field->system;
-		field->event = event_field->event;
+		field->event_name = event_field->event_name;
+		field->event = event;
 		field->field = field_name;
 	}
 
@@ -746,7 +834,7 @@ static void assign_match(const char *system, const char *event,
 	rval = &match->rval->field;
 
 	if (lval->system == system &&
-	    lval->event == event) {
+	    lval->event_name == event) {
 		*start_match = lval->field;
 		*end_match = rval->field;
 	} else {
@@ -772,7 +860,7 @@ static int build_compare(struct tracefs_synth *synth,
 	rval = &compare->rval->field;
 
 	if (lval->system == system &&
-	    lval->event == event) {
+	    lval->event_name == event) {
 		start_field = lval->field;
 		end_field = rval->field;
 		calc = TRACEFS_SYNTH_DELTA_START;
@@ -829,12 +917,12 @@ static int do_verify_filter(struct sqlhist_bison *sb, struct filter *filter,
 	 */
 	if (!*system && !*event) {
 		*system = filter->lval->field.system;
-		*event = filter->lval->field.event;
+		*event = filter->lval->field.event_name;
 		return 0;
 	}
 
 	if (filter->lval->field.system != *system ||
-	    filter->lval->field.event != *event)
+	    filter->lval->field.event_name != *event)
 		return verify_filter_error(sb, filter->lval, *event);
 
 	return 0;
@@ -1035,45 +1123,6 @@ static int build_filter(struct tep_handle *tep, struct sqlhist_bison *sb,
 	return ret;
 }
 
-static int test_event_exists(struct tep_handle *tep, struct sqlhist_bison *sb,
-			     struct expr *expr, struct tep_event **pevent)
-{
-	const char *system = expr->field.system;
-	const char *event_name = expr->field.event;
-	struct tep_event *event;
-
-	event = tep_find_event_by_name(tep, system, event_name);
-	if (pevent)
-		*pevent = event;
-	if (event)
-		return 0;
-
-	sb->line_no = expr->line;
-	sb->line_idx = expr->idx;
-
-	parse_error(sb, expr->field.raw, "event not found\n");
-	return -1;
-}
-
-static int test_field_exists(struct tep_handle *tep, struct sqlhist_bison *sb,
-			     struct expr *expr)
-{
-	struct tep_event *event;
-
-	if (test_event_exists(tep, sb, expr, &event))
-		return -1;
-
-	if (trace_verify_event_field(event, expr->field.field, NULL))
-		return 0;
-
-	sb->line_no = expr->line;
-	sb->line_idx = expr->idx;
-
-	parse_error(sb, expr->field.raw, "Field '%s' not part of event %s\n",
-		    expr->field.field, expr->field.event);
-	return -1;
-}
-
 static void *field_match_error(struct tep_handle *tep, struct sqlhist_bison *sb,
 			       struct match *match)
 {
@@ -1233,11 +1282,11 @@ static int verify_field_type(struct tep_handle *tep,
 	sb->line_no = expr->line;
 	sb->line_idx = expr->idx;
 
-	event = tep_find_event_by_name(tep, field->system, field->event);
+	event = tep_find_event_by_name(tep, field->system, field->event_name);
 	if (!event) {
 		parse_error(sb, field->raw,
 			    "Event '%s' not found\n",
-			    field->event ? : "(null)");
+			    field->event_name ? : "(null)");
 		return -1;
 	}
 
@@ -1339,12 +1388,12 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 
 	/* This could be a simple SQL statement to only build a histogram */
 	if (!table->to) {
-		ret = update_fields(tep, table, &table->from->field);
+		ret = update_fields(tep, table, table->from);
 		if (ret < 0)
 			return NULL;
 
 		start_system = table->from->field.system;
-		start_event = table->from->field.event;
+		start_event = table->from->field.event_name;
 
 		synth = synth_init_from(tep, start_system, start_event);
 		if (!synth)
@@ -1352,12 +1401,16 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 		goto hist_only;
 	}
 
-	ret = update_vars(table, &table->from->field);
+	ret = update_vars(tep, table, table->from);
+	if (ret < 0)
+		return NULL;
+
+	ret = update_vars(tep, table, table->to);
 	if (ret < 0)
 		return NULL;
 
 	start_system = table->from->field.system;
-	start_event = table->from->field.event;
+	start_event = table->from->field.event_name;
 
 	match = table->matches;
 	if (!match)
@@ -1368,7 +1421,7 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 		return NULL;
 
 	end_system = table->to->field.system;
-	end_event = table->to->field.event;
+	end_event = table->to->field.event_name;
 
 	assign_match(start_system, start_event, match,
 		     &start_match, &end_match);
@@ -1404,7 +1457,7 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 			ret = -1;
 			field = &expr->field;
 			if (field->system == start_system &&
-			    field->event == start_event) {
+			    field->event_name == start_event) {
 				int type;
 				type = verify_field_type(tep, table->sb, expr);
 				if (type < 0)
diff --git a/utest/tracefs-utest.c b/utest/tracefs-utest.c
index 645c84c..09bb8f2 100644
--- a/utest/tracefs-utest.c
+++ b/utest/tracefs-utest.c
@@ -51,7 +51,7 @@
 #define SQL_2_SQL	"select woke.next_pid as woke_pid, wake.common_pid as waking_pid from sched_waking as wake join sched_switch as woke on woke.next_pid = wake.pid"
 
 #define SQL_3_EVENT	"wakeup_lat"
-#define SQL_3_SQL	"select start.pid, end.next_prio as prio, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) as lat from sched_waking as start join sched_switch as end on start.pid = end.next_pid"
+#define SQL_3_SQL	"select sched_switch.next_prio as prio, end.prev_prio as pprio, (sched.sched_waking.common_timestamp.usecs - end.TIMESTAMP_USECS) as lat from sched_waking as start join sched_switch as end on start.pid = end.next_pid"
 
 #define SQL_4_EVENT	"wakeup_lat_2"
 #define SQL_4_SQL	"select start.pid, end.next_prio as prio, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) as lat from sched_waking as start join sched_switch as end on start.pid = end.next_pid where (start.prio >= 1 && start.prio < 100) || !(start.pid >= 0 && start.pid <= 1) && end.prev_pid != 0"

^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 01/22] libtracefs: Added new API tracefs_sql()
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
@ 2021-08-03 17:05 ` Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 02/22] tracefs: Add unit tests for tracefs_sql() Steven Rostedt
                   ` (20 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:05 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware),
	Lukas Bulwahn, Daniel Black

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

This adds the API tracefs_sql() that takes a tep_handle handler, a name,
and a SQL string and parses it to produce a tracefs_synth synthetic event
handler.

Currently it only supports simple SQL of the type:

  SELECT start.common_pid AS pid, end.common_timestamp.usecs AS usecs
    FROM sched_waking AS start JOIN sched_switch AS end
    ON start.pid = end.next_pid

Special thanks to:

  Lukas Bulwahn for first introducing the idea at RT Summit on the Summit
    Spring of 2019.

  Daniel Black for sorting out the syntax of mapping the synthetic
    event creations into SQL statements at Linux Plumbers, Fall of 2019.

Cc: Lukas Bulwahn <lukas.bulwahn@gmail.com>
Cc: Daniel Black <daniel@linux.ibm.com>
Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 include/tracefs.h     |   3 +
 src/Makefile          |  13 +
 src/sqlhist-parse.h   |  69 ++++
 src/sqlhist.l         |  88 +++++
 src/sqlhist.y         | 143 ++++++++
 src/tracefs-sqlhist.c | 756 ++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 1072 insertions(+)
 create mode 100644 src/sqlhist-parse.h
 create mode 100644 src/sqlhist.l
 create mode 100644 src/sqlhist.y
 create mode 100644 src/tracefs-sqlhist.c

diff --git a/include/tracefs.h b/include/tracefs.h
index 55f82392b004..219adba4b0ce 100644
--- a/include/tracefs.h
+++ b/include/tracefs.h
@@ -477,4 +477,7 @@ void tracefs_synth_free(struct tracefs_synth *synth);
 int tracefs_synth_show(struct trace_seq *seq, struct tracefs_instance *instance,
 		       struct tracefs_synth *synth);
 
+struct tracefs_synth *tracefs_sql(struct tep_handle *tep, const char *name,
+				  const char *sql_buffer, char **err);
+
 #endif /* _TRACE_FS_H */
diff --git a/src/Makefile b/src/Makefile
index 767af49034ee..9248efc5c7fd 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -12,6 +12,11 @@ OBJS += tracefs-kprobes.o
 OBJS += tracefs-hist.o
 OBJS += tracefs-filter.o
 
+# Order matters for the the three below
+OBJS += sqlhist-lex.o
+OBJS += sqlhist.tab.o
+OBJS += tracefs-sqlhist.o
+
 OBJS := $(OBJS:%.o=$(bdir)/%.o)
 DEPS := $(OBJS:$(bdir)/%.o=$(bdir)/.%.d)
 
@@ -32,6 +37,14 @@ $(LIBTRACEFS_SHARED_SO): $(LIBTRACEFS_SHARED_VERSION)
 
 libtracefs.so: $(LIBTRACEFS_SHARED_SO)
 
+# bison will create both sqlhist.tab.c and sqlhist.tab.h
+sqlhist.tab.h:
+sqlhist.tab.c: sqlhist.y sqlhist.tab.h
+	bison --debug -v --report-file=bison.report -d -o $@ $<
+
+sqlhist-lex.c: sqlhist.l sqlhist.tab.c
+	flex -o $@ $<
+
 $(bdir)/%.o: %.c
 	$(Q)$(call do_fpic_compile)
 
diff --git a/src/sqlhist-parse.h b/src/sqlhist-parse.h
new file mode 100644
index 000000000000..aa5232eea451
--- /dev/null
+++ b/src/sqlhist-parse.h
@@ -0,0 +1,69 @@
+#ifndef __SQLHIST_PARSE_H
+#define __SQLHIST_PARSE_H
+
+#include <stdarg.h>
+#include <tracefs.h>
+
+struct str_hash;
+#define HASH_BITS 10
+
+struct sql_table;
+
+struct sqlhist_bison {
+	const char		*buffer;
+	size_t			buffer_size;
+	size_t			buffer_idx;
+	int			line_no;
+	int			line_idx;
+	struct sql_table	*table;
+	char			*parse_error_str;
+	struct str_hash         *str_hash[1 << HASH_BITS];
+};
+
+extern struct sqlhist_bison *sb;
+
+#include "sqlhist.tab.h"
+
+enum filter_type {
+	FILTER_GROUP,
+	FILTER_NOT_GROUP,
+	FILTER_EQ,
+	FILTER_NE,
+	FILTER_LE,
+	FILTER_LT,
+	FILTER_GE,
+	FILTER_GT,
+	FILTER_BIN_AND,
+	FILTER_STR_CMP,
+	FILTER_AND,
+	FILTER_OR,
+};
+
+enum compare_type {
+	COMPARE_GROUP,
+	COMPARE_ADD,
+	COMPARE_SUB,
+	COMPARE_MUL,
+	COMPARE_DIV,
+	COMPARE_BIN_AND,
+	COMPARE_BIN_OR,
+	COMPARE_AND,
+	COMPARE_OR,
+};
+
+char * store_str(struct sqlhist_bison *sb, const char *str);
+
+int table_start(struct sqlhist_bison *sb);
+
+void *add_field(struct sqlhist_bison *sb, const char *field, const char *label);
+
+int add_match(struct sqlhist_bison *sb, void *A, void *B);
+
+int add_selection(struct sqlhist_bison *sb, void *item, const char *label);
+int add_from(struct sqlhist_bison *sb, void *item);
+int add_to(struct sqlhist_bison *sb, void *item);
+
+extern void sql_parse_error(struct sqlhist_bison *sb, const char *text,
+			    const char *fmt, va_list ap);
+
+#endif
diff --git a/src/sqlhist.l b/src/sqlhist.l
new file mode 100644
index 000000000000..5dfd17517c6e
--- /dev/null
+++ b/src/sqlhist.l
@@ -0,0 +1,88 @@
+%{
+/* code here */
+
+#include <stdarg.h>
+#include "sqlhist-parse.h"
+
+extern int my_yyinput(char *buf, int max);
+
+#undef YY_INPUT
+#define YY_INPUT(b, r, m) ({r = my_yyinput(b, m);})
+
+#define YY_NO_INPUT
+#define YY_NO_UNPUT
+
+#define YY_EXTRA_TYPE struct sqlhist_bison *
+
+#define HANDLE_COLUMN do { sb->line_idx += strlen(yytext); } while (0)
+
+%}
+
+%option caseless
+
+field		[a-z_][a-z0-9_\.]*
+qstring		\"[^\"]*\"
+hexnum		0x[0-9a-f]+
+number		[0-9a-f]+
+
+%%
+
+select { HANDLE_COLUMN; return SELECT; }
+as { HANDLE_COLUMN; return AS; }
+from { HANDLE_COLUMN; return FROM; }
+join { HANDLE_COLUMN; return JOIN; }
+on { HANDLE_COLUMN; return ON; }
+
+{qstring} {
+	HANDLE_COLUMN;
+	yylval.string = store_str(sb, yytext);
+	return STRING;
+}
+
+{field} {
+	HANDLE_COLUMN;
+	yylval.string = store_str(sb, yytext);
+	return FIELD;
+}
+
+{hexnum} {
+	HANDLE_COLUMN;
+	yylval.number = strtol(yytext, NULL, 0);
+	return NUMBER;
+}
+
+{number} {
+	HANDLE_COLUMN;
+	yylval.number = strtol(yytext, NULL, 0);
+	return NUMBER;
+}
+
+\!= { HANDLE_COLUMN; return NEQ; }
+\<= { HANDLE_COLUMN; return LE; }
+\>= { HANDLE_COLUMN; return GE; }
+== { HANDLE_COLUMN; return EQ; }
+&& { HANDLE_COLUMN; return AND; }
+"||" { HANDLE_COLUMN; return OR; }
+[<>&~] { HANDLE_COLUMN; return yytext[0]; }
+
+[\!()\-\+\*/,=] { HANDLE_COLUMN; return yytext[0]; }
+
+[ \t] { HANDLE_COLUMN; }
+\n { sb->line_idx = 0; sb->line_no++; }
+
+. { HANDLE_COLUMN; return PARSE_ERROR; }
+%%
+
+int yywrap(void)
+{
+	return 1;
+}
+
+void yyerror(const char *fmt, ...)
+{
+	va_list ap;
+
+	va_start(ap, fmt);
+	sql_parse_error(sb, yytext, fmt, ap);
+	va_end(ap);
+}
diff --git a/src/sqlhist.y b/src/sqlhist.y
new file mode 100644
index 000000000000..ecb0a0ed44b3
--- /dev/null
+++ b/src/sqlhist.y
@@ -0,0 +1,143 @@
+%{
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+
+#include "sqlhist-parse.h"
+
+extern int yylex(void);
+extern void yyerror(char *fmt, ...);
+
+#define CHECK_RETURN_PTR(x)					\
+	do {							\
+		if (!(x)) {					\
+			printf("FAILED MEMORY: %s\n", #x);	\
+			return -ENOMEM;				\
+		}						\
+	} while (0)
+
+#define CHECK_RETURN_VAL(x)					\
+	do {							\
+		if ((x) < 0) {					\
+			printf("FAILED MEMORY: %s\n", #x);	\
+			return -ENOMEM;				\
+		}						\
+	} while (0)
+
+%}
+
+%union {
+	int	s32;
+	char	*string;
+	long	number;
+	void	*expr;
+}
+
+%token AS SELECT FROM JOIN ON PARSE_ERROR
+%token <number> NUMBER
+%token <string> STRING
+%token <string> FIELD
+%token <string> LE GE EQ NEQ AND OR
+
+%left '+' '-'
+%left '*' '/'
+%left '<' '>'
+%left AND OR
+
+%type <string> name label
+
+%type <expr>  selection_expr field item named_field join_clause
+
+%%
+
+start :
+   select_statement
+ ;
+
+label : AS name { CHECK_RETURN_PTR($$ = store_str(sb, $2)); }
+ | name { CHECK_RETURN_PTR($$ = store_str(sb, $1)); }
+ ;
+
+select : SELECT  { table_start(sb); }
+  ;
+
+select_statement :
+    select selection_list table_exp
+  ;
+
+selection_list :
+   selection
+ | selection ',' selection_list
+ ;
+
+selection :
+    selection_expr
+				{
+					CHECK_RETURN_VAL(add_selection(sb, $1, NULL));
+				}
+  | selection_expr label
+				{
+					CHECK_RETURN_VAL(add_selection(sb, $1, $2));
+				}
+  ;
+
+selection_expr :
+   field
+ | '(' field ')'		{  $$ = $2; }
+ ;
+
+item :
+   named_field
+ | field
+ ;
+
+field :
+   FIELD	{ $$ = add_field(sb, $1, NULL); CHECK_RETURN_PTR($$); }
+ ;
+
+named_field :
+   FIELD label { $$ = add_field(sb, $1, $2); CHECK_RETURN_PTR($$); }
+ ;
+
+name :
+   FIELD
+ ;
+
+table_exp :
+   from_clause join_clause
+ ;
+
+from_clause :
+   FROM item		{ CHECK_RETURN_VAL(add_from(sb, $2)); }
+
+/*
+ * Select from a from clause confuses the variable parsing.
+ * disable it for now.
+
+   | FROM '(' select_statement ')' label
+				{
+					from_table_end($5);
+					$$ = store_printf("FROM (%s) AS %s", $3, $5);
+				}
+*/
+ ;
+
+join_clause :
+  JOIN item ON match_clause	{ add_to(sb, $2); }
+ ;
+
+match :
+   item '=' item { CHECK_RETURN_VAL(add_match(sb, $1, $3)); }
+ | item EQ item { CHECK_RETURN_VAL(add_match(sb, $1, $3)); }
+
+ ;
+
+match_clause :
+   match
+ | match ',' match_clause
+ ;
+
+%%
+
+
diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
new file mode 100644
index 000000000000..5f2729325c1c
--- /dev/null
+++ b/src/tracefs-sqlhist.c
@@ -0,0 +1,756 @@
+// SPDX-License-Identifier: LGPL-2.1
+/*
+ * Copyright (C) 2021 VMware Inc, Steven Rostedt <rostedt@goodmis.org>
+ *
+ * Updates:
+ * Copyright (C) 2021, VMware, Tzvetomir Stoyanov <tz.stoyanov@gmail.com>
+ *
+ */
+#include <trace-seq.h>
+#include <stdlib.h>
+#include <errno.h>
+
+#include "tracefs.h"
+#include "tracefs-local.h"
+#include "sqlhist-parse.h"
+
+struct sqlhist_bison *sb;
+
+extern int yylex_init(void* ptr_yy_globals);
+extern int yylex_init_extra(struct sqlhist_bison *sb, void* ptr_yy_globals);
+extern int yylex_destroy (void * yyscanner );
+
+struct str_hash {
+	struct str_hash		*next;
+	char			*str;
+};
+
+enum alias_type {
+	ALIAS_EVENT,
+	ALIAS_FIELD,
+};
+
+#define for_each_field(expr, field, table) \
+	for (expr = (table)->fields; expr; expr = (field)->next)
+
+struct field {
+	struct expr		*next;	/* private link list */
+	const char		*system;
+	const char		*event_name;
+	struct tep_event	*event;
+	const char		*raw;
+	const char		*label;
+	const char		*field;
+};
+
+struct match {
+	struct match		*next;
+	struct expr		*lval;
+	struct expr		*rval;
+};
+
+enum expr_type
+{
+	EXPR_NUMBER,
+	EXPR_STRING,
+	EXPR_FIELD,
+};
+
+struct expr {
+	struct expr		*free_list;
+	struct expr		*next;
+	enum expr_type		type;
+	union {
+		struct field	field;
+		const char	*string;
+		long		number;
+	};
+};
+
+struct sql_table {
+	struct sqlhist_bison	*sb;
+	const char		*name;
+	struct expr		*exprs;
+	struct expr		*fields;
+	struct expr		*from;
+	struct expr		*to;
+	struct match		*matches;
+	struct match		**next_match;
+	struct expr		*selections;
+	struct expr		**next_selection;
+};
+
+__hidden int my_yyinput(char *buf, int max)
+{
+	if (!sb || !sb->buffer)
+		return -1;
+
+	if (sb->buffer_idx + max > sb->buffer_size)
+		max = sb->buffer_size - sb->buffer_idx;
+
+	if (max)
+		memcpy(buf, sb->buffer + sb->buffer_idx, max);
+
+	sb->buffer_idx += max;
+
+	return max;
+}
+
+__hidden void sql_parse_error(struct sqlhist_bison *sb, const char *text,
+			      const char *fmt, va_list ap)
+{
+	const char *buffer = sb->buffer;
+	struct trace_seq s;
+	int line = sb->line_no;
+	int idx = sb->line_idx - strlen(text);
+	int i;
+
+	if (!buffer)
+		return;
+
+	trace_seq_init(&s);
+	if (!s.buffer) {
+		fprintf(stderr, "Error allocating internal buffer\n");
+		return;
+	}
+
+	for (i = 0; line && buffer[i]; i++) {
+		if (buffer[i] == '\n')
+			line--;
+	}
+	for (; buffer[i] && buffer[i] != '\n'; i++)
+		trace_seq_putc(&s, buffer[i]);
+	trace_seq_putc(&s, '\n');
+	for (i = idx; i > 0; i--)
+		trace_seq_putc(&s, ' ');
+	trace_seq_puts(&s, "^\n");
+	trace_seq_printf(&s, "ERROR: '%s'\n", text);
+	trace_seq_vprintf(&s, fmt, ap);
+
+	trace_seq_terminate(&s);
+
+	sb->parse_error_str = strdup(s.buffer);
+	trace_seq_destroy(&s);
+}
+
+static inline unsigned int quick_hash(const char *str)
+{
+	unsigned int val = 0;
+	int len = strlen(str);
+
+	for (; len >= 4; str += 4, len -= 4) {
+		val += str[0];
+		val += str[1] << 8;
+		val += str[2] << 16;
+		val += str[3] << 24;
+	}
+	for (; len > 0; str++, len--)
+		val += str[0] << (len * 8);
+
+        val *= 2654435761;
+
+        return val & ((1 << HASH_BITS) - 1);
+}
+
+
+static struct str_hash *find_string(struct sqlhist_bison *sb, const char *str)
+{
+	unsigned int key = quick_hash(str);
+	struct str_hash *hash = sb->str_hash[key];
+
+	for (; hash; hash = hash->next) {
+		if (!strcmp(hash->str, str))
+			return hash;
+	}
+	return NULL;
+}
+
+/*
+ * If @str is found, then return the hash string.
+ * This lets store_str() know to free str.
+ */
+static char **add_hash(struct sqlhist_bison *sb, const char *str)
+{
+	struct str_hash *hash;
+	unsigned int key;
+
+	if ((hash = find_string(sb, str))) {
+		return &hash->str;
+	}
+
+	hash = malloc(sizeof(*hash));
+	if (!hash)
+		return NULL;
+	key = quick_hash(str);
+	hash->next = sb->str_hash[key];
+	sb->str_hash[key] = hash;
+	hash->str = NULL;
+	return &hash->str;
+}
+
+__hidden char *store_str(struct sqlhist_bison *sb, const char *str)
+{
+	char **pstr = add_hash(sb, str);
+
+	if (!pstr)
+		return NULL;
+
+	if (!(*pstr))
+		*pstr = strdup(str);
+
+	return *pstr;
+}
+
+__hidden int add_selection(struct sqlhist_bison *sb, void *select,
+			   const char *name)
+{
+	struct sql_table *table = sb->table;
+	struct expr *expr = select;
+
+	switch (expr->type) {
+	case EXPR_FIELD:
+		break;
+	case EXPR_NUMBER:
+	case EXPR_STRING:
+	default:
+		return -1;
+	}
+
+	if (expr->next)
+		return -1;
+
+	*table->next_selection = expr;
+	table->next_selection = &expr->next;
+
+	return 0;
+}
+
+static struct expr *find_field(struct sqlhist_bison *sb,
+				const char *raw, const char *label)
+{
+	struct field *field;
+	struct expr *expr;
+
+	for_each_field(expr, field, sb->table) {
+		field = &expr->field;
+
+		if (!strcmp(field->raw, raw)) {
+			if (label && !field->label)
+				field->label = label;
+			return expr;
+		}
+
+		if (label && !strcmp(field->raw, label)) {
+			if (!field->label) {
+				field->label = label;
+				field->raw = raw;
+			}
+			return expr;
+		}
+
+		if (!field->label)
+			continue;
+
+		if (!strcmp(field->label, raw))
+			return expr;
+
+		if (label && !strcmp(field->label, label))
+			return expr;
+	}
+	return NULL;
+}
+
+static void *create_expr(enum expr_type type, struct expr **expr_p)
+{
+	struct expr *expr;
+
+	expr = calloc(1, sizeof(*expr));
+	if (!expr)
+		return NULL;
+
+	if (expr_p)
+		*expr_p = expr;
+
+	expr->free_list = sb->table->exprs;
+	sb->table->exprs = expr;
+
+	expr->type = type;
+
+	switch (type) {
+	case EXPR_FIELD:	return &expr->field;
+	case EXPR_NUMBER:	return &expr->number;
+	case EXPR_STRING:	return &expr->string;
+	}
+
+	return NULL;
+}
+
+#define __create_expr(var, type, ENUM, expr)			\
+	do {							\
+		var = (type *)create_expr(EXPR_##ENUM, expr);	\
+	} while(0)
+
+#define create_field(var, expr)				\
+	__create_expr(var, struct field, FIELD, expr)
+
+__hidden void *add_field(struct sqlhist_bison *sb,
+			 const char *field_name, const char *label)
+{
+	struct sql_table *table = sb->table;
+	struct expr *expr;
+	struct field *field;
+
+	expr = find_field(sb, field_name, label);
+	if (expr)
+		return expr;
+
+	create_field(field, &expr);
+
+	field->next = table->fields;
+	table->fields = expr;
+
+	field->raw = field_name;
+	field->label = label;
+
+	return expr;
+}
+
+__hidden int add_match(struct sqlhist_bison *sb, void *A, void *B)
+{
+	struct sql_table *table = sb->table;
+	struct match *match;
+
+	match = calloc(1, sizeof(*match));
+	if (!match)
+		return -1;
+
+	match->lval = A;
+	match->rval = B;
+
+	*table->next_match = match;
+	table->next_match = &match->next;
+
+	return 0;
+}
+
+__hidden int add_from(struct sqlhist_bison *sb, void *item)
+{
+	struct expr *expr = item;
+
+	if (expr->type != EXPR_FIELD)
+		return -1;
+
+	sb->table->from = expr;
+
+	return 0;
+}
+
+__hidden int add_to(struct sqlhist_bison *sb, void *item)
+{
+	struct expr *expr = item;
+
+	if (expr->type != EXPR_FIELD)
+		return -1;
+
+	sb->table->to = expr;
+
+	return 0;
+}
+
+__hidden int table_start(struct sqlhist_bison *sb)
+{
+	struct sql_table *table;
+
+	table = calloc(1, sizeof(*table));
+	if (!table)
+		return -ENOMEM;
+
+	table->sb = sb;
+	sb->table = table;
+
+	table->next_match = &table->matches;
+	table->next_selection = &table->selections;
+
+	return 0;
+}
+
+static int test_event_exists(struct tep_handle *tep,
+			     struct expr *expr, struct tep_event **pevent)
+{
+	struct field *field = &expr->field;
+	const char *system = field->system;
+	const char *event = field->event_name;
+
+	if (!field->event)
+		field->event = tep_find_event_by_name(tep, system, event);
+	if (pevent)
+		*pevent = field->event;
+	return field->event != NULL ? 0 : -1;
+}
+
+static int test_field_exists(struct expr *expr)
+{
+	struct field *field = &expr->field;
+	struct tep_format_field *tfield;
+	char *field_name;
+	const char *p;
+
+	if (!field->event)
+		return -1;
+
+	/* The field could have a conversion */
+	p = strchr(field->field, '.');
+	if (p)
+		field_name = strndup(field->field, p - field->field);
+	else
+		field_name = strdup(field->field);
+
+	if (!field_name)
+		return -1;
+
+	if (!strcmp(field_name, TRACEFS_TIMESTAMP) ||
+	    !strcmp(field->field, TRACEFS_TIMESTAMP_USECS))
+		tfield = (void *)1L;
+	else
+		tfield = tep_find_any_field(field->event, field_name);
+	free(field_name);
+
+	return tfield != NULL ? 0 : -1;
+}
+
+static int update_vars(struct tep_handle *tep,
+		       struct sql_table *table,
+		       struct expr *expr)
+{
+	struct sqlhist_bison *sb = table->sb;
+	struct field *event_field = &expr->field;
+	struct tep_event *event;
+	struct field *field;
+	const char *label;
+	const char *raw = event_field->raw;
+	const char *event_name;
+	const char *system;
+	const char *p;
+	int label_len = 0, event_len, system_len;
+
+	p = strchr(raw, '.');
+	if (p) {
+		char *str;
+
+		str = strndup(raw, p - raw);
+		if (!str)
+			return -1;
+		event_field->system = store_str(sb, str);
+		free(str);
+		if (!event_field->system)
+			return -1;
+		p++;
+	} else {
+		p = raw;
+	}
+
+	event_field->event_name = store_str(sb, p);
+	if (!event_field->event_name)
+		return -1;
+
+	if (test_event_exists(tep, expr, &event))
+		return -1;
+
+	if (!event_field->system)
+		event_field->system = store_str(sb, event->system);
+
+	if (!event_field->system)
+		return -1;
+
+	label = event_field->label;
+	if (label)
+		label_len = strlen(label);
+
+	system = event_field->system;
+	system_len = strlen(system);
+
+	event_name = event_field->event_name;
+	event_len = strlen(event_name);
+
+	for_each_field(expr, field, table) {
+		int len;
+
+		field = &expr->field;
+
+		if (field->event)
+			continue;
+
+		raw = field->raw;
+
+		/*
+		 * The field could be:
+		 *     system.event.field...
+		 *     event.field...
+		 *     label.field...
+		 * We check label first.
+		 */
+
+		len = label_len;
+		if (label && !strncmp(raw, label, len) &&
+		    raw[len] == '.') {
+			/* Label matches and takes precedence */
+			goto found;
+		}
+
+		if (!strncmp(raw, system, system_len) &&
+		    raw[system_len] == '.') {
+			raw += system_len + 1;
+			/* Check the event portion next */
+		}
+
+		len = event_len;
+		if (strncmp(raw, event_name, len) ||
+		    raw[len] != '.') {
+			/* Does not match */
+			continue;
+		}
+ found:
+		field->system = system;
+		field->event_name = event_name;
+		field->event = event;
+		field->field = raw + len + 1;
+
+		if (!strcmp(field->field, "TIMESTAMP"))
+			field->field = store_str(sb, TRACEFS_TIMESTAMP);
+		if (!strcmp(field->field, "TIMESTAMP_USECS"))
+			field->field = store_str(sb, TRACEFS_TIMESTAMP_USECS);
+		if (test_field_exists(expr))
+			return -1;
+	}
+
+	return 0;
+}
+
+static int test_match(struct sql_table *table, struct match *match)
+{
+	struct field *lval, *rval;
+	struct field *to, *from;
+
+	if (!match->lval || !match->rval)
+		return -1;
+
+	if (match->lval->type != EXPR_FIELD || match->rval->type != EXPR_FIELD)
+		return -1;
+
+	to = &table->to->field;
+	from = &table->from->field;
+
+	lval = &match->lval->field;
+	rval = &match->rval->field;
+
+	/*
+	 * Note, strings are stored in the string store, so all
+	 * duplicate strings are the same value, and we can use
+	 * normal "==" and "!=" instead of strcmp().
+	 *
+	 * Either lval == to and rval == from
+	 * or lval == from and rval == to.
+	 */
+	if ((lval->system != to->system) ||
+	    (lval->event != to->event)) {
+		if ((rval->system != to->system) ||
+		    (rval->event != to->event) ||
+		    (lval->system != from->system) ||
+		    (lval->event != from->event))
+			return -1;
+	} else {
+		if ((rval->system != from->system) ||
+		    (rval->event != from->event) ||
+		    (lval->system != to->system) ||
+		    (lval->event != to->event))
+			return -1;
+	}
+	return 0;
+}
+
+static void assign_match(const char *system, const char *event,
+			 struct match *match,
+			 const char **start_match, const char **end_match)
+{
+	struct field *lval, *rval;
+
+	lval = &match->lval->field;
+	rval = &match->rval->field;
+
+	if (lval->system == system &&
+	    lval->event_name == event) {
+		*start_match = lval->field;
+		*end_match = rval->field;
+	} else {
+		*start_match = rval->field;
+		*end_match = lval->field;
+	}
+}
+
+static struct tracefs_synth *build_synth(struct tep_handle *tep,
+					 const char *name,
+					 struct sql_table *table)
+{
+	struct tracefs_synth *synth;
+	struct field *field;
+	struct match *match;
+	struct expr *expr;
+	const char *start_system;
+	const char *start_event;
+	const char *end_system;
+	const char *end_event;
+	const char *start_match;
+	const char *end_match;
+	int ret;
+
+	if (!table->to || !table->from)
+		return NULL;
+
+	ret = update_vars(tep, table, table->to);
+	if (ret < 0)
+		return NULL;
+
+	ret = update_vars(tep, table, table->from);
+	if (ret < 0)
+		return NULL;
+
+	match = table->matches;
+	if (!match)
+		return NULL;
+
+	ret = test_match(table, match);
+	if (ret < 0)
+		return NULL;
+
+	start_system = table->from->field.system;
+	start_event = table->from->field.event_name;
+
+	end_system = table->to->field.system;
+	end_event = table->to->field.event_name;
+
+	assign_match(start_system, start_event, match,
+		     &start_match, &end_match);
+
+	synth = tracefs_synth_init(tep, name, start_system,
+				   start_event, end_system, end_event,
+				   start_match, end_match, NULL);
+	if (!synth)
+		return NULL;
+
+	for (match = match->next; match; match = match->next) {
+		ret = test_match(table, match);
+		if (ret < 0)
+			goto free;
+
+		assign_match(start_system, start_event, match,
+			     &start_match, &end_match);
+
+		ret = tracefs_synth_add_match_field(synth,
+						    start_match,
+						    end_match, NULL);
+		if (ret < 0)
+			goto free;
+	}
+
+	for (expr = table->selections; expr; expr = expr->next) {
+		if (expr->type == EXPR_FIELD) {
+			field = &expr->field;
+			if (field->system == start_system &&
+			    field->event_name == start_event) {
+				ret = tracefs_synth_add_start_field(synth,
+						field->field, field->label);
+			} else {
+				ret = tracefs_synth_add_end_field(synth,
+						field->field, field->label);
+			}
+			if (ret < 0)
+				goto free;
+			continue;
+		}
+		goto free;
+	}
+
+	return synth;
+ free:
+	tracefs_synth_free(synth);
+	return NULL;
+}
+
+static void free_sql_table(struct sql_table *table)
+{
+	struct match *match;
+	struct expr *expr;
+
+	if (!table)
+		return;
+
+	while ((expr = table->exprs)) {
+		table->exprs = expr->next;
+		free(expr);
+	}
+
+	while ((match = table->matches)) {
+		table->matches = match->next;
+		free(match);
+	}
+
+	free(table);
+}
+
+static void free_str_hash(struct str_hash **hash)
+{
+	struct str_hash *item;
+	int i;
+
+	for (i = 0; i < 1 << HASH_BITS; i++) {
+		while ((item = hash[i])) {
+			hash[i] = item->next;
+			free(item->str);
+			free(item);
+		}
+	}
+}
+
+static void free_sb(struct sqlhist_bison *sb)
+{
+	free_sql_table(sb->table);
+	free_str_hash(sb->str_hash);
+	free(sb->parse_error_str);
+}
+
+struct tracefs_synth *tracefs_sql(struct tep_handle *tep, const char *name,
+				  const char *sql_buffer, char **err)
+{
+	struct sqlhist_bison local_sb;
+	struct tracefs_synth *synth = NULL;
+	int ret;
+
+	if (!tep || !sql_buffer) {
+		errno = EINVAL;
+		return NULL;
+	}
+
+	memset(&local_sb, 0, sizeof(local_sb));
+
+	local_sb.buffer = sql_buffer;
+	local_sb.buffer_size = strlen(sql_buffer);
+	local_sb.buffer_idx = 0;
+
+	sb = &local_sb;
+	ret = yyparse();
+
+	if (ret)
+		goto free;
+
+	synth = build_synth(tep, name, sb->table);
+
+ free:
+	if (!synth) {
+		if (sb->parse_error_str && err) {
+			*err = sb->parse_error_str;
+			sb->parse_error_str = NULL;
+		}
+	}
+	free_sb(sb);
+	return synth;
+}
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 02/22] tracefs: Add unit tests for tracefs_sql()
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 01/22] libtracefs: Added new API tracefs_sql() Steven Rostedt
@ 2021-08-03 17:05 ` Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 03/22] libtracefs: Add comparing start and end fields in tracefs_sql() Steven Rostedt
                   ` (19 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:05 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

Add basic unit test for tracefs_sql()...

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 utest/tracefs-utest.c | 43 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)

diff --git a/utest/tracefs-utest.c b/utest/tracefs-utest.c
index 2a99217b938a..89bb1cce61f7 100644
--- a/utest/tracefs-utest.c
+++ b/utest/tracefs-utest.c
@@ -44,6 +44,12 @@
 #define KRETPROBE_ADDR	"do_sys_openat2"
 #define KRETPROBE_FMT	"ret=$retval"
 
+#define SQL_1_EVENT	"wakeup_1"
+#define SQL_1_SQL	"select sched_switch.next_pid as woke_pid, sched_waking.common_pid as waking_pid from sched_waking join sched_switch on sched_switch.next_pid = sched_waking.pid"
+
+#define SQL_2_EVENT	"wakeup_2"
+#define SQL_2_SQL	"select woke.next_pid as woke_pid, wake.common_pid as waking_pid from sched_waking as wake join sched_switch as woke on woke.next_pid = wake.pid"
+
 static struct tracefs_instance *test_instance;
 static struct tep_handle *test_tep;
 struct test_sample {
@@ -322,6 +328,41 @@ static void test_ftrace_marker(void)
 	test_instance_ftrace_marker(test_instance);
 }
 
+static void test_instance_trace_sql(struct tracefs_instance *instance)
+{
+	struct tracefs_synth *synth;
+	struct trace_seq seq;
+	struct tep_handle *tep;
+	int ret;
+
+	tep = tracefs_local_events(NULL);
+	CU_TEST(tep != NULL);
+
+	trace_seq_init(&seq);
+
+	synth = tracefs_sql(tep, SQL_1_EVENT, SQL_1_SQL, NULL);
+	CU_TEST(synth != NULL);
+	ret = tracefs_synth_show(&seq, instance, synth);
+	CU_TEST(ret == 0);
+	tracefs_synth_free(synth);
+	trace_seq_reset(&seq);
+
+	synth = tracefs_sql(tep, SQL_2_EVENT, SQL_2_SQL, NULL);
+	CU_TEST(synth != NULL);
+	ret = tracefs_synth_show(&seq, instance, synth);
+	CU_TEST(ret == 0);
+	tracefs_synth_free(synth);
+	trace_seq_reset(&seq);
+
+	tep_free(tep);
+	trace_seq_destroy(&seq);
+}
+
+static void test_trace_sql(void)
+{
+	test_instance_trace_sql(test_instance);
+}
+
 static void test_trace_file(void)
 {
 	const char *tmp = get_rand_str();
@@ -1338,6 +1379,8 @@ void test_tracefs_lib(void)
 		fprintf(stderr, "Suite \"%s\" cannot be ceated\n", TRACEFS_SUITE);
 		return;
 	}
+	CU_add_test(suite, "trace sql",
+		    test_trace_sql);
 	CU_add_test(suite, "tracing file / directory APIs",
 		    test_trace_file);
 	CU_add_test(suite, "instance file / directory APIs",
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 03/22] libtracefs: Add comparing start and end fields in tracefs_sql()
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 01/22] libtracefs: Added new API tracefs_sql() Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 02/22] tracefs: Add unit tests for tracefs_sql() Steven Rostedt
@ 2021-08-03 17:05 ` Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 04/22] libtracefs: Add unit test to test tracefs_sql() compare Steven Rostedt
                   ` (18 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:05 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

Add comparing a field and showing the differences between start and end
for tracefs_sql().

For example:

  SELECT (end.common_timestamp.usecs - start.common_timestamp.usecs) AS
    lat FROM sched_waking AS start JOIN sched_switch AS end ON
    start.pid = stop.next_pid

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 src/sqlhist-parse.h   |  1 +
 src/sqlhist.y         | 16 ++++++++
 src/tracefs-sqlhist.c | 85 ++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 101 insertions(+), 1 deletion(-)

diff --git a/src/sqlhist-parse.h b/src/sqlhist-parse.h
index aa5232eea451..aeb1738d693f 100644
--- a/src/sqlhist-parse.h
+++ b/src/sqlhist-parse.h
@@ -58,6 +58,7 @@ int table_start(struct sqlhist_bison *sb);
 void *add_field(struct sqlhist_bison *sb, const char *field, const char *label);
 
 int add_match(struct sqlhist_bison *sb, void *A, void *B);
+void *add_compare(struct sqlhist_bison *sb, void *A, void *B, enum compare_type type);
 
 int add_selection(struct sqlhist_bison *sb, void *item, const char *label);
 int add_from(struct sqlhist_bison *sb, void *item);
diff --git a/src/sqlhist.y b/src/sqlhist.y
index ecb0a0ed44b3..1ba3bf62ee43 100644
--- a/src/sqlhist.y
+++ b/src/sqlhist.y
@@ -48,6 +48,7 @@ extern void yyerror(char *fmt, ...);
 %type <string> name label
 
 %type <expr>  selection_expr field item named_field join_clause
+%type <expr>  selection_addition
 
 %%
 
@@ -85,6 +86,21 @@ selection :
 selection_expr :
    field
  | '(' field ')'		{  $$ = $2; }
+ | selection_addition
+ | '(' selection_addition ')'	{  $$ = $2; }
+ ;
+
+selection_addition :
+   field '+' field
+				{
+					$$ = add_compare(sb, $1, $3, COMPARE_ADD);
+					CHECK_RETURN_PTR($$);
+				}
+ | field '-' field
+				{
+					$$ = add_compare(sb, $1, $3, COMPARE_SUB);
+					CHECK_RETURN_PTR($$);
+				}
  ;
 
 item :
diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
index 5f2729325c1c..01c04211499a 100644
--- a/src/tracefs-sqlhist.c
+++ b/src/tracefs-sqlhist.c
@@ -43,17 +43,32 @@ struct field {
 	const char		*field;
 };
 
+struct filter {
+	enum filter_type	type;
+	struct expr		*lval;
+	struct expr		*rval;
+};
+
 struct match {
 	struct match		*next;
 	struct expr		*lval;
 	struct expr		*rval;
 };
 
+struct compare {
+	enum compare_type	type;
+	struct expr		*lval;
+	struct expr		*rval;
+	const char		*name;
+};
+
 enum expr_type
 {
 	EXPR_NUMBER,
 	EXPR_STRING,
 	EXPR_FIELD,
+	EXPR_FILTER,
+	EXPR_COMPARE,
 };
 
 struct expr {
@@ -62,6 +77,8 @@ struct expr {
 	enum expr_type		type;
 	union {
 		struct field	field;
+		struct filter	filter;
+		struct compare	compare;
 		const char	*string;
 		long		number;
 	};
@@ -210,8 +227,14 @@ __hidden int add_selection(struct sqlhist_bison *sb, void *select,
 	switch (expr->type) {
 	case EXPR_FIELD:
 		break;
+	case EXPR_COMPARE:
+		if (!name)
+			return -1;
+		expr->compare.name = name;
+		break;
 	case EXPR_NUMBER:
 	case EXPR_STRING:
+	case EXPR_FILTER:
 	default:
 		return -1;
 	}
@@ -278,8 +301,10 @@ static void *create_expr(enum expr_type type, struct expr **expr_p)
 
 	switch (type) {
 	case EXPR_FIELD:	return &expr->field;
+	case EXPR_COMPARE:	return &expr->compare;
 	case EXPR_NUMBER:	return &expr->number;
 	case EXPR_STRING:	return &expr->string;
+	case EXPR_FILTER:	return &expr->filter;
 	}
 
 	return NULL;
@@ -293,6 +318,9 @@ static void *create_expr(enum expr_type type, struct expr **expr_p)
 #define create_field(var, expr)				\
 	__create_expr(var, struct field, FIELD, expr)
 
+#define create_compare(var, expr)				\
+	__create_expr(var, struct compare, COMPARE, expr)
+
 __hidden void *add_field(struct sqlhist_bison *sb,
 			 const char *field_name, const char *label)
 {
@@ -332,6 +360,21 @@ __hidden int add_match(struct sqlhist_bison *sb, void *A, void *B)
 
 	return 0;
 }
+__hidden void *add_compare(struct sqlhist_bison *sb,
+			   void *A, void *B, enum compare_type type)
+{
+	struct compare *compare;
+	struct expr *expr;
+
+	create_compare(compare, &expr);
+
+	compare = &expr->compare;
+	compare->lval = A;
+	compare->rval = B;
+	compare->type = type;
+
+	return expr;
+}
 
 __hidden int add_from(struct sqlhist_bison *sb, void *item)
 {
@@ -587,6 +630,39 @@ static void assign_match(const char *system, const char *event,
 	}
 }
 
+static int build_compare(struct tracefs_synth *synth,
+			 const char *system, const char *event,
+			 struct compare *compare)
+{
+	const char *start_field;
+	const char *end_field;
+	struct field *lval, *rval;
+	enum tracefs_synth_calc calc;
+	int ret;
+
+	lval = &compare->lval->field;
+	rval = &compare->rval->field;
+
+	if (lval->system == system &&
+	    lval->event_name == event) {
+		start_field = lval->field;
+		end_field = rval->field;
+		calc = TRACEFS_SYNTH_DELTA_START;
+	} else {
+		start_field = rval->field;
+		end_field = lval->field;
+		calc = TRACEFS_SYNTH_DELTA_END;
+	}
+
+	if (compare->type == COMPARE_ADD)
+		calc = TRACEFS_SYNTH_ADD;
+
+	ret = tracefs_synth_add_compare_field(synth, start_field,
+					      end_field, calc,
+					      compare->name);
+	return ret;
+}
+
 static struct tracefs_synth *build_synth(struct tep_handle *tep,
 					 const char *name,
 					 struct sql_table *table)
@@ -667,7 +743,14 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 				goto free;
 			continue;
 		}
-		goto free;
+
+		if (expr->type != EXPR_COMPARE)
+			goto free;
+
+		ret = build_compare(synth, start_system, end_system,
+				    &expr->compare);
+		if (ret < 0)
+			goto free;
 	}
 
 	return synth;
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 04/22] libtracefs: Add unit test to test tracefs_sql() compare
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (2 preceding siblings ...)
  2021-08-03 17:05 ` [PATCH v3 03/22] libtracefs: Add comparing start and end fields in tracefs_sql() Steven Rostedt
@ 2021-08-03 17:05 ` Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 05/22] libtracefs: Add filtering for start and end events in tracefs_sql() Steven Rostedt
                   ` (17 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:05 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

Add a test to test passing time from sched_waking to sched_switch to
show wake up latency.

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 utest/tracefs-utest.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/utest/tracefs-utest.c b/utest/tracefs-utest.c
index 89bb1cce61f7..93bc6f16f99a 100644
--- a/utest/tracefs-utest.c
+++ b/utest/tracefs-utest.c
@@ -50,6 +50,9 @@
 #define SQL_2_EVENT	"wakeup_2"
 #define SQL_2_SQL	"select woke.next_pid as woke_pid, wake.common_pid as waking_pid from sched_waking as wake join sched_switch as woke on woke.next_pid = wake.pid"
 
+#define SQL_3_EVENT	"wakeup_lat"
+#define SQL_3_SQL	"select sched_switch.next_prio as prio, end.prev_prio as pprio, (sched.sched_waking.common_timestamp.usecs - end.TIMESTAMP_USECS) as lat from sched_waking as start join sched_switch as end on start.pid = end.next_pid"
+
 static struct tracefs_instance *test_instance;
 static struct tep_handle *test_tep;
 struct test_sample {
@@ -354,6 +357,13 @@ static void test_instance_trace_sql(struct tracefs_instance *instance)
 	tracefs_synth_free(synth);
 	trace_seq_reset(&seq);
 
+	synth = tracefs_sql(tep, SQL_3_EVENT, SQL_3_SQL, NULL);
+	CU_TEST(synth != NULL);
+	ret = tracefs_synth_show(&seq, instance, synth);
+	CU_TEST(ret == 0);
+	tracefs_synth_free(synth);
+	trace_seq_reset(&seq);
+
 	tep_free(tep);
 	trace_seq_destroy(&seq);
 }
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 05/22] libtracefs: Add filtering for start and end events in tracefs_sql()
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (3 preceding siblings ...)
  2021-08-03 17:05 ` [PATCH v3 04/22] libtracefs: Add unit test to test tracefs_sql() compare Steven Rostedt
@ 2021-08-03 17:05 ` Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 06/22] libtracefs: Add unit test to test tracefs_sql() where clause Steven Rostedt
                   ` (16 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:05 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

Allow the start and end events to have filters with the "WHERE" clause.

For example:

  SELECT (end.common_timestamp.usecs - start.common_timestamp.usecs) AS
    lat FROM sched_waking AS start JOIN sched_switch AS end ON
    start.pid = stop.next_pid WHERE start.prio < 100 &&
    end.prev_prio < 100

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 src/sqlhist-parse.h   |   6 +
 src/sqlhist.l         |   1 +
 src/sqlhist.y         |  70 +++++++++++-
 src/tracefs-sqlhist.c | 260 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 335 insertions(+), 2 deletions(-)

diff --git a/src/sqlhist-parse.h b/src/sqlhist-parse.h
index aeb1738d693f..0933bfe9a574 100644
--- a/src/sqlhist-parse.h
+++ b/src/sqlhist-parse.h
@@ -57,13 +57,19 @@ int table_start(struct sqlhist_bison *sb);
 
 void *add_field(struct sqlhist_bison *sb, const char *field, const char *label);
 
+void *add_filter(struct sqlhist_bison *sb, void *A, void *B, enum filter_type op);
+
 int add_match(struct sqlhist_bison *sb, void *A, void *B);
 void *add_compare(struct sqlhist_bison *sb, void *A, void *B, enum compare_type type);
+int add_where(struct sqlhist_bison *sb, void *expr);
 
 int add_selection(struct sqlhist_bison *sb, void *item, const char *label);
 int add_from(struct sqlhist_bison *sb, void *item);
 int add_to(struct sqlhist_bison *sb, void *item);
 
+void *add_string(struct sqlhist_bison *sb, const char *str);
+void *add_number(struct sqlhist_bison *sb, long val);
+
 extern void sql_parse_error(struct sqlhist_bison *sb, const char *text,
 			    const char *fmt, va_list ap);
 
diff --git a/src/sqlhist.l b/src/sqlhist.l
index 5dfd17517c6e..5baba0992afc 100644
--- a/src/sqlhist.l
+++ b/src/sqlhist.l
@@ -32,6 +32,7 @@ as { HANDLE_COLUMN; return AS; }
 from { HANDLE_COLUMN; return FROM; }
 join { HANDLE_COLUMN; return JOIN; }
 on { HANDLE_COLUMN; return ON; }
+where { HANDLE_COLUMN; return WHERE; }
 
 {qstring} {
 	HANDLE_COLUMN;
diff --git a/src/sqlhist.y b/src/sqlhist.y
index 1ba3bf62ee43..8dcc824bb9f1 100644
--- a/src/sqlhist.y
+++ b/src/sqlhist.y
@@ -34,7 +34,7 @@ extern void yyerror(char *fmt, ...);
 	void	*expr;
 }
 
-%token AS SELECT FROM JOIN ON PARSE_ERROR
+%token AS SELECT FROM JOIN ON WHERE PARSE_ERROR
 %token <number> NUMBER
 %token <string> STRING
 %token <string> FIELD
@@ -49,6 +49,9 @@ extern void yyerror(char *fmt, ...);
 
 %type <expr>  selection_expr field item named_field join_clause
 %type <expr>  selection_addition
+%type <expr>  compare compare_list compare_cmds compare_items
+%type <expr>  compare_and_or
+%type <expr>  str_val val
 
 %%
 
@@ -120,8 +123,71 @@ name :
    FIELD
  ;
 
+str_val :
+   STRING	{ $$ = add_string(sb, $1); CHECK_RETURN_PTR($$); }
+ ;
+
+val :
+   str_val
+ | NUMBER	{ $$ = add_number(sb, $1); CHECK_RETURN_PTR($$); }
+ ;
+
+
+compare :
+   field '<' val	{ $$ = add_filter(sb, $1, $3, FILTER_LT); CHECK_RETURN_PTR($$); }
+ | field '>' val	{ $$ = add_filter(sb, $1, $3, FILTER_GT); CHECK_RETURN_PTR($$); }
+ | field LE val	{ $$ = add_filter(sb, $1, $3, FILTER_LE); CHECK_RETURN_PTR($$); }
+ | field GE val	{ $$ = add_filter(sb, $1, $3, FILTER_GE); CHECK_RETURN_PTR($$); }
+ | field '=' val	{ $$ = add_filter(sb, $1, $3, FILTER_EQ); CHECK_RETURN_PTR($$); }
+ | field EQ val	{ $$ = add_filter(sb, $1, $3, FILTER_EQ); CHECK_RETURN_PTR($$); }
+ | field NEQ val	{ $$ = add_filter(sb, $1, $3, FILTER_NE); CHECK_RETURN_PTR($$); }
+ | field "!=" val	{ $$ = add_filter(sb, $1, $3, FILTER_NE); CHECK_RETURN_PTR($$); }
+ | field '&' val	{ $$ = add_filter(sb, $1, $3, FILTER_BIN_AND); CHECK_RETURN_PTR($$); }
+ | field '~' str_val	{ $$ = add_filter(sb, $1, $3, FILTER_STR_CMP); CHECK_RETURN_PTR($$); }
+;
+
+compare_and_or :
+   compare_and_or OR compare_and_or	{ $$ = add_filter(sb, $1, $3, FILTER_OR); CHECK_RETURN_PTR($$); }
+ | compare_and_or AND compare_and_or	{ $$ = add_filter(sb, $1, $3, FILTER_AND); CHECK_RETURN_PTR($$); }
+ | '!' '(' compare_and_or ')'		{ $$ = add_filter(sb, $3, NULL, FILTER_NOT_GROUP); CHECK_RETURN_PTR($$); }
+ | '!' compare				{ $$ = add_filter(sb, $2, NULL, FILTER_NOT_GROUP); CHECK_RETURN_PTR($$); }
+ | compare
+ ;
+
+compare_items :
+   compare_items OR compare_items	{ $$ = add_filter(sb, $1, $3, FILTER_OR); CHECK_RETURN_PTR($$); }
+ | '(' compare_and_or ')'		{ $$ = add_filter(sb, $2, NULL, FILTER_GROUP); CHECK_RETURN_PTR($$); }
+ | '!' '(' compare_and_or ')'		{ $$ = add_filter(sb, $3, NULL, FILTER_NOT_GROUP); CHECK_RETURN_PTR($$); }
+ | '!' compare				{ $$ = add_filter(sb, $2, NULL, FILTER_NOT_GROUP); CHECK_RETURN_PTR($$); }
+ | compare
+ ;
+
+compare_cmds :
+   compare_items		{ CHECK_RETURN_VAL(add_where(sb, $1)); }
+ ;
+
+/*
+ * Top level AND is equal to ',' but the compare_cmds in them must
+ * all be of for the same event (start or end exclusive).
+ * That is, OR is not to be used between start and end events.
+ */
+compare_list :
+   compare_cmds
+ | compare_cmds ',' compare_list
+ | compare_cmds AND compare_list
+ ;
+
+where_clause :
+   WHERE compare_list
+ ;
+
+opt_where_clause :
+   /* empty */
+ | where_clause
+;
+
 table_exp :
-   from_clause join_clause
+   from_clause join_clause opt_where_clause
  ;
 
 from_clause :
diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
index 01c04211499a..e17da39e1860 100644
--- a/src/tracefs-sqlhist.c
+++ b/src/tracefs-sqlhist.c
@@ -91,6 +91,8 @@ struct sql_table {
 	struct expr		*fields;
 	struct expr		*from;
 	struct expr		*to;
+	struct expr		*where;
+	struct expr		**next_where;
 	struct match		*matches;
 	struct match		**next_match;
 	struct expr		*selections;
@@ -318,9 +320,18 @@ static void *create_expr(enum expr_type type, struct expr **expr_p)
 #define create_field(var, expr)				\
 	__create_expr(var, struct field, FIELD, expr)
 
+#define create_filter(var, expr)			\
+	__create_expr(var, struct filter, FILTER, expr)
+
 #define create_compare(var, expr)				\
 	__create_expr(var, struct compare, COMPARE, expr)
 
+#define create_string(var, expr)			\
+	__create_expr(var, const char *, STRING, expr)
+
+#define create_number(var, expr)			\
+	__create_expr(var, long, NUMBER, expr)
+
 __hidden void *add_field(struct sqlhist_bison *sb,
 			 const char *field_name, const char *label)
 {
@@ -343,6 +354,22 @@ __hidden void *add_field(struct sqlhist_bison *sb,
 	return expr;
 }
 
+__hidden void *add_filter(struct sqlhist_bison *sb,
+			  void *A, void *B, enum filter_type op)
+{
+	struct filter *filter;
+	struct expr *expr;
+
+	create_filter(filter, &expr);
+
+	filter->lval = A;
+	filter->rval = B;
+
+	filter->type = op;
+
+	return expr;
+}
+
 __hidden int add_match(struct sqlhist_bison *sb, void *A, void *B)
 {
 	struct sql_table *table = sb->table;
@@ -376,6 +403,23 @@ __hidden void *add_compare(struct sqlhist_bison *sb,
 	return expr;
 }
 
+__hidden int add_where(struct sqlhist_bison *sb, void *item)
+{
+	struct expr *expr = item;
+	struct sql_table *table = sb->table;
+
+	if (expr->type != EXPR_FILTER)
+		return -1;
+
+	*table->next_where = expr;
+	table->next_where = &expr->next;
+
+	if (expr->next)
+		return -1;
+
+	return 0;
+}
+
 __hidden int add_from(struct sqlhist_bison *sb, void *item)
 {
 	struct expr *expr = item;
@@ -400,6 +444,34 @@ __hidden int add_to(struct sqlhist_bison *sb, void *item)
 	return 0;
 }
 
+__hidden void *add_string(struct sqlhist_bison *sb, const char *str)
+{
+	struct expr *expr;
+	const char **str_p;
+
+	create_string(str_p, &expr);
+	*str_p = str;
+	return expr;
+}
+
+__hidden void *add_number(struct sqlhist_bison *sb, long val)
+{
+	struct expr *expr;
+	long *num;
+
+	create_number(num, &expr);
+	*num = val;
+	return expr;
+
+	expr = calloc(1, sizeof(expr));
+	if (!expr)
+		return NULL;
+
+	expr->type = EXPR_NUMBER;
+	expr->number = val;
+	return expr;
+}
+
 __hidden int table_start(struct sqlhist_bison *sb)
 {
 	struct sql_table *table;
@@ -411,6 +483,7 @@ __hidden int table_start(struct sqlhist_bison *sb)
 	table->sb = sb;
 	sb->table = table;
 
+	table->next_where = &table->where;
 	table->next_match = &table->matches;
 	table->next_selection = &table->selections;
 
@@ -663,6 +736,167 @@ static int build_compare(struct tracefs_synth *synth,
 	return ret;
 }
 
+static int do_verify_filter(struct filter *filter,
+			    const char **system, const char **event)
+{
+	int ret;
+
+	if (filter->type == FILTER_OR ||
+	    filter->type == FILTER_AND) {
+		ret = do_verify_filter(&filter->lval->filter, system, event);
+		if (ret)
+			return ret;
+		return do_verify_filter(&filter->rval->filter, system, event);
+	}
+	if (filter->type == FILTER_GROUP ||
+	    filter->type == FILTER_NOT_GROUP) {
+		return do_verify_filter(&filter->lval->filter, system, event);
+	}
+
+	/*
+	 * system and event will be NULL until we find the left most
+	 * node. Then assign it, and compare on the way back up.
+	 */
+	if (!*system && !*event) {
+		*system = filter->lval->field.system;
+		*event = filter->lval->field.event_name;
+		return 0;
+	}
+
+	if (filter->lval->field.system != *system ||
+	    filter->lval->field.event_name != *event)
+		return -1;
+
+	return 0;
+}
+
+static int verify_filter(struct filter *filter,
+			 const char **system, const char **event)
+{
+	int ret;
+
+	switch (filter->type) {
+	case FILTER_OR:
+	case FILTER_AND:
+	case FILTER_GROUP:
+	case FILTER_NOT_GROUP:
+		break;
+	default:
+		return do_verify_filter(filter, system, event);
+	}
+
+	ret = do_verify_filter(&filter->lval->filter, system, event);
+	if (ret)
+		return ret;
+
+	switch (filter->type) {
+	case FILTER_OR:
+	case FILTER_AND:
+		return do_verify_filter(&filter->rval->filter, system, event);
+	default:
+		return 0;
+	}
+}
+
+static int build_filter(struct tracefs_synth *synth,
+			bool start, struct filter *filter, bool *started)
+{
+	int (*append_filter)(struct tracefs_synth *synth,
+			     enum tracefs_filter type,
+			     const char *field,
+			     enum tracefs_compare compare,
+			     const char *val);
+	enum tracefs_compare cmp;
+	const char *val;
+	int and_or = TRACEFS_FILTER_AND;
+	char num[64];
+	int ret;
+
+	if (start)
+		append_filter = tracefs_synth_append_start_filter;
+	else
+		append_filter = tracefs_synth_append_end_filter;
+
+	if (started && *started) {
+		ret = append_filter(synth, and_or, NULL, 0, NULL);
+		ret = append_filter(synth, TRACEFS_FILTER_OPEN_PAREN,
+				    NULL, 0, NULL);
+	}
+
+	switch (filter->type) {
+	case FILTER_NOT_GROUP:
+		ret = append_filter(synth, TRACEFS_FILTER_NOT,
+				    NULL, 0, NULL);
+		if (ret < 0)
+			goto out;
+		/* Fall through */
+	case FILTER_GROUP:
+		ret = append_filter(synth, TRACEFS_FILTER_OPEN_PAREN,
+				    NULL, 0, NULL);
+		if (ret < 0)
+			goto out;
+		ret = build_filter(synth, start, &filter->lval->filter, NULL);
+		if (ret < 0)
+			goto out;
+		ret = append_filter(synth, TRACEFS_FILTER_CLOSE_PAREN,
+				    NULL, 0, NULL);
+		goto out;
+
+	case FILTER_OR:
+		and_or = TRACEFS_FILTER_OR;
+		/* Fall through */
+	case FILTER_AND:
+		ret = build_filter(synth, start, &filter->lval->filter, NULL);
+		if (ret < 0)
+			goto out;
+		ret = append_filter(synth, and_or, NULL, 0, NULL);
+
+		if (ret)
+			goto out;
+		ret = build_filter(synth, start, &filter->rval->filter, NULL);
+		goto out;
+	default:
+		break;
+	}
+
+	switch (filter->rval->type) {
+	case EXPR_NUMBER:
+		sprintf(num, "%ld", filter->rval->number);
+		val = num;
+		break;
+	case EXPR_STRING:
+		val = filter->rval->string;
+		break;
+	default:
+		break;
+	}
+
+	switch (filter->type) {
+	case FILTER_EQ:		cmp = TRACEFS_COMPARE_EQ; break;
+	case FILTER_NE:		cmp = TRACEFS_COMPARE_NE; break;
+	case FILTER_LE:		cmp = TRACEFS_COMPARE_LE; break;
+	case FILTER_LT:		cmp = TRACEFS_COMPARE_LT; break;
+	case FILTER_GE:		cmp = TRACEFS_COMPARE_GE; break;
+	case FILTER_GT:		cmp = TRACEFS_COMPARE_GT; break;
+	case FILTER_BIN_AND:	cmp = TRACEFS_COMPARE_AND; break;
+	case FILTER_STR_CMP:	cmp = TRACEFS_COMPARE_RE; break;
+	default:
+		break;
+	}
+
+	ret = append_filter(synth, TRACEFS_FILTER_COMPARE,
+			    filter->lval->field.field, cmp, val);
+
+ out:
+	if (!ret && started) {
+		if (*started)
+			ret = append_filter(synth, TRACEFS_FILTER_CLOSE_PAREN,
+					    NULL, 0, NULL);
+		*started = true;
+	}
+	return ret;
+}
+
 static struct tracefs_synth *build_synth(struct tep_handle *tep,
 					 const char *name,
 					 struct sql_table *table)
@@ -677,6 +911,8 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 	const char *end_event;
 	const char *start_match;
 	const char *end_match;
+	bool started_start = false;
+	bool started_end = false;
 	int ret;
 
 	if (!table->to || !table->from)
@@ -753,6 +989,30 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 			goto free;
 	}
 
+	for (expr = table->where; expr; expr = expr->next) {
+		const char *filter_system = NULL;
+		const char *filter_event = NULL;
+		bool *started;
+		bool start;
+
+		ret = verify_filter(&expr->filter, &filter_system,
+				    &filter_event);
+		if (ret < 0)
+			goto free;
+
+		start = filter_system == start_system &&
+			filter_event == start_event;
+
+		if (start)
+			started = &started_start;
+		else
+			started = &started_end;
+
+		ret = build_filter(synth, start, &expr->filter, started);
+		if (ret < 0)
+			goto free;
+	}
+
 	return synth;
  free:
 	tracefs_synth_free(synth);
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 06/22] libtracefs: Add unit test to test tracefs_sql() where clause
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (4 preceding siblings ...)
  2021-08-03 17:05 ` [PATCH v3 05/22] libtracefs: Add filtering for start and end events in tracefs_sql() Steven Rostedt
@ 2021-08-03 17:05 ` Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 07/22] libtracefs: Make sqlhist parser reentrant Steven Rostedt
                   ` (15 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:05 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

Add a test to test filtering of events via the WHERE clause.

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 utest/tracefs-utest.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/utest/tracefs-utest.c b/utest/tracefs-utest.c
index 93bc6f16f99a..09bb8f29c191 100644
--- a/utest/tracefs-utest.c
+++ b/utest/tracefs-utest.c
@@ -53,6 +53,13 @@
 #define SQL_3_EVENT	"wakeup_lat"
 #define SQL_3_SQL	"select sched_switch.next_prio as prio, end.prev_prio as pprio, (sched.sched_waking.common_timestamp.usecs - end.TIMESTAMP_USECS) as lat from sched_waking as start join sched_switch as end on start.pid = end.next_pid"
 
+#define SQL_4_EVENT	"wakeup_lat_2"
+#define SQL_4_SQL	"select start.pid, end.next_prio as prio, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) as lat from sched_waking as start join sched_switch as end on start.pid = end.next_pid where (start.prio >= 1 && start.prio < 100) || !(start.pid >= 0 && start.pid <= 1) && end.prev_pid != 0"
+
+#define SQL_5_EVENT	"irq_lat"
+#define SQL_5_SQL	"select end.common_pid as pid, (end.common_timestamp.usecs - start.common_timestamp.usecs) as irq_lat from irq_disable as start join irq_enable as end on start.common_pid = end.common_pid, start.parent_offs == end.parent_offs where start.common_pid != 0"
+#define SQL_5_START	"irq_disable"
+
 static struct tracefs_instance *test_instance;
 static struct tep_handle *test_tep;
 struct test_sample {
@@ -336,6 +343,7 @@ static void test_instance_trace_sql(struct tracefs_instance *instance)
 	struct tracefs_synth *synth;
 	struct trace_seq seq;
 	struct tep_handle *tep;
+	struct tep_event *event;
 	int ret;
 
 	tep = tracefs_local_events(NULL);
@@ -364,6 +372,23 @@ static void test_instance_trace_sql(struct tracefs_instance *instance)
 	tracefs_synth_free(synth);
 	trace_seq_reset(&seq);
 
+	synth = tracefs_sql(tep, SQL_4_EVENT, SQL_4_SQL, NULL);
+	CU_TEST(synth != NULL);
+	ret = tracefs_synth_show(&seq, instance, synth);
+	CU_TEST(ret == 0);
+	tracefs_synth_free(synth);
+	trace_seq_reset(&seq);
+
+	event = tep_find_event_by_name(tep, NULL, SQL_5_START);
+	if (event) {
+		synth = tracefs_sql(tep, SQL_5_EVENT, SQL_5_SQL, NULL);
+		CU_TEST(synth != NULL);
+		ret = tracefs_synth_show(&seq, instance, synth);
+		CU_TEST(ret == 0);
+		tracefs_synth_free(synth);
+		trace_seq_reset(&seq);
+	}
+
 	tep_free(tep);
 	trace_seq_destroy(&seq);
 }
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 07/22] libtracefs: Make sqlhist parser reentrant
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (5 preceding siblings ...)
  2021-08-03 17:05 ` [PATCH v3 06/22] libtracefs: Add unit test to test tracefs_sql() where clause Steven Rostedt
@ 2021-08-03 17:05 ` Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 08/22] libtracefs: Make parser unique to libtracefs Steven Rostedt
                   ` (14 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:05 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

Update bison and flex to be reentrant, and not depend on any global
variables.

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 src/sqlhist-parse.h   |  3 +--
 src/sqlhist.l         | 26 ++++++++++++++++----------
 src/sqlhist.y         | 10 ++++++++--
 src/tracefs-sqlhist.c | 41 ++++++++++++++++++++++++-----------------
 4 files changed, 49 insertions(+), 31 deletions(-)

diff --git a/src/sqlhist-parse.h b/src/sqlhist-parse.h
index 0933bfe9a574..7c1b97ca65af 100644
--- a/src/sqlhist-parse.h
+++ b/src/sqlhist-parse.h
@@ -10,6 +10,7 @@ struct str_hash;
 struct sql_table;
 
 struct sqlhist_bison {
+	void			*scanner;
 	const char		*buffer;
 	size_t			buffer_size;
 	size_t			buffer_idx;
@@ -20,8 +21,6 @@ struct sqlhist_bison {
 	struct str_hash         *str_hash[1 << HASH_BITS];
 };
 
-extern struct sqlhist_bison *sb;
-
 #include "sqlhist.tab.h"
 
 enum filter_type {
diff --git a/src/sqlhist.l b/src/sqlhist.l
index 5baba0992afc..f9e0fcc17e63 100644
--- a/src/sqlhist.l
+++ b/src/sqlhist.l
@@ -4,21 +4,26 @@
 #include <stdarg.h>
 #include "sqlhist-parse.h"
 
-extern int my_yyinput(char *buf, int max);
+extern int my_yyinput(void *extra, char *buf, int max);
 
 #undef YY_INPUT
-#define YY_INPUT(b, r, m) ({r = my_yyinput(b, m);})
+#define YY_INPUT(b, r, m) ({r = my_yyinput(yyextra, b, m);})
 
 #define YY_NO_INPUT
 #define YY_NO_UNPUT
 
 #define YY_EXTRA_TYPE struct sqlhist_bison *
 
-#define HANDLE_COLUMN do { sb->line_idx += strlen(yytext); } while (0)
+#define yytext yyg->yytext_r
+
+#define TRACE_SB	((struct sqlhist_bison *)yyextra)
+#define HANDLE_COLUMN do { TRACE_SB->line_idx += strlen(yytext); } while (0)
 
 %}
 
 %option caseless
+%option reentrant
+%option bison-bridge
 
 field		[a-z_][a-z0-9_\.]*
 qstring		\"[^\"]*\"
@@ -36,25 +41,25 @@ where { HANDLE_COLUMN; return WHERE; }
 
 {qstring} {
 	HANDLE_COLUMN;
-	yylval.string = store_str(sb, yytext);
+	yylval->string = store_str(TRACE_SB, yyg->yytext_r);
 	return STRING;
 }
 
 {field} {
 	HANDLE_COLUMN;
-	yylval.string = store_str(sb, yytext);
+	yylval->string = store_str(TRACE_SB, yyg->yytext_r);
 	return FIELD;
 }
 
 {hexnum} {
 	HANDLE_COLUMN;
-	yylval.number = strtol(yytext, NULL, 0);
+	yylval->number = strtol(yyg->yytext_r, NULL, 0);
 	return NUMBER;
 }
 
 {number} {
 	HANDLE_COLUMN;
-	yylval.number = strtol(yytext, NULL, 0);
+	yylval->number = strtol(yyg->yytext_r, NULL, 0);
 	return NUMBER;
 }
 
@@ -69,18 +74,19 @@ where { HANDLE_COLUMN; return WHERE; }
 [\!()\-\+\*/,=] { HANDLE_COLUMN; return yytext[0]; }
 
 [ \t] { HANDLE_COLUMN; }
-\n { sb->line_idx = 0; sb->line_no++; }
+\n { TRACE_SB->line_idx = 0; TRACE_SB->line_no++; }
 
 . { HANDLE_COLUMN; return PARSE_ERROR; }
 %%
 
-int yywrap(void)
+int yywrap(void *data)
 {
 	return 1;
 }
 
-void yyerror(const char *fmt, ...)
+void yyerror(struct sqlhist_bison *sb, char *fmt, ...)
 {
+	struct yyguts_t * yyg = (struct yyguts_t*)sb->scanner;
 	va_list ap;
 
 	va_start(ap, fmt);
diff --git a/src/sqlhist.y b/src/sqlhist.y
index 8dcc824bb9f1..ce9cb58fb3a9 100644
--- a/src/sqlhist.y
+++ b/src/sqlhist.y
@@ -6,8 +6,10 @@
 
 #include "sqlhist-parse.h"
 
-extern int yylex(void);
-extern void yyerror(char *fmt, ...);
+#define scanner sb->scanner
+
+extern int yylex(YYSTYPE *yylval, void *);
+extern void yyerror(struct sqlhist_bison *, char *fmt, ...);
 
 #define CHECK_RETURN_PTR(x)					\
 	do {							\
@@ -27,6 +29,10 @@ extern void yyerror(char *fmt, ...);
 
 %}
 
+%define api.pure
+%lex-param {void *scanner}
+%parse-param {struct sqlhist_bison *sb}
+
 %union {
 	int	s32;
 	char	*string;
diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
index e17da39e1860..65577c9b7de3 100644
--- a/src/tracefs-sqlhist.c
+++ b/src/tracefs-sqlhist.c
@@ -14,8 +14,6 @@
 #include "tracefs-local.h"
 #include "sqlhist-parse.h"
 
-struct sqlhist_bison *sb;
-
 extern int yylex_init(void* ptr_yy_globals);
 extern int yylex_init_extra(struct sqlhist_bison *sb, void* ptr_yy_globals);
 extern int yylex_destroy (void * yyscanner );
@@ -99,8 +97,10 @@ struct sql_table {
 	struct expr		**next_selection;
 };
 
-__hidden int my_yyinput(char *buf, int max)
+__hidden int my_yyinput(void *extra, char *buf, int max)
 {
+	struct sqlhist_bison *sb = extra;
+
 	if (!sb || !sb->buffer)
 		return -1;
 
@@ -285,7 +285,8 @@ static struct expr *find_field(struct sqlhist_bison *sb,
 	return NULL;
 }
 
-static void *create_expr(enum expr_type type, struct expr **expr_p)
+static void *create_expr(struct sqlhist_bison *sb,
+			 enum expr_type type, struct expr **expr_p)
 {
 	struct expr *expr;
 
@@ -314,7 +315,7 @@ static void *create_expr(enum expr_type type, struct expr **expr_p)
 
 #define __create_expr(var, type, ENUM, expr)			\
 	do {							\
-		var = (type *)create_expr(EXPR_##ENUM, expr);	\
+		var = (type *)create_expr(sb, EXPR_##ENUM, expr);	\
 	} while(0)
 
 #define create_field(var, expr)				\
@@ -1064,8 +1065,8 @@ static void free_sb(struct sqlhist_bison *sb)
 struct tracefs_synth *tracefs_sql(struct tep_handle *tep, const char *name,
 				  const char *sql_buffer, char **err)
 {
-	struct sqlhist_bison local_sb;
 	struct tracefs_synth *synth = NULL;
+	struct sqlhist_bison sb;
 	int ret;
 
 	if (!tep || !sql_buffer) {
@@ -1073,27 +1074,33 @@ struct tracefs_synth *tracefs_sql(struct tep_handle *tep, const char *name,
 		return NULL;
 	}
 
-	memset(&local_sb, 0, sizeof(local_sb));
+	memset(&sb, 0, sizeof(sb));
+
+	sb.buffer = sql_buffer;
+	sb.buffer_size = strlen(sql_buffer);
+	sb.buffer_idx = 0;
 
-	local_sb.buffer = sql_buffer;
-	local_sb.buffer_size = strlen(sql_buffer);
-	local_sb.buffer_idx = 0;
+	ret = yylex_init_extra(&sb, &sb.scanner);
+	if (ret < 0) {
+		yylex_destroy(sb.scanner);
+		return NULL;
+	}
 
-	sb = &local_sb;
-	ret = yyparse();
+	ret = yyparse(&sb);
+	yylex_destroy(sb.scanner);
 
 	if (ret)
 		goto free;
 
-	synth = build_synth(tep, name, sb->table);
+	synth = build_synth(tep, name, sb.table);
 
  free:
 	if (!synth) {
-		if (sb->parse_error_str && err) {
-			*err = sb->parse_error_str;
-			sb->parse_error_str = NULL;
+		if (sb.parse_error_str && err) {
+			*err = sb.parse_error_str;
+			sb.parse_error_str = NULL;
 		}
 	}
-	free_sb(sb);
+	free_sb(&sb);
 	return synth;
 }
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 08/22] libtracefs: Make parser unique to libtracefs
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (6 preceding siblings ...)
  2021-08-03 17:05 ` [PATCH v3 07/22] libtracefs: Make sqlhist parser reentrant Steven Rostedt
@ 2021-08-03 17:05 ` Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 09/22] libtracefs: Add line number and index to expr structure Steven Rostedt
                   ` (13 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:05 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

Add %define api.prefix and defines to have the parser global variables use
tracefs_* instead of yy*, as without this, if a tool that links to this
library, and tries to use the synth sql parsing, it may end up using its
own yyparse() and friends functions.

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 src/sqlhist.y         | 10 ++++++++++
 src/tracefs-sqlhist.c |  2 +-
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/src/sqlhist.y b/src/sqlhist.y
index ce9cb58fb3a9..9d03a457ae84 100644
--- a/src/sqlhist.y
+++ b/src/sqlhist.y
@@ -30,6 +30,16 @@ extern void yyerror(struct sqlhist_bison *, char *fmt, ...);
 %}
 
 %define api.pure
+
+/* Change the globals to use tracefs_ prefix */
+%define api.prefix{tracefs_}
+%code provides
+{
+  #define YYSTYPE TRACEFS_STYPE
+  #define yylex tracefs_lex
+  #define yyerror tracefs_error
+}
+
 %lex-param {void *scanner}
 %parse-param {struct sqlhist_bison *sb}
 
diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
index 65577c9b7de3..70bcab14cb27 100644
--- a/src/tracefs-sqlhist.c
+++ b/src/tracefs-sqlhist.c
@@ -1086,7 +1086,7 @@ struct tracefs_synth *tracefs_sql(struct tep_handle *tep, const char *name,
 		return NULL;
 	}
 
-	ret = yyparse(&sb);
+	ret = tracefs_parse(&sb);
 	yylex_destroy(sb.scanner);
 
 	if (ret)
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 09/22] libtracefs: Add line number and index to expr structure
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (7 preceding siblings ...)
  2021-08-03 17:05 ` [PATCH v3 08/22] libtracefs: Make parser unique to libtracefs Steven Rostedt
@ 2021-08-03 17:05 ` Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 10/22] libtracefs: Add error message when match fields are not FROM and JOIN events Steven Rostedt
                   ` (12 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:05 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

In order to have better error messages, record the line number and index
when an expr structure is created. Then this can be used to show where in
the SQL sequence a problem was found if the building of the synth event
has issues.

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 src/tracefs-sqlhist.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
index 70bcab14cb27..2e301e024d13 100644
--- a/src/tracefs-sqlhist.c
+++ b/src/tracefs-sqlhist.c
@@ -73,6 +73,8 @@ struct expr {
 	struct expr		*free_list;
 	struct expr		*next;
 	enum expr_type		type;
+	int			line;
+	int			idx;
 	union {
 		struct field	field;
 		struct filter	filter;
@@ -301,6 +303,8 @@ static void *create_expr(struct sqlhist_bison *sb,
 	sb->table->exprs = expr;
 
 	expr->type = type;
+	expr->line = sb->line_no;
+	expr->idx = sb->line_idx;
 
 	switch (type) {
 	case EXPR_FIELD:	return &expr->field;
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 10/22] libtracefs: Add error message when match fields are not FROM and JOIN events
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (8 preceding siblings ...)
  2021-08-03 17:05 ` [PATCH v3 09/22] libtracefs: Add line number and index to expr structure Steven Rostedt
@ 2021-08-03 17:05 ` Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 11/22] libtracefs: Add error message when match or init fails from bad events Steven Rostedt
                   ` (11 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:05 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

It is required that the "match" content (the ON portion of the SQL
sequence) has a field from the FROM event and a field from the JOIN event.

If they do not, then give a better message about what went wrong.

To simplify addition of future errors, also add a parse_error() that calls into
sql_parse_error() with the appropriate "ap" argument. That is, parse_error()
takes a normal printf() form and then translates it to the vprintf from of
sql_parse_error.

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 src/tracefs-sqlhist.c | 43 +++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 41 insertions(+), 2 deletions(-)

diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
index 2e301e024d13..da4668f53ea1 100644
--- a/src/tracefs-sqlhist.c
+++ b/src/tracefs-sqlhist.c
@@ -8,6 +8,7 @@
  */
 #include <trace-seq.h>
 #include <stdlib.h>
+#include <stdarg.h>
 #include <errno.h>
 
 #include "tracefs.h"
@@ -154,6 +155,16 @@ __hidden void sql_parse_error(struct sqlhist_bison *sb, const char *text,
 	trace_seq_destroy(&s);
 }
 
+static void parse_error(struct sqlhist_bison *sb, const char *text,
+			const char *fmt, ...)
+{
+	va_list ap;
+
+	va_start(ap, fmt);
+	sql_parse_error(sb, text, fmt, ap);
+	va_end(ap);
+}
+
 static inline unsigned int quick_hash(const char *str)
 {
 	unsigned int val = 0;
@@ -647,6 +658,34 @@ static int update_vars(struct tep_handle *tep,
 	return 0;
 }
 
+static int match_error(struct sqlhist_bison *sb, struct match *match,
+		       struct field *lmatch, struct field *rmatch)
+{
+	struct field *lval = &match->lval->field;
+	struct field *rval = &match->rval->field;
+	struct field *field;
+	struct expr *expr;
+
+	if (lval->system != lmatch->system ||
+	    lval->event != lmatch->event) {
+		expr = match->lval;
+		field = lval;
+	} else {
+		expr = match->rval;
+		field = rval;
+	}
+
+	sb->line_no = expr->line;
+	sb->line_idx = expr->idx;
+
+	parse_error(sb, field->raw,
+		    "'%s' and '%s' must be a field for each event: '%s' and '%s'\n",
+		    lval->raw, rval->raw, sb->table->to->field.raw,
+		    sb->table->from->field.raw);
+
+	return -1;
+}
+
 static int test_match(struct sql_table *table, struct match *match)
 {
 	struct field *lval, *rval;
@@ -678,13 +717,13 @@ static int test_match(struct sql_table *table, struct match *match)
 		    (rval->event != to->event) ||
 		    (lval->system != from->system) ||
 		    (lval->event != from->event))
-			return -1;
+			return match_error(table->sb, match, from, to);
 	} else {
 		if ((rval->system != from->system) ||
 		    (rval->event != from->event) ||
 		    (lval->system != to->system) ||
 		    (lval->event != to->event))
-			return -1;
+			return match_error(table->sb, match, to, from);
 	}
 	return 0;
 }
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 11/22] libtracefs: Add error message when match or init fails from bad events
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (9 preceding siblings ...)
  2021-08-03 17:05 ` [PATCH v3 10/22] libtracefs: Add error message when match fields are not FROM and JOIN events Steven Rostedt
@ 2021-08-03 17:05 ` Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 12/22] libtracefs; Add error message for bad selections to SQL sequence Steven Rostedt
                   ` (10 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:05 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

If the tracefs_synth_init() or the matching fails due to events that do
not exist, or if the match fields are not compatible with each other. Add
an error message that reports this and where the problem is.

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 src/tracefs-sqlhist.c | 98 +++++++++++++++++++++++++++++++++++++++----
 1 file changed, 89 insertions(+), 9 deletions(-)

diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
index da4668f53ea1..ab190d123800 100644
--- a/src/tracefs-sqlhist.c
+++ b/src/tracefs-sqlhist.c
@@ -507,6 +507,7 @@ __hidden int table_start(struct sqlhist_bison *sb)
 }
 
 static int test_event_exists(struct tep_handle *tep,
+			     struct sqlhist_bison *sb,
 			     struct expr *expr, struct tep_event **pevent)
 {
 	struct field *field = &expr->field;
@@ -517,18 +518,30 @@ static int test_event_exists(struct tep_handle *tep,
 		field->event = tep_find_event_by_name(tep, system, event);
 	if (pevent)
 		*pevent = field->event;
-	return field->event != NULL ? 0 : -1;
+
+	if (field->event)
+		return 0;
+
+	sb->line_no = expr->line;
+	sb->line_idx = expr->idx;
+
+	parse_error(sb, field->raw, "event not found\n");
+	return -1;
 }
 
-static int test_field_exists(struct expr *expr)
+static int test_field_exists(struct tep_handle *tep,
+			     struct sqlhist_bison *sb,
+			     struct expr *expr)
 {
 	struct field *field = &expr->field;
 	struct tep_format_field *tfield;
 	char *field_name;
 	const char *p;
 
-	if (!field->event)
-		return -1;
+	if (!field->event) {
+		if (test_event_exists(tep, sb, expr, NULL))
+			return -1;
+	}
 
 	/* The field could have a conversion */
 	p = strchr(field->field, '.');
@@ -547,7 +560,16 @@ static int test_field_exists(struct expr *expr)
 		tfield = tep_find_any_field(field->event, field_name);
 	free(field_name);
 
-	return tfield != NULL ? 0 : -1;
+	if (tfield)
+		return 0;
+
+	sb->line_no = expr->line;
+	sb->line_idx = expr->idx;
+
+	parse_error(sb, field->raw,
+		    "Field '%s' not part of event %s\n",
+		    field->field, field->event_name);
+	return -1;
 }
 
 static int update_vars(struct tep_handle *tep,
@@ -585,7 +607,7 @@ static int update_vars(struct tep_handle *tep,
 	if (!event_field->event_name)
 		return -1;
 
-	if (test_event_exists(tep, expr, &event))
+	if (test_event_exists(tep, sb, expr, &event))
 		return -1;
 
 	if (!event_field->system)
@@ -651,7 +673,7 @@ static int update_vars(struct tep_handle *tep,
 			field->field = store_str(sb, TRACEFS_TIMESTAMP);
 		if (!strcmp(field->field, "TIMESTAMP_USECS"))
 			field->field = store_str(sb, TRACEFS_TIMESTAMP_USECS);
-		if (test_field_exists(expr))
+		if (test_field_exists(tep, sb, expr))
 			return -1;
 	}
 
@@ -941,6 +963,62 @@ static int build_filter(struct tracefs_synth *synth,
 	return ret;
 }
 
+static void *field_match_error(struct tep_handle *tep, struct sqlhist_bison *sb,
+			       struct match *match)
+{
+	switch (errno) {
+	case ENODEV:
+	case EBADE:
+		break;
+	default:
+		/* System error */
+		return NULL;
+	}
+
+	/* ENODEV means that an event or field does not exist */
+	if (errno == ENODEV) {
+		if (test_field_exists(tep, sb, match->lval))
+			return NULL;
+		if (test_field_exists(tep, sb, match->rval))
+			return NULL;
+		return NULL;
+	}
+
+	/* fields exist, but values are not compatible */
+	sb->line_no = match->lval->line;
+	sb->line_idx = match->lval->idx;
+
+	parse_error(sb, match->lval->field.raw,
+		    "Field '%s' is not compatible to match field '%s'\n",
+		    match->lval->field.raw, match->rval->field.raw);
+	return NULL;
+}
+
+static void *synth_init_error(struct tep_handle *tep, struct sql_table *table)
+{
+	struct sqlhist_bison *sb = table->sb;
+	struct match *match = table->matches;
+
+	switch (errno) {
+	case ENODEV:
+	case EBADE:
+		break;
+	default:
+		/* System error */
+		return NULL;
+	}
+
+	/* ENODEV could mean that start or end events do not exist */
+	if (errno == ENODEV) {
+		if (test_event_exists(tep, sb, table->from, NULL))
+			return NULL;
+		if (test_event_exists(tep, sb, table->to, NULL))
+			return NULL;
+	}
+
+	return field_match_error(tep, sb, match);
+}
+
 static struct tracefs_synth *build_synth(struct tep_handle *tep,
 					 const char *name,
 					 struct sql_table *table)
@@ -991,7 +1069,7 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 				   start_event, end_system, end_event,
 				   start_match, end_match, NULL);
 	if (!synth)
-		return NULL;
+		return synth_init_error(tep, table);
 
 	for (match = match->next; match; match = match->next) {
 		ret = test_match(table, match);
@@ -1004,8 +1082,10 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 		ret = tracefs_synth_add_match_field(synth,
 						    start_match,
 						    end_match, NULL);
-		if (ret < 0)
+		if (ret < 0) {
+			field_match_error(tep, table->sb, match);
 			goto free;
+		}
 	}
 
 	for (expr = table->selections; expr; expr = expr->next) {
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 12/22] libtracefs; Add error message for bad selections to SQL sequence
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (10 preceding siblings ...)
  2021-08-03 17:05 ` [PATCH v3 11/22] libtracefs: Add error message when match or init fails from bad events Steven Rostedt
@ 2021-08-03 17:05 ` Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 13/22] libtracefs: Add error message when compare fields fail Steven Rostedt
                   ` (9 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:05 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

If the building of the synthetic event fails on creating the selection,
report it properly.

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 src/tracefs-sqlhist.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
index ab190d123800..041d7077c3eb 100644
--- a/src/tracefs-sqlhist.c
+++ b/src/tracefs-sqlhist.c
@@ -1019,6 +1019,16 @@ static void *synth_init_error(struct tep_handle *tep, struct sql_table *table)
 	return field_match_error(tep, sb, match);
 }
 
+static void selection_error(struct tep_handle *tep,
+			    struct sqlhist_bison *sb, struct expr *expr)
+{
+	/* We just care about event not existing */
+	if (errno != ENODEV)
+		return;
+
+	test_field_exists(tep, sb, expr);
+}
+
 static struct tracefs_synth *build_synth(struct tep_handle *tep,
 					 const char *name,
 					 struct sql_table *table)
@@ -1099,8 +1109,10 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 				ret = tracefs_synth_add_end_field(synth,
 						field->field, field->label);
 			}
-			if (ret < 0)
+			if (ret < 0) {
+				selection_error(tep, table->sb, expr);
 				goto free;
+			}
 			continue;
 		}
 
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 13/22] libtracefs: Add error message when compare fields fail
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (11 preceding siblings ...)
  2021-08-03 17:05 ` [PATCH v3 12/22] libtracefs; Add error message for bad selections to SQL sequence Steven Rostedt
@ 2021-08-03 17:05 ` Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 14/22] libtracefs: Add error message for grouping events in SQL filter Steven Rostedt
                   ` (8 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:05 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

If the processing of comparing fields fail due to not existing or because
they are not compatible to compare, report a proper error message.

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 src/tracefs-sqlhist.c | 36 +++++++++++++++++++++++++++++++++++-
 1 file changed, 35 insertions(+), 1 deletion(-)

diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
index 041d7077c3eb..acd3e0242d65 100644
--- a/src/tracefs-sqlhist.c
+++ b/src/tracefs-sqlhist.c
@@ -1029,6 +1029,38 @@ static void selection_error(struct tep_handle *tep,
 	test_field_exists(tep, sb, expr);
 }
 
+static void compare_error(struct tep_handle *tep,
+			    struct sqlhist_bison *sb, struct expr *expr)
+{
+	struct compare *compare = &expr->compare;
+
+	switch (errno) {
+	case ENODEV:
+	case EBADE:
+		break;
+	default:
+		/* System error */
+		return;
+	}
+
+	/* ENODEV means that an event or field does not exist */
+	if (errno == ENODEV) {
+		if (test_field_exists(tep, sb, compare->lval))
+			return;
+		if (test_field_exists(tep, sb, compare->rval))
+			return;
+		return;
+	}
+
+	/* fields exist, but values are not compatible */
+	sb->line_no = compare->lval->line;
+	sb->line_idx = compare->lval->idx;
+
+	parse_error(sb, compare->lval->field.raw,
+		    "'%s' is not compatible to compare with '%s'\n",
+		    compare->lval->field.raw, compare->rval->field.raw);
+}
+
 static struct tracefs_synth *build_synth(struct tep_handle *tep,
 					 const char *name,
 					 struct sql_table *table)
@@ -1121,8 +1153,10 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 
 		ret = build_compare(synth, start_system, end_system,
 				    &expr->compare);
-		if (ret < 0)
+		if (ret < 0) {
+			compare_error(tep, table->sb, expr);
 			goto free;
+		}
 	}
 
 	for (expr = table->where; expr; expr = expr->next) {
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 14/22] libtracefs: Add error message for grouping events in SQL filter
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (12 preceding siblings ...)
  2021-08-03 17:05 ` [PATCH v3 13/22] libtracefs: Add error message when compare fields fail Steven Rostedt
@ 2021-08-03 17:05 ` Steven Rostedt
  2021-08-03 17:05 ` [PATCH v3 15/22] libtracefs: Add error message for bad filters in SQL statement Steven Rostedt
                   ` (7 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:05 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

One requirement for the SQL filter in tracefs_sql() is that the WHERE
clause (filter) can only filter the FROM and JOIN events with "&&".

That is, you can not have:

  sched_switch.next_pid == 0 || sched_waking.pid == 0

As the filtering one event stops the synthetic event, having an ||
conjunction makes no sense.

Add an error message that explains this when it is found.

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 src/tracefs-sqlhist.c | 35 +++++++++++++++++++++++++----------
 1 file changed, 25 insertions(+), 10 deletions(-)

diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
index acd3e0242d65..a0d934b4cd68 100644
--- a/src/tracefs-sqlhist.c
+++ b/src/tracefs-sqlhist.c
@@ -802,21 +802,36 @@ static int build_compare(struct tracefs_synth *synth,
 	return ret;
 }
 
-static int do_verify_filter(struct filter *filter,
+static int verify_filter_error(struct sqlhist_bison *sb, struct expr *expr,
+			       const char *event)
+{
+	struct field *field = &expr->field;
+
+	sb->line_no = expr->line;
+	sb->line_idx = expr->idx;
+
+	parse_error(sb, field->raw,
+		    "event '%s' can not be grouped or '||' together with '%s'\n"
+		    "All filters between '&&' must be for the same event\n",
+		    field->event, event);
+	return -1;
+}
+
+static int do_verify_filter(struct sqlhist_bison *sb, struct filter *filter,
 			    const char **system, const char **event)
 {
 	int ret;
 
 	if (filter->type == FILTER_OR ||
 	    filter->type == FILTER_AND) {
-		ret = do_verify_filter(&filter->lval->filter, system, event);
+		ret = do_verify_filter(sb, &filter->lval->filter, system, event);
 		if (ret)
 			return ret;
-		return do_verify_filter(&filter->rval->filter, system, event);
+		return do_verify_filter(sb, &filter->rval->filter, system, event);
 	}
 	if (filter->type == FILTER_GROUP ||
 	    filter->type == FILTER_NOT_GROUP) {
-		return do_verify_filter(&filter->lval->filter, system, event);
+		return do_verify_filter(sb, &filter->lval->filter, system, event);
 	}
 
 	/*
@@ -831,12 +846,12 @@ static int do_verify_filter(struct filter *filter,
 
 	if (filter->lval->field.system != *system ||
 	    filter->lval->field.event_name != *event)
-		return -1;
+		return verify_filter_error(sb, filter->lval, *event);
 
 	return 0;
 }
 
-static int verify_filter(struct filter *filter,
+static int verify_filter(struct sqlhist_bison *sb, struct filter *filter,
 			 const char **system, const char **event)
 {
 	int ret;
@@ -848,17 +863,17 @@ static int verify_filter(struct filter *filter,
 	case FILTER_NOT_GROUP:
 		break;
 	default:
-		return do_verify_filter(filter, system, event);
+		return do_verify_filter(sb, filter, system, event);
 	}
 
-	ret = do_verify_filter(&filter->lval->filter, system, event);
+	ret = do_verify_filter(sb, &filter->lval->filter, system, event);
 	if (ret)
 		return ret;
 
 	switch (filter->type) {
 	case FILTER_OR:
 	case FILTER_AND:
-		return do_verify_filter(&filter->rval->filter, system, event);
+		return do_verify_filter(sb, &filter->rval->filter, system, event);
 	default:
 		return 0;
 	}
@@ -1165,7 +1180,7 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 		bool *started;
 		bool start;
 
-		ret = verify_filter(&expr->filter, &filter_system,
+		ret = verify_filter(table->sb, &expr->filter, &filter_system,
 				    &filter_event);
 		if (ret < 0)
 			goto free;
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 15/22] libtracefs: Add error message for bad filters in SQL statement
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (13 preceding siblings ...)
  2021-08-03 17:05 ` [PATCH v3 14/22] libtracefs: Add error message for grouping events in SQL filter Steven Rostedt
@ 2021-08-03 17:05 ` Steven Rostedt
  2021-08-03 17:06 ` [PATCH v3 16/22] libtracefs: Add error message when calculation has no label Steven Rostedt
                   ` (6 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:05 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

If a filter has a bad event, or incompatibility with the value assigned to
it, have the error message display it.

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 src/tracefs-sqlhist.c | 80 +++++++++++++++++++++++++++++++++++++++----
 1 file changed, 74 insertions(+), 6 deletions(-)

diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
index a0d934b4cd68..d7829d4804b1 100644
--- a/src/tracefs-sqlhist.c
+++ b/src/tracefs-sqlhist.c
@@ -879,14 +879,80 @@ static int verify_filter(struct sqlhist_bison *sb, struct filter *filter,
 	}
 }
 
-static int build_filter(struct tracefs_synth *synth,
-			bool start, struct filter *filter, bool *started)
+static int test_field_exists(struct tep_handle *tep, struct sqlhist_bison *sb,
+			     struct expr *expr);
+
+static void filter_compare_error(struct tep_handle *tep,
+				 struct sqlhist_bison *sb,
+				 struct expr *expr)
+{
+	struct field *field = &expr->field;
+
+	switch (errno) {
+	case ENODEV:
+	case EBADE:
+		break;
+	case EINVAL:
+		parse_error(sb, field->raw, "Invalid compare\n");
+		break;
+	default:
+		parse_error(sb, field->raw, "System error?\n");
+		return;
+	}
+
+	/* ENODEV means that an event or field does not exist */
+	if (errno == ENODEV) {
+		if (test_field_exists(tep, sb, expr))
+			return;
+		if (test_field_exists(tep, sb, expr))
+			return;
+		return;
+	}
+
+	/* fields exist, but values are not compatible */
+	sb->line_no = expr->line;
+	sb->line_idx = expr->idx;
+
+	parse_error(sb, field->raw,
+		    "Field '%s' is not compatible to be compared with the given value\n",
+		    field->field);
+}
+
+static void filter_error(struct tep_handle *tep,
+			 struct sqlhist_bison *sb, struct expr *expr)
+{
+	struct filter *filter = &expr->filter;
+
+	sb->line_no = expr->line;
+	sb->line_idx = expr->idx;
+
+	switch (filter->type) {
+	case FILTER_NOT_GROUP:
+	case FILTER_GROUP:
+	case FILTER_OR:
+	case FILTER_AND:
+		break;
+	default:
+		filter_compare_error(tep, sb, filter->lval);
+		return;
+	}
+
+	sb->line_no = expr->line;
+	sb->line_idx = expr->idx;
+
+	parse_error(sb, "", "Problem with filter entry?\n");
+}
+
+static int build_filter(struct tep_handle *tep, struct sqlhist_bison *sb,
+			struct tracefs_synth *synth,
+			bool start, struct expr *expr, bool *started)
 {
 	int (*append_filter)(struct tracefs_synth *synth,
 			     enum tracefs_filter type,
 			     const char *field,
 			     enum tracefs_compare compare,
 			     const char *val);
+	struct filter *filter = &expr->filter;
 	enum tracefs_compare cmp;
 	const char *val;
 	int and_or = TRACEFS_FILTER_AND;
@@ -916,7 +982,7 @@ static int build_filter(struct tracefs_synth *synth,
 				    NULL, 0, NULL);
 		if (ret < 0)
 			goto out;
-		ret = build_filter(synth, start, &filter->lval->filter, NULL);
+		ret = build_filter(tep, sb, synth, start, filter->lval, NULL);
 		if (ret < 0)
 			goto out;
 		ret = append_filter(synth, TRACEFS_FILTER_CLOSE_PAREN,
@@ -927,14 +993,14 @@ static int build_filter(struct tracefs_synth *synth,
 		and_or = TRACEFS_FILTER_OR;
 		/* Fall through */
 	case FILTER_AND:
-		ret = build_filter(synth, start, &filter->lval->filter, NULL);
+		ret = build_filter(tep, sb, synth, start, filter->lval, NULL);
 		if (ret < 0)
 			goto out;
 		ret = append_filter(synth, and_or, NULL, 0, NULL);
 
 		if (ret)
 			goto out;
-		ret = build_filter(synth, start, &filter->rval->filter, NULL);
+		ret = build_filter(tep, sb, synth, start, filter->rval, NULL);
 		goto out;
 	default:
 		break;
@@ -968,6 +1034,8 @@ static int build_filter(struct tracefs_synth *synth,
 	ret = append_filter(synth, TRACEFS_FILTER_COMPARE,
 			    filter->lval->field.field, cmp, val);
 
+	if (ret)
+		filter_error(tep, sb, expr);
  out:
 	if (!ret && started) {
 		if (*started)
@@ -1193,7 +1261,7 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 		else
 			started = &started_end;
 
-		ret = build_filter(synth, start, &expr->filter, started);
+		ret = build_filter(tep, table->sb, synth, start, expr, started);
 		if (ret < 0)
 			goto free;
 	}
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 16/22] libtracefs: Add error message when calculation has no label
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (14 preceding siblings ...)
  2021-08-03 17:05 ` [PATCH v3 15/22] libtracefs: Add error message for bad filters in SQL statement Steven Rostedt
@ 2021-08-03 17:06 ` Steven Rostedt
  2021-08-03 17:06 ` [PATCH v3 17/22] libtracefs: Add man page for tracefs_sql() Steven Rostedt
                   ` (5 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:06 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

If a calculation between event fields is performed and there's no label
(name) for it, it errors out, causing the bison parser to give a strange
error:

  FAILED MEMORY: add_selection(sb, (yyvsp[0].expr), NULL)
  Failed creating synthetic event!: No such file or directory

Instead, just set the compare->name field to NULL, and report a better
error later on in the processing.

  ERROR: 'no name'
  Field calculations must be labeled 'AS name'

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 src/tracefs-sqlhist.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
index d7829d4804b1..81a0cd1a908b 100644
--- a/src/tracefs-sqlhist.c
+++ b/src/tracefs-sqlhist.c
@@ -243,8 +243,6 @@ __hidden int add_selection(struct sqlhist_bison *sb, void *select,
 	case EXPR_FIELD:
 		break;
 	case EXPR_COMPARE:
-		if (!name)
-			return -1;
 		expr->compare.name = name;
 		break;
 	case EXPR_NUMBER:
@@ -779,6 +777,9 @@ static int build_compare(struct tracefs_synth *synth,
 	enum tracefs_synth_calc calc;
 	int ret;
 
+	if (!compare->name)
+		return -1;
+
 	lval = &compare->lval->field;
 	rval = &compare->rval->field;
 
@@ -1117,6 +1118,14 @@ static void compare_error(struct tep_handle *tep,
 {
 	struct compare *compare = &expr->compare;
 
+	if (!compare->name) {
+		sb->line_no = expr->line;
+		sb->line_idx = expr->idx + strlen("no name");
+
+		parse_error(sb, "no name",
+		    "Field calculations must be labeled 'AS name'\n");
+	}
+
 	switch (errno) {
 	case ENODEV:
 	case EBADE:
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 17/22] libtracefs: Add man page for tracefs_sql()
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (15 preceding siblings ...)
  2021-08-03 17:06 ` [PATCH v3 16/22] libtracefs: Add error message when calculation has no label Steven Rostedt
@ 2021-08-03 17:06 ` Steven Rostedt
  2021-08-03 17:06 ` [PATCH v3 18/22] libtracefs: Add Makefile rule to create sqlhist Steven Rostedt
                   ` (4 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:06 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

Add a man page for tracefs_sql(). Included in that man page is a full
working sql parser example program that can allow you to create synthetic
events from writing SQL on the command line.

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 Documentation/libtracefs-sql.txt | 389 +++++++++++++++++++++++++++++++
 1 file changed, 389 insertions(+)
 create mode 100644 Documentation/libtracefs-sql.txt

diff --git a/Documentation/libtracefs-sql.txt b/Documentation/libtracefs-sql.txt
new file mode 100644
index 000000000000..e10a22cd531b
--- /dev/null
+++ b/Documentation/libtracefs-sql.txt
@@ -0,0 +1,389 @@
+libtracefs(3)
+=============
+
+NAME
+----
+tracefs_sql - Create a synthetitc event via an SQL statement
+
+SYNOPSIS
+--------
+[verse]
+--
+*#include <tracefs.h>*
+
+struct tracefs_synth *tracefs_sql(struct tep_handle pass:[*]tep, const char pass:[*]name,
+				  const char pass:[*]sql_buffer, char pass:[**]err);
+--
+
+DESCRIPTION
+-----------
+Synthetic events are dynamically created events that attach two existing events
+together via one or more matching fields between the two events. It can be used
+to find the latency between the events, or to simply pass fields of the first event
+on to the second event to display as one event.
+
+The Linux kernel interface to create synthetic events is complex, and there needs
+to be a better way to create synthetic events that is easy and can be understood
+via existing technology.
+
+If you think of each event as a table, where the fields are the column of the table
+and each instance of the event as a row, you can understand how SQL can be used
+to attach two events together and form another event (table). Utilizing the
+SQL *SELECT* *FROM* *JOIN* *ON* [ *WHERE* ] syntax, a synthetic event can easily
+be created from two different events.
+
+
+*tracefs_sql*() takes in a *tep* handler (See _tep_local_events_(3)) that is used to
+verify the events within the _sql_buffer_ expression. The _name_ is the name of the
+synthetic event to create. If _err_ points to an address of a string, it will be filled
+with a detailed message on any type of parsing error, including fields that do not belong
+to an event, or if the events or fields are not properly compared.
+
+The example program below is a fully functional parser where it will create a synthetic
+event from a SQL syntax passed in via the command line or a file.
+
+The SQL format is as follows:
+
+*SELECT* <fields> *FROM* <start-event> *JOIN* <end-event> *ON* <matching-fields> *WHERE* <filter>
+
+Note, although the examples show the SQL commands in uppercase, they are not required to
+be so. That is, you can use "SELECT" or "select" or "sElEct".
+
+For example:
+[source,c]
+--
+SELECT syscalls.sys_enter_read.fd, syscalls.sys_exit_read.ret FROM syscalls.sys_enter_read
+   JOIN syscalls.sys_exit_read
+   ON syscalls.sys_enter_read.common_pid = syscalls.sys_exit_write.common_pid
+--
+
+Will create a synthetic event that with the fields:
+
+  u64 fd; s64 ret;
+
+Because the function takes a _tep_ handle, and usually all event names are unique, you can
+leave off the system (group) name of the event, and *tracefs_sql*() will discover the
+system for you.
+
+That is, the above statement would work with:
+
+[source,c]
+--
+SELECT sys_enter_read.fd, sys_exit_read.ret FROM sys_enter_read JOIN sys_exit_read
+   ON sys_enter_read.common_pid = sys_exit_write.common_pid
+--
+
+The *AS* keyword can be used to name the fields as well as to give an alias to the
+events, such that the above can be simplified even more as:
+
+[source,c]
+--
+SELECT start.fd, end.ret FROM sys_enter_read AS start JOIN sys_exit_read AS end ON start.common_pid = end.common_pid
+--
+
+The above aliases _sys_enter_read_ as *start* and _sys_exit_read_ as *end* and uses
+those aliases to reference the event throughout the statement.
+
+Using the *AS* keyword in the selection portion of the SQL statement will define what
+those fields will be called in the synthetic event.
+
+[source,c]
+--
+SELECT start.fd AS filed, end.ret AS return FROM sys_enter_read AS start JOIN sys_exit_read AS end
+   ON start.common_pid = end.common_pid
+--
+
+The above labels the _fd_ of _start_ as *filed* and the _ret_ of _end_ as *return* where
+the synthetic event that is created will now have the fields:
+
+  u64 filed; s64 return;
+
+The fields can also be calculated with results passed to the synthetic event:
+
+[source,c]
+--
+select start.truesize, end.len, (start.truesize - end.len) as diff from napi_gro_receive_entry as start
+   JOIN netif_receive_skb as end ON start.skbaddr = end.skbaddr
+--
+
+Which would show the *truesize* of the _napi_gro_receive_entry_ event, the actual
+_len_ of the content, shown by the _netif_receive_skb_, and the delta between
+the two and expressed by the field *diff*.
+
+The code also supports recording the timestamps at either event, and performing calculations
+on them. For wakeup latency, you have:
+
+[source,c]
+--
+select start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) as lat from sched_waking as start
+   JOIN sched_switch as end ON start.pid = end.next_pid
+--
+
+The above will create a synthetic event that records the _pid_ of the task being woken up,
+and the time difference between the _sched_waking_ event and the _sched_switch_ event.
+The *TIMESTAMP_USECS* will truncate the time down to microseconds as the timestamp usually
+recorded in the tracing buffer has nanosecond resolution. If you do not want that
+truncation, use *TIMESTAMP* instead of *TIMESTAMP_USECS*.
+
+Finally, the *WHERE* clause can be added, that will let you add filters on either or both events.
+
+[source,c]
+--
+select start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) as lat from sched_waking as start
+   JOIN sched_switch as end ON start.pid = end.next_pid
+   WHERE start.prio < 100 && (!(end.prev_pid < 1 || end.prev_prio > 100) || end.prev_pid == 0)
+--
+
+*NOTE*
+
+Although both events can be used together in the *WHERE* clause, they must not be mixed outside
+the top most "&&" statements. You can not OR (||) the events together, where a filter of one
+event is OR'd to a filter of the other event. This does not make sense, as the synthetic event
+requires both events to take place to be recorded. If one is filtered out, then the synthetic
+event does not execute.
+
+[source,c]
+--
+select start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) as lat from sched_waking as start
+   JOIN sched_switch as end ON start.pid = end.next_pid
+   WHERE start.prio < 100 && end.prev_prio < 100
+--
+
+The above is valid.
+
+Where as the below is not.
+
+[source,c]
+--
+select start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) as lat from sched_waking as start
+   JOIN sched_switch as end ON start.pid = end.next_pid
+   WHERE start.prio < 100 || end.prev_prio < 100
+--
+
+
+RETURN VALUE
+------------
+Returns 0 on success and -1 on failure. On failure, if _err_ is defined, it will be
+allocated to hold a detailed description of what went wrong if it the error was caused
+by a parsing error, or that an event, field does not exist or is not compatible with
+what it was combined with.
+
+CREATE A TOOL
+-------------
+
+The below example is a functional program that can be used to parse SQL commands into
+synthetic events.
+
+[source, c]
+--
+   man tracefs_sql | sed -ne '/^EXAMPLE/,/FILES/ { /EXAMPLE/d ; /FILES/d ; p}' > sqlhist.c
+   gcc -o sqlhist sqlhist.c `pkg-config --cflags --libs libtracefs`
+--
+
+Then you can run the above examples:
+
+[source, c]
+--
+  sudo ./sqlhist 'select start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) as lat from sched_waking as start
+                  JOIN sched_switch as end ON start.pid = end.next_pid
+                  WHERE start.prio < 100 || end.prev_prio < 100'
+--
+
+EXAMPLE
+-------
+[source,c]
+--
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdarg.h>
+#include <string.h>
+#include <errno.h>
+#include <unistd.h>
+#include <tracefs.h>
+
+static void usage(char **argv)
+{
+	fprintf(stderr, "usage: %s [-ed][-n name][-t dir][-f file | sql-command-line]\n"
+		"  -n name - name of synthetic event 'Anonymous' if left off\n"
+		"  -t dir - use dir instead of /sys/kernel/tracing\n"
+		"  -e - execute the commands to create the synthetic event\n"
+		"  -d - delete the synthetic event that would be created\n"
+		"  -f file - read sql lines from file otherwise from the command line\n"
+		"            if file is '-' then read from standard input.\n",
+		argv[0]);
+	exit(-1);
+}
+
+static int do_sql(const char *buffer, const char *name,
+		  const char *trace_dir, bool execute)
+{
+	struct tracefs_synth *synth;
+	struct tep_handle *tep;
+	struct trace_seq seq;
+	char *err;
+
+	if (!name)
+		name = "Anonymous";
+
+	trace_seq_init(&seq);
+	tep = tracefs_local_events(trace_dir);
+	if (!tep) {
+		if (!trace_dir)
+			trace_dir = "tracefs directory";
+		perror(trace_dir);
+		exit(-1);
+	}
+
+	synth = tracefs_sql(tep, name, buffer, &err);
+	if (!synth) {
+		perror("Failed creating synthetic event!");
+		if (err)
+			fprintf(stderr, "%s", err);
+		free(err);
+		exit(-1);
+	}
+
+	tracefs_synth_show(&seq, NULL, synth);
+	if (execute)
+		tracefs_synth_create(NULL, synth);
+	tracefs_synth_free(synth);
+
+	trace_seq_do_printf(&seq);
+	trace_seq_destroy(&seq);
+	return 0;
+}
+
+int main (int argc, char **argv)
+{
+	char *trace_dir = NULL;
+	char *buffer = NULL;
+	char buf[BUFSIZ];
+	int buffer_size = 0;
+	const char *file = NULL;
+	bool execute = false;
+	const char *name;
+	FILE *fp;
+	size_t r;
+	int c;
+	int i;
+
+	for (;;) {
+		c = getopt(argc, argv, "ht:f:edn:");
+		if (c == -1)
+			break;
+
+		switch(c) {
+		case 'h':
+			usage(argv);
+		case 't':
+			trace_dir = optarg;
+			break;
+		case 'f':
+			file = optarg;
+			break;
+		case 'e':
+			execute = true;
+			break;
+		case 'n':
+			name = optarg;
+			break;
+		}
+	}
+
+	if (file) {
+		if (!strcmp(file, "-"))
+			fp = stdin;
+		else
+			fp = fopen(file, "r");
+		if (!fp) {
+			perror(file);
+			exit(-1);
+		}
+		while ((r = fread(buf, 1, BUFSIZ, fp)) > 0) {
+			buffer = realloc(buffer, buffer_size + r + 1);
+			strncpy(buffer + buffer_size, buf, r);
+			buffer_size += r;
+		}
+		fclose(fp);
+		if (buffer_size)
+			buffer[buffer_size] = '\0';
+	} else if (argc == optind) {
+		usage(argv);
+	} else {
+		for (i = optind; i < argc; i++) {
+			r = strlen(argv[i]);
+			buffer = realloc(buffer, buffer_size + r + 2);
+			if (i != optind)
+				buffer[buffer_size++] = ' ';
+			strcpy(buffer + buffer_size, argv[i]);
+			buffer_size += r;
+		}
+	}
+
+	do_sql(buffer, name, trace_dir, execute);
+	free(buffer);
+
+	return 0;
+}
+--
+
+FILES
+-----
+[verse]
+--
+*tracefs.h*
+	Header file to include in order to have access to the library APIs.
+*-ltracefs*
+	Linker switch to add when building a program that uses the library.
+--
+
+SEE ALSO
+--------
+_libtracefs(3)_,
+_libtraceevent(3)_,
+_trace-cmd(1)_,
+_tracefs_synth_init(3)_,
+_tracefs_synth_add_match_field(3)_,
+_tracefs_synth_add_compare_field(3)_,
+_tracefs_synth_add_start_field(3)_,
+_tracefs_synth_add_end_field(3)_,
+_tracefs_synth_append_start_filter(3)_,
+_tracefs_synth_append_end_filter(3)_,
+_tracefs_synth_create(3)_,
+_tracefs_synth_destroy(3)_,
+_tracefs_synth_free(3)_,
+_tracefs_synth_show(3)_,
+_tracefs_hist_alloc(3)_,
+_tracefs_hist_free(3)_,
+_tracefs_hist_add_key(3)_,
+_tracefs_hist_add_value(3)_,
+_tracefs_hist_add_name(3)_,
+_tracefs_hist_start(3)_,
+_tracefs_hist_destory(3)_,
+_tracefs_hist_add_sort_key(3)_,
+_tracefs_hist_sort_key_direction(3)_
+
+AUTHOR
+------
+[verse]
+--
+*Steven Rostedt* <rostedt@goodmis.org>
+*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>
+*sameeruddin shaik* <sameeruddin.shaik8@gmail.com>
+--
+REPORTING BUGS
+--------------
+Report bugs to  <linux-trace-devel@vger.kernel.org>
+
+LICENSE
+-------
+libtracefs is Free Software licensed under the GNU LGPL 2.1
+
+RESOURCES
+---------
+https://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git/
+
+COPYING
+-------
+Copyright \(C) 2020 VMware, Inc. Free use of this software is granted under
+the terms of the GNU Public License (GPL).
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 18/22] libtracefs: Add Makefile rule to create sqlhist
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (16 preceding siblings ...)
  2021-08-03 17:06 ` [PATCH v3 17/22] libtracefs: Add man page for tracefs_sql() Steven Rostedt
@ 2021-08-03 17:06 ` Steven Rostedt
  2021-08-03 17:06 ` [PATCH v3 19/22] libtracefs: Allow for simple SQL statements to create a histogram Steven Rostedt
                   ` (3 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:06 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

Add a Makefile rule to extract the example from the man page and create a
sqlhist executable from it.

  make sqlhist

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 Makefile | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/Makefile b/Makefile
index 96d7f5528a56..7f382c304b9b 100644
--- a/Makefile
+++ b/Makefile
@@ -364,11 +364,21 @@ $(bdir)/libtracefs.so.$(TRACEFS_VERSION): force
 	$(Q)mkdir -p $(bdir)
 	$(Q)$(MAKE) -C $(src)/src libtracefs.so
 
+$(bdir)/sqlhist.c: Documentation/libtracefs-sql.txt
+	cat $< | sed -ne '/^EXAMPLE/,/FILES/ { /EXAMPLE/,+2d ; /^FILES/d ;  /^--/d ; p}' > $@
+
+$(bdir)/sqlhist.o: $(bdir)/sqlhist.c
+	$(CC) -g -Wall -c -o $@ $^ -Iinclude/ $(LIBTRACEEVENT_INCLUDES)
+
+sqlhist: $(bdir)/sqlhist.o $(LIBTRACEFS_STATIC)
+	$(CC) -o $@ $^ $(LIBTRACEEVENT_LIBS)
+
 clean:
 	$(MAKE) -C $(src)/utest clean
 	$(MAKE) -C $(src)/src clean
 	$(RM) $(TARGETS) $(bdir)/*.a $(bdir)/*.so $(bdir)/*.so.* $(bdir)/*.o $(bdir)/.*.d
 	$(RM) $(PKG_CONFIG_FILE)
 	$(RM) $(VERSION_FILE)
+	$(RM) $(bdir)/sqlhist.o $(bdir)/sqlhist.c sqlhist
 
 .PHONY: clean
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 19/22] libtracefs: Allow for simple SQL statements to create a histogram
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (17 preceding siblings ...)
  2021-08-03 17:06 ` [PATCH v3 18/22] libtracefs: Add Makefile rule to create sqlhist Steven Rostedt
@ 2021-08-03 17:06 ` Steven Rostedt
  2021-08-03 17:06 ` [PATCH v3 20/22] libtracefs: Allow trace_sql() to take keywords for fields with backslash Steven Rostedt
                   ` (2 subsequent siblings)
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:06 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware),
	Ahmed S . Darwish

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

Allow tracefs_sql() to take a simple select statement without the
JOIN .. ON clause, that will simply update the start event. This, along
with tracefs_synth_get_start_hist(), will allow a user to utilize
tracefs_sql() to create a synthetic event.

Link: https://lore.kernel.org/linux-rt-users/YQakDYRnId+bK+ue@lx-t490/

Suggested-by: Ahmed S. Darwish <a.darwish@linutronix.de>
Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 Documentation/libtracefs-sql.txt |  27 +++++-
 include/tracefs-local.h          |   3 +
 src/sqlhist.y                    |  11 ++-
 src/tracefs-hist.c               |  60 +++++++++----
 src/tracefs-sqlhist.c            | 144 +++++++++++++++++++++++++++++--
 5 files changed, 216 insertions(+), 29 deletions(-)

diff --git a/Documentation/libtracefs-sql.txt b/Documentation/libtracefs-sql.txt
index e10a22cd531b..ee8d5c1d63c7 100644
--- a/Documentation/libtracefs-sql.txt
+++ b/Documentation/libtracefs-sql.txt
@@ -32,6 +32,8 @@ to attach two events together and form another event (table). Utilizing the
 SQL *SELECT* *FROM* *JOIN* *ON* [ *WHERE* ] syntax, a synthetic event can easily
 be created from two different events.
 
+For simple SQL queries to make a histogram instead of a synthetic event, see
+HISTOGRAMS below.
 
 *tracefs_sql*() takes in a *tep* handler (See _tep_local_events_(3)) that is used to
 verify the events within the _sql_buffer_ expression. The _name_ is the name of the
@@ -160,6 +162,12 @@ select start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) as lat from sche
    WHERE start.prio < 100 || end.prev_prio < 100
 --
 
+HISTOGRAMS
+----------
+
+Simple SQL statements without the *JOIN* *ON* may also be used, which will create a histogram
+instead. When doing this, the struct tracefs_hist descriptor can be retrieved from the
+returned synthetic event descriptor via the *tracefs_synth_get_start_hist*(3).
 
 RETURN VALUE
 ------------
@@ -243,9 +251,22 @@ static int do_sql(const char *buffer, const char *name,
 		exit(-1);
 	}
 
-	tracefs_synth_show(&seq, NULL, synth);
-	if (execute)
-		tracefs_synth_create(NULL, synth);
+	if (tracefs_synth_complete(synth)) {
+		tracefs_synth_show(&seq, NULL, synth);
+		if (execute)
+			tracefs_synth_create(NULL, synth);
+	} else {
+		struct tracefs_hist *hist;
+		hist = tracefs_synth_get_start_hist(synth);
+		if (!hist) {
+			perror("get_start_hist");
+			exit(-1);
+		}
+		tracefs_hist_show(&seq, NULL, hist, 0);
+		if (execute)
+			tracefs_hist_start(NULL, hist);
+	}
+
 	tracefs_synth_free(synth);
 
 	trace_seq_do_printf(&seq);
diff --git a/include/tracefs-local.h b/include/tracefs-local.h
index 41fbcc0faa95..09288aeac521 100644
--- a/include/tracefs-local.h
+++ b/include/tracefs-local.h
@@ -85,4 +85,7 @@ int trace_append_filter(char **filter, unsigned int *state,
 			enum tracefs_compare compare,
 			 const char *val);
 
+struct tracefs_synth *synth_init_from(struct tep_handle *tep,
+				      const char *start_system,
+				      const char *start_event);
 #endif /* _TRACE_FS_LOCAL_H */
diff --git a/src/sqlhist.y b/src/sqlhist.y
index 9d03a457ae84..d5cbecc7bf92 100644
--- a/src/sqlhist.y
+++ b/src/sqlhist.y
@@ -63,7 +63,7 @@ extern void yyerror(struct sqlhist_bison *, char *fmt, ...);
 
 %type <string> name label
 
-%type <expr>  selection_expr field item named_field join_clause
+%type <expr>  selection_expr field item named_field
 %type <expr>  selection_addition
 %type <expr>  compare compare_list compare_cmds compare_items
 %type <expr>  compare_and_or
@@ -202,8 +202,13 @@ opt_where_clause :
  | where_clause
 ;
 
+opt_join_clause :
+  /* empty set */
+  | join_clause
+ ;
+
 table_exp :
-   from_clause join_clause opt_where_clause
+   from_clause opt_join_clause opt_where_clause
  ;
 
 from_clause :
@@ -222,7 +227,7 @@ from_clause :
  ;
 
 join_clause :
-  JOIN item ON match_clause	{ add_to(sb, $2); }
+ JOIN item ON match_clause	{ add_to(sb, $2); }
  ;
 
 match :
diff --git a/src/tracefs-hist.c b/src/tracefs-hist.c
index 305e3e720341..301abc255d5f 100644
--- a/src/tracefs-hist.c
+++ b/src/tracefs-hist.c
@@ -724,6 +724,33 @@ static int add_var(char ***list, const char *name, const char *var, bool is_var)
 	return 0;
 }
 
+__hidden struct tracefs_synth *
+synth_init_from(struct tep_handle *tep, const char *start_system,
+		const char *start_event_name)
+{
+	struct tep_event *start_event;
+	struct tracefs_synth *synth;
+
+	start_event = tep_find_event_by_name(tep, start_system,
+					     start_event_name);
+	if (!start_event) {
+		errno = ENODEV;
+		return NULL;
+	}
+
+	synth = calloc(1, sizeof(*synth));
+	if (!synth)
+		return NULL;
+
+	synth->start_event = start_event;
+
+	/* Hold onto a reference to this handler */
+	tep_ref(tep);
+	synth->tep = tep;
+
+	return synth;
+}
+
 /**
  * tracefs_synth_init - create a new tracefs_synth instance
  * @tep: The tep handle that holds the events to work on
@@ -778,7 +805,6 @@ struct tracefs_synth *tracefs_synth_init(struct tep_handle *tep,
 					 const char *end_match_field,
 					 const char *match_name)
 {
-	struct tep_event *start_event;
 	struct tep_event *end_event;
 	struct tracefs_synth *synth;
 	int ret = 0;
@@ -789,25 +815,18 @@ struct tracefs_synth *tracefs_synth_init(struct tep_handle *tep,
 		return NULL;
 	}
 
-	start_event = tep_find_event_by_name(tep, start_system,
-					     start_event_name);
-	if (!start_event) {
-		errno = ENODEV;
+	synth = synth_init_from(tep, start_system, start_event_name);
+	if (!synth)
 		return NULL;
-	}
 
 	end_event = tep_find_event_by_name(tep, end_system,
 					   end_event_name);
 	if (!end_event) {
+		tep_unref(tep);
 		errno = ENODEV;
 		return NULL;
 	}
 
-	synth = calloc(1, sizeof(*synth));
-	if (!synth)
-		return NULL;
-
-	synth->start_event = start_event;
 	synth->end_event = end_event;
 
 	synth->name = strdup(name);
@@ -815,10 +834,6 @@ struct tracefs_synth *tracefs_synth_init(struct tep_handle *tep,
 	ret = tracefs_synth_add_match_field(synth, start_match_field,
 					    end_match_field, match_name);
 
-	/* Hold onto a reference to this handler */
-	tep_ref(tep);
-	synth->tep = tep;
-
 	if (!synth->name || !synth->start_keys || !synth->end_keys || ret) {
 		tracefs_synth_free(synth);
 		synth = NULL;
@@ -1458,6 +1473,11 @@ int tracefs_synth_create(struct tracefs_instance *instance,
 		return -1;
 	}
 
+	if (!synth->name || !synth->end_event) {
+		errno = EUNATCH;
+		return -1;
+	}
+
 	if (verify_state(synth) < 0)
 		return -1;
 
@@ -1540,6 +1560,11 @@ int tracefs_synth_destroy(struct tracefs_instance *instance,
 		return -1;
 	}
 
+	if (!synth->name || !synth->end_event) {
+		errno = EUNATCH;
+		return -1;
+	}
+
 	/* Try to disable the event if possible */
 	tracefs_event_disable(instance, "synthetic", synth->name);
 
@@ -1596,6 +1621,11 @@ int tracefs_synth_show(struct trace_seq *seq,
 		return -1;
 	}
 
+	if (!synth->name || !synth->end_event) {
+		errno = EUNATCH;
+		return -1;
+	}
+
 	synthetic_event = create_synthetic_event(synth);
 	if (!synthetic_event)
 		return -1;
diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
index 81a0cd1a908b..d9ebe2eab411 100644
--- a/src/tracefs-sqlhist.c
+++ b/src/tracefs-sqlhist.c
@@ -678,6 +678,70 @@ static int update_vars(struct tep_handle *tep,
 	return 0;
 }
 
+/*
+ * Called when there's a FROM but no JOIN(to), which means that the
+ * selections can be fields and not mention the event itself.
+ */
+static int update_fields(struct tep_handle *tep,
+			 struct sql_table *table,
+			 struct expr *expr)
+{
+	struct field *event_field = &expr->field;
+	struct sqlhist_bison *sb = table->sb;
+	struct tep_format_field *tfield;
+	struct tep_event *event;
+	struct field *field;
+	const char *p;
+	int len;
+
+	/* First update fields with aliases an such and add event */
+	update_vars(tep, table, expr);
+
+	/*
+	 * If event is not found, the creation of the synth will
+	 * add a proper error, so return "success".
+	*/
+	if (!event_field->event)
+		return 0;
+
+	event = event_field->event;
+
+	for_each_field(expr, field, table) {
+		const char *field_name;
+
+		field = &expr->field;
+
+		if (field->event)
+			continue;
+
+		field_name = field->raw;
+
+		p = strchr(field_name, '.');
+		if (p) {
+			len = p - field_name;
+			p = strndup(field_name, len);
+			if (!p)
+				return -1;
+			field_name = store_str(sb, p);
+			if (!field_name)
+				return -1;
+			free((char *)p);
+		}
+
+		tfield = tep_find_any_field(event, field_name);
+		/* Let it error properly later */
+		if (!tfield)
+			continue;
+
+		field->system = event_field->system;
+		field->event_name = event_field->event_name;
+		field->event = event;
+		field->field = field_name;
+	}
+
+	return 0;
+}
+
 static int match_error(struct sqlhist_bison *sb, struct match *match,
 		       struct field *lmatch, struct field *rmatch)
 {
@@ -1153,6 +1217,42 @@ static void compare_error(struct tep_handle *tep,
 		    compare->lval->field.raw, compare->rval->field.raw);
 }
 
+static void compare_no_to_error(struct sqlhist_bison *sb, struct expr *expr)
+{
+	struct compare *compare = &expr->compare;
+
+	sb->line_no = compare->lval->line;
+	sb->line_idx = compare->lval->idx;
+
+	parse_error(sb, compare->lval->field.raw,
+		    "Simple SQL (without JOIN/ON) do not allow comparisons\n",
+		    compare->lval->field.raw, compare->rval->field.raw);
+}
+
+static void where_no_to_error(struct sqlhist_bison *sb, struct expr *expr,
+			      const char *from_event, const char *event)
+{
+	while (expr) {
+		switch (expr->filter.type) {
+		case FILTER_OR:
+		case FILTER_AND:
+		case FILTER_GROUP:
+		case FILTER_NOT_GROUP:
+			expr = expr->filter.lval;
+			continue;
+		default:
+			break;
+		}
+		break;
+	}
+	sb->line_no = expr->filter.lval->line;
+	sb->line_idx = expr->filter.lval->idx;
+
+	parse_error(sb, expr->filter.lval->field.raw,
+		    "Event '%s' does not match FROM event '%s'\n",
+		    event, from_event);
+}
+
 static struct tracefs_synth *build_synth(struct tep_handle *tep,
 					 const char *name,
 					 struct sql_table *table)
@@ -1171,17 +1271,35 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 	bool started_end = false;
 	int ret;
 
-	if (!table->to || !table->from)
+	if (!table->from)
 		return NULL;
 
-	ret = update_vars(tep, table, table->to);
+	/* This could be a simple SQL statement to only build a histogram */
+	if (!table->to) {
+		ret = update_fields(tep, table, table->from);
+		if (ret < 0)
+			return NULL;
+
+		start_system = table->from->field.system;
+		start_event = table->from->field.event_name;
+
+		synth = synth_init_from(tep, start_system, start_event);
+		if (!synth)
+			return synth_init_error(tep, table);
+		goto hist_only;
+	}
+
+	ret = update_vars(tep, table, table->from);
 	if (ret < 0)
 		return NULL;
 
-	ret = update_vars(tep, table, table->from);
+	ret = update_vars(tep, table, table->to);
 	if (ret < 0)
 		return NULL;
 
+	start_system = table->from->field.system;
+	start_event = table->from->field.event_name;
+
 	match = table->matches;
 	if (!match)
 		return NULL;
@@ -1190,9 +1308,6 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 	if (ret < 0)
 		return NULL;
 
-	start_system = table->from->field.system;
-	start_event = table->from->field.event_name;
-
 	end_system = table->to->field.system;
 	end_event = table->to->field.event_name;
 
@@ -1222,14 +1337,18 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 		}
 	}
 
+ hist_only:
+	/* table->to may be NULL here */
+
 	for (expr = table->selections; expr; expr = expr->next) {
 		if (expr->type == EXPR_FIELD) {
+			ret = -1;
 			field = &expr->field;
 			if (field->system == start_system &&
 			    field->event_name == start_event) {
 				ret = tracefs_synth_add_start_field(synth,
 						field->field, field->label);
-			} else {
+			} else if (table->to) {
 				ret = tracefs_synth_add_end_field(synth,
 						field->field, field->label);
 			}
@@ -1240,6 +1359,11 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 			continue;
 		}
 
+		if (!table->to) {
+			compare_no_to_error(table->sb, expr);
+			goto free;
+		}
+
 		if (expr->type != EXPR_COMPARE)
 			goto free;
 
@@ -1267,7 +1391,11 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 
 		if (start)
 			started = &started_start;
-		else
+		else if (!table->to) {
+			where_no_to_error(table->sb, expr, start_event,
+					  filter_event);
+			goto free;
+		} else
 			started = &started_end;
 
 		ret = build_filter(tep, table->sb, synth, start, expr, started);
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 20/22] libtracefs: Allow trace_sql() to take keywords for fields with backslash
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (18 preceding siblings ...)
  2021-08-03 17:06 ` [PATCH v3 19/22] libtracefs: Allow for simple SQL statements to create a histogram Steven Rostedt
@ 2021-08-03 17:06 ` Steven Rostedt
  2021-08-03 17:06 ` [PATCH v3 21/22] libtracefs: Add CAST() syntax to SQL parsing for histogram types Steven Rostedt
  2021-08-03 17:06 ` [PATCH v3 22/22] libtracefs: Add CAST(x AS _COUNTER_) syntax to create values in histograms Steven Rostedt
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:06 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

Some events have fields with keywords, and the parsing will not let it
work. For example, you can have:

  select from from regcache_drop_region

And that will produce a syntax error, as the from will confuse the parser.
Allow fields to start with a backslash, which will allow the parser to see
"from" as a field and not as a keyword. That is:

  select \from from regcache_drop_region

will work as expected. Note, any field can start with a backslash, and the
starting backslash will be ignored.

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 Documentation/libtracefs-sql.txt | 20 ++++++++++++++++++++
 src/sqlhist.l                    |  6 ++++--
 2 files changed, 24 insertions(+), 2 deletions(-)

diff --git a/Documentation/libtracefs-sql.txt b/Documentation/libtracefs-sql.txt
index ee8d5c1d63c7..bc2811cfe08b 100644
--- a/Documentation/libtracefs-sql.txt
+++ b/Documentation/libtracefs-sql.txt
@@ -162,6 +162,26 @@ select start.pid, (end.TIMESTAMP_USECS - start.TIMESTAMP_USECS) as lat from sche
    WHERE start.prio < 100 || end.prev_prio < 100
 --
 
+
+KEYWORDS AS EVENT FIELDS
+------------------------
+
+In some cases, an event may have a keyword. For example, regcache_drop_region has "from"
+as a field and the following will not work
+
+[source,c]
+--
+  select from from regcache_drop_region
+--
+
+In such cases, add a backslash to the conflicting field, and this will tell the parser
+that the "from" is a field and not a keyword:
+
+[source,c]
+--
+  select \from from regcache_drop_region
+--
+
 HISTOGRAMS
 ----------
 
diff --git a/src/sqlhist.l b/src/sqlhist.l
index f9e0fcc17e63..897daac7d2a8 100644
--- a/src/sqlhist.l
+++ b/src/sqlhist.l
@@ -25,7 +25,7 @@ extern int my_yyinput(void *extra, char *buf, int max);
 %option reentrant
 %option bison-bridge
 
-field		[a-z_][a-z0-9_\.]*
+field		\\?[a-z_][a-z0-9_\.]*
 qstring		\"[^\"]*\"
 hexnum		0x[0-9a-f]+
 number		[0-9a-f]+
@@ -46,8 +46,10 @@ where { HANDLE_COLUMN; return WHERE; }
 }
 
 {field} {
+	const char *str = yyg->yytext_r;
 	HANDLE_COLUMN;
-	yylval->string = store_str(TRACE_SB, yyg->yytext_r);
+	if (str[0] == '\\') { str++; };
+	yylval->string = store_str(TRACE_SB, str);
 	return FIELD;
 }
 
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 21/22] libtracefs: Add CAST() syntax to SQL parsing for histogram types
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (19 preceding siblings ...)
  2021-08-03 17:06 ` [PATCH v3 20/22] libtracefs: Allow trace_sql() to take keywords for fields with backslash Steven Rostedt
@ 2021-08-03 17:06 ` Steven Rostedt
  2021-08-03 17:06 ` [PATCH v3 22/22] libtracefs: Add CAST(x AS _COUNTER_) syntax to create values in histograms Steven Rostedt
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:06 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

Use CAST() command of SQL to define the type of a field for a histogram.

For example:

  SELECT common_pid, id FROM sys_enter

Will only create:

  echo 'hist:keys=common_pid,id' > events/raw_syscalls/sys_enter

that produces the uninteresting:

{ common_pid:       1428 } hitcount:          2
{ common_pid:       1427 } hitcount:          2
{ common_pid:      13684 } hitcount:          2
{ common_pid:       1534 } hitcount:          3
{ common_pid:       1715 } hitcount:          3
{ common_pid:       1334 } hitcount:          4

Use CAST to add the conversions:

  SELECT CAST(common_pid AS comm), CAST(id as syscall) FROM sys_enter

That creates:

  echo 'hist:keys=common_pid.execname,id.syscall' > events/raw_syscalls/sys_enter

Which produces the much more informative histogram:

{ common_pid: bash            [     18116], id: sys_setpgid                   [109] } hitcount:          1
{ common_pid: cat             [     18116], id: sys_munmap                    [ 11] } hitcount:          1
{ common_pid: cat             [     18115], id: sys_exit_group                [231] } hitcount:          1
{ common_pid: cat             [     18115], id: sys_fadvise64                 [221] } hitcount:          1
{ common_pid: wpa_supplicant  [      1437], id: sys_select                    [ 23] } hitcount:          1
{ common_pid: gmain           [      1715], id: sys_poll                      [  7] } hitcount:          1
{ common_pid: cat             [     18116], id: sys_fadvise64                 [221] } hitcount:          1
{ common_pid: gmain           [      1534], id: sys_poll                      [  7] } hitcount:          1
{ common_pid: bash            [     18117], id: sys_getpid                    [ 39] } hitcount:          1
{ common_pid: gmain           [     13684], id: sys_poll                      [  7] } hitcount:          1
{ common_pid: bash            [      1768], id: sys_dup2                      [ 33] } hitcount:          1

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 Documentation/libtracefs-sql.txt |  50 ++++++++++++++
 include/tracefs-local.h          |   4 ++
 src/sqlhist-parse.h              |   1 +
 src/sqlhist.l                    |   3 +-
 src/sqlhist.y                    |   8 ++-
 src/tracefs-hist.c               |  79 +++++++++++++++-------
 src/tracefs-sqlhist.c            | 108 ++++++++++++++++++++++++++++++-
 7 files changed, 224 insertions(+), 29 deletions(-)

diff --git a/Documentation/libtracefs-sql.txt b/Documentation/libtracefs-sql.txt
index bc2811cfe08b..b7d5c1b4f658 100644
--- a/Documentation/libtracefs-sql.txt
+++ b/Documentation/libtracefs-sql.txt
@@ -189,6 +189,56 @@ Simple SQL statements without the *JOIN* *ON* may also be used, which will creat
 instead. When doing this, the struct tracefs_hist descriptor can be retrieved from the
 returned synthetic event descriptor via the *tracefs_synth_get_start_hist*(3).
 
+In order to utilize the histogram types (see xxx) the CAST command of SQL can be used.
+
+That is:
+
+[source,c]
+--
+  select CAST(common_pid AS comm, CAST(id AS syscall) FROM sys_enter
+--
+
+Which produces:
+
+[source,c]
+--
+ # echo 'hist:keys=common_pid.execname,id.syscall' > events/raw_syscalls/sys_enter/trigger
+
+ # cat events/raw_syscalls/sys_enter/hist
+
+{ common_pid: bash            [     18248], id: sys_setpgid                   [109] } hitcount:          1
+{ common_pid: sendmail        [      1812], id: sys_read                      [  0] } hitcount:          1
+{ common_pid: bash            [     18247], id: sys_getpid                    [ 39] } hitcount:          1
+{ common_pid: bash            [     18247], id: sys_dup2                      [ 33] } hitcount:          1
+{ common_pid: gmain           [     13684], id: sys_inotify_add_watch         [254] } hitcount:          1
+{ common_pid: cat             [     18247], id: sys_access                    [ 21] } hitcount:          1
+{ common_pid: bash            [     18248], id: sys_getpid                    [ 39] } hitcount:          1
+{ common_pid: cat             [     18247], id: sys_fadvise64                 [221] } hitcount:          1
+{ common_pid: sendmail        [      1812], id: sys_openat                    [257] } hitcount:          1
+{ common_pid: less            [     18248], id: sys_munmap                    [ 11] } hitcount:          1
+{ common_pid: sendmail        [      1812], id: sys_close                     [  3] } hitcount:          1
+{ common_pid: gmain           [      1534], id: sys_poll                      [  7] } hitcount:          1
+{ common_pid: bash            [     18247], id: sys_execve                    [ 59] } hitcount:          1
+--
+
+Note, string fields may not be cast.
+
+The possible types to cast to are:
+
+*HEX* - convert the value to use hex and not decimal
+
+*SYM* - convert a pointer to symbolic (kallsyms values)
+
+*SYM-OFFSET* - convert a pointer to symbolic and include the offset.
+
+*SYSCALL* - convert the number to the mapped system call name
+
+*EXECNAME* or *COMM* - can only be used with the common_pid field. Will show the task
+name of the process.
+
+*LOG* or *LOG2* - bucket the key values in a log 2 values (1, 2, 3-4, 5-8, 9-16, 17-32, ...)
+
+
 RETURN VALUE
 ------------
 Returns 0 on success and -1 on failure. On failure, if _err_ is defined, it will be
diff --git a/include/tracefs-local.h b/include/tracefs-local.h
index 09288aeac521..07d40b2fae4f 100644
--- a/include/tracefs-local.h
+++ b/include/tracefs-local.h
@@ -88,4 +88,8 @@ int trace_append_filter(char **filter, unsigned int *state,
 struct tracefs_synth *synth_init_from(struct tep_handle *tep,
 				      const char *start_system,
 				      const char *start_event);
+int synth_add_start_field(struct tracefs_synth *synth,
+			  const char *start_field,
+			  const char *name,
+			  enum tracefs_hist_key_type type);
 #endif /* _TRACE_FS_LOCAL_H */
diff --git a/src/sqlhist-parse.h b/src/sqlhist-parse.h
index 7c1b97ca65af..d5b86cacd8ce 100644
--- a/src/sqlhist-parse.h
+++ b/src/sqlhist-parse.h
@@ -65,6 +65,7 @@ int add_where(struct sqlhist_bison *sb, void *expr);
 int add_selection(struct sqlhist_bison *sb, void *item, const char *label);
 int add_from(struct sqlhist_bison *sb, void *item);
 int add_to(struct sqlhist_bison *sb, void *item);
+void *add_cast(struct sqlhist_bison *sb, void *field, const char *type);
 
 void *add_string(struct sqlhist_bison *sb, const char *str);
 void *add_number(struct sqlhist_bison *sb, long val);
diff --git a/src/sqlhist.l b/src/sqlhist.l
index 897daac7d2a8..db0ae1101a58 100644
--- a/src/sqlhist.l
+++ b/src/sqlhist.l
@@ -27,9 +27,9 @@ extern int my_yyinput(void *extra, char *buf, int max);
 
 field		\\?[a-z_][a-z0-9_\.]*
 qstring		\"[^\"]*\"
+
 hexnum		0x[0-9a-f]+
 number		[0-9a-f]+
-
 %%
 
 select { HANDLE_COLUMN; return SELECT; }
@@ -38,6 +38,7 @@ from { HANDLE_COLUMN; return FROM; }
 join { HANDLE_COLUMN; return JOIN; }
 on { HANDLE_COLUMN; return ON; }
 where { HANDLE_COLUMN; return WHERE; }
+cast { HANDLE_COLUMN; return CAST; }
 
 {qstring} {
 	HANDLE_COLUMN;
diff --git a/src/sqlhist.y b/src/sqlhist.y
index d5cbecc7bf92..49dac8370359 100644
--- a/src/sqlhist.y
+++ b/src/sqlhist.y
@@ -50,8 +50,8 @@ extern void yyerror(struct sqlhist_bison *, char *fmt, ...);
 	void	*expr;
 }
 
-%token AS SELECT FROM JOIN ON WHERE PARSE_ERROR
-%token <number> NUMBER
+%token AS SELECT FROM JOIN ON WHERE PARSE_ERROR CAST
+%token <number> NUMBER field_type
 %token <string> STRING
 %token <string> FIELD
 %token <string> LE GE EQ NEQ AND OR
@@ -107,6 +107,10 @@ selection_expr :
  | '(' field ')'		{  $$ = $2; }
  | selection_addition
  | '(' selection_addition ')'	{  $$ = $2; }
+ | CAST '(' field AS FIELD ')'	{
+					 $$ = add_cast(sb, $3, $5);
+					 CHECK_RETURN_PTR($$);
+				}
  ;
 
 selection_addition :
diff --git a/src/tracefs-hist.c b/src/tracefs-hist.c
index 301abc255d5f..344e2525c823 100644
--- a/src/tracefs-hist.c
+++ b/src/tracefs-hist.c
@@ -554,6 +554,7 @@ struct tracefs_synth {
 	unsigned int		start_state;
 	unsigned int		end_parens;
 	unsigned int		end_state;
+	int			*start_type;
 	int			arg_cnt;
 };
 
@@ -1056,28 +1057,16 @@ int tracefs_synth_add_compare_field(struct tracefs_synth *synth,
 	return ret ? -1 : 0;
 }
 
-/**
- * tracefs_synth_add_start_field - add a start field to save
- * @synth: The tracefs_synth descriptor
- * @start_field: The field of the start event to save
- * @name: The name to show in the synthetic event (if NULL @start_field is used)
- *
- * This adds a field named by @start_field of the start event to
- * record in the synthetic event.
- *
- * Returns 0 on succes and -1 on error.
- * On error, errno is set to:
- * ENOMEM - memory allocation failure.
- * ENIVAL - a parameter is passed as NULL that should not be
- * ENODEV - could not find a field
- */
-int tracefs_synth_add_start_field(struct tracefs_synth *synth,
-				  const char *start_field,
-				  const char *name)
+__hidden int synth_add_start_field(struct tracefs_synth *synth,
+				   const char *start_field,
+				   const char *name,
+				   enum tracefs_hist_key_type type)
 {
 	const struct tep_format_field *field;
 	char *start_arg;
 	char **tmp;
+	int *types;
+	int len;
 	int ret;
 
 	if (!synth || !start_field) {
@@ -1108,15 +1097,53 @@ int tracefs_synth_add_start_field(struct tracefs_synth *synth,
 		goto out_free;
 
 	tmp = tracefs_list_add(synth->start_selection, start_field);
-	if (tmp)
-		synth->start_selection = tmp;
-	else
+	if (!tmp) {
 		ret = -1;
+		goto out_free;
+	}
+	synth->start_selection = tmp;
+
+	len = tracefs_list_size(tmp);
+	if (len <= 0) { /* ?? */
+		ret = -1;
+		goto out_free;
+	}
+
+	types = realloc(synth->start_type, sizeof(*types) * len);
+	if (!types) {
+		ret = -1;
+		goto out_free;
+	}
+	synth->start_type = types;
+	synth->start_type[len - 1] = type;
+
  out_free:
 	free(start_arg);
 	return ret;
 }
 
+/**
+ * tracefs_synth_add_start_field - add a start field to save
+ * @synth: The tracefs_synth descriptor
+ * @start_field: The field of the start event to save
+ * @name: The name to show in the synthetic event (if NULL @start_field is used)
+ *
+ * This adds a field named by @start_field of the start event to
+ * record in the synthetic event.
+ *
+ * Returns 0 on succes and -1 on error.
+ * On error, errno is set to:
+ * ENOMEM - memory allocation failure.
+ * ENIVAL - a parameter is passed as NULL that should not be
+ * ENODEV - could not find a field
+ */
+int tracefs_synth_add_start_field(struct tracefs_synth *synth,
+				  const char *start_field,
+				  const char *name)
+{
+	return synth_add_start_field(synth, start_field, name, 0);
+}
+
 /**
  * tracefs_synth_add_end_field - add a end field to save
  * @synth: The tracefs_synth descriptor
@@ -1399,6 +1426,7 @@ tracefs_synth_get_start_hist(struct tracefs_synth *synth)
 	const char *event;
 	const char *key;
 	char **keys;
+	int *types;
 	int ret;
 	int i;
 
@@ -1409,6 +1437,7 @@ tracefs_synth_get_start_hist(struct tracefs_synth *synth)
 
 	system = synth->start_event->system;
 	event = synth->start_event->name;
+	types = synth->start_type;
 	keys = synth->start_keys;
 	tep = synth->tep;
 
@@ -1419,23 +1448,25 @@ tracefs_synth_get_start_hist(struct tracefs_synth *synth)
 		return NULL;
 
 	for (i = 0; keys[i]; i++) {
+		int type = types ? types[i] : 0;
+
 		key = keys[i];
 
 		if (i) {
-			ret = tracefs_hist_add_key(hist, key, 0);
+			ret = tracefs_hist_add_key(hist, key, type);
 			if (ret < 0) {
 				tracefs_hist_free(hist);
 				return NULL;
 			}
 		} else {
 			hist = tracefs_hist_alloc(tep, system, event,
-						  key, 0);
+						  key, type);
 			if (!hist)
 				return NULL;
 		}
 	}
 
-	if (synth->start_filter) {
+	if (hist && synth->start_filter) {
 		hist->filter = strdup(synth->start_filter);
 		if (!hist->filter) {
 			tracefs_hist_free(hist);
diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
index d9ebe2eab411..621310400959 100644
--- a/src/tracefs-sqlhist.c
+++ b/src/tracefs-sqlhist.c
@@ -9,6 +9,7 @@
 #include <trace-seq.h>
 #include <stdlib.h>
 #include <stdarg.h>
+#include <ctype.h>
 #include <errno.h>
 
 #include "tracefs.h"
@@ -40,6 +41,7 @@ struct field {
 	const char		*raw;
 	const char		*label;
 	const char		*field;
+	const char		*type;
 };
 
 struct filter {
@@ -233,6 +235,16 @@ __hidden char *store_str(struct sqlhist_bison *sb, const char *str)
 	return *pstr;
 }
 
+__hidden void *add_cast(struct sqlhist_bison *sb,
+			void *data, const char *type)
+{
+	struct expr *expr = data;
+	struct field *field = &expr->field;
+
+	field->type = type;
+	return expr;
+}
+
 __hidden int add_selection(struct sqlhist_bison *sb, void *select,
 			   const char *name)
 {
@@ -1253,6 +1265,93 @@ static void where_no_to_error(struct sqlhist_bison *sb, struct expr *expr,
 		    event, from_event);
 }
 
+static int verify_field_type(struct tep_handle *tep,
+			     struct sqlhist_bison *sb,
+			     struct expr *expr)
+{
+	struct field *field = &expr->field;
+	struct tep_event *event;
+	struct tep_format_field *tfield;
+	char *type;
+	int ret;
+	int i;
+
+	if (!field->type)
+		return 0;
+
+	sb->line_no = expr->line;
+	sb->line_idx = expr->idx;
+
+	event = tep_find_event_by_name(tep, field->system, field->event_name);
+	if (!event) {
+		parse_error(sb, field->raw,
+			    "Event '%s' not found\n",
+			    field->event_name ? : "(null)");
+		return -1;
+	}
+
+	tfield = tep_find_any_field(event, field->field);
+	if (!tfield) {
+		parse_error(sb, field->raw,
+			    "Field '%s' not part of event '%s'\n",
+			    field->field ? : "(null)", field->event);
+		return -1;
+	}
+
+	type = strdup(field->type);
+	if (!type)
+		return -1;
+
+	for (i = 0; type[i]; i++)
+		type[i] = tolower(type[i]);
+
+	if (!strcmp(type, "hex")) {
+		if (tfield->flags & (TEP_FIELD_IS_STRING | TEP_FIELD_IS_ARRAY))
+			goto fail_type;
+		ret = TRACEFS_HIST_KEY_HEX;
+	} else if (!strcmp(type, "sym")) {
+		if (tfield->flags & (TEP_FIELD_IS_STRING | TEP_FIELD_IS_ARRAY))
+			goto fail_type;
+		ret = TRACEFS_HIST_KEY_SYM;
+	} else if (!strcmp(type, "sym-offset")) {
+		if (tfield->flags & (TEP_FIELD_IS_STRING | TEP_FIELD_IS_ARRAY))
+			goto fail_type;
+		ret = TRACEFS_HIST_KEY_SYM_OFFSET;
+	} else if (!strcmp(type, "syscall")) {
+		if (tfield->flags & (TEP_FIELD_IS_STRING | TEP_FIELD_IS_ARRAY))
+			goto fail_type;
+		ret = TRACEFS_HIST_KEY_SYSCALL;
+	} else if (!strcmp(type, "execname") ||
+		   !strcmp(type, "comm")) {
+		ret = TRACEFS_HIST_KEY_EXECNAME;
+		if (strcmp(field->field, "common_pid")) {
+			parse_error(sb, field->raw,
+				    "'%s' is only allowed for common_pid\n",
+				    type);
+			ret = -1;
+		}
+	} else if (!strcmp(type, "log") ||
+		   !strcmp(type, "log2")) {
+		if (tfield->flags & (TEP_FIELD_IS_STRING | TEP_FIELD_IS_ARRAY))
+			goto fail_type;
+		ret = TRACEFS_HIST_KEY_LOG;
+	} else {
+		parse_error(sb, field->raw,
+			    "Cast of '%s' to unknown type '%s'\n",
+			    field->raw, type);
+		ret = -1;
+	}
+	free(type);
+	return ret;
+ fail_type:
+	parse_error(sb, field->raw,
+		    "Field '%s' cast to '%s' but is of type %s\n",
+		    field->field, type, tfield->flags & TEP_FIELD_IS_STRING ?
+		    "string" : "array");
+	free(type);
+	return -1;
+}
+
 static struct tracefs_synth *build_synth(struct tep_handle *tep,
 					 const char *name,
 					 struct sql_table *table)
@@ -1346,8 +1445,13 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 			field = &expr->field;
 			if (field->system == start_system &&
 			    field->event_name == start_event) {
-				ret = tracefs_synth_add_start_field(synth,
-						field->field, field->label);
+				int type;
+				type = verify_field_type(tep, table->sb, expr);
+				if (type < 0)
+					goto free;
+				ret = synth_add_start_field(synth,
+						field->field, field->label,
+						type);
 			} else if (table->to) {
 				ret = tracefs_synth_add_end_field(synth,
 						field->field, field->label);
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [PATCH v3 22/22] libtracefs: Add CAST(x AS _COUNTER_) syntax to create values in histograms
  2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
                   ` (20 preceding siblings ...)
  2021-08-03 17:06 ` [PATCH v3 21/22] libtracefs: Add CAST() syntax to SQL parsing for histogram types Steven Rostedt
@ 2021-08-03 17:06 ` Steven Rostedt
  21 siblings, 0 replies; 23+ messages in thread
From: Steven Rostedt @ 2021-08-03 17:06 UTC (permalink / raw)
  To: linux-trace-devel
  Cc: linux-kernel, Tom Zanussi, Daniel Bristot de Oliveira,
	Masami Hiramatsu, Namhyung Kim, linux-rt-users, Clark Williams,
	Steven Rostedt (VMware)

From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>

Use the CAST() command of SQL to define which items in the select should
be cast as values and not keys. By casting the field as the special value
_COUNTER_, it will turn the selection item into a value.

For example:

   SELECT common_pid, CAST(bytes_req AS _COUNTER_) FROM kmalloc

Will create:

  echo 'hist:keys=common_pid:vals=bytes_req' > events/kmem/kmalloc/trigger

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
 Documentation/libtracefs-sql.txt | 24 ++++++++++++++++++++++++
 include/tracefs-local.h          |  2 ++
 include/tracefs.h                |  3 +++
 src/tracefs-hist.c               | 28 ++++++++++++++++++++++++++--
 src/tracefs-sqlhist.c            | 23 +++++++++++++++++++++++
 5 files changed, 78 insertions(+), 2 deletions(-)

diff --git a/Documentation/libtracefs-sql.txt b/Documentation/libtracefs-sql.txt
index b7d5c1b4f658..91c99d7969c3 100644
--- a/Documentation/libtracefs-sql.txt
+++ b/Documentation/libtracefs-sql.txt
@@ -238,6 +238,30 @@ name of the process.
 
 *LOG* or *LOG2* - bucket the key values in a log 2 values (1, 2, 3-4, 5-8, 9-16, 17-32, ...)
 
+The above fields are not case sensitive, and "LOG2" works as good as "log".
+
+A special CAST to _COUNTER_ or __COUNTER__ will make the field a value and not
+a key. For example:
+
+[source,c]
+--
+  SELECT common_pid, CAST(bytes_req AS _COUNTER_) FROM kmalloc
+--
+
+Which will create
+
+[source,c]
+--
+  echo 'hist:keys=common_pid:vals=bytes_req' > events/kmem/kmalloc/trigger
+
+  cat events/kmem/kmalloc/hist
+
+{ common_pid:       1812 } hitcount:          1  bytes_req:         32
+{ common_pid:       9111 } hitcount:          2  bytes_req:        272
+{ common_pid:       1768 } hitcount:          3  bytes_req:       1112
+{ common_pid:          0 } hitcount:          4  bytes_req:        512
+{ common_pid:      18297 } hitcount:         11  bytes_req:       2004
+--
 
 RETURN VALUE
 ------------
diff --git a/include/tracefs-local.h b/include/tracefs-local.h
index 07d40b2fae4f..684eccffafee 100644
--- a/include/tracefs-local.h
+++ b/include/tracefs-local.h
@@ -88,6 +88,8 @@ int trace_append_filter(char **filter, unsigned int *state,
 struct tracefs_synth *synth_init_from(struct tep_handle *tep,
 				      const char *start_system,
 				      const char *start_event);
+
+#define HIST_COUNTER_TYPE	(TRACEFS_HIST_KEY_MAX + 100)
 int synth_add_start_field(struct tracefs_synth *synth,
 			  const char *start_field,
 			  const char *name,
diff --git a/include/tracefs.h b/include/tracefs.h
index 219adba4b0ce..17020de0108a 100644
--- a/include/tracefs.h
+++ b/include/tracefs.h
@@ -264,6 +264,7 @@ enum tracefs_hist_key_type {
 	TRACEFS_HIST_KEY_EXECNAME,
 	TRACEFS_HIST_KEY_LOG,
 	TRACEFS_HIST_KEY_USECS,
+	TRACEFS_HIST_KEY_MAX
 };
 
 enum tracefs_hist_sort_direction {
@@ -275,6 +276,8 @@ enum tracefs_hist_sort_direction {
 #define TRACEFS_HIST_TIMESTAMP_USECS	"common_timestamp.usecs"
 #define TRACEFS_HIST_CPU		"cpu"
 
+#define TRACEFS_HIST_COUNTER		"__COUNTER__"
+
 #define TRACEFS_HIST_HITCOUNT		"hitcount"
 
 struct tracefs_hist;
diff --git a/src/tracefs-hist.c b/src/tracefs-hist.c
index 344e2525c823..7f9cf3820611 100644
--- a/src/tracefs-hist.c
+++ b/src/tracefs-hist.c
@@ -256,7 +256,7 @@ int tracefs_hist_add_key(struct tracefs_hist *hist, const char *key,
 	bool use_key = false;
 	char *key_type = NULL;
 	char **new_list;
-	int ret;
+	int ret = -1;
 
 	switch (type) {
 	case TRACEFS_HIST_KEY_NORMAL:
@@ -284,6 +284,9 @@ int tracefs_hist_add_key(struct tracefs_hist *hist, const char *key,
 	case TRACEFS_HIST_KEY_USECS:
 		ret = asprintf(&key_type, "%s.usecs", key);
 		break;
+	case TRACEFS_HIST_KEY_MAX:
+		/* error */
+		break;
 	}
 
 	if (ret < 0)
@@ -1450,6 +1453,9 @@ tracefs_synth_get_start_hist(struct tracefs_synth *synth)
 	for (i = 0; keys[i]; i++) {
 		int type = types ? types[i] : 0;
 
+		if (type == HIST_COUNTER_TYPE)
+			continue;
+
 		key = keys[i];
 
 		if (i) {
@@ -1466,7 +1472,25 @@ tracefs_synth_get_start_hist(struct tracefs_synth *synth)
 		}
 	}
 
-	if (hist && synth->start_filter) {
+	if (!hist)
+		return NULL;
+
+	for (i = 0; keys[i]; i++) {
+		int type = types ? types[i] : 0;
+
+		if (type != HIST_COUNTER_TYPE)
+			continue;
+
+		key = keys[i];
+
+		ret = tracefs_hist_add_value(hist, key);
+		if (ret < 0) {
+			tracefs_hist_free(hist);
+			return NULL;
+		}
+	}
+
+	if (synth->start_filter) {
 		hist->filter = strdup(synth->start_filter);
 		if (!hist->filter) {
 			tracefs_hist_free(hist);
diff --git a/src/tracefs-sqlhist.c b/src/tracefs-sqlhist.c
index 621310400959..622467745a94 100644
--- a/src/tracefs-sqlhist.c
+++ b/src/tracefs-sqlhist.c
@@ -1302,6 +1302,17 @@ static int verify_field_type(struct tep_handle *tep,
 	if (!type)
 		return -1;
 
+	if (!strcmp(type, TRACEFS_HIST_COUNTER) ||
+	    !strcmp(type, "_COUNTER_")) {
+		ret = HIST_COUNTER_TYPE;
+		if (tfield->flags & (TEP_FIELD_IS_STRING | TEP_FIELD_IS_ARRAY)) {
+			parse_error(sb, field->raw,
+				    "'%s' is a string, and counters may only be used with numbers\n");
+			ret = -1;
+		}
+		goto out;
+	}
+
 	for (i = 0; type[i]; i++)
 		type[i] = tolower(type[i]);
 
@@ -1341,6 +1352,7 @@ static int verify_field_type(struct tep_handle *tep,
 			    field->raw, type);
 		ret = -1;
 	}
+ out:
 	free(type);
 	return ret;
  fail_type:
@@ -1368,6 +1380,7 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 	const char *end_match;
 	bool started_start = false;
 	bool started_end = false;
+	bool non_val = false;
 	int ret;
 
 	if (!table->from)
@@ -1449,6 +1462,8 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 				type = verify_field_type(tep, table->sb, expr);
 				if (type < 0)
 					goto free;
+				if (type != HIST_COUNTER_TYPE)
+					non_val = true;
 				ret = synth_add_start_field(synth,
 						field->field, field->label,
 						type);
@@ -1479,6 +1494,14 @@ static struct tracefs_synth *build_synth(struct tep_handle *tep,
 		}
 	}
 
+	if (!non_val && !table->to) {
+		table->sb->line_no = 0;
+		table->sb->line_idx = 10;
+		parse_error(table->sb, "CAST",
+			    "Not all SELECT items can be of type _COUNTER_\n");
+		goto free;
+	}
+
 	for (expr = table->where; expr; expr = expr->next) {
 		const char *filter_system = NULL;
 		const char *filter_event = NULL;
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 23+ messages in thread

end of thread, other threads:[~2021-08-03 17:07 UTC | newest]

Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-03 17:05 [PATCH v3 00/22] libtracefs: Introducing tracefs_sql() to create synthetice events with an SQL line Steven Rostedt
2021-08-03 17:05 ` [PATCH v3 01/22] libtracefs: Added new API tracefs_sql() Steven Rostedt
2021-08-03 17:05 ` [PATCH v3 02/22] tracefs: Add unit tests for tracefs_sql() Steven Rostedt
2021-08-03 17:05 ` [PATCH v3 03/22] libtracefs: Add comparing start and end fields in tracefs_sql() Steven Rostedt
2021-08-03 17:05 ` [PATCH v3 04/22] libtracefs: Add unit test to test tracefs_sql() compare Steven Rostedt
2021-08-03 17:05 ` [PATCH v3 05/22] libtracefs: Add filtering for start and end events in tracefs_sql() Steven Rostedt
2021-08-03 17:05 ` [PATCH v3 06/22] libtracefs: Add unit test to test tracefs_sql() where clause Steven Rostedt
2021-08-03 17:05 ` [PATCH v3 07/22] libtracefs: Make sqlhist parser reentrant Steven Rostedt
2021-08-03 17:05 ` [PATCH v3 08/22] libtracefs: Make parser unique to libtracefs Steven Rostedt
2021-08-03 17:05 ` [PATCH v3 09/22] libtracefs: Add line number and index to expr structure Steven Rostedt
2021-08-03 17:05 ` [PATCH v3 10/22] libtracefs: Add error message when match fields are not FROM and JOIN events Steven Rostedt
2021-08-03 17:05 ` [PATCH v3 11/22] libtracefs: Add error message when match or init fails from bad events Steven Rostedt
2021-08-03 17:05 ` [PATCH v3 12/22] libtracefs; Add error message for bad selections to SQL sequence Steven Rostedt
2021-08-03 17:05 ` [PATCH v3 13/22] libtracefs: Add error message when compare fields fail Steven Rostedt
2021-08-03 17:05 ` [PATCH v3 14/22] libtracefs: Add error message for grouping events in SQL filter Steven Rostedt
2021-08-03 17:05 ` [PATCH v3 15/22] libtracefs: Add error message for bad filters in SQL statement Steven Rostedt
2021-08-03 17:06 ` [PATCH v3 16/22] libtracefs: Add error message when calculation has no label Steven Rostedt
2021-08-03 17:06 ` [PATCH v3 17/22] libtracefs: Add man page for tracefs_sql() Steven Rostedt
2021-08-03 17:06 ` [PATCH v3 18/22] libtracefs: Add Makefile rule to create sqlhist Steven Rostedt
2021-08-03 17:06 ` [PATCH v3 19/22] libtracefs: Allow for simple SQL statements to create a histogram Steven Rostedt
2021-08-03 17:06 ` [PATCH v3 20/22] libtracefs: Allow trace_sql() to take keywords for fields with backslash Steven Rostedt
2021-08-03 17:06 ` [PATCH v3 21/22] libtracefs: Add CAST() syntax to SQL parsing for histogram types Steven Rostedt
2021-08-03 17:06 ` [PATCH v3 22/22] libtracefs: Add CAST(x AS _COUNTER_) syntax to create values in histograms Steven Rostedt

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).