[PATCH 0/1] tools: perf: Expose sample ID / stream ID to python scripts

[PATCH 0/1] tools: perf: Expose sample ID / stream ID to python scripts

[Ben Gainey]

This patch modifies the perf python scripting engine so that the ID and
STREAM_ID are exposed as part of the sample so that they may be
correlated to the corresponding throttle/unthrottle event (for example).

NB: For scripts where perf_db_export_mode = True, this may be a breaking
change depending on how the script is constructed. Each field is passed
to `sample_table` as an argument so any script that is written as:

    def sample_table(db_id, evsel_id, machine_id, ..., cyc_cnt, flags)

will now fail due to the changed number of arguments with:

    TypeError: sample_table() takes 25 positional arguments but 27 were given

Scripts that use:

    def sample_table(*args)

or some variation thereof will not be affected.

When `perf_db_export_mode = False`, the script should be unaffected as
all the arguments are inserted into a dictionary.

The export-to-xxx.py scripts use the (..., *x) form so are not affected.


Ben Gainey (1):
  tools: perf: Expose sample ID / stream ID to python scripts

 tools/perf/Documentation/perf-script-python.txt        | 4 ++--
 tools/perf/util/scripting-engines/trace-event-python.c | 8 +++++++-
 2 files changed, 9 insertions(+), 3 deletions(-)

-- 
2.43.0

[PATCH 1/1] tools: perf: Expose sample ID / stream ID to python scripts

[Ben Gainey]

perf script exposes the evsel_name to python scripts as part of the data
passed to the sample or tracepoint handler function, and it passes the id and
stream_id to the throttled/unthrottled handler functions. This makes matching
throttle events and samples difficult.

To make this possible, this change exposes the sample id and stream_id values
to the script.

Signed-off-by: Ben Gainey <ben.gainey@arm.com>
---
 tools/perf/Documentation/perf-script-python.txt        | 4 ++--
 tools/perf/util/scripting-engines/trace-event-python.c | 8 +++++++-
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/tools/perf/Documentation/perf-script-python.txt b/tools/perf/Documentation/perf-script-python.txt
index 6a8581012e162..13e37e9385ee4 100644
--- a/tools/perf/Documentation/perf-script-python.txt
+++ b/tools/perf/Documentation/perf-script-python.txt
@@ -642,8 +642,8 @@ SUPPORTED FIELDS
 
 Currently supported fields:
 
-ev_name, comm, pid, tid, cpu, ip, time, period, phys_addr, addr,
-symbol, symoff, dso, time_enabled, time_running, values, callchain,
+ev_name, comm, id, stream_id, pid, tid, cpu, ip, time, period, phys_addr,
+addr, symbol, symoff, dso, time_enabled, time_running, values, callchain,
 brstack, brstacksym, datasrc, datasrc_decode, iregs, uregs,
 weight, transaction, raw_buf, attr, cpumode.
 
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 860e1837ba969..d88966645b2f4 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -858,6 +858,10 @@ static PyObject *get_perf_sample_dict(struct perf_sample *sample,
 	pydict_set_item_string_decref(dict, "ev_name", _PyUnicode_FromString(evsel__name(evsel)));
 	pydict_set_item_string_decref(dict, "attr", _PyBytes_FromStringAndSize((const char *)&evsel->core.attr, sizeof(evsel->core.attr)));
 
+	pydict_set_item_string_decref(dict_sample, "id",
+			PyLong_FromUnsignedLongLong(sample->id));
+	pydict_set_item_string_decref(dict_sample, "stream_id",
+			PyLong_FromUnsignedLongLong(sample->stream_id));
 	pydict_set_item_string_decref(dict_sample, "pid",
 			_PyLong_FromLong(sample->pid));
 	pydict_set_item_string_decref(dict_sample, "tid",
@@ -1306,7 +1310,7 @@ static void python_export_sample_table(struct db_export *dbe,
 	struct tables *tables = container_of(dbe, struct tables, dbe);
 	PyObject *t;
 
-	t = tuple_new(25);
+	t = tuple_new(27);
 
 	tuple_set_d64(t, 0, es->db_id);
 	tuple_set_d64(t, 1, es->evsel->db_id);
@@ -1333,6 +1337,8 @@ static void python_export_sample_table(struct db_export *dbe,
 	tuple_set_d64(t, 22, es->sample->insn_cnt);
 	tuple_set_d64(t, 23, es->sample->cyc_cnt);
 	tuple_set_s32(t, 24, es->sample->flags);
+	tuple_set_d64(t, 25, es->sample->id);
+	tuple_set_d64(t, 26, es->sample->stream_id);
 
 	call_object(tables->sample_handler, t, "sample_table");
 
-- 
2.43.0

Re: [PATCH 1/1] tools: perf: Expose sample ID / stream ID to python scripts

[Adrian Hunter]

On 23/01/24 12:31, Ben Gainey wrote:
> perf script exposes the evsel_name to python scripts as part of the data
> passed to the sample or tracepoint handler function, and it passes the id and
> stream_id to the throttled/unthrottled handler functions. This makes matching
> throttle events and samples difficult.
> 
> To make this possible, this change exposes the sample id and stream_id values
> to the script.
> 
> Signed-off-by: Ben Gainey <ben.gainey@arm.com>
> ---
>  tools/perf/Documentation/perf-script-python.txt        | 4 ++--
>  tools/perf/util/scripting-engines/trace-event-python.c | 8 +++++++-
>  2 files changed, 9 insertions(+), 3 deletions(-)
> 
> diff --git a/tools/perf/Documentation/perf-script-python.txt b/tools/perf/Documentation/perf-script-python.txt
> index 6a8581012e162..13e37e9385ee4 100644
> --- a/tools/perf/Documentation/perf-script-python.txt
> +++ b/tools/perf/Documentation/perf-script-python.txt
> @@ -642,8 +642,8 @@ SUPPORTED FIELDS
>  
>  Currently supported fields:
>  
> -ev_name, comm, pid, tid, cpu, ip, time, period, phys_addr, addr,
> -symbol, symoff, dso, time_enabled, time_running, values, callchain,
> +ev_name, comm, id, stream_id, pid, tid, cpu, ip, time, period, phys_addr,
> +addr, symbol, symoff, dso, time_enabled, time_running, values, callchain,
>  brstack, brstacksym, datasrc, datasrc_decode, iregs, uregs,
>  weight, transaction, raw_buf, attr, cpumode.
>  
> diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
> index 860e1837ba969..d88966645b2f4 100644
> --- a/tools/perf/util/scripting-engines/trace-event-python.c
> +++ b/tools/perf/util/scripting-engines/trace-event-python.c
> @@ -858,6 +858,10 @@ static PyObject *get_perf_sample_dict(struct perf_sample *sample,
>  	pydict_set_item_string_decref(dict, "ev_name", _PyUnicode_FromString(evsel__name(evsel)));
>  	pydict_set_item_string_decref(dict, "attr", _PyBytes_FromStringAndSize((const char *)&evsel->core.attr, sizeof(evsel->core.attr)));
>  
> +	pydict_set_item_string_decref(dict_sample, "id",
> +			PyLong_FromUnsignedLongLong(sample->id));
> +	pydict_set_item_string_decref(dict_sample, "stream_id",
> +			PyLong_FromUnsignedLongLong(sample->stream_id));
>  	pydict_set_item_string_decref(dict_sample, "pid",
>  			_PyLong_FromLong(sample->pid));
>  	pydict_set_item_string_decref(dict_sample, "tid",
> @@ -1306,7 +1310,7 @@ static void python_export_sample_table(struct db_export *dbe,
>  	struct tables *tables = container_of(dbe, struct tables, dbe);
>  	PyObject *t;
>  
> -	t = tuple_new(25);
> +	t = tuple_new(27);
>  
>  	tuple_set_d64(t, 0, es->db_id);
>  	tuple_set_d64(t, 1, es->evsel->db_id);
> @@ -1333,6 +1337,8 @@ static void python_export_sample_table(struct db_export *dbe,
>  	tuple_set_d64(t, 22, es->sample->insn_cnt);
>  	tuple_set_d64(t, 23, es->sample->cyc_cnt);
>  	tuple_set_s32(t, 24, es->sample->flags);
> +	tuple_set_d64(t, 25, es->sample->id);
> +	tuple_set_d64(t, 26, es->sample->stream_id);

Unless you actually plan to use the db_export interface with
these, there is no need to add them at this time.

>  
>  	call_object(tables->sample_handler, t, "sample_table");
>  

Re: [PATCH 1/1] tools: perf: Expose sample ID / stream ID to python scripts

[Ben Gainey]

On Tue, 2024-01-23 at 13:04 +0200, Adrian Hunter wrote:
> On 23/01/24 12:31, Ben Gainey wrote:
> > perf script exposes the evsel_name to python scripts as part of the
> > data
> > passed to the sample or tracepoint handler function, and it passes
> > the id and
> > stream_id to the throttled/unthrottled handler functions. This
> > makes matching
> > throttle events and samples difficult.
> > 
> > To make this possible, this change exposes the sample id and
> > stream_id values
> > to the script.
> > 
> > Signed-off-by: Ben Gainey <ben.gainey@arm.com>
> > ---
> >  tools/perf/Documentation/perf-script-python.txt        | 4 ++--
> >  tools/perf/util/scripting-engines/trace-event-python.c | 8
> > +++++++-
> >  2 files changed, 9 insertions(+), 3 deletions(-)
> > 
> > diff --git a/tools/perf/Documentation/perf-script-python.txt
> > b/tools/perf/Documentation/perf-script-python.txt
> > index 6a8581012e162..13e37e9385ee4 100644
> > --- a/tools/perf/Documentation/perf-script-python.txt
> > +++ b/tools/perf/Documentation/perf-script-python.txt
> > @@ -642,8 +642,8 @@ SUPPORTED FIELDS
> >  
> >  Currently supported fields:
> >  
> > -ev_name, comm, pid, tid, cpu, ip, time, period, phys_addr, addr,
> > -symbol, symoff, dso, time_enabled, time_running, values,
> > callchain,
> > +ev_name, comm, id, stream_id, pid, tid, cpu, ip, time, period,
> > phys_addr,
> > +addr, symbol, symoff, dso, time_enabled, time_running, values,
> > callchain,
> >  brstack, brstacksym, datasrc, datasrc_decode, iregs, uregs,
> >  weight, transaction, raw_buf, attr, cpumode.
> >  
> > diff --git a/tools/perf/util/scripting-engines/trace-event-python.c
> > b/tools/perf/util/scripting-engines/trace-event-python.c
> > index 860e1837ba969..d88966645b2f4 100644
> > --- a/tools/perf/util/scripting-engines/trace-event-python.c
> > +++ b/tools/perf/util/scripting-engines/trace-event-python.c
> > @@ -858,6 +858,10 @@ static PyObject *get_perf_sample_dict(struct
> > perf_sample *sample,
> >         pydict_set_item_string_decref(dict, "ev_name",
> > _PyUnicode_FromString(evsel__name(evsel)));
> >         pydict_set_item_string_decref(dict, "attr",
> > _PyBytes_FromStringAndSize((const char *)&evsel->core.attr,
> > sizeof(evsel->core.attr)));
> >  
> > +       pydict_set_item_string_decref(dict_sample, "id",
> > +                       PyLong_FromUnsignedLongLong(sample->id));
> > +       pydict_set_item_string_decref(dict_sample, "stream_id",
> > +                       PyLong_FromUnsignedLongLong(sample-
> > >stream_id));
> >         pydict_set_item_string_decref(dict_sample, "pid",
> >                         _PyLong_FromLong(sample->pid));
> >         pydict_set_item_string_decref(dict_sample, "tid",
> > @@ -1306,7 +1310,7 @@ static void python_export_sample_table(struct
> > db_export *dbe,
> >         struct tables *tables = container_of(dbe, struct tables,
> > dbe);
> >         PyObject *t;
> >  
> > -       t = tuple_new(25);
> > +       t = tuple_new(27);
> >  
> >         tuple_set_d64(t, 0, es->db_id);
> >         tuple_set_d64(t, 1, es->evsel->db_id);
> > @@ -1333,6 +1337,8 @@ static void python_export_sample_table(struct
> > db_export *dbe,
> >         tuple_set_d64(t, 22, es->sample->insn_cnt);
> >         tuple_set_d64(t, 23, es->sample->cyc_cnt);
> >         tuple_set_s32(t, 24, es->sample->flags);
> > +       tuple_set_d64(t, 25, es->sample->id);
> > +       tuple_set_d64(t, 26, es->sample->stream_id);
> 
> Unless you actually plan to use the db_export interface with
> these, there is no need to add them at this time.
> 

For the particular script I was using the db export interface.


> >  
> >         call_object(tables->sample_handler, t, "sample_table");
> >  
> 

Re: [PATCH 0/1] tools: perf: Expose sample ID / stream ID to python scripts

[Adrian Hunter]

On 23/01/24 12:31, Ben Gainey wrote:
> This patch modifies the perf python scripting engine so that the ID and
> STREAM_ID are exposed as part of the sample so that they may be
> correlated to the corresponding throttle/unthrottle event (for example).
> 
> NB: For scripts where perf_db_export_mode = True, this may be a breaking
> change depending on how the script is constructed. Each field is passed
> to `sample_table` as an argument so any script that is written as:
> 
>     def sample_table(db_id, evsel_id, machine_id, ..., cyc_cnt, flags)
> 
> will now fail due to the changed number of arguments with:
> 
>     TypeError: sample_table() takes 25 positional arguments but 27 were given
> 
> Scripts that use:
> 
>     def sample_table(*args)
> 
> or some variation thereof will not be affected.

That is documented in tools/perf/Documentation/db-export.txt.  It was
anticipated that more arguments could be added, so that should be OK.

> 
> When `perf_db_export_mode = False`, the script should be unaffected as
> all the arguments are inserted into a dictionary.
> 
> The export-to-xxx.py scripts use the (..., *x) form so are not affected.
> 
> 
> Ben Gainey (1):
>   tools: perf: Expose sample ID / stream ID to python scripts
> 
>  tools/perf/Documentation/perf-script-python.txt        | 4 ++--
>  tools/perf/util/scripting-engines/trace-event-python.c | 8 +++++++-
>  2 files changed, 9 insertions(+), 3 deletions(-)
> 

Re: [PATCH 1/1] tools: perf: Expose sample ID / stream ID to python scripts

[Adrian Hunter]

On 23/01/24 12:31, Ben Gainey wrote:
> perf script exposes the evsel_name to python scripts as part of the data
> passed to the sample or tracepoint handler function, and it passes the id and
> stream_id to the throttled/unthrottled handler functions. This makes matching
> throttle events and samples difficult.
> 
> To make this possible, this change exposes the sample id and stream_id values
> to the script.
> 
> Signed-off-by: Ben Gainey <ben.gainey@arm.com>

Reviewed-by: Adrian Hunter <adrian.hunter@intel.com>

> ---
>  tools/perf/Documentation/perf-script-python.txt        | 4 ++--
>  tools/perf/util/scripting-engines/trace-event-python.c | 8 +++++++-
>  2 files changed, 9 insertions(+), 3 deletions(-)
> 
> diff --git a/tools/perf/Documentation/perf-script-python.txt b/tools/perf/Documentation/perf-script-python.txt
> index 6a8581012e162..13e37e9385ee4 100644
> --- a/tools/perf/Documentation/perf-script-python.txt
> +++ b/tools/perf/Documentation/perf-script-python.txt
> @@ -642,8 +642,8 @@ SUPPORTED FIELDS
>  
>  Currently supported fields:
>  
> -ev_name, comm, pid, tid, cpu, ip, time, period, phys_addr, addr,
> -symbol, symoff, dso, time_enabled, time_running, values, callchain,
> +ev_name, comm, id, stream_id, pid, tid, cpu, ip, time, period, phys_addr,
> +addr, symbol, symoff, dso, time_enabled, time_running, values, callchain,
>  brstack, brstacksym, datasrc, datasrc_decode, iregs, uregs,
>  weight, transaction, raw_buf, attr, cpumode.
>  
> diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
> index 860e1837ba969..d88966645b2f4 100644
> --- a/tools/perf/util/scripting-engines/trace-event-python.c
> +++ b/tools/perf/util/scripting-engines/trace-event-python.c
> @@ -858,6 +858,10 @@ static PyObject *get_perf_sample_dict(struct perf_sample *sample,
>  	pydict_set_item_string_decref(dict, "ev_name", _PyUnicode_FromString(evsel__name(evsel)));
>  	pydict_set_item_string_decref(dict, "attr", _PyBytes_FromStringAndSize((const char *)&evsel->core.attr, sizeof(evsel->core.attr)));
>  
> +	pydict_set_item_string_decref(dict_sample, "id",
> +			PyLong_FromUnsignedLongLong(sample->id));
> +	pydict_set_item_string_decref(dict_sample, "stream_id",
> +			PyLong_FromUnsignedLongLong(sample->stream_id));
>  	pydict_set_item_string_decref(dict_sample, "pid",
>  			_PyLong_FromLong(sample->pid));
>  	pydict_set_item_string_decref(dict_sample, "tid",
> @@ -1306,7 +1310,7 @@ static void python_export_sample_table(struct db_export *dbe,
>  	struct tables *tables = container_of(dbe, struct tables, dbe);
>  	PyObject *t;
>  
> -	t = tuple_new(25);
> +	t = tuple_new(27);
>  
>  	tuple_set_d64(t, 0, es->db_id);
>  	tuple_set_d64(t, 1, es->evsel->db_id);
> @@ -1333,6 +1337,8 @@ static void python_export_sample_table(struct db_export *dbe,
>  	tuple_set_d64(t, 22, es->sample->insn_cnt);
>  	tuple_set_d64(t, 23, es->sample->cyc_cnt);
>  	tuple_set_s32(t, 24, es->sample->flags);
> +	tuple_set_d64(t, 25, es->sample->id);
> +	tuple_set_d64(t, 26, es->sample->stream_id);
>  
>  	call_object(tables->sample_handler, t, "sample_table");
>  

Re: [PATCH 0/1] tools: perf: Expose sample ID / stream ID to python scripts

[Namhyung Kim]

Hello,

On Tue, Jan 23, 2024 at 2:32 AM Ben Gainey <ben.gainey@arm.com> wrote:
>
> This patch modifies the perf python scripting engine so that the ID and
> STREAM_ID are exposed as part of the sample so that they may be
> correlated to the corresponding throttle/unthrottle event (for example).
>
> NB: For scripts where perf_db_export_mode = True, this may be a breaking
> change depending on how the script is constructed. Each field is passed
> to `sample_table` as an argument so any script that is written as:
>
>     def sample_table(db_id, evsel_id, machine_id, ..., cyc_cnt, flags)
>
> will now fail due to the changed number of arguments with:
>
>     TypeError: sample_table() takes 25 positional arguments but 27 were given
>
> Scripts that use:
>
>     def sample_table(*args)
>
> or some variation thereof will not be affected.
>
> When `perf_db_export_mode = False`, the script should be unaffected as
> all the arguments are inserted into a dictionary.
>
> The export-to-xxx.py scripts use the (..., *x) form so are not affected.

As long as all in-tree users are fine, I think it's ok.

Thanks,
Namhyung

>
>
> Ben Gainey (1):
>   tools: perf: Expose sample ID / stream ID to python scripts
>
>  tools/perf/Documentation/perf-script-python.txt        | 4 ++--
>  tools/perf/util/scripting-engines/trace-event-python.c | 8 +++++++-
>  2 files changed, 9 insertions(+), 3 deletions(-)
>
> --
> 2.43.0
>

Re: [PATCH 0/1] tools: perf: Expose sample ID / stream ID to python scripts

[Namhyung Kim]

On Tue, 23 Jan 2024 10:31:36 +0000, Ben Gainey wrote:
> This patch modifies the perf python scripting engine so that the ID and
> STREAM_ID are exposed as part of the sample so that they may be
> correlated to the corresponding throttle/unthrottle event (for example).
> 
> NB: For scripts where perf_db_export_mode = True, this may be a breaking
> change depending on how the script is constructed. Each field is passed
> to `sample_table` as an argument so any script that is written as:
> 
> [...]

Applied to perf-tools-next, thanks!

Best regards,
-- 
Namhyung Kim <namhyung@kernel.org>