perf & rasd integration plan

perf & rasd integration plan

[Jean Pihet]

The RAS Deamon (rasd) as available at [1] and the development version
at [2], allows to enable kernel tracepoints and outputs the
tracepoints fields according to the kernel format.

[1] https://git.kernel.org/cgit/utils/ras/rasd.git/
[2] https://git.linaro.org/people/jean.pihet/rasd.git

rasd currently is a duplicate of the perf tool code, ultimately perf
and rasd will use the same common code. The goal is to factor out the
common code from perf and place it in small librairies in tools/lib.

Here is the code that rasd currently uses and that should be moved to tools/lib:

- debugfs: already in tools/lib/api/fs
  . mount and retrieve path

- evlist: tools/perf/util/evlist.[ch]
  . create and init new evlist,
  . set cpu and thread maps,
  . add events to evlist,
  . init and use internal event id,
  . alloc and mmap events buffers, manage file descriptors,
  . enable events,
  . read events buffers, parse data,
  . unmap and free buffers

- evsel: tools/perf/util/evsel.[ch]
  . create and init new tracepoints events,
  . init and use internal event id,
  . open events, manage fds,
  . close and free events

- trace-event: tools/perf/util/trace-event.[ch] and
  tools/perf/util/trace-event-parse.c
  . retrieve and parse events format (using event-parse),
  . print out events fields

- event-parse: already in tools/lib/traceevent/event-parse.[ch]
  . retrieve and parse events format,
  . parse events format and print out events fields

- trace-seq: already in tools/lib/traceevent/trace-seq.c
  . format output string for event fields

- events plugins: already in tools/lib/traceevent/event-plugin.c

- util: tools/perf/util/util.[ch]
  . files open/read,
  . manage events attributes,
  . various macros

- test events attributes: tools/perf/tests/attr.c
  . test_attr__open()

- thread: tools/perf/util/thread_map.[ch] and
- cpu: tools/perf/util/cpumap.[ch]
  . init and manage process maps

- xyarray: tools/perf/util/xyarray.[ch]

- syscall: tools/perf/perf-sys.h

- cgroup: tools/perf/util/cgroup.[ch]

The plan is to move the small and generic functions first: util,
xyarray, cpumap, thread_map etc; then evlist, evsel, trace-event,
trace-event-parse; and finally integrate rasd into the tools/ dir.

Any thought? Can evlist, evsel etc. be moved at once?

Patches should come soon, when time allows.

Regards,
Jean

Re: perf & rasd integration plan

[Arnaldo Carvalho de Melo]

Em Tue, Sep 30, 2014 at 11:06:21AM +0200, Jean Pihet escreveu:
> The RAS Deamon (rasd) as available at [1] and the development version
> at [2], allows to enable kernel tracepoints and outputs the
> tracepoints fields according to the kernel format.
> 
> [1] https://git.kernel.org/cgit/utils/ras/rasd.git/
> [2] https://git.linaro.org/people/jean.pihet/rasd.git
> 
> rasd currently is a duplicate of the perf tool code, ultimately perf
> and rasd will use the same common code. The goal is to factor out the
> common code from perf and place it in small librairies in tools/lib.
> 
> Here is the code that rasd currently uses and that should be moved to tools/lib:
> 
> - debugfs: already in tools/lib/api/fs
>   . mount and retrieve path
> 
> - evlist: tools/perf/util/evlist.[ch]
>   . create and init new evlist,
>   . set cpu and thread maps,
>   . add events to evlist,
>   . init and use internal event id,
>   . alloc and mmap events buffers, manage file descriptors,
>   . enable events,
>   . read events buffers, parse data,
>   . unmap and free buffers
> 
> - evsel: tools/perf/util/evsel.[ch]
>   . create and init new tracepoints events,
>   . init and use internal event id,
>   . open events, manage fds,
>   . close and free events
> 
> - trace-event: tools/perf/util/trace-event.[ch] and
>   tools/perf/util/trace-event-parse.c
>   . retrieve and parse events format (using event-parse),
>   . print out events fields
> 
> - event-parse: already in tools/lib/traceevent/event-parse.[ch]
>   . retrieve and parse events format,
>   . parse events format and print out events fields
> 
> - trace-seq: already in tools/lib/traceevent/trace-seq.c
>   . format output string for event fields
> 
> - events plugins: already in tools/lib/traceevent/event-plugin.c
> 
> - util: tools/perf/util/util.[ch]
>   . files open/read,
>   . manage events attributes,
>   . various macros
> 
> - test events attributes: tools/perf/tests/attr.c
>   . test_attr__open()
> 
> - thread: tools/perf/util/thread_map.[ch] and
> - cpu: tools/perf/util/cpumap.[ch]
>   . init and manage process maps
> 
> - xyarray: tools/perf/util/xyarray.[ch]
> 
> - syscall: tools/perf/perf-sys.h
> 
> - cgroup: tools/perf/util/cgroup.[ch]
> 
> The plan is to move the small and generic functions first: util,
> xyarray, cpumap, thread_map etc; then evlist, evsel, trace-event,
> trace-event-parse; and finally integrate rasd into the tools/ dir.
> 
> Any thought? Can evlist, evsel etc. be moved at once?
> 
> Patches should come soon, when time allows.

Why don't you add it to tools/rasd/ and in tools/rasd/Makefile you just
go on and add tools/perf/util/evlist.o et all to be linked directly, as
a first step.

Then, as a second step, we can create a tools/lib/perf/evlist.c having
what is currently used by both tools/perf/ and tools/rasd/, i.e. what is
proven to be useful for something other than perf.

As the need arises, we go on moving things into tools/lib/perf/evlist.c
et all from wherever it appeared first, be it from tools/rasd/,
tools/perf/util/evlist.c or anywhere else.

Initial rule being that once it is used by multiple tools living in
tools/, then it deserves a place in tools/lib/perf/.

Ditto for other stuff currently living in tools/perf/util/.

- Arnaldo

Re: perf & rasd integration plan

[Borislav Petkov]

Top-posting on purpose:

Btw, jolsa, if you get your LCE proposal for the perf splitting
approved, please post the time here so people can come.

Thanks.

On Tue, Sep 30, 2014 at 10:24:16AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Tue, Sep 30, 2014 at 11:06:21AM +0200, Jean Pihet escreveu:
> > The RAS Deamon (rasd) as available at [1] and the development version
> > at [2], allows to enable kernel tracepoints and outputs the
> > tracepoints fields according to the kernel format.
> > 
> > [1] https://git.kernel.org/cgit/utils/ras/rasd.git/
> > [2] https://git.linaro.org/people/jean.pihet/rasd.git
> > 
> > rasd currently is a duplicate of the perf tool code, ultimately perf
> > and rasd will use the same common code. The goal is to factor out the
> > common code from perf and place it in small librairies in tools/lib.
> > 
> > Here is the code that rasd currently uses and that should be moved to tools/lib:
> > 
> > - debugfs: already in tools/lib/api/fs
> >   . mount and retrieve path
> > 
> > - evlist: tools/perf/util/evlist.[ch]
> >   . create and init new evlist,
> >   . set cpu and thread maps,
> >   . add events to evlist,
> >   . init and use internal event id,
> >   . alloc and mmap events buffers, manage file descriptors,
> >   . enable events,
> >   . read events buffers, parse data,
> >   . unmap and free buffers
> > 
> > - evsel: tools/perf/util/evsel.[ch]
> >   . create and init new tracepoints events,
> >   . init and use internal event id,
> >   . open events, manage fds,
> >   . close and free events
> > 
> > - trace-event: tools/perf/util/trace-event.[ch] and
> >   tools/perf/util/trace-event-parse.c
> >   . retrieve and parse events format (using event-parse),
> >   . print out events fields
> > 
> > - event-parse: already in tools/lib/traceevent/event-parse.[ch]
> >   . retrieve and parse events format,
> >   . parse events format and print out events fields
> > 
> > - trace-seq: already in tools/lib/traceevent/trace-seq.c
> >   . format output string for event fields
> > 
> > - events plugins: already in tools/lib/traceevent/event-plugin.c
> > 
> > - util: tools/perf/util/util.[ch]
> >   . files open/read,
> >   . manage events attributes,
> >   . various macros
> > 
> > - test events attributes: tools/perf/tests/attr.c
> >   . test_attr__open()
> > 
> > - thread: tools/perf/util/thread_map.[ch] and
> > - cpu: tools/perf/util/cpumap.[ch]
> >   . init and manage process maps
> > 
> > - xyarray: tools/perf/util/xyarray.[ch]
> > 
> > - syscall: tools/perf/perf-sys.h
> > 
> > - cgroup: tools/perf/util/cgroup.[ch]
> > 
> > The plan is to move the small and generic functions first: util,
> > xyarray, cpumap, thread_map etc; then evlist, evsel, trace-event,
> > trace-event-parse; and finally integrate rasd into the tools/ dir.
> > 
> > Any thought? Can evlist, evsel etc. be moved at once?
> > 
> > Patches should come soon, when time allows.
> 
> Why don't you add it to tools/rasd/ and in tools/rasd/Makefile you just
> go on and add tools/perf/util/evlist.o et all to be linked directly, as
> a first step.
> 
> Then, as a second step, we can create a tools/lib/perf/evlist.c having
> what is currently used by both tools/perf/ and tools/rasd/, i.e. what is
> proven to be useful for something other than perf.
> 
> As the need arises, we go on moving things into tools/lib/perf/evlist.c
> et all from wherever it appeared first, be it from tools/rasd/,
> tools/perf/util/evlist.c or anywhere else.
> 
> Initial rule being that once it is used by multiple tools living in
> tools/, then it deserves a place in tools/lib/perf/.
> 
> Ditto for other stuff currently living in tools/perf/util/.
> 
> - Arnaldo
> 

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: perf & rasd integration plan

[Jiri Olsa]

On Sun, Oct 05, 2014 at 07:48:01PM +0200, Borislav Petkov wrote:
> Top-posting on purpose:
> 
> Btw, jolsa, if you get your LCE proposal for the perf splitting
> approved, please post the time here so people can come.

yep, it got accepted, ther schedule is:
  Friday, October 17, 2014 from 9:00am – Noon
  Room 2

https://pdxplumbers.osuosl.org/2014/ocw/events/LPC2014/tracks/351

jirka

Re: perf & rasd integration plan

[Borislav Petkov]

On Sun, Oct 05, 2014 at 08:24:42PM +0200, Jiri Olsa wrote:
> On Sun, Oct 05, 2014 at 07:48:01PM +0200, Borislav Petkov wrote:
> > Top-posting on purpose:
> > 
> > Btw, jolsa, if you get your LCE proposal for the perf splitting
> > approved, please post the time here so people can come.
> 
> yep, it got accepted, ther schedule is:
>   Friday, October 17, 2014 from 9:00am – Noon
>   Room 2
> 
> https://pdxplumbers.osuosl.org/2014/ocw/events/LPC2014/tracks/351

Cool, thanks.

I'll add it to the calendar - lot to do this conf :-)

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: perf & rasd integration plan

[Chuck Ebbert]

On Sun, 5 Oct 2014 20:24:42 +0200
Jiri Olsa <jolsa@redhat.com> wrote:

> On Sun, Oct 05, 2014 at 07:48:01PM +0200, Borislav Petkov wrote:
> > Top-posting on purpose:
> > 
> > Btw, jolsa, if you get your LCE proposal for the perf splitting
> > approved, please post the time here so people can come.
> 
> yep, it got accepted, ther schedule is:
>   Friday, October 17, 2014 from 9:00am – Noon
>   Room 2
> 
> https://pdxplumbers.osuosl.org/2014/ocw/events/LPC2014/tracks/351
> 

The SSL cert for that site is comic.

1. Expired over three years ago.

2. Was issued to a different domain.

3. Issued by an un-verifiable CA.

This is a joke, right?

Re: perf & rasd integration plan

[Jiri Olsa]

On Sun, Oct 05, 2014 at 02:24:01PM -0500, Chuck Ebbert wrote:
> On Sun, 5 Oct 2014 20:24:42 +0200
> Jiri Olsa <jolsa@redhat.com> wrote:
> 
> > On Sun, Oct 05, 2014 at 07:48:01PM +0200, Borislav Petkov wrote:
> > > Top-posting on purpose:
> > > 
> > > Btw, jolsa, if you get your LCE proposal for the perf splitting
> > > approved, please post the time here so people can come.
> > 
> > yep, it got accepted, ther schedule is:
> >   Friday, October 17, 2014 from 9:00am – Noon
> >   Room 2
> > 
> > https://pdxplumbers.osuosl.org/2014/ocw/events/LPC2014/tracks/351
> > 
> 
> The SSL cert for that site is comic.
> 
> 1. Expired over three years ago.
> 
> 2. Was issued to a different domain.
> 
> 3. Issued by an un-verifiable CA.
> 
> This is a joke, right?

;-) it's the link from this page:
http://www.linuxplumbersconf.org/2014/ocw/events/LPC2014/schedule

jirka

Re: perf & rasd integration plan

[Jean Pihet]

Hi Jiri,


On 5 October 2014 20:24, Jiri Olsa <jolsa@redhat.com> wrote:
> On Sun, Oct 05, 2014 at 07:48:01PM +0200, Borislav Petkov wrote:
>> Top-posting on purpose:
>>
>> Btw, jolsa, if you get your LCE proposal for the perf splitting
>> approved, please post the time here so people can come.
>
> yep, it got accepted, ther schedule is:
>   Friday, October 17, 2014 from 9:00am – Noon
>   Room 2
>
> https://pdxplumbers.osuosl.org/2014/ocw/events/LPC2014/tracks/351
Great! Are there slides that we can download?

I have a couple of weeks to work on this, I can prepare the first set
of patches.

Note: I will not be at ELCE/LPC but could attend remotely or have a
call to discuss the points.

Cheers,
Jean

>
> jirka

Re: perf & rasd integration plan

[Robert Richter]

On 30.09.14 10:24:16, Arnaldo Carvalho de Melo wrote:
> Em Tue, Sep 30, 2014 at 11:06:21AM +0200, Jean Pihet escreveu:
> > The plan is to move the small and generic functions first: util,
> > xyarray, cpumap, thread_map etc; then evlist, evsel, trace-event,
> > trace-event-parse; and finally integrate rasd into the tools/ dir.
> > 
> > Any thought? Can evlist, evsel etc. be moved at once?
> > 
> > Patches should come soon, when time allows.
> 
> Why don't you add it to tools/rasd/ and in tools/rasd/Makefile you just
> go on and add tools/perf/util/evlist.o et all to be linked directly, as
> a first step.
> 
> Then, as a second step, we can create a tools/lib/perf/evlist.c having
> what is currently used by both tools/perf/ and tools/rasd/, i.e. what is
> proven to be useful for something other than perf.

It would be good to have tools/lib/perf or so with some base
implementation to setup and connect to perf buffers. This is useful
for tools not only in tools/. The rasd would be a good reference for
this regardless if it is in tools/ or not. I am not sure whether and
when rasd will be moved there.

> As the need arises, we go on moving things into tools/lib/perf/evlist.c
> et all from wherever it appeared first, be it from tools/rasd/,
> tools/perf/util/evlist.c or anywhere else.
> 
> Initial rule being that once it is used by multiple tools living in
> tools/, then it deserves a place in tools/lib/perf/.

So this wouldn't quite work well as it excludes tools not in tools/.

-Robert

> 
> Ditto for other stuff currently living in tools/perf/util/.

Re: perf & rasd integration plan

[Jean Pihet]

Hi,

On 6 October 2014 11:07, Robert Richter <rric@kernel.org> wrote:
> On 30.09.14 10:24:16, Arnaldo Carvalho de Melo wrote:
>> Em Tue, Sep 30, 2014 at 11:06:21AM +0200, Jean Pihet escreveu:
>> > The plan is to move the small and generic functions first: util,
>> > xyarray, cpumap, thread_map etc; then evlist, evsel, trace-event,
>> > trace-event-parse; and finally integrate rasd into the tools/ dir.
>> >
>> > Any thought? Can evlist, evsel etc. be moved at once?
>> >
>> > Patches should come soon, when time allows.
>>
>> Why don't you add it to tools/rasd/ and in tools/rasd/Makefile you just
>> go on and add tools/perf/util/evlist.o et all to be linked directly, as
>> a first step.
>>
>> Then, as a second step, we can create a tools/lib/perf/evlist.c having
>> what is currently used by both tools/perf/ and tools/rasd/, i.e. what is
>> proven to be useful for something other than perf.
>
> It would be good to have tools/lib/perf or so with some base
> implementation to setup and connect to perf buffers. This is useful
> for tools not only in tools/. The rasd would be a good reference for
> this regardless if it is in tools/ or not. I am not sure whether and
> when rasd will be moved there.

Agree. Having a lib (or a set of libs) that external tools can use is
a must. This will encourage tools developers to use this API instead
of re-invening the wheel every time, and to provide standard tools
that can be merged in the kernel source if needed.

>
>> As the need arises, we go on moving things into tools/lib/perf/evlist.c
>> et all from wherever it appeared first, be it from tools/rasd/,
>> tools/perf/util/evlist.c or anywhere else.
>>
>> Initial rule being that once it is used by multiple tools living in
>> tools/, then it deserves a place in tools/lib/perf/.
>
> So this wouldn't quite work well as it excludes tools not in tools/.
Ditto

I am willing to propose a first set of patches to factor out the
common code from perf, but an agreement is needed on the direction to
take.
My plan is to provide patches for the proposed integration plan (as in
the initial e-mail), is it worth doing so or is it purely wasted time
and effort?

What do you think?

Thx,
Jean

>
> -Robert
>
>>
>> Ditto for other stuff currently living in tools/perf/util/.

Re: perf & rasd integration plan

[Arnaldo Carvalho de Melo]

Em Mon, Oct 06, 2014 at 11:07:26AM +0200, Robert Richter escreveu:
> On 30.09.14 10:24:16, Arnaldo Carvalho de Melo wrote:
> > Em Tue, Sep 30, 2014 at 11:06:21AM +0200, Jean Pihet escreveu:
> > > The plan is to move the small and generic functions first: util,
> > > xyarray, cpumap, thread_map etc; then evlist, evsel, trace-event,
> > > trace-event-parse; and finally integrate rasd into the tools/ dir.

> > > Any thought? Can evlist, evsel etc. be moved at once?

> > > Patches should come soon, when time allows.

> > Why don't you add it to tools/rasd/ and in tools/rasd/Makefile you just
> > go on and add tools/perf/util/evlist.o et all to be linked directly, as
> > a first step.

> > Then, as a second step, we can create a tools/lib/perf/evlist.c having
> > what is currently used by both tools/perf/ and tools/rasd/, i.e. what is
> > proven to be useful for something other than perf.
 
> It would be good to have tools/lib/perf or so with some base
> implementation to setup and connect to perf buffers. This is useful
> for tools not only in tools/. The rasd would be a good reference for

Agreed, we can use rasd as a reference, I'm now looking for the git repo
url to see what it is using to prototype how I think this should be
done.

- Arnaldo

> this regardless if it is in tools/ or not. I am not sure whether and
> when rasd will be moved there.

- Arnaldo

Re: perf & rasd integration plan

[Borislav Petkov]

On Mon, Oct 06, 2014 at 11:58:56AM -0300, Arnaldo Carvalho de Melo wrote:
> Agreed, we can use rasd as a reference, I'm now looking for the git repo
> url to see what it is using to prototype how I think this should be
> done.

https://git.kernel.org/cgit/utils/ras/rasd.git

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: perf & rasd integration plan

[Jean Pihet]

Hi Arnaldo,

On 6 October 2014 16:58, Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> Em Mon, Oct 06, 2014 at 11:07:26AM +0200, Robert Richter escreveu:
>> On 30.09.14 10:24:16, Arnaldo Carvalho de Melo wrote:
>> > Em Tue, Sep 30, 2014 at 11:06:21AM +0200, Jean Pihet escreveu:
>> > > The plan is to move the small and generic functions first: util,
>> > > xyarray, cpumap, thread_map etc; then evlist, evsel, trace-event,
>> > > trace-event-parse; and finally integrate rasd into the tools/ dir.
>
>> > > Any thought? Can evlist, evsel etc. be moved at once?
>
>> > > Patches should come soon, when time allows.
>
>> > Why don't you add it to tools/rasd/ and in tools/rasd/Makefile you just
>> > go on and add tools/perf/util/evlist.o et all to be linked directly, as
>> > a first step.
>
>> > Then, as a second step, we can create a tools/lib/perf/evlist.c having
>> > what is currently used by both tools/perf/ and tools/rasd/, i.e. what is
>> > proven to be useful for something other than perf.
>
>> It would be good to have tools/lib/perf or so with some base
>> implementation to setup and connect to perf buffers. This is useful
>> for tools not only in tools/. The rasd would be a good reference for
>
> Agreed, we can use rasd as a reference, I'm now looking for the git repo
> url to see what it is using to prototype how I think this should be
> done.
The latest patches have been submitted to linux-edac [1] and also are
available in the dev git tree [2].
[1] http://marc.info/?l=linux-edac&m=141234802311271&w=2
[2] https://git.linaro.org/people/jean.pihet/rasd.git

Thx,
Jean

>
> - Arnaldo
>
>> this regardless if it is in tools/ or not. I am not sure whether and
>> when rasd will be moved there.
>
> - Arnaldo

Re: perf & rasd integration plan

[Arnaldo Carvalho de Melo]

Em Tue, Sep 30, 2014 at 11:06:21AM +0200, Jean Pihet escreveu:
> The RAS Deamon (rasd) as available at [1] and the development version
> at [2], allows to enable kernel tracepoints and outputs the
> tracepoints fields according to the kernel format.
> 
> [1] https://git.kernel.org/cgit/utils/ras/rasd.git/
> [2] https://git.linaro.org/people/jean.pihet/rasd.git
> 
> rasd currently is a duplicate of the perf tool code, ultimately perf
> and rasd will use the same common code. The goal is to factor out the
> common code from perf and place it in small librairies in tools/lib.
> 
> Here is the code that rasd currently uses and that should be moved to tools/lib:
> 
> - debugfs: already in tools/lib/api/fs
>   . mount and retrieve path
> 
> - evlist: tools/perf/util/evlist.[ch]
>   . create and init new evlist,
>   . set cpu and thread maps,
>   . add events to evlist,
>   . init and use internal event id,
>   . alloc and mmap events buffers, manage file descriptors,
>   . enable events,
>   . read events buffers, parse data,
>   . unmap and free buffers
> 
> - evsel: tools/perf/util/evsel.[ch]
>   . create and init new tracepoints events,
>   . init and use internal event id,
>   . open events, manage fds,
>   . close and free events

So, why have you commented out the perf_missing_features fallbacks? Are
they getting in the way somehow, what can we do upstream, i.e. in
tools/perf/util/ so that it gets closer to what you want to have?

Ditto for all the other ifdef'ed code, can you elaborate on why each is
needed? That will help in moving what is used by rasd from
tools/perf/util/ to tools/perf/lib/ so that we can eliminate the
duplicity.

- Arnaldo

Re: perf & rasd integration plan

[Arnaldo Carvalho de Melo]

Em Mon, Oct 06, 2014 at 05:01:47PM +0200, Borislav Petkov escreveu:
> On Mon, Oct 06, 2014 at 11:58:56AM -0300, Arnaldo Carvalho de Melo wrote:
> > Agreed, we can use rasd as a reference, I'm now looking for the git repo
> > url to see what it is using to prototype how I think this should be
> > done.
> 
> https://git.kernel.org/cgit/utils/ras/rasd.git

I got Jean's "development" repo, should I get the g.k.o one?

- Arnaldo

Re: perf & rasd integration plan

[Borislav Petkov]

On Mon, Oct 06, 2014 at 12:07:41PM -0300, Arnaldo Carvalho de Melo wrote:
> So, why have you commented out the perf_missing_features fallbacks? Are
> they getting in the way somehow, what can we do upstream, i.e. in
> tools/perf/util/ so that it gets closer to what you want to have?
> 
> Ditto for all the other ifdef'ed code, can you elaborate on why each is
> needed? That will help in moving what is used by rasd from
> tools/perf/util/ to tools/perf/lib/ so that we can eliminate the
> duplicity.

Oh, the reason is very simple: so that it builds. We don't need those
facilities yet. We will enable them when needed but we wanted to
concentrate on rasd first and later clean up stuff as we go.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: perf & rasd integration plan

[Borislav Petkov]

On Mon, Oct 06, 2014 at 12:08:38PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Mon, Oct 06, 2014 at 05:01:47PM +0200, Borislav Petkov escreveu:
> > On Mon, Oct 06, 2014 at 11:58:56AM -0300, Arnaldo Carvalho de Melo wrote:
> > > Agreed, we can use rasd as a reference, I'm now looking for the git repo
> > > url to see what it is using to prototype how I think this should be
> > > done.
> > 
> > https://git.kernel.org/cgit/utils/ras/rasd.git
> 
> I got Jean's "development" repo, should I get the g.k.o one?

Right, this one above is the official rasd one and Jean's stuff goes
ontop. So you can look at both. :-)

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: perf & rasd integration plan

[Arnaldo Carvalho de Melo]

Em Mon, Oct 06, 2014 at 05:16:06PM +0200, Borislav Petkov escreveu:
> On Mon, Oct 06, 2014 at 12:07:41PM -0300, Arnaldo Carvalho de Melo wrote:
> > So, why have you commented out the perf_missing_features fallbacks? Are
> > they getting in the way somehow, what can we do upstream, i.e. in
> > tools/perf/util/ so that it gets closer to what you want to have?
> > 
> > Ditto for all the other ifdef'ed code, can you elaborate on why each is
> > needed? That will help in moving what is used by rasd from
> > tools/perf/util/ to tools/perf/lib/ so that we can eliminate the
> > duplicity.
> 
> Oh, the reason is very simple: so that it builds. We don't need those
> facilities yet. We will enable them when needed but we wanted to
> concentrate on rasd first and later clean up stuff as we go.

Right, and it is being a great exercise, thanks for the patience so far
;-)

Looking at those ifdefs we see things that are specific to tools/perf/,
like perf_evsel having a struct hists embedded... I.e. that is of no
interest whatsoever (so far) to rasd, and in turn pulls other object
files.

So I think that right now we need to look at those ifdefs and go on
making what is in tools/perf/util/ stop using it somehow, so that what
then gets moved to tools/lib/api/perf/ (I guess this is where it should
be, opinions?) have that sorted out.

I.e. what goes to tools/lib/api/perf/ is what is common to the needs of
tools/perf/ and rasd (wherever it may roam).

Perhaps something like I did for sock/inet_sock/inet_connection_sock
ages ago... I.e. the tools that want a hists will have a 

struct hists_evsel {
	struct perf_evsel evsel;
	struct hists	  hists;
};

etc.

Experimenting with that, as it is the only thing ifdefed out in rasd's
copy of evsel.h...

- Arnaldo

Re: perf & rasd integration plan

[Borislav Petkov]

On Mon, Oct 06, 2014 at 04:12:27PM -0300, Arnaldo Carvalho de Melo wrote:
> Right, and it is being a great exercise, thanks for the patience so far
> ;-)

I know, this is your secret agenda to keep me at bay working on this! :-P

> Looking at those ifdefs we see things that are specific to tools/perf/,
> like perf_evsel having a struct hists embedded... I.e. that is of no
> interest whatsoever (so far) to rasd, and in turn pulls other object
> files.
> 
> So I think that right now we need to look at those ifdefs and go on
> making what is in tools/perf/util/ stop using it somehow, so that what
> then gets moved to tools/lib/api/perf/ (I guess this is where it should
> be, opinions?) have that sorted out.
> 
> I.e. what goes to tools/lib/api/perf/ is what is common to the needs of
> tools/perf/ and rasd (wherever it may roam).
> 
> Perhaps something like I did for sock/inet_sock/inet_connection_sock
> ages ago... I.e. the tools that want a hists will have a 
> 
> struct hists_evsel {
> 	struct perf_evsel evsel;
> 	struct hists	  hists;
> };
> 
> etc.
> 
> Experimenting with that, as it is the only thing ifdefed out in rasd's
> copy of evsel.h...

Right, this all reads like it is going in the right direction but the
more important question IMO is are we doing a libperf or are we still
doing tools/lib/api/perf/ of single object files to which people can
link?

Because if it is the second - and it sounds to me like it is - how do
you propose we link with external projects? IOW, if I want to have rasd
build *without* the kernel tree, is a simple

cp -r tools/lib/api/ <my_project>/path/to/perf/lib/

work?

I mean, I don't know and I'm just asking. Is that the proposed way? Are
we fine with that? Do we want something different, maybe even a lib? Is
it time for a lib even?

Because the whole perf functionality is being cravingly ogled by other
users - I know Matt wants it for cache QoS or whatever it is called and
there are probably a whole lot of projects which would like to process
events programatically in userspace.

So what I'm saying is, we probably should have a nice clean way
to be able to share that code with external projects instead of
external projects duplicating and building a whole library around
perf_event_open() and the likes...

Just a couple of thoughts...

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: perf & rasd integration plan

[Arnaldo Carvalho de Melo]

Em Mon, Oct 06, 2014 at 09:53:49PM +0200, Borislav Petkov escreveu:
> On Mon, Oct 06, 2014 at 04:12:27PM -0300, Arnaldo Carvalho de Melo wrote:
> > Right, and it is being a great exercise, thanks for the patience so far
> > ;-)
> 
> I know, this is your secret agenda to keep me at bay working on this! :-P
> 
> > Looking at those ifdefs we see things that are specific to tools/perf/,
> > like perf_evsel having a struct hists embedded... I.e. that is of no
> > interest whatsoever (so far) to rasd, and in turn pulls other object
> > files.
> > 
> > So I think that right now we need to look at those ifdefs and go on
> > making what is in tools/perf/util/ stop using it somehow, so that what
> > then gets moved to tools/lib/api/perf/ (I guess this is where it should
> > be, opinions?) have that sorted out.

> > I.e. what goes to tools/lib/api/perf/ is what is common to the needs of
> > tools/perf/ and rasd (wherever it may roam).

> > Perhaps something like I did for sock/inet_sock/inet_connection_sock
> > ages ago... I.e. the tools that want a hists will have a 

> > struct hists_evsel {
> > 	struct perf_evsel evsel;
> > 	struct hists	  hists;
> > };

> > etc.

> > Experimenting with that, as it is the only thing ifdefed out in rasd's
> > copy of evsel.h...
 
> Right, this all reads like it is going in the right direction but the
> more important question IMO is are we doing a libperf or are we still
> doing tools/lib/api/perf/ of single object files to which people can
> link?

My preference would be for single object files, but the pressure to have
a written in stone library seems to just build up...
 
> Because if it is the second - and it sounds to me like it is - how do
> you propose we link with external projects? IOW, if I want to have rasd
> build *without* the kernel tree, is a simple
 
> cp -r tools/lib/api/ <my_project>/path/to/perf/lib/
 
> work?

It should, no matter if we end up with a library.

What I'm doing these days to test if perf builds on multiple distros is:

#!/bin/sh
remotebuild() {
	target=$1
	perfpkg=$2
  	cmd="rm -rf ${perfpkg} ; tar xf ${perfpkg}.tar.gz && cd ${perfpkg} && rm -rf /tmp/build/perf ; mkdir -p /tmp/build/perf && make -C tools/perf O=/tmp/build/perf install && make -C tools/perf build-test"
	scp ${perfpkg}.tar.gz gita@${target}:. && \
  	ssh gita@${target} "${cmd}"
}

targets="fedora14 opensuse12.3 fedora19 ubuntu13.10"
perfpkg=perf-3.17.0-rc4

for target in ${targets} ; do
	remotebuild ${target} ${perfpkg}
done

After doing a 'make perf-targz-src-pkg'

I.e. no kernel sources involved on the machines where I build test.

IOW, it is untangled from the kernel sources. As tools/lib/api/ should
as well.

> I mean, I don't know and I'm just asking. Is that the proposed way? Are
> we fine with that? Do we want something different, maybe even a lib? Is
> it time for a lib even?

Well, the rasd experience is serving to show areas where there is
unnecessary entanglement (hists inside perf_event, etc, the ifdefs you
put in place).

I'm working to remove the ones that are in rasd.c, aiming to have a
tools/lib/api/ tree that can be used to build rasd and tools/perf/.

What I don't want to do is to simply straight more
tools/perf/util/evlist.c to tools/lib/api/perf/, some untanglement work
is needed.
 
> Because the whole perf functionality is being cravingly ogled by other
> users - I know Matt wants it for cache QoS or whatever it is called and
> there are probably a whole lot of projects which would like to process
> events programatically in userspace.
> 
> So what I'm saying is, we probably should have a nice clean way
> to be able to share that code with external projects instead of
> external projects duplicating and building a whole library around
> perf_event_open() and the likes...

Agreed, hopefully we'll get that, finally.
 
> Just a couple of thoughts...
> 
> -- 
> Regards/Gruss,
>     Boris.
> 
> Sent from a fat crate under my desk. Formatting is fine.
> --

[PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[Arnaldo Carvalho de Melo]

Heya, please check if this is OK.

This was while looking the set of methods used by rasd, trying to reduce
it to the bare minimum.

Perhaps even that cpu_map__new() one can be ditched, leaving to use the
default of a NULL cpumap that will end up being one with -1, i.e. all
cpus.

- Arnaldo

>From 8dc34bd2824c7843182f4fc6deabaf573e42e806 Mon Sep 17 00:00:00 2001
From: Arnaldo Carvalho de Melo <acme@redhat.com>
Date: Mon, 6 Oct 2014 15:43:42 -0300
Subject: [PATCH] rasd: Use perf_evlist__open() instead of open coded
 equivalent

Cc: Borislav Petkov <bp@suse.de>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 src/rasd.c | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/src/rasd.c b/src/rasd.c
index 06ccdcdd9d56..fb33fca131fa 100644
--- a/src/rasd.c
+++ b/src/rasd.c
@@ -241,7 +241,6 @@ static void daemonize(void)
 
 int main()
 {
-	struct perf_evsel *c;
 	struct thread_map *threads;
 	struct cpu_map *cpus;
 	int i;
@@ -278,12 +277,9 @@ int main()
 	perf_evlist__set_maps(evlist, cpus, threads);
 
 	/* Open events */
-	evlist__for_each(evlist, c) {
-		/* On all online cpus by default, system wide tracing */
-		if (perf_evsel__open(c, evlist->cpus, NULL) < 0)
-			err("opening tracepoint, are you root?");
-	}
-	perf_evlist__set_id_pos(evlist);
+	/* On all online cpus by default, system wide tracing */
+	if (perf_evlist__open(evlist) < 0)
+		err("opening tracepoint, are you root?");
 
 	/* mmap buffers */
 	if (perf_evlist__mmap(evlist, 4 /* opts->mmap_pages */, false) < 0)
-- 
1.9.3

Re: [PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[Jean Pihet]

Hi Arnaldo,

On 6 October 2014 23:26, Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> Heya, please check if this is OK.
>
> This was while looking the set of methods used by rasd, trying to reduce
> it to the bare minimum.
>
> Perhaps even that cpu_map__new() one can be ditched, leaving to use the
> default of a NULL cpumap that will end up being one with -1, i.e. all
> cpus.
>
> - Arnaldo
>
> From 8dc34bd2824c7843182f4fc6deabaf573e42e806 Mon Sep 17 00:00:00 2001
> From: Arnaldo Carvalho de Melo <acme@redhat.com>
> Date: Mon, 6 Oct 2014 15:43:42 -0300
> Subject: [PATCH] rasd: Use perf_evlist__open() instead of open coded
>  equivalent
>
> Cc: Borislav Petkov <bp@suse.de>
> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
> ---
>  src/rasd.c | 10 +++-------
>  1 file changed, 3 insertions(+), 7 deletions(-)
>
> diff --git a/src/rasd.c b/src/rasd.c
> index 06ccdcdd9d56..fb33fca131fa 100644
> --- a/src/rasd.c
> +++ b/src/rasd.c
> @@ -241,7 +241,6 @@ static void daemonize(void)
>
>  int main()
>  {
> -       struct perf_evsel *c;
>         struct thread_map *threads;
>         struct cpu_map *cpus;
>         int i;
> @@ -278,12 +277,9 @@ int main()
>         perf_evlist__set_maps(evlist, cpus, threads);
>
>         /* Open events */
> -       evlist__for_each(evlist, c) {
> -               /* On all online cpus by default, system wide tracing */
> -               if (perf_evsel__open(c, evlist->cpus, NULL) < 0)
> -                       err("opening tracepoint, are you root?");
> -       }
> -       perf_evlist__set_id_pos(evlist);
> +       /* On all online cpus by default, system wide tracing */
> +       if (perf_evlist__open(evlist) < 0)
> +               err("opening tracepoint, are you root?");

That way the system wide tracing does not work. perf_evlist__open uses
a non-NULL thread mapping and so it only traces the events generated
by the daemon itself.

perf top uses a similar approach but uses a lot of code in machine.c
(machine__synthesize_threads) and util/event.c
(perf_event__synthesize_threads) to synthesize the threads etc., which
we want to avoid in the minimalistic approach of rasd.

Maybe I missed something about the system wide tracing, any suggestion
is welcome.

Thanks for looking!

Jean

>
>         /* mmap buffers */
>         if (perf_evlist__mmap(evlist, 4 /* opts->mmap_pages */, false) < 0)
> --
> 1.9.3
>

Re: perf & rasd integration plan

[Borislav Petkov]

On Mon, Oct 06, 2014 at 06:22:43PM -0300, Arnaldo Carvalho de Melo wrote:
> My preference would be for single object files, but the pressure to have
> a written in stone library seems to just build up...

Yeah, I think if we have an alternative way to easily integrate that
functionality into external projects, we can avoid the need for a lib.

...

> After doing a 'make perf-targz-src-pkg'
> 
> I.e. no kernel sources involved on the machines where I build test.
> 
> IOW, it is untangled from the kernel sources. As tools/lib/api/ should
> as well.

Ok, that actually sounds like something we should try:

$ make perf-api-src-pkg

which would spit a tgz with all the generic stuff perf carries. People
would simply unpack it in their projects and build it.

That would require somewhat stable function names so that updating is
relatively painless and if visible functions change, some sort of a
README file should say what and why.

Sounds like a good alternative to a lib to me...

> Well, the rasd experience is serving to show areas where there is
> unnecessary entanglement (hists inside perf_event, etc, the ifdefs you
> put in place).

Yep.

> I'm working to remove the ones that are in rasd.c, aiming to have a
> tools/lib/api/ tree that can be used to build rasd and tools/perf/.
> 
> What I don't want to do is to simply straight more
> tools/perf/util/evlist.c to tools/lib/api/perf/, some untanglement work
> is needed.

Yep.

Thanks acme!

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[Arnaldo Carvalho de Melo]

Em Tue, Oct 07, 2014 at 10:45:14AM +0200, Jean Pihet escreveu:
> > --- a/src/rasd.c
> > -       evlist__for_each(evlist, c) {
> > -               /* On all online cpus by default, system wide tracing */
> > -               if (perf_evsel__open(c, evlist->cpus, NULL) < 0)
> > -                       err("opening tracepoint, are you root?");
> > -       }
> > -       perf_evlist__set_id_pos(evlist);
> > +       /* On all online cpus by default, system wide tracing */
> > +       if (perf_evlist__open(evlist) < 0)
> > +               err("opening tracepoint, are you root?");
 
> That way the system wide tracing does not work. perf_evlist__open uses
> a non-NULL thread mapping and so it only traces the events generated
> by the daemon itself.
 
> perf top uses a similar approach but uses a lot of code in machine.c
> (machine__synthesize_threads) and util/event.c
> (perf_event__synthesize_threads) to synthesize the threads etc., which
> we want to avoid in the minimalistic approach of rasd.

Humm, I will look into making it support this usecase, the point of
evsel/evlist is to try to abstract away as much as possible, leaving
just a few methods to be used by tools.

I think the default for perf_evlist__open() should be the most useful
for the majority of tools, which I thought would serve rasd well. It is
not the case, so I'll into how the tools that currently use
perf_evlist__open() behave and try to get a sane default in place.

Hopefully we will completely remove the need to set up any thread or cpu
map, as what you want is syswide tracing, right?

- Arnaldo
 
> Maybe I missed something about the system wide tracing, any suggestion
> is welcome.
> 
> Thanks for looking!
> 
> Jean
> 
> >
> >         /* mmap buffers */
> >         if (perf_evlist__mmap(evlist, 4 /* opts->mmap_pages */, false) < 0)
> > --
> > 1.9.3
> >

Re: perf & rasd integration plan

[Arnaldo Carvalho de Melo]

Em Tue, Oct 07, 2014 at 01:23:42PM +0200, Borislav Petkov escreveu:
> On Mon, Oct 06, 2014 at 06:22:43PM -0300, Arnaldo Carvalho de Melo wrote:
> > My preference would be for single object files, but the pressure to have
> > a written in stone library seems to just build up...
 
> Yeah, I think if we have an alternative way to easily integrate that
> functionality into external projects, we can avoid the need for a lib.
 
> ...
 
> > After doing a 'make perf-targz-src-pkg'

> > I.e. no kernel sources involved on the machines where I build test.

> > IOW, it is untangled from the kernel sources. As tools/lib/api/ should
> > as well.

> Ok, that actually sounds like something we should try:

> $ make perf-api-src-pkg

> which would spit a tgz with all the generic stuff perf carries. People
> would simply unpack it in their projects and build it.

> That would require somewhat stable function names so that updating is
> relatively painless and if visible functions change, some sort of a

Right, we now should always think that the great advantage of having
this code and its users in the same repo tree is going away, i.e. we
will not be able to improve the code by fixing mistakes that require
changing its users.

So we should export the bare minimum that tools advocate exporting, like
you guys are doing now by providing the code that shows how you use it.

> README file should say what and why.

Well, at that point we can't do that anymore, we will need instead to
add new methods while keeping the existing ones in place and with the
same semantics, at most for something really annoying to maintain we
will have to add annoying warnings telling that in a year or two
something will go away, it was deprecated, annoy the maintainer of your
tool to update their project.

> Sounds like a good alternative to a lib to me...

Yes, should be interesting to experiment with this, at the very least it
can be considered a step into the direction of a full blown library, if
we ever feel that it is strictly needed.
 
> > Well, the rasd experience is serving to show areas where there is
> > unnecessary entanglement (hists inside perf_event, etc, the ifdefs you
> > put in place).
 
> Yep.
 
> > I'm working to remove the ones that are in rasd.c, aiming to have a
> > tools/lib/api/ tree that can be used to build rasd and tools/perf/.

> > What I don't want to do is to simply straight more

       s/more/move/g

> > tools/perf/util/evlist.c to tools/lib/api/perf/, some untanglement work
> > is needed.
 
> Yep.

- Arnaldo

Re: perf & rasd integration plan

[Borislav Petkov]

On Tue, Oct 07, 2014 at 10:40:21AM -0300, Arnaldo Carvalho de Melo wrote:
> Right, we now should always think that the great advantage of having
> this code and its users in the same repo tree is going away, i.e. we
> will not be able to improve the code by fixing mistakes that require
> changing its users.

Why? I mean, in the rasd case, changing function calling conventions as
part of updating the perf libraries is no act. I mean, we won't link
against a shared lib but build a static executable and thus the perf src
lib (let's call it that - a source library) we're using is a don't care.

Thanks.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: perf & rasd integration plan

[Arnaldo Carvalho de Melo]

Em Tue, Oct 07, 2014 at 03:49:12PM +0200, Borislav Petkov escreveu:
> On Tue, Oct 07, 2014 at 10:40:21AM -0300, Arnaldo Carvalho de Melo wrote:
> > Right, we now should always think that the great advantage of having
> > this code and its users in the same repo tree is going away, i.e. we
> > will not be able to improve the code by fixing mistakes that require
> > changing its users.
> 
> Why? I mean, in the rasd case, changing function calling conventions as
> part of updating the perf libraries is no act. I mean, we won't link
> against a shared lib but build a static executable and thus the perf src
> lib (let's call it that - a source library) we're using is a don't care.

So you say that if we change a method, say, perf_evlist__open() to
accept a new argument, or change the type of one of its arguments,
with a good reason, it is acceptable and with just a change in the
README explaining why the build fails, which helps in finding how to fix
it in a particular project using these "source libraries" then it would
be ok?

That would be fine with me.

We should take care to make sure that the build _fails_ in such cases,
i.e.avoid changing the semantic of the Nth argument but keeping its type kind
of changes.

At some point there will be no more reasons to change things, and that
will be noticed by how long since the last change was made to a
particular class, at that point we may well think about making library
type promises.

- Arnaldo

Re: perf & rasd integration plan

[Borislav Petkov]

On Tue, Oct 07, 2014 at 10:55:31AM -0300, Arnaldo Carvalho de Melo wrote:
> So you say that if we change a method, say, perf_evlist__open() to
> accept a new argument, or change the type of one of its arguments,
> with a good reason, it is acceptable and with just a change in the
> README explaining why the build fails, which helps in finding how to fix
> it in a particular project using these "source libraries" then it would
> be ok?

Right, exactly.

Updating the perf bits used in another project will require a
recompilation anyway so adjusting stuff here and there should not be of
issue.

> That would be fine with me.

Yeah.

> We should take care to make sure that the build _fails_ in such cases,
> i.e.avoid changing the semantic of the Nth argument but keeping its
> type kind of changes.

Yep.

> At some point there will be no more reasons to change things, and that
> will be noticed by how long since the last change was made to a
> particular class, at that point we may well think about making library
> type promises.

I was about to say the same thing - once the code doesn't change anymore
and gets even boring, doing the library promise is probably pretty easy.

See, no need for you to get over to LCE: we're practically done :-)

We'll run the proposal in jolsa's timeslot again and make sure to poke
holes in it. But it looks good from where I'm standing.

Btw, did we just solve the compatibility problem of libraries by
delivering source libraries first!?!

Haha.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[Borislav Petkov]

On Tue, Oct 07, 2014 at 10:32:36AM -0300, Arnaldo Carvalho de Melo wrote:
> Hopefully we will completely remove the need to set up any thread or cpu
> map, as what you want is syswide tracing, right?

Yeah, I was questioning the need to open at least one thread even for
system-wide tracing.

See, source libraries are the new great thing for collaborative
development - f*ck "cloud"!

:-D

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: perf & rasd integration plan

[Arnaldo Carvalho de Melo]

Em Tue, Oct 07, 2014 at 04:02:29PM +0200, Borislav Petkov escreveu:
> On Tue, Oct 07, 2014 at 10:55:31AM -0300, Arnaldo Carvalho de Melo wrote:
> > So you say that if we change a method, say, perf_evlist__open() to
> > accept a new argument, or change the type of one of its arguments,
> > with a good reason, it is acceptable and with just a change in the
> > README explaining why the build fails, which helps in finding how to fix
> > it in a particular project using these "source libraries" then it would
> > be ok?
 
> Right, exactly.
 
> Updating the perf bits used in another project will require a
> recompilation anyway so adjusting stuff here and there should not be of
> issue.
 
> > That would be fine with me.
 
> Yeah.
 
> > We should take care to make sure that the build _fails_ in such cases,
> > i.e.avoid changing the semantic of the Nth argument but keeping its
> > type kind of changes.
 
> Yep.
 
> > At some point there will be no more reasons to change things, and that
> > will be noticed by how long since the last change was made to a
> > particular class, at that point we may well think about making library
> > type promises.
 
> I was about to say the same thing - once the code doesn't change anymore
> and gets even boring, doing the library promise is probably pretty easy.

Ok, at some point we need to make sure this gets into tools/lib/api/README.
 
> See, no need for you to get over to LCE: we're practically done :-)

Nah, there is much more than this at LCE, I'm sure, I'll try again next
time.
 
> We'll run the proposal in jolsa's timeslot again and make sure to poke
> holes in it. But it looks good from where I'm standing.

Thanks!
 
> Btw, did we just solve the compatibility problem of libraries by
> delivering source libraries first!?!

For me one of the points of having tools/ is to try and make kernel
developers try to do userspace muck, dirtying the hands trying to use
the same discipline used in the kernel proper, using code as unchanged
as possible from the kernel proper, etc.

If we now are moving to share code in a way that keeps what is nice in
having code and its users in the same repo, like the kernel guys have
with subsystems and its drivers while allowing to do it in different
repos, that is a good thing indeed.
 
> Haha.

While having fun even! Good deal.

- Arnaldo

Re: [PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[Arnaldo Carvalho de Melo]

Em Tue, Oct 07, 2014 at 04:04:33PM +0200, Borislav Petkov escreveu:
> On Tue, Oct 07, 2014 at 10:32:36AM -0300, Arnaldo Carvalho de Melo wrote:
> > Hopefully we will completely remove the need to set up any thread or cpu
> > map, as what you want is syswide tracing, right?
 
> Yeah, I was questioning the need to open at least one thread even for
> system-wide tracing.

Right, evsel/evlist evolved out of tons of refactoring steps moving
stuff out of open coded tools (record, report, top) while trying to make
it work for those tools, sometimes this resulted in sub-optimal defaults
(or plain bugs) like needing a thread_map and/or a cpu_map to do syswide
tracing, need to get that clarified and sorted out.

- Arnaldo
 
> See, source libraries are the new great thing for collaborative
> development - f*ck "cloud"!
 
> :-D

:-)

- Arnaldo

Re: perf & rasd integration plan

[Jiri Olsa]

On Mon, Oct 06, 2014 at 08:53:51AM +0200, Jean Pihet wrote:
> Hi Jiri,
> 
> 
> On 5 October 2014 20:24, Jiri Olsa <jolsa@redhat.com> wrote:
> > On Sun, Oct 05, 2014 at 07:48:01PM +0200, Borislav Petkov wrote:
> >> Top-posting on purpose:
> >>
> >> Btw, jolsa, if you get your LCE proposal for the perf splitting
> >> approved, please post the time here so people can come.
> >
> > yep, it got accepted, ther schedule is:
> >   Friday, October 17, 2014 from 9:00am – Noon
> >   Room 2
> >
> > https://pdxplumbers.osuosl.org/2014/ocw/events/LPC2014/tracks/351
> Great! Are there slides that we can download?

there'll be just one.. I'm on vacation this week,
so I'll try send some till end of the week

jirka

Re: [PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[Arnaldo Carvalho de Melo]

Em Tue, Oct 07, 2014 at 04:04:33PM +0200, Borislav Petkov escreveu:
> On Tue, Oct 07, 2014 at 10:32:36AM -0300, Arnaldo Carvalho de Melo wrote:
> > Hopefully we will completely remove the need to set up any thread or cpu
> > map, as what you want is syswide tracing, right?
> 
> Yeah, I was questioning the need to open at least one thread even for
> system-wide tracing.

Ok, so what is now at my perf/hists branch at:

git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git

should do the trick, the patch below makes rasd.c use it. I'll try
building rasd.c with it and checking if it works.

Never having tried this, what are the requisites to test it? Some
specific hardware and a kernel with the right tracepoint? I guess some
recent 3.17-rc kernel is all that I need?

- Arnaldo

---

[PATCH] rasd: Use perf_evlist__open() instead of open coded equivalent

Cc: Borislav Petkov <bp@suse.de>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 src/rasd.c | 23 +++--------------------
 1 file changed, 3 insertions(+), 20 deletions(-)

diff --git a/src/rasd.c b/src/rasd.c
index 06ccdcdd9d56..d3df9868388c 100644
--- a/src/rasd.c
+++ b/src/rasd.c
@@ -241,9 +241,6 @@ static void daemonize(void)
 
 int main()
 {
-	struct perf_evsel *c;
-	struct thread_map *threads;
-	struct cpu_map *cpus;
 	int i;
 
 	/* Run the process as a daemon */
@@ -266,24 +263,10 @@ int main()
 	if (read_config_file())
 		err("error reading config file");
 
-	/* Set cpu and threads maps */
-	threads = thread_map__new(-1, getpid(), UINT_MAX);
-	if (threads == NULL)
-		err("allocating threads_map\n");
-
-	cpus = cpu_map__new(NULL);
-	if (cpus == NULL)
-		err("allocating cpu_map\n");
-
-	perf_evlist__set_maps(evlist, cpus, threads);
-
 	/* Open events */
-	evlist__for_each(evlist, c) {
-		/* On all online cpus by default, system wide tracing */
-		if (perf_evsel__open(c, evlist->cpus, NULL) < 0)
-			err("opening tracepoint, are you root?");
-	}
-	perf_evlist__set_id_pos(evlist);
+	/* On all online cpus by default, system wide tracing */
+	if (perf_evlist__open(evlist) < 0)
+		err("opening tracepoint, are you root?");
 
 	/* mmap buffers */
 	if (perf_evlist__mmap(evlist, 4 /* opts->mmap_pages */, false) < 0)
-- 
1.9.3

Re: [PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[Borislav Petkov]

On Fri, Oct 10, 2014 at 05:07:08PM -0300, Arnaldo Carvalho de Melo wrote:
> Ok, so what is now at my perf/hists branch at:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git
> 
> should do the trick, the patch below makes rasd.c use it.

Cool.

> I'll try  lding rasd.c with it and checking if it works.
> 
> Never having tried this, what are the requisites to test it? Some
> specific hardware and a kernel with the right tracepoint? I guess some
> recent 3.17-rc kernel is all that I need?

Well, you'd need the part of Jean's patches which adds the event to evlist:

https://lkml.kernel.org/r/1412933690-25576-1-git-send-email-jean.pihet@linaro.org

AFAICT, you could apply patches 1-5 and replace 6 with yours. Now,
rasd.cfg has the mce:mce_record tracepoint which rasd opens but you
probably want to put a tracepoint which is much easier to exercise,
maybe some syscall or whatever.

I think that should do it but we won't know until we've tried it.

HTH and thanks a lot for doing this!

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[Arnaldo Carvalho de Melo]

Em Fri, Oct 10, 2014 at 10:28:54PM +0200, Borislav Petkov escreveu:
> On Fri, Oct 10, 2014 at 05:07:08PM -0300, Arnaldo Carvalho de Melo wrote:
> > I'll try  lding rasd.c with it and checking if it works.

> > Never having tried this, what are the requisites to test it? Some
> > specific hardware and a kernel with the right tracepoint? I guess some
> > recent 3.17-rc kernel is all that I need?

> Well, you'd need the part of Jean's patches which adds the event to evlist:
 
> https://lkml.kernel.org/r/1412933690-25576-1-git-send-email-jean.pihet@linaro.org
 
> AFAICT, you could apply patches 1-5 and replace 6 with yours. Now,
> rasd.cfg has the mce:mce_record tracepoint which rasd opens but you
> probably want to put a tracepoint which is much easier to exercise,
> maybe some syscall or whatever.

Right, stoopid me, no need for some specific tracepoint, just to see
that whatever tp it is, it will show up in "rasd"'s event loop. Ok, I'll
try that later.

Next stuff I probably will do is to move the bare minimum used by rasd
to tools/lib/api/perf/, i.e. there will be:

tools/lib/api/perf/evsel.c
tools/perf/util/evsel.c

Both will share the perf_evsel__ namespace (which I thought at some
point to make just: evsel__<METHOD_NAME>, wdyt?).

That way we just make public the bare minimum that already proved to be
useful outside tools/perf/ and over time we move stuff from
tools/perf/util/evsel.c (and from other tools in or out perf's repo)
into the lib.
 
> I think that should do it but we won't know until we've tried it.
> 
> HTH and thanks a lot for doing this!

Np, had to be done at some point :)

- Arnaldo
 
> -- 
> Regards/Gruss,
>     Boris.
> 
> Sent from a fat crate under my desk. Formatting is fine.
> --

Re: [PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[Borislav Petkov]

On Fri, Oct 10, 2014 at 05:41:58PM -0300, Arnaldo Carvalho de Melo wrote:
> Right, stoopid me, no need for some specific tracepoint, just to see
> that whatever tp it is, it will show up in "rasd"'s event loop. Ok, I'll
> try that later.
> 
> Next stuff I probably will do is to move the bare minimum used by rasd
> to tools/lib/api/perf/, i.e. there will be:
> 
> tools/lib/api/perf/evsel.c
> tools/perf/util/evsel.c
> 
> Both will share the perf_evsel__ namespace (which I thought at some
> point to make just: evsel__<METHOD_NAME>, wdyt?).
> 
> That way we just make public the bare minimum that already proved to be
> useful outside tools/perf/ and over time we move stuff from
> tools/perf/util/evsel.c (and from other tools in or out perf's repo)
> into the lib.

Abso-f*cking-lutely!

:-D

> Np, had to be done at some point :)

Yeah, cool. Let me know what I should do to help out.

Thanks.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[Jean Pihet]

Hi Arnaldo, Borislav,

On 10 October 2014 22:44, Borislav Petkov <bp@alien8.de> wrote:
> On Fri, Oct 10, 2014 at 05:41:58PM -0300, Arnaldo Carvalho de Melo wrote:
>> Right, stoopid me, no need for some specific tracepoint, just to see
>> that whatever tp it is, it will show up in "rasd"'s event loop. Ok, I'll
>> try that later.
>>
>> Next stuff I probably will do is to move the bare minimum used by rasd
>> to tools/lib/api/perf/, i.e. there will be:
>>
>> tools/lib/api/perf/evsel.c
>> tools/perf/util/evsel.c
>>
>> Both will share the perf_evsel__ namespace (which I thought at some
>> point to make just: evsel__<METHOD_NAME>, wdyt?).
Makes perfectly sense

>>
>> That way we just make public the bare minimum that already proved to be
>> useful outside tools/perf/ and over time we move stuff from
>> tools/perf/util/evsel.c (and from other tools in or out perf's repo)
>> into the lib.
That is great!

About rasd usage, there are some details at
https://wiki.linaro.org/LEG/Engineering/Kernel/perfAndRAS#rasd_implementation.

Thx,
Jean

>
> Abso-f*cking-lutely!
>
> :-D
>
>> Np, had to be done at some point :)
>
> Yeah, cool. Let me know what I should do to help out.
>
> Thanks.
>
> --
> Regards/Gruss,
>     Boris.
>
> Sent from a fat crate under my desk. Formatting is fine.
> --

Re: [PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[Jiri Olsa]

On Mon, Oct 13, 2014 at 09:29:57AM +0200, Jean Pihet wrote:
> Hi Arnaldo, Borislav,
> 
> On 10 October 2014 22:44, Borislav Petkov <bp@alien8.de> wrote:
> > On Fri, Oct 10, 2014 at 05:41:58PM -0300, Arnaldo Carvalho de Melo wrote:
> >> Right, stoopid me, no need for some specific tracepoint, just to see
> >> that whatever tp it is, it will show up in "rasd"'s event loop. Ok, I'll
> >> try that later.
> >>
> >> Next stuff I probably will do is to move the bare minimum used by rasd
> >> to tools/lib/api/perf/, i.e. there will be:
> >>
> >> tools/lib/api/perf/evsel.c
> >> tools/perf/util/evsel.c
> >>
> >> Both will share the perf_evsel__ namespace (which I thought at some
> >> point to make just: evsel__<METHOD_NAME>, wdyt?).
> Makes perfectly sense
> 
> >>
> >> That way we just make public the bare minimum that already proved to be
> >> useful outside tools/perf/ and over time we move stuff from
> >> tools/perf/util/evsel.c (and from other tools in or out perf's repo)
> >> into the lib.
> That is great!
> 
> About rasd usage, there are some details at
> https://wiki.linaro.org/LEG/Engineering/Kernel/perfAndRAS#rasd_implementation.

heya,
sorry for late reply.. I was on vacation last week..

I read the rasd sources and realized we could poke this
from another angle..  AFAIU the work the rasd does is following:
   - reads config file and opens configured tracepoints
   - reads samples comming from those tracepoints and displays/writes
     this data to the console/file
   - is there more?

If I'm not missing anything, this is quite usefull/common usage
pattern which would deserve new perf command.

I can see the analogy with ftrace debugfs interface
  - choose/enable tracepoints
  - cat .../tracing/trace-pipe

and there could be '-d' for the command to act as daemon.

thoughts? ;-)

jirka

Re: [PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[Arnaldo Carvalho de Melo]

Em Tue, Oct 14, 2014 at 03:56:31PM +0200, Jiri Olsa escreveu:
> On Mon, Oct 13, 2014 at 09:29:57AM +0200, Jean Pihet wrote:
> > On 10 October 2014 22:44, Borislav Petkov <bp@alien8.de> wrote:
> > > On Fri, Oct 10, 2014 at 05:41:58PM -0300, Arnaldo Carvalho de Melo wrote:
> > >> That way we just make public the bare minimum that already proved to be
> > >> useful outside tools/perf/ and over time we move stuff from
> > >> tools/perf/util/evsel.c (and from other tools in or out perf's repo)
> > >> into the lib.
> > That is great!

> > About rasd usage, there are some details at
> > https://wiki.linaro.org/LEG/Engineering/Kernel/perfAndRAS#rasd_implementation.
 
> sorry for late reply.. I was on vacation last week..
 
> I read the rasd sources and realized we could poke this
> from another angle..  AFAIU the work the rasd does is following:
>    - reads config file and opens configured tracepoints
>    - reads samples comming from those tracepoints and displays/writes
>      this data to the console/file
>    - is there more?
 
> If I'm not missing anything, this is quite usefull/common usage
> pattern which would deserve new perf command.
 
> I can see the analogy with ftrace debugfs interface
>   - choose/enable tracepoints
>   - cat .../tracing/trace-pipe
 
> and there could be '-d' for the command to act as daemon.

Right, this is to be integrated into 'trace', i.e. to be able to ask for
more events, some with callchains, some without, etc.

We may even have something really bare bones that does what rasd.c does
right now, to show how one can write a tool using the exported
interfaces in tools/lib/api/, but I think that what rasd is _right now_,
is just an attempt to use the libraries using as few as possible apis,
right? I.e. more "meat" will be added there, no?

Anyway, its beeing an interesting exercise and will result in a lot of
untanglement, which is good in any case.

- Arnaldo

Re: [PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[David Ahern]

On 10/14/14, 7:56 AM, Jiri Olsa wrote:
> I read the rasd sources and realized we could poke this
> from another angle..  AFAIU the work the rasd does is following:
>     - reads config file and opens configured tracepoints
>     - reads samples comming from those tracepoints and displays/writes
>       this data to the console/file
>     - is there more?
>
> If I'm not missing anything, this is quite usefull/common usage
> pattern which would deserve new perf command.
>
> I can see the analogy with ftrace debugfs interface
>    - choose/enable tracepoints
>    - cat .../tracing/trace-pipe
>
> and there could be '-d' for the command to act as daemon.

Yes, this parallels a new use case on my end. Right now I am running 
perf record ... | perf script. With the tracepoints and filters involved 
it is a LOT of typing - and still collects more than is needed (I don't 
need MMAP events for example, only COMM events). I am leaning towards a 
new perf sub-command but from my scheduling timehist and daemon commands 
I know there is a lot of overhead that goes with that. A perf library 
with a stable API would make this a lot easier. (AFAIK the python 
bindings do not currently support opening events, it is mainly an 
analysis option.)

David

Re: [PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[Jiri Olsa]

On Tue, Oct 14, 2014 at 11:02:23AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Tue, Oct 14, 2014 at 03:56:31PM +0200, Jiri Olsa escreveu:
> > On Mon, Oct 13, 2014 at 09:29:57AM +0200, Jean Pihet wrote:
> > > On 10 October 2014 22:44, Borislav Petkov <bp@alien8.de> wrote:
> > > > On Fri, Oct 10, 2014 at 05:41:58PM -0300, Arnaldo Carvalho de Melo wrote:
> > > >> That way we just make public the bare minimum that already proved to be
> > > >> useful outside tools/perf/ and over time we move stuff from
> > > >> tools/perf/util/evsel.c (and from other tools in or out perf's repo)
> > > >> into the lib.
> > > That is great!
> 
> > > About rasd usage, there are some details at
> > > https://wiki.linaro.org/LEG/Engineering/Kernel/perfAndRAS#rasd_implementation.
>  
> > sorry for late reply.. I was on vacation last week..
>  
> > I read the rasd sources and realized we could poke this
> > from another angle..  AFAIU the work the rasd does is following:
> >    - reads config file and opens configured tracepoints
> >    - reads samples comming from those tracepoints and displays/writes
> >      this data to the console/file
> >    - is there more?
>  
> > If I'm not missing anything, this is quite usefull/common usage
> > pattern which would deserve new perf command.
>  
> > I can see the analogy with ftrace debugfs interface
> >   - choose/enable tracepoints
> >   - cat .../tracing/trace-pipe
>  
> > and there could be '-d' for the command to act as daemon.
> 
> Right, this is to be integrated into 'trace', i.e. to be able to ask for
> more events, some with callchains, some without, etc.
> 
> We may even have something really bare bones that does what rasd.c does
> right now, to show how one can write a tool using the exported
> interfaces in tools/lib/api/, but I think that what rasd is _right now_,
> is just an attempt to use the libraries using as few as possible apis,
> right? I.e. more "meat" will be added there, no?

right.. is there more planned for RAS daemon to do? because at this
point it looks to me more like new perf functionality (in trace or
some other new command)

> 
> Anyway, its beeing an interesting exercise and will result in a lot of
> untanglement, which is good in any case.

yea, I dont mind untangling the interface.. I noticed in the thread
people want this anyway - simple 'events open' interface.. I often
need something simple for test programs

jirka

Re: [PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[Borislav Petkov]

On Tue, Oct 14, 2014 at 04:22:13PM +0200, Jiri Olsa wrote:
> right.. is there more planned for RAS daemon to do? because at this
> point it looks to me more like new perf functionality (in trace or
> some other new command)
> 
> > 
> > Anyway, its beeing an interesting exercise and will result in a lot of
> > untanglement, which is good in any case.
> 
> yea, I dont mind untangling the interface.. I noticed in the thread
> people want this anyway - simple 'events open' interface.. I often
> need something simple for test programs

Right, we simply need a way to get the tracepoint data into userspace.
Nothing else. And right now I cannot think about needing any other perf
tool functionality. But who knows.., if we need more, we'll untangle
more.

:-)

-- 
Regards/Gruss,
    Boris.
--

Re: [PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[Jean Pihet]

On 14 October 2014 17:17, Borislav Petkov <bp@alien8.de> wrote:
> On Tue, Oct 14, 2014 at 04:22:13PM +0200, Jiri Olsa wrote:
>> right.. is there more planned for RAS daemon to do? because at this
>> point it looks to me more like new perf functionality (in trace or
>> some other new command)
>>
>> >
>> > Anyway, its beeing an interesting exercise and will result in a lot of
>> > untanglement, which is good in any case.
>>
>> yea, I dont mind untangling the interface.. I noticed in the thread
>> people want this anyway - simple 'events open' interface.. I often
>> need something simple for test programs
>
> Right, we simply need a way to get the tracepoint data into userspace.
> Nothing else. And right now I cannot think about needing any other perf
> tool functionality. But who knows.., if we need more, we'll untangle
> more.
The persistent events stuff still needs some work and is needed by rasd.

>
> :-)
;-P

>
> --
> Regards/Gruss,
>     Boris.
> --

Cheers,
Jean

Re: [PATCH 1/1] rasd: Use perf_evlist__open() instead of open coded

[Arnaldo Carvalho de Melo]

Em Tue, Oct 14, 2014 at 08:19:52AM -0600, David Ahern escreveu:
> On 10/14/14, 7:56 AM, Jiri Olsa wrote:
> >I read the rasd sources and realized we could poke this
> >from another angle..  AFAIU the work the rasd does is following:
> >    - reads config file and opens configured tracepoints
> >    - reads samples comming from those tracepoints and displays/writes
> >      this data to the console/file
> >    - is there more?
> >
> >If I'm not missing anything, this is quite usefull/common usage
> >pattern which would deserve new perf command.
> >
> >I can see the analogy with ftrace debugfs interface
> >   - choose/enable tracepoints
> >   - cat .../tracing/trace-pipe
> >
> >and there could be '-d' for the command to act as daemon.
> 
> Yes, this parallels a new use case on my end. Right now I am running perf
> record ... | perf script. With the tracepoints and filters involved it is a
> LOT of typing - and still collects more than is needed (I don't need MMAP
> events for example, only COMM events). I am leaning towards a new perf
> sub-command but from my scheduling timehist and daemon commands I know there
> is a lot of overhead that goes with that. A perf library with a stable API
> would make this a lot easier. (AFAIK the python bindings do not currently
> support opening events, it is mainly an analysis option.)

Yeah, please take a look at this branch:

https://git.kernel.org/cgit/linux/kernel/git/acme/linux.git/log/?h=perf/hists

Trying to use the rasd.c opportunity to go over the boilerplate to
diminish it by having defaults for syswide tracing, etc.

I'm now working on moving the most minimalistic set of evsel.c/evlist.c
into tools/lib/api/perf/

- Arnaldo