[RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Steven Rostedt]

First, I hope everyone had a Happy New Year!

Next, Sudip has been working to get the libtraceevent library into
Debian. As this has been happening, I've been working at how to get all
the projects that use this, to use the library installed on the system
if it does exist. I'm hoping that once it's in Debian, the other
distros will follow suit.

Currently, the home of libtraceevent lives in the Linux kernel source
tree under tools/lib/traceevent. This was because perf uses it to parse
the events, and it seemed logical (at the time) to use this location as
the main source tree for the distributions.

The problem I'm now having is that I'm looking at fixing and updating
some of the code in this library, and since library versioning is
critical for applications that depend on it, we need to have a way to
update the versions, and this does not correspond with the Linux
versions.

For example, we currently have:

 libtraceevent.so.1.1.0

If I make some bug fixes, I probably want to change it to:

 libtraceevent.so.1.1.1 or libtraceevent.so.1.2.0

But if I change the API, which I plan on doing soon, I would probably
need to update the major version.

 libtraceevent.so.2.0.0

The thing is, we shouldn't be making these changes for every update
that we send to the main kernel. I would like to have a minimum of tags
to state what the version should be, and perhaps even branches for
working on a development version.

This is a problem with living in the Linux source tree as tags and
branches in Linus's tree are for only the Linux kernel source itself.
This may work fine for perf, as it's not a library and there's not
tools depending on the version of it. But it is a problem when it comes
to shared libraries.

Should we move libtraceevent into a stand alone git repo (on
kernel.org), that can have tags and branches specifically for it? We
can keep a copy in the Linux source tree for perf to use till it
becomes something that is reliably in all distributions. It's not like
perf doesn't depend on other libraries today anyway.

Thoughts or suggestions?

Thanks!

-- Steve

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Arnaldo Carvalho de Melo]

Em Thu, Jan 02, 2020 at 12:20:04PM -0500, Steven Rostedt escreveu:
> This is a problem with living in the Linux source tree as tags and
> branches in Linus's tree are for only the Linux kernel source itself.
> This may work fine for perf, as it's not a library and there's not

[acme@quaco perf]$ git log --oneline tools/lib/perf
6364e2891a4c libperf: Add man pages
19e0154effb4 libperf: Move to tools/lib/perf
[acme@quaco perf]$

[acme@quaco perf]$ git log --follow --oneline tools/lib/perf/Makefile
6364e2891a4c libperf: Add man pages
19e0154effb4 libperf: Move to tools/lib/perf
395e62cde10d libperf: Link static tests with libapi.a
7728fa0cfaeb libperf: Adopt perf_mmap__consume() function from tools/perf
fb4bf51fcc15 libperf: Add libperf dependency for tests targets
d80a5540bccb libperf: Link libapi.a in libperf.so
227cb129858a libperf: Add missing event.h file to install rule
b81d39c7a1ef (tag: perf-core-for-mingo-5.4-20190820) libperf: Fix arch include paths
6a94b52a71b7 libperf: Add tests support
0a64d7091efd libperf: Add install targets
47f9bccc79cb libperf: Add build version support
314350491810 libperf: Make libperf.a part of the perf build
[acme@quaco perf]$

libperf adopted the versioning and packaging practices introduced by
tools/lib/bpf, perhaps you could do the same for tools/lib/traceevent
and then we would have a standard for these cases?

The problem of having libperf, libbpf in distros is already being
tackled for quite a while, would be interesting to follow what has
happened in that area as well, Jiri knows more about this, Jiri?

- Arnaldo

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Steven Rostedt]

[ Added the BPF folks ]

On Thu, 2 Jan 2020 15:42:52 -0300
Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote:

> Em Thu, Jan 02, 2020 at 12:20:04PM -0500, Steven Rostedt escreveu:
> > This is a problem with living in the Linux source tree as tags and
> > branches in Linus's tree are for only the Linux kernel source itself.
> > This may work fine for perf, as it's not a library and there's not  
> 

> 
> libperf adopted the versioning and packaging practices introduced by
> tools/lib/bpf, perhaps you could do the same for tools/lib/traceevent
> and then we would have a standard for these cases?

I don't see libperf in my Debian testing installation, but I did find
libbpf.

> 
> The problem of having libperf, libbpf in distros is already being
> tackled for quite a while, would be interesting to follow what has
> happened in that area as well, Jiri knows more about this, Jiri?

Looking at the libbpf Makefile, it is getting the version from the
libbpf.map file.

What's the standard way for distributions to know when to use the
number? Only from Linux stable trees? That way, we can make fixes to the
library, but still need to remember to update the version number before
the release.

How does libbpf handle the version numbers with bug fixes? Does it get
updated one a kernel release if there are changes?

Obviously, we need to update the major number if the API changes in
anyway that is not compatible for a new application, which includes
adding new functions. Right?

-- Steve

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Sudip Mukherjee]

On Thu, Jan 2, 2020 at 5:20 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> First, I hope everyone had a Happy New Year!

Happy New Year to you too.

>
> Next, Sudip has been working to get the libtraceevent library into
> Debian. As this has been happening, I've been working at how to get all
> the projects that use this, to use the library installed on the system
> if it does exist. I'm hoping that once it's in Debian, the other
> distros will follow suit.

I have sent you another patch for libtraceevent. And, assuming that
you will not have any objection to that patch libtraceevent has been
merged in Debian and is now available in Debian Sid releases. Thanks
to Ben for all his suggestion and help.

The packages are at:
https://packages.debian.org/unstable/libtraceevent1
https://packages.debian.org/unstable/libtraceevent-dev
https://packages.debian.org/unstable/libtraceevent1-plugin


-- 
Regards
Sudip

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Steven Rostedt]

On Thu, 2 Jan 2020 22:43:34 +0000
Sudip Mukherjee <sudipm.mukherjee@gmail.com> wrote:

> On Thu, Jan 2, 2020 at 5:20 PM Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > First, I hope everyone had a Happy New Year!  
> 
> Happy New Year to you too.
> 
> >
> > Next, Sudip has been working to get the libtraceevent library into
> > Debian. As this has been happening, I've been working at how to get all
> > the projects that use this, to use the library installed on the system
> > if it does exist. I'm hoping that once it's in Debian, the other
> > distros will follow suit.  
> 
> I have sent you another patch for libtraceevent. And, assuming that
> you will not have any objection to that patch libtraceevent has been
> merged in Debian and is now available in Debian Sid releases. Thanks
> to Ben for all his suggestion and help.
> 
> The packages are at:
> https://packages.debian.org/unstable/libtraceevent1
> https://packages.debian.org/unstable/libtraceevent-dev
> https://packages.debian.org/unstable/libtraceevent1-plugin
> 
> 

BTW, while doing some minor fixes, I realized I still have generic
names for "warning", "pr_stat" and "vpr_stat" and thought they should
be changed to "tep_warning", "tep_pr_stat" and "tep_vpr_stat" for
namespace reasons even though they are weak functions. Would this
require a major version change, or perhaps its early enough to get this
in with a minor version change as the libraries are probably not being
used yet?

-- Steve

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Namhyung Kim]

Hello Steve,

Happy new year!

On Sat, Jan 4, 2020 at 8:19 AM Steven Rostedt <rostedt@goodmis.org> wrote:
> BTW, while doing some minor fixes, I realized I still have generic
> names for "warning", "pr_stat" and "vpr_stat" and thought they should
> be changed to "tep_warning", "tep_pr_stat" and "tep_vpr_stat" for
> namespace reasons even though they are weak functions. Would this
> require a major version change, or perhaps its early enough to get this
> in with a minor version change as the libraries are probably not being
> used yet?

Hmm.. I don't think it requires a major version change as they are not
public APIs actually.  They are internal functions but can be overridden
by external ones, right?

I guess it's because there was a concern with printing messages in
a library.  What about adding a mechanism to register callbacks
(and maybe defaults to discard all messages)?

Thanks
Namhyung

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Jiri Olsa]

On Thu, Jan 02, 2020 at 12:20:04PM -0500, Steven Rostedt wrote:
> First, I hope everyone had a Happy New Year!

heya, Happy New Year! ;-)

> 
> Next, Sudip has been working to get the libtraceevent library into
> Debian. As this has been happening, I've been working at how to get all
> the projects that use this, to use the library installed on the system
> if it does exist. I'm hoping that once it's in Debian, the other
> distros will follow suit.
> 
> Currently, the home of libtraceevent lives in the Linux kernel source
> tree under tools/lib/traceevent. This was because perf uses it to parse
> the events, and it seemed logical (at the time) to use this location as
> the main source tree for the distributions.
> 
> The problem I'm now having is that I'm looking at fixing and updating
> some of the code in this library, and since library versioning is
> critical for applications that depend on it, we need to have a way to
> update the versions, and this does not correspond with the Linux
> versions.
> 
> For example, we currently have:
> 
>  libtraceevent.so.1.1.0
> 
> If I make some bug fixes, I probably want to change it to:
> 
>  libtraceevent.so.1.1.1 or libtraceevent.so.1.2.0
> 
> But if I change the API, which I plan on doing soon, I would probably
> need to update the major version.
> 
>  libtraceevent.so.2.0.0
> 
> The thing is, we shouldn't be making these changes for every update
> that we send to the main kernel. I would like to have a minimum of tags
> to state what the version should be, and perhaps even branches for
> working on a development version.
> 
> This is a problem with living in the Linux source tree as tags and
> branches in Linus's tree are for only the Linux kernel source itself.
> This may work fine for perf, as it's not a library and there's not
> tools depending on the version of it. But it is a problem when it comes
> to shared libraries.
> 
> Should we move libtraceevent into a stand alone git repo (on
> kernel.org), that can have tags and branches specifically for it? We
> can keep a copy in the Linux source tree for perf to use till it

so libbpf 'moved' for this reason to github repo,
but keeping the kernel as the true/first source,
and updating github repo when release is ready

libbpf github repo is then source for fedora (and others)
package


> becomes something that is reliably in all distributions. It's not like
> perf doesn't depend on other libraries today anyway.

yep, we already have a way to link libbpf dynamicaly from package,
will work the same for libtraceevent

jirka

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Steven Rostedt]

On Fri, 3 Jan 2020 00:49:50 +0100
Jiri Olsa <jolsa@redhat.com> wrote:

> > Should we move libtraceevent into a stand alone git repo (on
> > kernel.org), that can have tags and branches specifically for it? We
> > can keep a copy in the Linux source tree for perf to use till it  
> 
> so libbpf 'moved' for this reason to github repo,
> but keeping the kernel as the true/first source,
> and updating github repo when release is ready
> 
> libbpf github repo is then source for fedora (and others)
> package

Ah, so perhaps I should follow this? I could keep it a kernel.org repo
(as I rather have it there anyway).

We can have the tools/lib/traceevent be the main source, but then just
copy it to the stand alone for releases.

Sudip, would this work for you too? (and yes, I plan on acking that
patch for the -ldl change, after looking at it a little bit more).

-- Steve

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Sudip Mukherjee]

On Thu, Jan 2, 2020 at 11:58 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Fri, 3 Jan 2020 00:49:50 +0100
> Jiri Olsa <jolsa@redhat.com> wrote:
>
> > > Should we move libtraceevent into a stand alone git repo (on
> > > kernel.org), that can have tags and branches specifically for it? We
> > > can keep a copy in the Linux source tree for perf to use till it
> >
> > so libbpf 'moved' for this reason to github repo,
> > but keeping the kernel as the true/first source,
> > and updating github repo when release is ready
> >
> > libbpf github repo is then source for fedora (and others)
> > package
>
> Ah, so perhaps I should follow this? I could keep it a kernel.org repo
> (as I rather have it there anyway).

Debian is building libbpf from kernel tree.

>
> We can have the tools/lib/traceevent be the main source, but then just
> copy it to the stand alone for releases.
>
> Sudip, would this work for you too? (and yes, I plan on acking that
> patch for the -ldl change, after looking at it a little bit more).

If it moves to a separate repo then I guess we will need to move it
out of the debian kernel and create a separate package. Might be
better if we do for libbpf also.

(Adding Ben).
Ben?


-- 
Regards
Sudip

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Jiri Olsa]

On Thu, Jan 02, 2020 at 06:58:53PM -0500, Steven Rostedt wrote:
> On Fri, 3 Jan 2020 00:49:50 +0100
> Jiri Olsa <jolsa@redhat.com> wrote:
> 
> > > Should we move libtraceevent into a stand alone git repo (on
> > > kernel.org), that can have tags and branches specifically for it? We
> > > can keep a copy in the Linux source tree for perf to use till it  
> > 
> > so libbpf 'moved' for this reason to github repo,
> > but keeping the kernel as the true/first source,
> > and updating github repo when release is ready
> > 
> > libbpf github repo is then source for fedora (and others)
> > package
> 
> Ah, so perhaps I should follow this? I could keep it a kernel.org repo
> (as I rather have it there anyway).

sounds good, and if it works out, we'll follow you with libperf :-)

if you want to check on the libbpf:
  https://github.com/libbpf/libbpf

there might be some syncs scripts worth checking

jirka

> 
> We can have the tools/lib/traceevent be the main source, but then just
> copy it to the stand alone for releases.
> 
> Sudip, would this work for you too? (and yes, I plan on acking that
> patch for the -ldl change, after looking at it a little bit more).
> 
> -- Steve
> 

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Sudip Mukherjee]

On Fri, Jan 3, 2020 at 1:36 PM Jiri Olsa <jolsa@redhat.com> wrote:
>
> On Thu, Jan 02, 2020 at 06:58:53PM -0500, Steven Rostedt wrote:
> > On Fri, 3 Jan 2020 00:49:50 +0100
> > Jiri Olsa <jolsa@redhat.com> wrote:
> >
> > > > Should we move libtraceevent into a stand alone git repo (on
> > > > kernel.org), that can have tags and branches specifically for it? We
> > > > can keep a copy in the Linux source tree for perf to use till it
> > >
> > > so libbpf 'moved' for this reason to github repo,
> > > but keeping the kernel as the true/first source,
> > > and updating github repo when release is ready
> > >
> > > libbpf github repo is then source for fedora (and others)
> > > package
> >
> > Ah, so perhaps I should follow this? I could keep it a kernel.org repo
> > (as I rather have it there anyway).
>
> sounds good, and if it works out, we'll follow you with libperf :-)
>
> if you want to check on the libbpf:
>   https://github.com/libbpf/libbpf

fwiw, I have opened a bug report in Debian requesting to package
libbpf from the github repo.

-- 
Regards
Sudip

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Steven Rostedt]

[ Added Konstantin and kernel.org users mailing list ]

On Fri, 3 Jan 2020 14:36:40 +0100
Jiri Olsa <jolsa@redhat.com> wrote:

> On Thu, Jan 02, 2020 at 06:58:53PM -0500, Steven Rostedt wrote:
> > On Fri, 3 Jan 2020 00:49:50 +0100
> > Jiri Olsa <jolsa@redhat.com> wrote:
> >   
> > > > Should we move libtraceevent into a stand alone git repo (on
> > > > kernel.org), that can have tags and branches specifically for it? We
> > > > can keep a copy in the Linux source tree for perf to use till it    
> > > 
> > > so libbpf 'moved' for this reason to github repo,
> > > but keeping the kernel as the true/first source,
> > > and updating github repo when release is ready
> > > 
> > > libbpf github repo is then source for fedora (and others)
> > > package  
> > 
> > Ah, so perhaps I should follow this? I could keep it a kernel.org repo
> > (as I rather have it there anyway).  
> 
> sounds good, and if it works out, we'll follow you with libperf :-)
> 
> if you want to check on the libbpf:
>   https://github.com/libbpf/libbpf
> 
> there might be some syncs scripts worth checking

I wonder if there should be a:

  git://git.kernel.org/pub/scm/utils/lib/

directory to have:

  git://git.kernel.org/pub/scm/utils/lib/traceevent/
  git://git.kernel.org/pub/scm/utils/lib/libbpf/
  git://git.kernel.org/pub/scm/utils/lib/libperf/

That could hold the libraries that are tight to the kernel?

-- Steve


> 
> jirka
> 
> > 
> > We can have the tools/lib/traceevent be the main source, but then just
> > copy it to the stand alone for releases.
> > 
> > Sudip, would this work for you too? (and yes, I plan on acking that
> > patch for the -ldl change, after looking at it a little bit more).
> > 
> > -- Steve
> >   

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Jiri Olsa]

On Fri, Jan 03, 2020 at 06:16:14PM -0500, Steven Rostedt wrote:
> 
> [ Added Konstantin and kernel.org users mailing list ]
> 
> On Fri, 3 Jan 2020 14:36:40 +0100
> Jiri Olsa <jolsa@redhat.com> wrote:
> 
> > On Thu, Jan 02, 2020 at 06:58:53PM -0500, Steven Rostedt wrote:
> > > On Fri, 3 Jan 2020 00:49:50 +0100
> > > Jiri Olsa <jolsa@redhat.com> wrote:
> > >   
> > > > > Should we move libtraceevent into a stand alone git repo (on
> > > > > kernel.org), that can have tags and branches specifically for it? We
> > > > > can keep a copy in the Linux source tree for perf to use till it    
> > > > 
> > > > so libbpf 'moved' for this reason to github repo,
> > > > but keeping the kernel as the true/first source,
> > > > and updating github repo when release is ready
> > > > 
> > > > libbpf github repo is then source for fedora (and others)
> > > > package  
> > > 
> > > Ah, so perhaps I should follow this? I could keep it a kernel.org repo
> > > (as I rather have it there anyway).  
> > 
> > sounds good, and if it works out, we'll follow you with libperf :-)
> > 
> > if you want to check on the libbpf:
> >   https://github.com/libbpf/libbpf
> > 
> > there might be some syncs scripts worth checking
> 
> I wonder if there should be a:
> 
>   git://git.kernel.org/pub/scm/utils/lib/
> 
> directory to have:
> 
>   git://git.kernel.org/pub/scm/utils/lib/traceevent/
>   git://git.kernel.org/pub/scm/utils/lib/libbpf/
>   git://git.kernel.org/pub/scm/utils/lib/libperf/

we could loose the 'lib' and just have:

    git://git.kernel.org/pub/scm/utils/lib/perf/

> 
> That could hold the libraries that are tight to the kernel?

I don't think libbpf will change now after they are settled in github,
but we could consider this for libperf

jirka

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Arnaldo Carvalho de Melo]

Em Mon, Jan 06, 2020 at 04:19:02PM +0100, Jiri Olsa escreveu:
> On Fri, Jan 03, 2020 at 06:16:14PM -0500, Steven Rostedt wrote:
> > 
> > [ Added Konstantin and kernel.org users mailing list ]
> > 
> > On Fri, 3 Jan 2020 14:36:40 +0100
> > Jiri Olsa <jolsa@redhat.com> wrote:
> > 
> > > On Thu, Jan 02, 2020 at 06:58:53PM -0500, Steven Rostedt wrote:
> > > > On Fri, 3 Jan 2020 00:49:50 +0100
> > > > Jiri Olsa <jolsa@redhat.com> wrote:
> > > >   
> > > > > > Should we move libtraceevent into a stand alone git repo (on
> > > > > > kernel.org), that can have tags and branches specifically for it? We
> > > > > > can keep a copy in the Linux source tree for perf to use till it    
> > > > > 
> > > > > so libbpf 'moved' for this reason to github repo,
> > > > > but keeping the kernel as the true/first source,
> > > > > and updating github repo when release is ready
> > > > > 
> > > > > libbpf github repo is then source for fedora (and others)
> > > > > package  
> > > > 
> > > > Ah, so perhaps I should follow this? I could keep it a kernel.org repo
> > > > (as I rather have it there anyway).  
> > > 
> > > sounds good, and if it works out, we'll follow you with libperf :-)
> > > 
> > > if you want to check on the libbpf:
> > >   https://github.com/libbpf/libbpf
> > > 
> > > there might be some syncs scripts worth checking
> > 
> > I wonder if there should be a:
> > 
> >   git://git.kernel.org/pub/scm/utils/lib/
> > 
> > directory to have:
> > 
> >   git://git.kernel.org/pub/scm/utils/lib/traceevent/
> >   git://git.kernel.org/pub/scm/utils/lib/libbpf/
> >   git://git.kernel.org/pub/scm/utils/lib/libperf/
> 
> we could loose the 'lib' and just have:
> 
>     git://git.kernel.org/pub/scm/utils/lib/perf/

So, we have:

https://www.kernel.org/pub/linux/kernel/tools/perf/

trying to mimic the kernel sources tree structure, so perhaps we could
have:

https://www.kernel.org/pub/linux/kernel/tools/lib/{perf,traceevent}/

To continue that directory tree mirror?

> > That could hold the libraries that are tight to the kernel?

> I don't think libbpf will change now after they are settled in github,
> but we could consider this for libperf
> 
> jirka

-- 

- Arnaldo

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Steven Rostedt]

On Mon, 6 Jan 2020 13:26:23 -0300
Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote:

> So, we have:
> 
> https://www.kernel.org/pub/linux/kernel/tools/perf/
> 
> trying to mimic the kernel sources tree structure, so perhaps we could
> have:
> 
> https://www.kernel.org/pub/linux/kernel/tools/lib/{perf,traceevent}/
> 
> To continue that directory tree mirror?

Wouldn't that become a bit of manual work. Unlike perf, the versions
will not correspond to the Linux kernel versions. They would need to
follow library versioning.

It would at a minimum require new scripting to get this right.

-- Steve

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Arnaldo Carvalho de Melo]

Em Mon, Jan 06, 2020 at 11:36:15AM -0500, Steven Rostedt escreveu:
> On Mon, 6 Jan 2020 13:26:23 -0300
> Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote:
 
> > So, we have:

> > https://www.kernel.org/pub/linux/kernel/tools/perf/

> > trying to mimic the kernel sources tree structure, so perhaps we could
> > have:

> > https://www.kernel.org/pub/linux/kernel/tools/lib/{perf,traceevent}/

> > To continue that directory tree mirror?
 
> Wouldn't that become a bit of manual work. Unlike perf, the versions
> will not correspond to the Linux kernel versions. They would need to
> follow library versioning.

It doesn't have to correspond, the versions you use there are entirely
up to libtraceevent developers, no?  I.e. when you decide to cut some
version, tag it in the linux kernel git repo, create a tarball,
something like:

make help | grep perf

but using whatever versioning you decide to use, which would be the same
regardless of where you develop it, and make it available via
https://www.kernel.org/pub/linux/kernel/tools/lib/traceevent/
 
> It would at a minimum require new scripting to get this right.

Sure, regardless of where you do source code control you will need to
tag, create a tarball, signatures (which kup helps with) for kernel.org,
for instance I use:

  kup put perf-${VER}.tar.xz perf-${VER}.tar.sign /pub/linux/kernel/tools/perf/v${VER}/perf-${VER}.tar.xz

What is in ${VER} is entirely up to me, its just that perf has a very
active development process with lots of patches each release and we
try to stop getting features when the kernel closes the window, have a
-next for new features, etc, so we end up with perf-ver == kernel-ver,
but that never was a requirement, its just convenient and we got used to
it.

- Arnaldo

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Konstantin Ryabitsev]

On Mon, Jan 06, 2020 at 04:47:11PM -0300, Arnaldo Carvalho de Melo wrote:
> Sure, regardless of where you do source code control you will need to
> tag, create a tarball, signatures (which kup helps with) for kernel.org,
> for instance I use:
> 
>   kup put perf-${VER}.tar.xz perf-${VER}.tar.sign /pub/linux/kernel/tools/perf/v${VER}/perf-${VER}.tar.xz

It's worth noting that you don't have to use kup if you don't want to -- 
we have a mechanism to create tarball releases directly from tag 
signatures. You just have to add a special note to the tag and the 
backend does the rest automatically -- we have a handy script [^1] to 
make it easier.

Greg KH has been using this process for a while now.

If you would like to switch to that instead of using kup directly, just 
let me know.

-K

[^1]: https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/tree/git-archive-signer

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Arnaldo Carvalho de Melo]

Em Mon, Jan 06, 2020 at 03:14:01PM -0500, Konstantin Ryabitsev escreveu:
> On Mon, Jan 06, 2020 at 04:47:11PM -0300, Arnaldo Carvalho de Melo wrote:
> > Sure, regardless of where you do source code control you will need to
> > tag, create a tarball, signatures (which kup helps with) for kernel.org,
> > for instance I use:
> > 
> >   kup put perf-${VER}.tar.xz perf-${VER}.tar.sign /pub/linux/kernel/tools/perf/v${VER}/perf-${VER}.tar.xz
> 
> It's worth noting that you don't have to use kup if you don't want to -- 
> we have a mechanism to create tarball releases directly from tag 
> signatures. You just have to add a special note to the tag and the 
> backend does the rest automatically -- we have a handy script [^1] to 
> make it easier.
> 
> Greg KH has been using this process for a while now.
> 
> If you would like to switch to that instead of using kup directly, just 
> let me know.

Sure I wanna know, will read [^1], thanks for the pointer.

On a side note since this is a library/tool that is hosted in the
kernel, like perf, using that /pub/linux/kernel/tools/lib/ path seems
appropriate, i.e. it provides tarballs for things in the tools/
directory of the linux/kernel/ :-)

- Arnaldo
 
> -K
 
> [^1]: https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/tree/git-archive-signer

Re: [kernel.org users] [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Jason Gunthorpe]

On Mon, Jan 06, 2020 at 11:36:15AM -0500, Steven Rostedt wrote:
> On Mon, 6 Jan 2020 13:26:23 -0300
> Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote:
> 
> > So, we have:
> > 
> > https://www.kernel.org/pub/linux/kernel/tools/perf/
> > 
> > trying to mimic the kernel sources tree structure, so perhaps we could
> > have:
> > 
> > https://www.kernel.org/pub/linux/kernel/tools/lib/{perf,traceevent}/
> > 
> > To continue that directory tree mirror?
> 
> Wouldn't that become a bit of manual work. Unlike perf, the versions
> will not correspond to the Linux kernel versions. They would need to
> follow library versioning.
> 
> It would at a minimum require new scripting to get this right.

If it is not tightly linked to the kernel and is just a normal
library, you might consider using github. This is what we do for the
rdma user space and it works well. We still take patches from the
mailing list flow, but do use a fair amount of the github stuff too:

https://github.com/linux-rdma/rdma-core

With github actions now able to provide a quite good CI it covers a
lot of required stuff for a library in one place, in a way that
doesn't silo all the build infrastucture.

If you are interested in how we set it up I can write a longer email.

Regards,
Jason

Re: [kernel.org users] [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Steven Rostedt]

On Mon, 6 Jan 2020 16:47:15 -0400
Jason Gunthorpe <jgg@ziepe.ca> wrote:

> If it is not tightly linked to the kernel and is just a normal

Well, it's used by perf, trace-cmd, power-top and rasdaemon (and
perhaps even more). It lives in the kernel tree mainly because of perf.

> library, you might consider using github. This is what we do for the
> rdma user space and it works well. We still take patches from the
> mailing list flow, but do use a fair amount of the github stuff too:
> 
> https://github.com/linux-rdma/rdma-core
> 
> With github actions now able to provide a quite good CI it covers a
> lot of required stuff for a library in one place, in a way that
> doesn't silo all the build infrastucture.

Github has ways to help with libraries? I'm totally clueless about
this. I'm interested in hearing more.

Thanks,

-- Steve


> 
> If you are interested in how we set it up I can write a longer email.

Re: [kernel.org users] [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Jason Gunthorpe]

On Mon, Jan 06, 2020 at 03:52:32PM -0500, Steven Rostedt wrote:
> On Mon, 6 Jan 2020 16:47:15 -0400
> Jason Gunthorpe <jgg@ziepe.ca> wrote:
> 
> > If it is not tightly linked to the kernel and is just a normal
> 
> Well, it's used by perf, trace-cmd, power-top and rasdaemon (and
> perhaps even more). It lives in the kernel tree mainly because of perf.

I see
 
> > With github actions now able to provide a quite good CI it covers a
> > lot of required stuff for a library in one place, in a way that
> > doesn't silo all the build infrastucture.
> 
> Github has ways to help with libraries? I'm totally clueless about
> this. I'm interested in hearing more.

These days it is a lot of work to get a library ready for the
distributions and github now has a built-in CI (git hub actions) that
lets projects run through all the build and in some cases runtime
tests needed often and automatically.

For instance we build rdma-core for Centos 6,78, Fedora 31, Ubuntu,
SuSe, cross compile on ARM and PPC, all automatically and all drive
from a fairly short script in the source tree, so anyone can
contribute.

The release process to .tar.gz (and distro packages if we wanted) is
also automated via the same. Push a tag and all the release stuff is
done and the right .tar.gz appears automatically in the right place.

It is nothing so unique, but having everything nicely integrated in
one place makes it possible for a project to spend a small amount of
time on CI and administration stuff instead of a large amount :)

Jason

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Konstantin Ryabitsev]

On Mon, Jan 06, 2020 at 04:19:02PM +0100, Jiri Olsa wrote:
> > I wonder if there should be a:
> > 
> >   git://git.kernel.org/pub/scm/utils/lib/
> > 
> > directory to have:
> > 
> >   git://git.kernel.org/pub/scm/utils/lib/traceevent/
> >   git://git.kernel.org/pub/scm/utils/lib/libbpf/
> >   git://git.kernel.org/pub/scm/utils/lib/libperf/

We already have 'pub/scm/libs' so I suggest we just place things there:

pub/scm/libs/traceevent
pub/scm/libs/libbpf
pub/scm/libs/libperf

Normally, we do a whole subdir, but that can be unnecessary, especially 
to avoid repetitions like:

pub/scm/libs/libgpiod/libgpiod.git

If the suggested locations work for you, just email helpdesk@kernel.org 
requesting their creation, and please specify who should have push 
access to them.

-K

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Masami Hiramatsu]

On Thu, 2 Jan 2020 12:20:04 -0500
Steven Rostedt <rostedt@goodmis.org> wrote:

> First, I hope everyone had a Happy New Year!

Hope you to have a Happy New Year too!

> 
> Next, Sudip has been working to get the libtraceevent library into
> Debian. As this has been happening, I've been working at how to get all
> the projects that use this, to use the library installed on the system
> if it does exist. I'm hoping that once it's in Debian, the other
> distros will follow suit.
> 
> Currently, the home of libtraceevent lives in the Linux kernel source
> tree under tools/lib/traceevent. This was because perf uses it to parse
> the events, and it seemed logical (at the time) to use this location as
> the main source tree for the distributions.
> 
> The problem I'm now having is that I'm looking at fixing and updating
> some of the code in this library, and since library versioning is
> critical for applications that depend on it, we need to have a way to
> update the versions, and this does not correspond with the Linux
> versions.
> 
> For example, we currently have:
> 
>  libtraceevent.so.1.1.0
> 
> If I make some bug fixes, I probably want to change it to:
> 
>  libtraceevent.so.1.1.1 or libtraceevent.so.1.2.0
> 
> But if I change the API, which I plan on doing soon, I would probably
> need to update the major version.
> 
>  libtraceevent.so.2.0.0
> 
> The thing is, we shouldn't be making these changes for every update
> that we send to the main kernel. I would like to have a minimum of tags
> to state what the version should be, and perhaps even branches for
> working on a development version.
> 
> This is a problem with living in the Linux source tree as tags and
> branches in Linus's tree are for only the Linux kernel source itself.
> This may work fine for perf, as it's not a library and there's not
> tools depending on the version of it. But it is a problem when it comes
> to shared libraries.
> 
> Should we move libtraceevent into a stand alone git repo (on
> kernel.org), that can have tags and branches specifically for it? We
> can keep a copy in the Linux source tree for perf to use till it
> becomes something that is reliably in all distributions. It's not like
> perf doesn't depend on other libraries today anyway.

+1. It sounds reasonable to move the main part of libtraceevent out
of kernel tree. BTW, the plugins seems depending on the kernel. Maybe
we can provide it under
/lib/modules/<kversion>/shared/libtraceevent/plugins/ ? :)

Thank you,

-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Steven Rostedt]

On Fri, 3 Jan 2020 21:17:43 +0900
Masami Hiramatsu <mhiramat@kernel.org> wrote:

> +1. It sounds reasonable to move the main part of libtraceevent out
> of kernel tree. BTW, the plugins seems depending on the kernel. Maybe
> we can provide it under
> /lib/modules/<kversion>/shared/libtraceevent/plugins/ ? :)

They really shouldn't be. They should be able to be used between
different versions of the kernel. Which plugins do you see an issue
with?

-- Steve

Re: [RFC] tools lib traceevent: How to do library versioning being in the Linux kernel source?

[Masami Hiramatsu]

On Fri, 3 Jan 2020 18:12:42 -0500
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Fri, 3 Jan 2020 21:17:43 +0900
> Masami Hiramatsu <mhiramat@kernel.org> wrote:
> 
> > +1. It sounds reasonable to move the main part of libtraceevent out
> > of kernel tree. BTW, the plugins seems depending on the kernel. Maybe
> > we can provide it under
> > /lib/modules/<kversion>/shared/libtraceevent/plugins/ ? :)
> 
> They really shouldn't be. They should be able to be used between
> different versions of the kernel. Which plugins do you see an issue
> with?

I meant tools/lib/traceevent/plugins.
It seems that those plugins provide equivarent codes of some macros
or values in event formats. In that case, when a new kernel modifies
the event definitions, I think some of those needs to be updated too.
Or, would those events be considered as a stable ABI?

Thank you,

-- 
Masami Hiramatsu <mhiramat@kernel.org>