btrfs-progs and libbtrfsutil versioning

btrfs-progs and libbtrfsutil versioning

[Neal Gompa]

Hey all,

I've been working on a set of patches to synchronize the version
numbers across everything produced by btrfs-progs and add pkgconfig
files to make it easier for software to consume the libraries, but as
I worked through it, I started realizing that btrfs-progs actually has
*two* distinctly separate projects in one package.

This is somewhat problematic since it makes properly representing that
versioning in the packages I ship in Fedora rather difficult.

Would it be possible to consider splitting libbtrfsutil into its own
project with its own releases? If not, is there a reason that we
*can't* synchronize the versions across btrfs-progs, libbtrfs, and
libbtrfsutil? We could still make sure that the library sonames are
versioned properly, but having the user-facing versions actually sync
up makes life a lot easier...

-- 
真実はいつも一つ！/ Always, there's only one truth!

Re: btrfs-progs and libbtrfsutil versioning

[David Sterba]

On Wed, Sep 23, 2020 at 09:48:44PM -0400, Neal Gompa wrote:
> I've been working on a set of patches to synchronize the version
> numbers across everything produced by btrfs-progs and add pkgconfig
> files to make it easier for software to consume the libraries, but as
> I worked through it, I started realizing that btrfs-progs actually has
> *two* distinctly separate projects in one package.

Yes, we ended up doing it like that as it makes development easier. The
util library is there for 3rd party tools to use but otherwise is's
internally used by btrfs-progs too. Synchronizing btrfs-progs
development that would need new interfaces from the util library would
require separate release and deployment.

> This is somewhat problematic since it makes properly representing that
> versioning in the packages I ship in Fedora rather difficult.
> 
> Would it be possible to consider splitting libbtrfsutil into its own
> project with its own releases?

There's probably not a hard reason why to have them both in one project,
but like it is now is saving my time spent on annoying tasks.

> If not, is there a reason that we
> *can't* synchronize the versions across btrfs-progs, libbtrfs, and
> libbtrfsutil? We could still make sure that the library sonames are
> versioned properly, but having the user-facing versions actually sync
> up makes life a lot easier...

I'm a bit lost in which versions are not in sync. The shared library
version is an ABI and that changes only when new symbols appear. The
whole project release versions follow the kernel scheme. So what are you
suggesting? Eg. that libbtrfsutil should not be 5.9 but follow the
soname version which is 1.2.0?

Can this be solved on the packaging side by 3 projects that are built
from the same sources but produce only a subset of all the built files?
This is slightly inefficient as you'd have 3 copies of btrfs-progs.tar
but otherwise it's one time job, comparted to me having to update and
deploy 2 projects just to progress with progs development?

Re: btrfs-progs and libbtrfsutil versioning

[Neal Gompa]

On Fri, Oct 2, 2020 at 1:19 PM David Sterba <dsterba@suse.cz> wrote:
>
> On Wed, Sep 23, 2020 at 09:48:44PM -0400, Neal Gompa wrote:
> > I've been working on a set of patches to synchronize the version
> > numbers across everything produced by btrfs-progs and add pkgconfig
> > files to make it easier for software to consume the libraries, but as
> > I worked through it, I started realizing that btrfs-progs actually has
> > *two* distinctly separate projects in one package.
>
> Yes, we ended up doing it like that as it makes development easier. The
> util library is there for 3rd party tools to use but otherwise is's
> internally used by btrfs-progs too. Synchronizing btrfs-progs
> development that would need new interfaces from the util library would
> require separate release and deployment.
>
> > This is somewhat problematic since it makes properly representing that
> > versioning in the packages I ship in Fedora rather difficult.
> >
> > Would it be possible to consider splitting libbtrfsutil into its own
> > project with its own releases?
>
> There's probably not a hard reason why to have them both in one project,
> but like it is now is saving my time spent on annoying tasks.
>

That's fine with me.

> > If not, is there a reason that we
> > *can't* synchronize the versions across btrfs-progs, libbtrfs, and
> > libbtrfsutil? We could still make sure that the library sonames are
> > versioned properly, but having the user-facing versions actually sync
> > up makes life a lot easier...
>
> I'm a bit lost in which versions are not in sync. The shared library
> version is an ABI and that changes only when new symbols appear. The
> whole project release versions follow the kernel scheme. So what are you
> suggesting? Eg. that libbtrfsutil should not be 5.9 but follow the
> soname version which is 1.2.0?
>

No, I'm suggesting the other way around. The libbtrfsutil "public
version" (that is, not the ABI version) would be synchronized with the
kernel version like the rest of btrfs-progs. Right now it's not,
evidenced by python-btrfsutil giving me 1.2.0 instead of 5.9.

Basically, I want to add pkgconfig files and fix the version emitted
by the Python metadata to use the project version rather than whatever
soname version is used. The soname version would still be preserved
and work as a way to track the ABI changes, but everything that reads
metadata would get the btrfs-progs parent version consistently.

> Can this be solved on the packaging side by 3 projects that are built
> from the same sources but produce only a subset of all the built files?
> This is slightly inefficient as you'd have 3 copies of btrfs-progs.tar
> but otherwise it's one time job, comparted to me having to update and
> deploy 2 projects just to progress with progs development?

It _could_, but I'd rather avoid that if I can, because it makes
keeping everything up to date painful.


-- 
真実はいつも一つ！/ Always, there's only one truth!

Re: btrfs-progs and libbtrfsutil versioning

[David Sterba]

On Fri, Oct 02, 2020 at 04:56:24PM -0400, Neal Gompa wrote:
> On Fri, Oct 2, 2020 at 1:19 PM David Sterba <dsterba@suse.cz> wrote:
> > On Wed, Sep 23, 2020 at 09:48:44PM -0400, Neal Gompa wrote:
> > > If not, is there a reason that we
> > > *can't* synchronize the versions across btrfs-progs, libbtrfs, and
> > > libbtrfsutil? We could still make sure that the library sonames are
> > > versioned properly, but having the user-facing versions actually sync
> > > up makes life a lot easier...
> >
> > I'm a bit lost in which versions are not in sync. The shared library
> > version is an ABI and that changes only when new symbols appear. The
> > whole project release versions follow the kernel scheme. So what are you
> > suggesting? Eg. that libbtrfsutil should not be 5.9 but follow the
> > soname version which is 1.2.0?
> 
> No, I'm suggesting the other way around. The libbtrfsutil "public
> version" (that is, not the ABI version) would be synchronized with the
> kernel version like the rest of btrfs-progs. Right now it's not,
> evidenced by python-btrfsutil giving me 1.2.0 instead of 5.9.

Oh, so this sounds easier than I thought. If the "main" verions is
derived from progs, then it's just the python-btrfsutil that needs to be
fixed:

libbtrfsutil/python/setup.py:

 28 def get_version():
 29     with open('../btrfsutil.h', 'r') as f:
 30         btrfsutil_h = f.read()
 31     major = re.search(r'^#define BTRFS_UTIL_VERSION_MAJOR ([0-9]+)$',
 32                       btrfsutil_h, flags=re.MULTILINE).group(1)
 33     minor = re.search(r'^#define BTRFS_UTIL_VERSION_MINOR ([0-9]+)$',
 34                       btrfsutil_h, flags=re.MULTILINE).group(1)
 35     patch = re.search(r'^#define BTRFS_UTIL_VERSION_PATCH ([0-9]+)$',
 36                       btrfsutil_h, flags=re.MULTILINE).group(1)
 37     return major + '.' + minor + '.' + patch

IOW, the python package version is derived from the library .so version.

> Basically, I want to add pkgconfig files and fix the version emitted
> by the Python metadata to use the project version rather than whatever
> soname version is used. The soname version would still be preserved
> and work as a way to track the ABI changes, but everything that reads
> metadata would get the btrfs-progs parent version consistently.

And with something using the VERSION file instead it should do what you
expect, right?

Re: btrfs-progs and libbtrfsutil versioning

[Neal Gompa]

On Mon, Oct 12, 2020 at 6:51 PM David Sterba <dsterba@suse.cz> wrote:
>
> On Fri, Oct 02, 2020 at 04:56:24PM -0400, Neal Gompa wrote:
> > On Fri, Oct 2, 2020 at 1:19 PM David Sterba <dsterba@suse.cz> wrote:
> > > On Wed, Sep 23, 2020 at 09:48:44PM -0400, Neal Gompa wrote:
> > > > If not, is there a reason that we
> > > > *can't* synchronize the versions across btrfs-progs, libbtrfs, and
> > > > libbtrfsutil? We could still make sure that the library sonames are
> > > > versioned properly, but having the user-facing versions actually sync
> > > > up makes life a lot easier...
> > >
> > > I'm a bit lost in which versions are not in sync. The shared library
> > > version is an ABI and that changes only when new symbols appear. The
> > > whole project release versions follow the kernel scheme. So what are you
> > > suggesting? Eg. that libbtrfsutil should not be 5.9 but follow the
> > > soname version which is 1.2.0?
> >
> > No, I'm suggesting the other way around. The libbtrfsutil "public
> > version" (that is, not the ABI version) would be synchronized with the
> > kernel version like the rest of btrfs-progs. Right now it's not,
> > evidenced by python-btrfsutil giving me 1.2.0 instead of 5.9.
>
> Oh, so this sounds easier than I thought. If the "main" verions is
> derived from progs, then it's just the python-btrfsutil that needs to be
> fixed:
>
> libbtrfsutil/python/setup.py:
>
>  28 def get_version():
>  29     with open('../btrfsutil.h', 'r') as f:
>  30         btrfsutil_h = f.read()
>  31     major = re.search(r'^#define BTRFS_UTIL_VERSION_MAJOR ([0-9]+)$',
>  32                       btrfsutil_h, flags=re.MULTILINE).group(1)
>  33     minor = re.search(r'^#define BTRFS_UTIL_VERSION_MINOR ([0-9]+)$',
>  34                       btrfsutil_h, flags=re.MULTILINE).group(1)
>  35     patch = re.search(r'^#define BTRFS_UTIL_VERSION_PATCH ([0-9]+)$',
>  36                       btrfsutil_h, flags=re.MULTILINE).group(1)
>  37     return major + '.' + minor + '.' + patch
>
> IOW, the python package version is derived from the library .so version.
>
> > Basically, I want to add pkgconfig files and fix the version emitted
> > by the Python metadata to use the project version rather than whatever
> > soname version is used. The soname version would still be preserved
> > and work as a way to track the ABI changes, but everything that reads
> > metadata would get the btrfs-progs parent version consistently.
>
> And with something using the VERSION file instead it should do what you
> expect, right?

Yeah, basically! This issue blocks me from enabling the Python
bindings because version updates get weird and confusing...


-- 
真実はいつも一つ！/ Always, there's only one truth!

Re: btrfs-progs and libbtrfsutil versioning

[David Sterba]

On Mon, Oct 12, 2020 at 08:16:36PM -0400, Neal Gompa wrote:
> On Mon, Oct 12, 2020 at 6:51 PM David Sterba <dsterba@suse.cz> wrote:
> > On Fri, Oct 02, 2020 at 04:56:24PM -0400, Neal Gompa wrote:
> > > On Fri, Oct 2, 2020 at 1:19 PM David Sterba <dsterba@suse.cz> wrote:
> > > > On Wed, Sep 23, 2020 at 09:48:44PM -0400, Neal Gompa wrote:
> >  33     minor = re.search(r'^#define BTRFS_UTIL_VERSION_MINOR ([0-9]+)$',
> >  34                       btrfsutil_h, flags=re.MULTILINE).group(1)
> >  35     patch = re.search(r'^#define BTRFS_UTIL_VERSION_PATCH ([0-9]+)$',
> >  36                       btrfsutil_h, flags=re.MULTILINE).group(1)
> >  37     return major + '.' + minor + '.' + patch
> >
> > IOW, the python package version is derived from the library .so version.
> >
> > > Basically, I want to add pkgconfig files and fix the version emitted
> > > by the Python metadata to use the project version rather than whatever
> > > soname version is used. The soname version would still be preserved
> > > and work as a way to track the ABI changes, but everything that reads
> > > metadata would get the btrfs-progs parent version consistently.
> >
> > And with something using the VERSION file instead it should do what you
> > expect, right?
> 
> Yeah, basically! This issue blocks me from enabling the Python
> bindings because version updates get weird and confusing...

So the python will now follow the package version. One thing I'm not
sure about is the pkg-config .pc file for libbtrfsutil: is the Version:
supposed to be the library ABI or the main package as well? Right now
it's the ABI.

Re: btrfs-progs and libbtrfsutil versioning

[Neal Gompa]

On Mon, Oct 19, 2020 at 2:34 PM David Sterba <dsterba@suse.cz> wrote:
>
> On Mon, Oct 12, 2020 at 08:16:36PM -0400, Neal Gompa wrote:
> > On Mon, Oct 12, 2020 at 6:51 PM David Sterba <dsterba@suse.cz> wrote:
> > > On Fri, Oct 02, 2020 at 04:56:24PM -0400, Neal Gompa wrote:
> > > > On Fri, Oct 2, 2020 at 1:19 PM David Sterba <dsterba@suse.cz> wrote:
> > > > > On Wed, Sep 23, 2020 at 09:48:44PM -0400, Neal Gompa wrote:
> > >  33     minor = re.search(r'^#define BTRFS_UTIL_VERSION_MINOR ([0-9]+)$',
> > >  34                       btrfsutil_h, flags=re.MULTILINE).group(1)
> > >  35     patch = re.search(r'^#define BTRFS_UTIL_VERSION_PATCH ([0-9]+)$',
> > >  36                       btrfsutil_h, flags=re.MULTILINE).group(1)
> > >  37     return major + '.' + minor + '.' + patch
> > >
> > > IOW, the python package version is derived from the library .so version.
> > >
> > > > Basically, I want to add pkgconfig files and fix the version emitted
> > > > by the Python metadata to use the project version rather than whatever
> > > > soname version is used. The soname version would still be preserved
> > > > and work as a way to track the ABI changes, but everything that reads
> > > > metadata would get the btrfs-progs parent version consistently.
> > >
> > > And with something using the VERSION file instead it should do what you
> > > expect, right?
> >
> > Yeah, basically! This issue blocks me from enabling the Python
> > bindings because version updates get weird and confusing...
>
> So the python will now follow the package version. One thing I'm not
> sure about is the pkg-config .pc file for libbtrfsutil: is the Version:
> supposed to be the library ABI or the main package as well? Right now
> it's the ABI.

Main package version too. That's the "user facing" version. Not the ABI version.


-- 
真実はいつも一つ！/ Always, there's only one truth!