pahole and libdwarves linking/versioning issues

pahole and libdwarves linking/versioning issues

[Andrii Nakryiko]

Hey Arnaldo,

We use pahole pretty extensively here at Facebook, as you might
imagine, given it's a required tool to compile Linux kernel with BTF.

There are a few problems we periodically run into, all of which seems
to be related to the fact that pahole is just a thin wrapper binary on
top of libdwarves shared library.

One very common problem is when we upgrade the dwarves package from
one minor version to another one. Typically people would do

yum upgrade dwarves

And that would succeed to update the dwarves package (say 1.13 to
1.16). But when you run

pahole --version

You'll still see 1.13 (and still have 1.13 functionality, of
course)... So what happens is that pahole *the binary* was updated,
but libdwarves1 package, which is a dependency of dwarves package,
wasn't updated. And the actual version string all pretty much all of
the functionality is coming from libdwarves1. So users have to

yum upgrade libdwarves1

explicitly. Which is, of course, not an obvious and frustrating experience.

Also, whenever pahole is using some new API from libdwarves and
libdwarves is outdated, that results in spectacular failure during
dynamic linking time. Which makes sense because libdwarves doesn't
maintain API stability per se and is supposed to match 1:1 w/ pahole
version. But it all causes real logistics issues.

So I have a few questions:
1. Is there anything on the RPM spec side that could be done to
trigger libdwarves1 update when dwarves is updated and keep them in
sync down to the minor version?
2. If not, can we add an option of statically linking libdwarves into
pahole and other tools? That would eliminate all the problems
altogether at the expense of a pretty negligible slight increase in
the binary sizes.

Thanks!

-- Andrii

Re: pahole and libdwarves linking/versioning issues

[Arnaldo]

On November 10, 2020 9:14:16 PM GMT-03:00, Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
>Hey Arnaldo,
>
>We use pahole pretty extensively here at Facebook, as you might
>imagine, given it's a required tool to compile Linux kernel with BTF.
>
>There are a few problems we periodically run into, all of which seems
>to be related to the fact that pahole is just a thin wrapper binary on
>top of libdwarves shared library.
>
>One very common problem is when we upgrade the dwarves package from
>one minor version to another one. Typically people would do
>
>yum upgrade dwarves


I'll reply to this more thoroughly tomorrow morning.

But yes, the spec file um the pahole git repo has a explicit Requires: tag to ask for the latest libdwarves.

- Arnaldo

>
>And that would succeed to update the dwarves package (say 1.13 to
>1.16). But when you run
>
>pahole --version
>
>You'll still see 1.13 (and still have 1.13 functionality, of

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: pahole and libdwarves linking/versioning issues

[Arnaldo Carvalho de Melo]

Em Tue, Nov 10, 2020 at 04:14:16PM -0800, Andrii Nakryiko escreveu:
> Hey Arnaldo,

Hi Andrii,
 
> We use pahole pretty extensively here at Facebook, as you might
> imagine, given it's a required tool to compile Linux kernel with BTF.
 
> There are a few problems we periodically run into, all of which seems
> to be related to the fact that pahole is just a thin wrapper binary on
> top of libdwarves shared library.
 
> One very common problem is when we upgrade the dwarves package from
> one minor version to another one. Typically people would do
 
> yum upgrade dwarves
 
> And that would succeed to update the dwarves package (say 1.13 to
> 1.16). But when you run
 
> pahole --version
 
> You'll still see 1.13 (and still have 1.13 functionality, of
> course)... So what happens is that pahole *the binary* was updated,
> but libdwarves1 package, which is a dependency of dwarves package,
> wasn't updated. And the actual version string all pretty much all of
> the functionality is coming from libdwarves1. So users have to
 
> yum upgrade libdwarves1
 
> explicitly. Which is, of course, not an obvious and frustrating experience.
 
> Also, whenever pahole is using some new API from libdwarves and
> libdwarves is outdated, that results in spectacular failure during
> dynamic linking time. Which makes sense because libdwarves doesn't
> maintain API stability per se and is supposed to match 1:1 w/ pahole
> version. But it all causes real logistics issues.
 
> So I have a few questions:
> 1. Is there anything on the RPM spec side that could be done to
> trigger libdwarves1 update when dwarves is updated and keep them in
> sync down to the minor version?

So, there is this in the reference .spec file that is in pahole's git
repo:

[acme@five pahole]$ egrep '^(Requires|Name)' rpm/SPECS/dwarves.spec
Name: dwarves
Requires: %{libname}%{libver} = %{version}-%{release}
Requires: %{libname}%{libver} = %{version}-%{release}
[acme@five pahole]$ egrep '^(Requires|Name|%package)' rpm/SPECS/dwarves.spec
Name: dwarves
Requires: %{libname}%{libver} = %{version}-%{release}
%package -n %{libname}%{libver}
%package -n %{libname}%{libver}-devel
Requires: %{libname}%{libver} = %{version}-%{release}
[acme@five pahole]$

Someone else also reported this recently... Fedora has:

[acme@five pahole]$ rpm -qR dwarves | grep ^libdwarves1
libdwarves1 = 1.17-4.fc33
[acme@five pahole]$

So it includes both ${version} and %{release}, which should get it in
lockstep.

Having said that I'm not aware of anything using libdwarves and it was
intended just to share those among the tools that comes in the
'dwarves' package, so I should have really made it internal, installed
in /usr/libexec/dwarves/ :-\

> 2. If not, can we add an option of statically linking libdwarves into
> pahole and other tools? That would eliminate all the problems
> altogether at the expense of a pretty negligible slight increase in
> the binary sizes.

Yeah, this looks like the way to proceed, i.e. we statically link it
while keeping the .so around in case someone uses it, while adding some
warning that that mode is deprecated and is going away.

- Arnaldo

Re: pahole and libdwarves linking/versioning issues

[Andrii Nakryiko]

On Fri, Nov 13, 2020 at 3:45 AM Arnaldo Carvalho de Melo
<arnaldo.melo@gmail.com> wrote:
>
> Em Tue, Nov 10, 2020 at 04:14:16PM -0800, Andrii Nakryiko escreveu:
> > Hey Arnaldo,
>
> Hi Andrii,
>
> > We use pahole pretty extensively here at Facebook, as you might
> > imagine, given it's a required tool to compile Linux kernel with BTF.
>
> > There are a few problems we periodically run into, all of which seems
> > to be related to the fact that pahole is just a thin wrapper binary on
> > top of libdwarves shared library.
>
> > One very common problem is when we upgrade the dwarves package from
> > one minor version to another one. Typically people would do
>
> > yum upgrade dwarves
>
> > And that would succeed to update the dwarves package (say 1.13 to
> > 1.16). But when you run
>
> > pahole --version
>
> > You'll still see 1.13 (and still have 1.13 functionality, of
> > course)... So what happens is that pahole *the binary* was updated,
> > but libdwarves1 package, which is a dependency of dwarves package,
> > wasn't updated. And the actual version string all pretty much all of
> > the functionality is coming from libdwarves1. So users have to
>
> > yum upgrade libdwarves1
>
> > explicitly. Which is, of course, not an obvious and frustrating experience.
>
> > Also, whenever pahole is using some new API from libdwarves and
> > libdwarves is outdated, that results in spectacular failure during
> > dynamic linking time. Which makes sense because libdwarves doesn't
> > maintain API stability per se and is supposed to match 1:1 w/ pahole
> > version. But it all causes real logistics issues.
>
> > So I have a few questions:
> > 1. Is there anything on the RPM spec side that could be done to
> > trigger libdwarves1 update when dwarves is updated and keep them in
> > sync down to the minor version?
>
> So, there is this in the reference .spec file that is in pahole's git
> repo:
>
> [acme@five pahole]$ egrep '^(Requires|Name)' rpm/SPECS/dwarves.spec
> Name: dwarves
> Requires: %{libname}%{libver} = %{version}-%{release}
> Requires: %{libname}%{libver} = %{version}-%{release}
> [acme@five pahole]$ egrep '^(Requires|Name|%package)' rpm/SPECS/dwarves.spec
> Name: dwarves
> Requires: %{libname}%{libver} = %{version}-%{release}
> %package -n %{libname}%{libver}
> %package -n %{libname}%{libver}-devel
> Requires: %{libname}%{libver} = %{version}-%{release}

Ok, so I would have never noticed it, but Matt Mullins has an eagle
eye! This Requirement applies only to libdwarves1-devel package!

%package -n %{libname}%{libver} doesn't have this requirement!

Arnaldo, if you can roll it into 1.19 along the changelog timestamp
fix, that would be awesome!

> [acme@five pahole]$
>
> Someone else also reported this recently... Fedora has:
>
> [acme@five pahole]$ rpm -qR dwarves | grep ^libdwarves1
> libdwarves1 = 1.17-4.fc33
> [acme@five pahole]$
>
> So it includes both ${version} and %{release}, which should get it in
> lockstep.
>
> Having said that I'm not aware of anything using libdwarves and it was
> intended just to share those among the tools that comes in the
> 'dwarves' package, so I should have really made it internal, installed
> in /usr/libexec/dwarves/ :-\
>
> > 2. If not, can we add an option of statically linking libdwarves into
> > pahole and other tools? That would eliminate all the problems
> > altogether at the expense of a pretty negligible slight increase in
> > the binary sizes.
>
> Yeah, this looks like the way to proceed, i.e. we statically link it
> while keeping the .so around in case someone uses it, while adding some
> warning that that mode is deprecated and is going away.
>
> - Arnaldo