[PATCH v3] glibc: move ld.so.conf back to main package

[PATCH v3] glibc: move ld.so.conf back to main package

[Rasmus Villemoes]

There are cases where one doesn't want ldconfig on target (e.g. for
read-only root filesystems, it's rather pointless), yet one still
needs ld.so.conf to be present at image build time:

When some recipe installs libraries to a non-standard location, and
dutifully drops in a file in /etc/ld.so.conf.d/foo.conf, we need the
ld.so.conf containing the

  include /etc/ld.so.conf.d/*.conf

stanza to get those other locations picked up.

So change the packaging logic so that there's always an ld.so.conf
present when the build-time ldconfig runs.

The ld.so.conf and ld.so.conf.d/*.conf files don't take up much
room (at least not compared to the 700K binary ldconfig), and they
might be needed in case ldconfig is installable, so leave them
alone.

In case of a read-only rootfs, one could add some logic to remove them
if one really wants to shave those few dozens of bytes off.

While here, fix typos in the bb.note (add spaces) so one can just
copy-paste the line from the log-file and redo the command.

Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
---

v3: rebase to master

v2: don't delete the configuration file(s) - that's also a much
simpler patch.

 meta/lib/oe/rootfs.py                     | 2 +-
 meta/recipes-core/glibc/glibc-package.inc | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index cd65e62030..a0ac33ada6 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -297,7 +297,7 @@ class Rootfs(object, metaclass=ABCMeta):
 
     def _run_ldconfig(self):
         if self.d.getVar('LDCONFIGDEPEND'):
-            bb.note("Executing: ldconfig -r" + self.image_rootfs + "-c new -v")
+            bb.note("Executing: ldconfig -r " + self.image_rootfs + " -c new -v")
             self._exec_shell_cmd(['ldconfig', '-r', self.image_rootfs, '-c',
                                   'new', '-v'])
 
diff --git a/meta/recipes-core/glibc/glibc-package.inc b/meta/recipes-core/glibc/glibc-package.inc
index ff25fd4187..5f1ab60dc6 100644
--- a/meta/recipes-core/glibc/glibc-package.inc
+++ b/meta/recipes-core/glibc/glibc-package.inc
@@ -23,9 +23,9 @@ ARCH_DYNAMIC_LOADER_aarch64 = "ld-linux-${TARGET_ARCH}.so.1"
 libc_baselibs_append = " ${@oe.utils.conditional('ARCH_DYNAMIC_LOADER', '', '', '${root_prefix}/lib/${ARCH_DYNAMIC_LOADER}', d)}"
 INSANE_SKIP_${PN}_append_aarch64 = " libdir"
 
-FILES_${PN} = "${libc_baselibs} ${libexecdir}/*"
+FILES_${PN} = "${libc_baselibs} ${libexecdir}/* ${sysconfdir}/ld.so.conf"
 RRECOMMENDS_${PN} = "${@bb.utils.contains('DISTRO_FEATURES', 'ldconfig', '${MLPREFIX}ldconfig', '', d)}"
-FILES_ldconfig = "${base_sbindir}/ldconfig ${sysconfdir}/ld.so.conf"
+FILES_ldconfig = "${base_sbindir}/ldconfig"
 FILES_ldd = "${bindir}/ldd"
 FILES_libsegfault = "${base_libdir}/libSegFault*"
 FILES_libcidn = "${base_libdir}/libcidn-*.so ${base_libdir}/libcidn.so.*"
-- 
2.23.0

Re: [PATCH v3] glibc: move ld.so.conf back to main package

[Khem Raj]

On Tue, Jun 2, 2020 at 5:17 AM Rasmus Villemoes
<rasmus.villemoes@prevas.dk> wrote:
>
> There are cases where one doesn't want ldconfig on target (e.g. for
> read-only root filesystems, it's rather pointless), yet one still
> needs ld.so.conf to be present at image build time:
>
> When some recipe installs libraries to a non-standard location, and
> dutifully drops in a file in /etc/ld.so.conf.d/foo.conf, we need the
> ld.so.conf containing the
>
>   include /etc/ld.so.conf.d/*.conf
>
> stanza to get those other locations picked up.
>
> So change the packaging logic so that there's always an ld.so.conf
> present when the build-time ldconfig runs.
>
> The ld.so.conf and ld.so.conf.d/*.conf files don't take up much
> room (at least not compared to the 700K binary ldconfig), and they
> might be needed in case ldconfig is installable, so leave them
> alone.
>
> In case of a read-only rootfs, one could add some logic to remove them
> if one really wants to shave those few dozens of bytes off.
>
> While here, fix typos in the bb.note (add spaces) so one can just
> copy-paste the line from the log-file and redo the command.
>

The patch, in essence, looks ok, have you considered the feeds based
distro's case
will it keep working for them, or if there are changes e.g. they must
include ldconfig
compulsory etc. then it should be documented.

> Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
> ---
>
> v3: rebase to master
>
> v2: don't delete the configuration file(s) - that's also a much
> simpler patch.
>
>  meta/lib/oe/rootfs.py                     | 2 +-
>  meta/recipes-core/glibc/glibc-package.inc | 4 ++--
>  2 files changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
> index cd65e62030..a0ac33ada6 100644
> --- a/meta/lib/oe/rootfs.py
> +++ b/meta/lib/oe/rootfs.py
> @@ -297,7 +297,7 @@ class Rootfs(object, metaclass=ABCMeta):
>
>      def _run_ldconfig(self):
>          if self.d.getVar('LDCONFIGDEPEND'):
> -            bb.note("Executing: ldconfig -r" + self.image_rootfs + "-c new -v")
> +            bb.note("Executing: ldconfig -r " + self.image_rootfs + " -c new -v")
>              self._exec_shell_cmd(['ldconfig', '-r', self.image_rootfs, '-c',
>                                    'new', '-v'])
>
> diff --git a/meta/recipes-core/glibc/glibc-package.inc b/meta/recipes-core/glibc/glibc-package.inc
> index ff25fd4187..5f1ab60dc6 100644
> --- a/meta/recipes-core/glibc/glibc-package.inc
> +++ b/meta/recipes-core/glibc/glibc-package.inc
> @@ -23,9 +23,9 @@ ARCH_DYNAMIC_LOADER_aarch64 = "ld-linux-${TARGET_ARCH}.so.1"
>  libc_baselibs_append = " ${@oe.utils.conditional('ARCH_DYNAMIC_LOADER', '', '', '${root_prefix}/lib/${ARCH_DYNAMIC_LOADER}', d)}"
>  INSANE_SKIP_${PN}_append_aarch64 = " libdir"
>
> -FILES_${PN} = "${libc_baselibs} ${libexecdir}/*"
> +FILES_${PN} = "${libc_baselibs} ${libexecdir}/* ${sysconfdir}/ld.so.conf"
>  RRECOMMENDS_${PN} = "${@bb.utils.contains('DISTRO_FEATURES', 'ldconfig', '${MLPREFIX}ldconfig', '', d)}"
> -FILES_ldconfig = "${base_sbindir}/ldconfig ${sysconfdir}/ld.so.conf"
> +FILES_ldconfig = "${base_sbindir}/ldconfig"
>  FILES_ldd = "${bindir}/ldd"
>  FILES_libsegfault = "${base_libdir}/libSegFault*"
>  FILES_libcidn = "${base_libdir}/libcidn-*.so ${base_libdir}/libcidn.so.*"
> --
> 2.23.0
>

Re: [PATCH v3] glibc: move ld.so.conf back to main package

[Rasmus Villemoes]

On 02/06/2020 19.58, Khem Raj wrote:
> On Tue, Jun 2, 2020 at 5:17 AM Rasmus Villemoes
> <rasmus.villemoes@prevas.dk> wrote:
>>
>> There are cases where one doesn't want ldconfig on target (e.g. for
>> read-only root filesystems, it's rather pointless), yet one still
>> needs ld.so.conf to be present at image build time:
>>
>> When some recipe installs libraries to a non-standard location, and
>> dutifully drops in a file in /etc/ld.so.conf.d/foo.conf, we need the
>> ld.so.conf containing the
>>
>>   include /etc/ld.so.conf.d/*.conf
>>
>> stanza to get those other locations picked up.
>>
>> So change the packaging logic so that there's always an ld.so.conf
>> present when the build-time ldconfig runs.
>>
>> The ld.so.conf and ld.so.conf.d/*.conf files don't take up much
>> room (at least not compared to the 700K binary ldconfig), and they
>> might be needed in case ldconfig is installable, so leave them
>> alone.
>>
>> In case of a read-only rootfs, one could add some logic to remove them
>> if one really wants to shave those few dozens of bytes off.
>>
>> While here, fix typos in the bb.note (add spaces) so one can just
>> copy-paste the line from the log-file and redo the command.
>>
> 
> The patch, in essence, looks ok, have you considered the feeds based
> distro's case
> will it keep working for them, or if there are changes e.g. they must
> include ldconfig
> compulsory etc. then it should be documented.

I don't think I understand the question. All I'm doing (now) is making
sure the rootfs always (well, it comes bundled with glibc, so for all
glibc-based systems) has a ld.so.conf. If there's an update of that
ld.so.conf, that will change the hash of the glibc recipe (it's a
SRC_URI) and hence of both the glibc and ldconfig packages. A change of
the provided ld.so.conf will lead to glibc being rebuilt (no?), which I
expect in turn will cause glibc on target to (need to) be updated (no?).
That seems to be the case with or without this patch.

But maybe I have no clue what a "feeds based distro" is, I've only ever
done wholesale replace-the-entire-image (in an A/B setup).

Rasmus

Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package

[Phil Blundell]

On Tue, Jun 02, 2020 at 02:17:22PM +0200, Rasmus Villemoes wrote:
> In case of a read-only rootfs, one could add some logic to remove them
> if one really wants to shave those few dozens of bytes off.

Note that there's also a slight runtime performance penalty associated
with having these files in place if they're not doing anything useful.
It'd be nice to try to make sure that nothing regresses in that area.

p.

Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package

[Richard Purdie]

On Tue, 2020-06-02 at 22:00 +0200, Phil Blundell wrote:
> On Tue, Jun 02, 2020 at 02:17:22PM +0200, Rasmus Villemoes wrote:
> > In case of a read-only rootfs, one could add some logic to remove
> > them
> > if one really wants to shave those few dozens of bytes off.
> 
> Note that there's also a slight runtime performance penalty
> associated
> with having these files in place if they're not doing anything
> useful.
> It'd be nice to try to make sure that nothing regresses in that area.

I understand the concern, I am a little torn on this as adding in too
many different controls and options complicates the test matrix and
makes things harder for users.

You're effectively suggesting a new DISTRO_FEATURE to control this? or
maybe better, perhaps a glibc PACKAGECONFIG?

Cheers,

Richard

Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package

[Phil Blundell]

On Tue, Jun 02, 2020 at 09:17:44PM +0100, Richard Purdie wrote:
> I understand the concern, I am a little torn on this as adding in too
> many different controls and options complicates the test matrix and
> makes things harder for users.
> 
> You're effectively suggesting a new DISTRO_FEATURE to control this? or
> maybe better, perhaps a glibc PACKAGECONFIG?

Yes, right.  I don't think it need be a DISTRO_FEATURE because nothing
outside glibc needs to care.

Here are the scenarios that I think matter, ordered from simplest to
most complex:

1. Immutable filesystem, every library installed in the place where
ld.so would first look for it anyway (i.e. everything in {/usr}/lib).
In this case, we don't want ldconfig (because it can never do anything
useful), we don't want ld.so.cache and we don't want ld.so.conf
because they would cause ld.so to do extra file loads and computation
but end up with the same result that it would anyway.

2. Immutable filesystem but some libraries are in places that ld.so
wouldn't automatically know about.  In this case we do want ld.so.conf
and ld.so.cache, but we still don't want ldconfig.

3. Mutable filesystem where arbitrary binaries can be installed in
arbitrary places.  It's probably debatable whether ldconfig is needed
in all these cases, but clearly it's needed in some and I think at
this point it can be left to a DISTRO decision how exactly they want
to optimize things.

I think right now oe-core supports #1 and #3.  The proposed patch
seems to be aimed at #2, which is a completely valid usecase, but
my concern is that we don't want to pessimize #1 in the process.

There is a whole parallel discussion to be had about the merits
or otherwise of ld.so's hwcaps mechanism.  meta-micro has been
carrying a patch for about a decade to turn all that stuff off,
on the grounds that it's just a loss on any kind of embedded
environment.  Maybe I should make some effort to get it upstream
as an option.

p.

Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package

[Andre McCurdy]

On Tue, Jun 2, 2020 at 2:15 PM Phil Blundell via
lists.openembedded.org <pb=pbcl.net@lists.openembedded.org> wrote:
>
> On Tue, Jun 02, 2020 at 09:17:44PM +0100, Richard Purdie wrote:
> > I understand the concern, I am a little torn on this as adding in too
> > many different controls and options complicates the test matrix and
> > makes things harder for users.
> >
> > You're effectively suggesting a new DISTRO_FEATURE to control this? or
> > maybe better, perhaps a glibc PACKAGECONFIG?
>
> Yes, right.  I don't think it need be a DISTRO_FEATURE because nothing
> outside glibc needs to care.
>
> Here are the scenarios that I think matter, ordered from simplest to
> most complex:
>
> 1. Immutable filesystem, every library installed in the place where
> ld.so would first look for it anyway (i.e. everything in {/usr}/lib).
> In this case, we don't want ldconfig (because it can never do anything
> useful), we don't want ld.so.cache and we don't want ld.so.conf
> because they would cause ld.so to do extra file loads and computation
> but end up with the same result that it would anyway.
>
> 2. Immutable filesystem but some libraries are in places that ld.so
> wouldn't automatically know about.  In this case we do want ld.so.conf
> and ld.so.cache, but we still don't want ldconfig.
>
> 3. Mutable filesystem where arbitrary binaries can be installed in
> arbitrary places.  It's probably debatable whether ldconfig is needed
> in all these cases, but clearly it's needed in some and I think at
> this point it can be left to a DISTRO decision how exactly they want
> to optimize things.
>
> I think right now oe-core supports #1 and #3.  The proposed patch
> seems to be aimed at #2, which is a completely valid usecase,

It seems like a weird corner case to me. Where do these libraries come
from and why can't they be moved or symlinked into a standard path?

If they are somehow special and only used by special applications then
setting rpath or using LD_LIBRARY_PATH just for those applications
would seem to be a better solution than enabling them globally.

> but
> my concern is that we don't want to pessimize #1 in the process.
>
> There is a whole parallel discussion to be had about the merits
> or otherwise of ld.so's hwcaps mechanism.  meta-micro has been
> carrying a patch for about a decade to turn all that stuff off,
> on the grounds that it's just a loss on any kind of embedded
> environment.  Maybe I should make some effort to get it upstream
> as an option.

Note also that glibc already has a build time option to completely
disable support for ldconfig:

  https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/configure.ac#l89

Sadly it's not usable in recent versions of glibc (setting
use_ldconfig=no results in a build failure due to some seemingly
unrelated missing build dependency which I haven't had time to dig
into yet) but it worked well for OE 2.2 and gave a noticeable
reduction in glibc code size.

Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package

[Rasmus Villemoes]

On 02/06/2020 23.46, Andre McCurdy wrote:
> On Tue, Jun 2, 2020 at 2:15 PM Phil Blundell via
> lists.openembedded.org <pb=pbcl.net@lists.openembedded.org> wrote:
>>
>> On Tue, Jun 02, 2020 at 09:17:44PM +0100, Richard Purdie wrote:
>>> I understand the concern, I am a little torn on this as adding in too
>>> many different controls and options complicates the test matrix and
>>> makes things harder for users.
>>>
>>> You're effectively suggesting a new DISTRO_FEATURE to control this? or
>>> maybe better, perhaps a glibc PACKAGECONFIG?
>>
>> Yes, right.  I don't think it need be a DISTRO_FEATURE because nothing
>> outside glibc needs to care.
>>
>> Here are the scenarios that I think matter, ordered from simplest to
>> most complex:
>>
>> 1. Immutable filesystem, every library installed in the place where
>> ld.so would first look for it anyway (i.e. everything in {/usr}/lib).
>> In this case, we don't want ldconfig (because it can never do anything
>> useful), we don't want ld.so.cache and we don't want ld.so.conf
>> because they would cause ld.so to do extra file loads and computation
>> but end up with the same result that it would anyway.
>>
>> 2. Immutable filesystem but some libraries are in places that ld.so
>> wouldn't automatically know about.  In this case we do want ld.so.conf
>> and ld.so.cache, but we still don't want ldconfig.
>>
>> 3. Mutable filesystem where arbitrary binaries can be installed in
>> arbitrary places.  It's probably debatable whether ldconfig is needed
>> in all these cases, but clearly it's needed in some and I think at
>> this point it can be left to a DISTRO decision how exactly they want
>> to optimize things.
>>
>> I think right now oe-core supports #1 and #3.  The proposed patch
>> seems to be aimed at #2, which is a completely valid usecase,
> 
> It seems like a weird corner case to me. Where do these libraries come
> from and why can't they be moved or symlinked into a standard path?

Pre-compiled proprietary binaries/libraries that go in
/opt/<vendor>/..., just as /opt is meant for (and no, we can't just
install them to /usr/lib, /usr/bin etc - partly because they contain
hard-coded absolute paths, but also due to some not-entirely-techical
reasons). So there's an /etc/ld.so.conf.d/<vendor>.conf that needs to be
picked up.

> If they are somehow special and only used by special applications then
> setting rpath or using LD_LIBRARY_PATH just for those applications
> would seem to be a better solution than enabling them globally.

We don't control the compilation, so rpath is out. We did consider
LD_LIBRARY_PATH, but figured that the ld.conf solution is more robust
(no problems with setuid binaries or an application that misguidedly
sanitizes the environment for subprocesses).

Rasmus

Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package

[Andre McCurdy]

On Wed, Jun 3, 2020 at 12:38 AM Rasmus Villemoes
<rasmus.villemoes@prevas.dk> wrote:
> On 02/06/2020 23.46, Andre McCurdy wrote:
> > On Tue, Jun 2, 2020 at 2:15 PM Phil Blundell via
> > lists.openembedded.org <pb=pbcl.net@lists.openembedded.org> wrote:
> >>
> >> On Tue, Jun 02, 2020 at 09:17:44PM +0100, Richard Purdie wrote:
> >>> I understand the concern, I am a little torn on this as adding in too
> >>> many different controls and options complicates the test matrix and
> >>> makes things harder for users.
> >>>
> >>> You're effectively suggesting a new DISTRO_FEATURE to control this? or
> >>> maybe better, perhaps a glibc PACKAGECONFIG?
> >>
> >> Yes, right.  I don't think it need be a DISTRO_FEATURE because nothing
> >> outside glibc needs to care.
> >>
> >> Here are the scenarios that I think matter, ordered from simplest to
> >> most complex:
> >>
> >> 1. Immutable filesystem, every library installed in the place where
> >> ld.so would first look for it anyway (i.e. everything in {/usr}/lib).
> >> In this case, we don't want ldconfig (because it can never do anything
> >> useful), we don't want ld.so.cache and we don't want ld.so.conf
> >> because they would cause ld.so to do extra file loads and computation
> >> but end up with the same result that it would anyway.
> >>
> >> 2. Immutable filesystem but some libraries are in places that ld.so
> >> wouldn't automatically know about.  In this case we do want ld.so.conf
> >> and ld.so.cache, but we still don't want ldconfig.
> >>
> >> 3. Mutable filesystem where arbitrary binaries can be installed in
> >> arbitrary places.  It's probably debatable whether ldconfig is needed
> >> in all these cases, but clearly it's needed in some and I think at
> >> this point it can be left to a DISTRO decision how exactly they want
> >> to optimize things.
> >>
> >> I think right now oe-core supports #1 and #3.  The proposed patch
> >> seems to be aimed at #2, which is a completely valid usecase,
> >
> > It seems like a weird corner case to me. Where do these libraries come
> > from and why can't they be moved or symlinked into a standard path?
>
> Pre-compiled proprietary binaries/libraries that go in
> /opt/<vendor>/..., just as /opt is meant for (and no, we can't just
> install them to /usr/lib, /usr/bin etc - partly because they contain
> hard-coded absolute paths, but also due to some not-entirely-techical
> reasons). So there's an /etc/ld.so.conf.d/<vendor>.conf that needs to be
> picked up.
>
> > If they are somehow special and only used by special applications then
> > setting rpath or using LD_LIBRARY_PATH just for those applications
> > would seem to be a better solution than enabling them globally.
>
> We don't control the compilation, so rpath is out. We did consider
> LD_LIBRARY_PATH, but figured that the ld.conf solution is more robust
> (no problems with setuid binaries or an application that misguidedly
> sanitizes the environment for subprocesses).

Unless the supplier of your binaries is completely unresponsive (if
so, bad luck) then educating them on the usage of rpath and asking for
a new release is probably the cleanest solution.

If you start your special application via a wrapper script which sets
LD_LIBRARY_PATH then it won't be seen by anything else, so the
concerns about other apps messing with it etc should not apply?

Relying on /etc/ld.so.conf.d/*.conf will certainly work too, but has
two disadvantages: it will add the custom library path globally
instead of just for the special apps and it's not portable (it's not
supported by musl etc - although if you only care about your
proprietary binaries pre-compiled for glibc then that won't matter to
you).

Anyway, the original patch has been merged, so this now just a side discussion.

Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package

[Rasmus Villemoes]

On 03/06/2020 22.44, Andre McCurdy wrote:
> On Wed, Jun 3, 2020 at 12:38 AM Rasmus Villemoes
> <rasmus.villemoes@prevas.dk> wrote:
>> On 02/06/2020 23.46, Andre McCurdy wrote:
>>> On Tue, Jun 2, 2020 at 2:15 PM Phil Blundell via
>>> lists.openembedded.org <pb=pbcl.net@lists.openembedded.org> wrote:
>>>>
>>>
>>> It seems like a weird corner case to me. Where do these libraries come
>>> from and why can't they be moved or symlinked into a standard path?
>>
>> Pre-compiled proprietary binaries/libraries that go in
>> /opt/<vendor>/..., just as /opt is meant for (and no, we can't just
>> install them to /usr/lib, /usr/bin etc - partly because they contain
>> hard-coded absolute paths, but also due to some not-entirely-techical
>> reasons). So there's an /etc/ld.so.conf.d/<vendor>.conf that needs to be
>> picked up.
>>
>>> If they are somehow special and only used by special applications then
>>> setting rpath or using LD_LIBRARY_PATH just for those applications
>>> would seem to be a better solution than enabling them globally.
>>
>> We don't control the compilation, so rpath is out. We did consider
>> LD_LIBRARY_PATH, but figured that the ld.conf solution is more robust
>> (no problems with setuid binaries or an application that misguidedly
>> sanitizes the environment for subprocesses).
> 
> Unless the supplier of your binaries is completely unresponsive (if
> so, bad luck) then educating them on the usage of rpath and asking for
> a new release is probably the cleanest solution.

For various reasons that is not a viable path.

> If you start your special application via a wrapper script which sets
> LD_LIBRARY_PATH then it won't be seen by anything else, so the
> concerns about other apps messing with it etc should not apply?

"Other apps" could be the apps in /opt themselves, that start other
binaries from under /opt, perhaps via some maze of shell scripts or
whatnot (the usual mess one sees with proprietary stuff) - which is also
the reason I'm worried they might have done some "value-add"
sanitization somewhere.

> Relying on /etc/ld.so.conf.d/*.conf will certainly work too, but has
> two disadvantages: it will add the custom library path globally
> instead of just for the special apps and it's not portable (it's not
> supported by musl etc - although if you only care about your
> proprietary binaries pre-compiled for glibc then that won't matter to
> you).

One could also ask the question: If one is not supposed to use the
drop-in dir ld.so.conf.d/ for exactly what I'm using it for, then why
does Yocto provide a ld.so.conf that mentions it? Sure, for glibc-based
distros (of course, it's from the glibc recipe), but that's fine.

Please remember that this is all just about ensuring that the ldconfig
run at image build time produces the same result, regardless of whether
one has opted to have ldconfig-the-binary in the rootfs or not (and
everybody seems to agree there are valid reasons not to have that).

> Anyway, the original patch has been merged, so this now just a side discussion.

Indeed.

Rasmus

Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package

[Richard Purdie]

On Wed, 2020-06-03 at 13:44 -0700, Andre McCurdy wrote:
> On Wed, Jun 3, 2020 at 12:38 AM Rasmus Villemoes
> <rasmus.villemoes@prevas.dk> wrote:
> > On 02/06/2020 23.46, Andre McCurdy wrote:
> > > On Tue, Jun 2, 2020 at 2:15 PM Phil Blundell via
> > > lists.openembedded.org <pb=pbcl.net@lists.openembedded.org>
> > > wrote:
> > > > On Tue, Jun 02, 2020 at 09:17:44PM +0100, Richard Purdie wrote:
> > > > > I understand the concern, I am a little torn on this as
> > > > > adding in too
> > > > > many different controls and options complicates the test
> > > > > matrix and
> > > > > makes things harder for users.
> > > > > 
> > > > > You're effectively suggesting a new DISTRO_FEATURE to control
> > > > > this? or
> > > > > maybe better, perhaps a glibc PACKAGECONFIG?
> > > > 
> > > > Yes, right.  I don't think it need be a DISTRO_FEATURE because
> > > > nothing
> > > > outside glibc needs to care.
> > > > 
> > > > Here are the scenarios that I think matter, ordered from
> > > > simplest to
> > > > most complex:
> > > > 
> > > > 1. Immutable filesystem, every library installed in the place
> > > > where
> > > > ld.so would first look for it anyway (i.e. everything in
> > > > {/usr}/lib).
> > > > In this case, we don't want ldconfig (because it can never do
> > > > anything
> > > > useful), we don't want ld.so.cache and we don't want ld.so.conf
> > > > because they would cause ld.so to do extra file loads and
> > > > computation
> > > > but end up with the same result that it would anyway.
> > > > 
> > > > 2. Immutable filesystem but some libraries are in places that
> > > > ld.so
> > > > wouldn't automatically know about.  In this case we do want
> > > > ld.so.conf
> > > > and ld.so.cache, but we still don't want ldconfig.
> > > > 
> > > > 3. Mutable filesystem where arbitrary binaries can be installed
> > > > in
> > > > arbitrary places.  It's probably debatable whether ldconfig is
> > > > needed
> > > > in all these cases, but clearly it's needed in some and I think
> > > > at
> > > > this point it can be left to a DISTRO decision how exactly they
> > > > want
> > > > to optimize things.
> > > > 
> > > > I think right now oe-core supports #1 and #3.  The proposed
> > > > patch
> > > > seems to be aimed at #2, which is a completely valid usecase,
> > > 
> > > It seems like a weird corner case to me. Where do these libraries
> > > come
> > > from and why can't they be moved or symlinked into a standard
> > > path?
> > 
> > Pre-compiled proprietary binaries/libraries that go in
> > /opt/<vendor>/..., just as /opt is meant for (and no, we can't just
> > install them to /usr/lib, /usr/bin etc - partly because they
> > contain
> > hard-coded absolute paths, but also due to some not-entirely-
> > techical
> > reasons). So there's an /etc/ld.so.conf.d/<vendor>.conf that needs
> > to be
> > picked up.
> > 
> > > If they are somehow special and only used by special applications
> > > then
> > > setting rpath or using LD_LIBRARY_PATH just for those
> > > applications
> > > would seem to be a better solution than enabling them globally.
> > 
> > We don't control the compilation, so rpath is out. We did consider
> > LD_LIBRARY_PATH, but figured that the ld.conf solution is more
> > robust
> > (no problems with setuid binaries or an application that
> > misguidedly
> > sanitizes the environment for subprocesses).
> 
> Unless the supplier of your binaries is completely unresponsive (if
> so, bad luck) then educating them on the usage of rpath and asking
> for
> a new release is probably the cleanest solution.
> 
> If you start your special application via a wrapper script which sets
> LD_LIBRARY_PATH then it won't be seen by anything else, so the
> concerns about other apps messing with it etc should not apply?
> 
> Relying on /etc/ld.so.conf.d/*.conf will certainly work too, but has
> two disadvantages: it will add the custom library path globally
> instead of just for the special apps and it's not portable (it's not
> supported by musl etc - although if you only care about your
> proprietary binaries pre-compiled for glibc then that won't matter to
> you).
> 
> Anyway, the original patch has been merged, so this now just a side
> discussion.

It hasn't, its still in master-next waiting for us to reach a
conclusion...

Cheers,

Richard

Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package

[Andre McCurdy]

On Thu, Jun 4, 2020 at 5:43 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Wed, 2020-06-03 at 13:44 -0700, Andre McCurdy wrote:
> > On Wed, Jun 3, 2020 at 12:38 AM Rasmus Villemoes
> > <rasmus.villemoes@prevas.dk> wrote:
> > > On 02/06/2020 23.46, Andre McCurdy wrote:
> > > > On Tue, Jun 2, 2020 at 2:15 PM Phil Blundell via
> > > > lists.openembedded.org <pb=pbcl.net@lists.openembedded.org>
> > > > wrote:
> > > > > On Tue, Jun 02, 2020 at 09:17:44PM +0100, Richard Purdie wrote:
> > > > > > I understand the concern, I am a little torn on this as
> > > > > > adding in too
> > > > > > many different controls and options complicates the test
> > > > > > matrix and
> > > > > > makes things harder for users.
> > > > > >
> > > > > > You're effectively suggesting a new DISTRO_FEATURE to control
> > > > > > this? or
> > > > > > maybe better, perhaps a glibc PACKAGECONFIG?
> > > > >
> > > > > Yes, right.  I don't think it need be a DISTRO_FEATURE because
> > > > > nothing
> > > > > outside glibc needs to care.
> > > > >
> > > > > Here are the scenarios that I think matter, ordered from
> > > > > simplest to
> > > > > most complex:
> > > > >
> > > > > 1. Immutable filesystem, every library installed in the place
> > > > > where
> > > > > ld.so would first look for it anyway (i.e. everything in
> > > > > {/usr}/lib).
> > > > > In this case, we don't want ldconfig (because it can never do
> > > > > anything
> > > > > useful), we don't want ld.so.cache and we don't want ld.so.conf
> > > > > because they would cause ld.so to do extra file loads and
> > > > > computation
> > > > > but end up with the same result that it would anyway.
> > > > >
> > > > > 2. Immutable filesystem but some libraries are in places that
> > > > > ld.so
> > > > > wouldn't automatically know about.  In this case we do want
> > > > > ld.so.conf
> > > > > and ld.so.cache, but we still don't want ldconfig.
> > > > >
> > > > > 3. Mutable filesystem where arbitrary binaries can be installed
> > > > > in
> > > > > arbitrary places.  It's probably debatable whether ldconfig is
> > > > > needed
> > > > > in all these cases, but clearly it's needed in some and I think
> > > > > at
> > > > > this point it can be left to a DISTRO decision how exactly they
> > > > > want
> > > > > to optimize things.
> > > > >
> > > > > I think right now oe-core supports #1 and #3.  The proposed
> > > > > patch
> > > > > seems to be aimed at #2, which is a completely valid usecase,
> > > >
> > > > It seems like a weird corner case to me. Where do these libraries
> > > > come
> > > > from and why can't they be moved or symlinked into a standard
> > > > path?
> > >
> > > Pre-compiled proprietary binaries/libraries that go in
> > > /opt/<vendor>/..., just as /opt is meant for (and no, we can't just
> > > install them to /usr/lib, /usr/bin etc - partly because they
> > > contain
> > > hard-coded absolute paths, but also due to some not-entirely-
> > > techical
> > > reasons). So there's an /etc/ld.so.conf.d/<vendor>.conf that needs
> > > to be
> > > picked up.
> > >
> > > > If they are somehow special and only used by special applications
> > > > then
> > > > setting rpath or using LD_LIBRARY_PATH just for those
> > > > applications
> > > > would seem to be a better solution than enabling them globally.
> > >
> > > We don't control the compilation, so rpath is out. We did consider
> > > LD_LIBRARY_PATH, but figured that the ld.conf solution is more
> > > robust
> > > (no problems with setuid binaries or an application that
> > > misguidedly
> > > sanitizes the environment for subprocesses).
> >
> > Unless the supplier of your binaries is completely unresponsive (if
> > so, bad luck) then educating them on the usage of rpath and asking
> > for
> > a new release is probably the cleanest solution.
> >
> > If you start your special application via a wrapper script which sets
> > LD_LIBRARY_PATH then it won't be seen by anything else, so the
> > concerns about other apps messing with it etc should not apply?
> >
> > Relying on /etc/ld.so.conf.d/*.conf will certainly work too, but has
> > two disadvantages: it will add the custom library path globally
> > instead of just for the special apps and it's not portable (it's not
> > supported by musl etc - although if you only care about your
> > proprietary binaries pre-compiled for glibc then that won't matter to
> > you).
> >
> > Anyway, the original patch has been merged, so this now just a side
> > discussion.
>
> It hasn't, its still in master-next waiting for us to reach a
> conclusion...

OK. The discussion has moved on to the details of Rasmus's use case,
but I can try to summarise the details related to the actual patch...

The patch fixes the problem that if the ldconfig distro feature is
disabled, then the config file needed by ldconfig will not be present
in the rootfs when ldconfig is run at build time (and so ld.so.cache
may not be correctly generated). The change to move ldconfig and its
config file into a separate package didn't account for the fact that
the config file is used at build time, but the underlying problem was
there before that too (before the separate package, the config file
was simply deleted).

The risk of an incorrectly generated ld.so.cache is a bug, so it
should be fixing unconditionally (ie it should not be tied to any
PACKAGECONFIG or new distro config option).

The fix being proposed is simply to always leave the config file in
the rootfs, so it's always there when ldconfig is run at build time.
Doing so seems safe enough. Since the config file is only parsed by
ldconfig, there's no run time performance penalty of leaving it around
- it will just be ignored if there's no ldconfig in the rootfs to use
it.

There's a genuine bug and I don't see an obviously better fix, so I
think the patch should be merged.

Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package

[Phil Blundell]

On Thu, Jun 04, 2020 at 01:05:30PM -0700, Andre McCurdy wrote:
> The fix being proposed is simply to always leave the config file in
> the rootfs, so it's always there when ldconfig is run at build time.
> Doing so seems safe enough. Since the config file is only parsed by
> ldconfig, there's no run time performance penalty of leaving it around
> - it will just be ignored if there's no ldconfig in the rootfs to use
> it.

Yes, agreed.  For some reason I had previously got it into my head
that ld.so itself would read ld.so.conf if there is no ld.so.cache,
but I just checked the glibc sources again and clearly it doesn't.
I'm not sure if my previous understanding was outdated, mixed up
with some other libc, or just plain wrong, but either way you're
right that there's no performance penalty.

I also checked with strace which reveals that, on the other hand,
ld.so _is_ trying to open /etc/ld.so.preload which is clearly a
waste of time as well.  But that's a separate issue...

p.

Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package

[Rasmus Villemoes]

On 04/06/2020 22.05, Andre McCurdy wrote:

> but I can try to summarise the details related to the actual patch...

[snip]

> There's a genuine bug and I don't see an obviously better fix, so I
> think the patch should be merged.

Thank you for that summary, that's much better than what I could have
written, and also thanks for the conclusion.

Rasmus

Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package

[Khem Raj]

On Tue, Jun 2, 2020 at 2:46 PM Andre McCurdy <armccurdy@gmail.com> wrote:
>
> On Tue, Jun 2, 2020 at 2:15 PM Phil Blundell via
> lists.openembedded.org <pb=pbcl.net@lists.openembedded.org> wrote:
> >
> > On Tue, Jun 02, 2020 at 09:17:44PM +0100, Richard Purdie wrote:
> > > I understand the concern, I am a little torn on this as adding in too
> > > many different controls and options complicates the test matrix and
> > > makes things harder for users.
> > >
> > > You're effectively suggesting a new DISTRO_FEATURE to control this? or
> > > maybe better, perhaps a glibc PACKAGECONFIG?
> >
> > Yes, right.  I don't think it need be a DISTRO_FEATURE because nothing
> > outside glibc needs to care.
> >
> > Here are the scenarios that I think matter, ordered from simplest to
> > most complex:
> >
> > 1. Immutable filesystem, every library installed in the place where
> > ld.so would first look for it anyway (i.e. everything in {/usr}/lib).
> > In this case, we don't want ldconfig (because it can never do anything
> > useful), we don't want ld.so.cache and we don't want ld.so.conf
> > because they would cause ld.so to do extra file loads and computation
> > but end up with the same result that it would anyway.
> >
> > 2. Immutable filesystem but some libraries are in places that ld.so
> > wouldn't automatically know about.  In this case we do want ld.so.conf
> > and ld.so.cache, but we still don't want ldconfig.
> >
> > 3. Mutable filesystem where arbitrary binaries can be installed in
> > arbitrary places.  It's probably debatable whether ldconfig is needed
> > in all these cases, but clearly it's needed in some and I think at
> > this point it can be left to a DISTRO decision how exactly they want
> > to optimize things.
> >
> > I think right now oe-core supports #1 and #3.  The proposed patch
> > seems to be aimed at #2, which is a completely valid usecase,
>
> It seems like a weird corner case to me. Where do these libraries come
> from and why can't they be moved or symlinked into a standard path?
>
> If they are somehow special and only used by special applications then
> setting rpath or using LD_LIBRARY_PATH just for those applications
> would seem to be a better solution than enabling them globally.
>
> > but
> > my concern is that we don't want to pessimize #1 in the process.
> >
> > There is a whole parallel discussion to be had about the merits
> > or otherwise of ld.so's hwcaps mechanism.  meta-micro has been
> > carrying a patch for about a decade to turn all that stuff off,
> > on the grounds that it's just a loss on any kind of embedded
> > environment.  Maybe I should make some effort to get it upstream
> > as an option.
>
> Note also that glibc already has a build time option to completely
> disable support for ldconfig:
>
>   https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/configure.ac#l89
>
> Sadly it's not usable in recent versions of glibc (setting
> use_ldconfig=no results in a build failure due to some seemingly
> unrelated missing build dependency which I haven't had time to dig
> into yet) but it worked well for OE 2.2 and gave a noticeable
> reduction in glibc code size.


If we can get this working and perhaps as a packageconfig this would address
concern #1 nicely per distro policies.

Re: [OE-core] [PATCH v3] glibc: move ld.so.conf back to main package

[Rasmus Villemoes]

On 02/06/2020 23.15, Phil Blundell wrote:
> On Tue, Jun 02, 2020 at 09:17:44PM +0100, Richard Purdie wrote:
>> I understand the concern, I am a little torn on this as adding in too
>> many different controls and options complicates the test matrix and
>> makes things harder for users.
>>
>> You're effectively suggesting a new DISTRO_FEATURE to control this? or
>> maybe better, perhaps a glibc PACKAGECONFIG?
> 
> Yes, right.  I don't think it need be a DISTRO_FEATURE because nothing
> outside glibc needs to care.
> 
> Here are the scenarios that I think matter, ordered from simplest to
> most complex:
> 
> 1. Immutable filesystem, every library installed in the place where
> ld.so would first look for it anyway (i.e. everything in {/usr}/lib).
> In this case, we don't want ldconfig (because it can never do anything
> useful), we don't want ld.so.cache and we don't want ld.so.conf
> because they would cause ld.so to do extra file loads and computation
> but end up with the same result that it would anyway.
> 
> I think right now oe-core supports #1 and #3.  The proposed patch
> seems to be aimed at #2, which is a completely valid usecase, but
> my concern is that we don't want to pessimize #1 in the process.

With or without my patch, there's an ldconfig invocation at image build
time, and lacking an /etc/ld.so.conf in the -r filesystem, ldconfig just
looks at the default locations (/lib and /usr/lib, perhaps a few more
depending on $arch). But you still get an ld.so.cache generated listing
the libs in /lib and /usr/lib. So I don't understand this performance
concern at all - at the very most, the presence of the config file
causes a microscopic performance hit at image build time when ldconfig
goes looking for the /etc/ld.so.conf.d/* files and (usually) find none.

Also, IME, the cache file usually speed things up since ld.so avoids a
filesystem access to figure out that
/{usr,}/lib/<arch-specific-path>/libfoo.so doesn't exist (of course
there's an upfront cost of mmapping and parsing the ld.so.cache, so for
programs with very few dynlib dependencies, it might be faster to not
have the cache). See below.

If you really don't want the ld.so.cache, ldconfig has the -N option.
But adding some knob to put that in is completely orthogonal to this patch.

Rasmus

Here's the first few lines of "strace -o /tmp/cache.txt factor 12345":

execve("/usr/bin/factor", ["factor", "12345"], 0xbff6cf3c /* 14 vars */) = 0
brk(NULL)                               = 0x10173000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or
directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=6527, ...}) = 0
mmap2(NULL, 6527, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7ef2000
close(3)                                = 0
openat(AT_FDCWD, "/lib/librt.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3,
"\177ELF\1\2\1\3\0\0\0\0\0\0\0\0\0\3\0\24\0\0\0\1\0\0\36\300\0\0\0004"...,
512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=38596, ...}) = 0
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7ef0000
mmap2(0xffdf000, 65768, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
3, 0) = 0xffdf000
mprotect(0xffe7000, 28672, PROT_NONE)   = 0
mmap2(0xffee000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0xffee000
close(3)                                = 0
openat(AT_FDCWD, "/lib/libcap.so.2", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3,
"\177ELF\1\2\1\0\0\0\0\0\0\0\0\0\0\3\0\24\0\0\0\1\0\0\21P\0\0\0004"...,
512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=22060, ...}) = 0
mmap2(0xffbe000, 65820, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
3, 0) = 0xffbe000
mprotect(0xffc3000, 40960, PROT_NONE)   = 0
mmap2(0xffcd000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0xffcd000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libgmp.so.10",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\2\1\0\0\0\0\0\0\0\0\0\0\3\0\24\0\0\0\1\0\0\226
\0\0\0004"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=551740, ...}) = 0
mmap2(0xff1d000, 591268, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
3, 0) = 0xff1d000
mprotect(0xffa3000, 36864, PROT_NONE)   = 0
mmap2(0xffac000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x85000) = 0xffac000
close(3)                                = 0

Compare that to what happens when I move the ld.so.cache out of the way:

execve("/usr/bin/factor", ["factor", "12345"], 0xbfc7c3bc /* 14 vars */) = 0
brk(NULL)                               = 0x10173000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or
directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE|O_CLOEXEC) =
-1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/tls/ppc603/librt.so.1",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/ppc603", 0xbf98df78)   = -1 ENOENT (No such file or
directory)
openat(AT_FDCWD, "/lib/tls/librt.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC)
= -1 ENOENT (No such file or directory)
stat64("/lib/tls", 0xbf98df78)          = -1 ENOENT (No such file or
directory)
openat(AT_FDCWD, "/lib/ppc603/librt.so.1",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/ppc603", 0xbf98df78)       = -1 ENOENT (No such file or
directory)
openat(AT_FDCWD, "/lib/librt.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3,
"\177ELF\1\2\1\3\0\0\0\0\0\0\0\0\0\3\0\24\0\0\0\1\0\0\36\300\0\0\0004"...,
512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=38596, ...}) = 0
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7cd0000
mmap2(0xffdf000, 65768, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
3, 0) = 0xffdf000
mprotect(0xffe7000, 28672, PROT_NONE)   = 0
mmap2(0xffee000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0xffee000
close(3)                                = 0
openat(AT_FDCWD, "/lib/libcap.so.2", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3,
"\177ELF\1\2\1\0\0\0\0\0\0\0\0\0\0\3\0\24\0\0\0\1\0\0\21P\0\0\0004"...,
512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=22060, ...}) = 0
mmap2(0xffbe000, 65820, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
3, 0) = 0xffbe000
mprotect(0xffc3000, 40960, PROT_NONE)   = 0
mmap2(0xffcd000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0xffcd000
close(3)                                = 0
openat(AT_FDCWD, "/lib/libgmp.so.10", O_RDONLY|O_LARGEFILE|O_CLOEXEC) =
-1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/tls/ppc603/libgmp.so.10",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/ppc603", 0xbf98df38) = -1 ENOENT (No such file or
directory)
openat(AT_FDCWD, "/usr/lib/tls/libgmp.so.10",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls", 0xbf98df38)      = -1 ENOENT (No such file or
directory)
openat(AT_FDCWD, "/usr/lib/ppc603/libgmp.so.10",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/ppc603", 0xbf98df38)   = -1 ENOENT (No such file or
directory)
openat(AT_FDCWD, "/usr/lib/libgmp.so.10",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\2\1\0\0\0\0\0\0\0\0\0\0\3\0\24\0\0\0\1\0\0\226
\0\0\0004"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=551740, ...}) = 0
mmap2(0xff1d000, 591268, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
3, 0) = 0xff1d000
mprotect(0xffa3000, 36864, PROT_NONE)   = 0
mmap2(0xffac000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x85000) = 0xffac000
close(3)                                = 0

So ld.so is smart enough to say, "hm, ok, there was no
/lib/tls/libfoo.so, is there even a /lib/tls or do I not need bother
looking there for the next few libraries". But (1) we still end up doing
10 more system calls compared to the case of having a cache and (2) this
is the good case, if some library (maybe totally unrelated to this
binary) did exist in one of those arch-specific dirs, the directory
would of course exist, and ld.so would keep searching that dir before
looking in /lib or /usr/lib.