Very large size of libraries in core-image-minimal rootfs

Very large size of libraries in core-image-minimal rootfs

[Ankur Tyagi]

[-- Attachment #1: Type: text/plain, Size: 5758 bytes --]

Hi,

Based upon "thud" branch, I created core-image-minimal for am335x-evm board
and resulting image size is very big (71M)

/lib dir is 39M and /usr/lib is 24M.

How can the libraries be trimmed down to fit image inside 40M partition? I
can see duplicacy in /usr/lib and symlink should help but that would reduce
12M. I need to reduce further 20M.

When I was using "daisy" branch, resulting image was much tiny but world
has changed since then. Any help is much appreciated as I am now blocked
and looking for some help.

$ ls -al lib/
total 38712
drwxr-xr-x  4 ankur ankur     4096 Nov 26 07:13 .
drwxr-xr-x 17 ankur ankur     4096 Nov 26 07:13 ..
-rwxr-xr-x  1 ankur ankur  1347940 Nov 26 06:41 ld-2.28.so
lrwxrwxrwx  1 ankur ankur       10 Nov 26 06:41 ld-linux-armhf.so.3 ->
ld-2.28.so
-rwxr-xr-x  1 ankur ankur   133528 Nov 26 06:41 libanl.so.1
lrwxrwxrwx  1 ankur ankur       16 Nov 26 06:41 libattr.so.1 ->
libattr.so.1.1.0
-rw-r--r--  1 ankur ankur    13736 Nov 26 06:41 libattr.so.1.1.0
lrwxrwxrwx  1 ankur ankur       17 Nov 26 06:44 libblkid.so.1 ->
libblkid.so.1.1.0
-rwxr-xr-x  1 ankur ankur   195404 Nov 26 06:44 libblkid.so.1.1.0
-rwxr-xr-x  1 ankur ankur    25392 Nov 26 06:41 libBrokenLocale.so.1
-rwxr-xr-x  1 ankur ankur   153372 Nov 26 06:41 libcrypt.so.1
-rwxr-xr-x  1 ankur ankur *16537400* Nov 26 06:41 libc.so.6
-rwxr-xr-x  1 ankur ankur   225508 Nov 26 06:41 libdl.so.2
-rw-r--r--  1 ankur ankur      132 Nov 26 06:40 libgcc_s.so
-rw-r--r--  1 ankur ankur  *9063520* Nov 26 06:40 libgcc_s.so.1
-rwxr-xr-x  1 ankur ankur  *6416640* Nov 26 06:41 libm.so.6
-rwxr-xr-x  1 ankur ankur   722176 Nov 26 06:41 libnsl.so.1
-rwxr-xr-x  1 ankur ankur   178932 Nov 26 06:41 libnss_compat.so.2
-rwxr-xr-x  1 ankur ankur   122236 Nov 26 06:41 libnss_dns.so.2
-rwxr-xr-x  1 ankur ankur   373084 Nov 26 06:41 libnss_files.so.2
-rwxr-xr-x  1 ankur ankur   115836 Nov 26 06:41 libnss_hesiod.so.2
lrwxrwxrwx  1 ankur ankur       17 Nov 26 06:44 libpamc.so.0 ->
libpamc.so.0.82.1
-rwxr-xr-x  1 ankur ankur     9592 Nov 26 06:44 libpamc.so.0.82.1
lrwxrwxrwx  1 ankur ankur       21 Nov 26 06:44 libpam_misc.so.0 ->
libpam_misc.so.0.82.1
-rwxr-xr-x  1 ankur ankur     9628 Nov 26 06:44 libpam_misc.so.0.82.1
lrwxrwxrwx  1 ankur ankur       16 Nov 26 06:44 libpam.so.0 ->
libpam.so.0.84.2
-rwxr-xr-x  1 ankur ankur    34396 Nov 26 06:44 libpam.so.0.84.2
-rwxr-xr-x  1 ankur ankur  *2742208* Nov 26 06:41 libpthread.so.0
-rwxr-xr-x  1 ankur ankur   436644 Nov 26 06:41 libresolv.so.2
-rwxr-xr-x  1 ankur ankur   424948 Nov 26 06:41 librt.so.1
lrwxrwxrwx  1 ankur ankur       12 Nov 26 06:45 libudev.so.0 -> libudev.so.1
lrwxrwxrwx  1 ankur ankur       16 Nov 26 06:45 libudev.so.1 ->
libudev.so.1.6.3
-rwxr-xr-x  1 ankur ankur    88176 Nov 26 06:45 libudev.so.1.6.3
lrwxrwxrwx  1 ankur ankur       19 Nov 26 07:13 libusb-1.0.so.0 ->
libusb-1.0.so.0.1.0
-rwxr-xr-x  1 ankur ankur    67292 Nov 26 07:13 libusb-1.0.so.0.1.0
-rwxr-xr-x  1 ankur ankur    40320 Nov 26 06:41 libutil.so.1
lrwxrwxrwx  1 ankur ankur       16 Nov 26 06:44 libuuid.so.1 ->
libuuid.so.1.3.0
-rwxr-xr-x  1 ankur ankur    22108 Nov 26 06:44 libuuid.so.1.3.0
lrwxrwxrwx  1 ankur ankur       14 Nov 26 06:40 libz.so.1 -> libz.so.1.2.11
-rwxr-xr-x  1 ankur ankur    63012 Nov 26 06:40 libz.so.1.2.11
drwxr-xr-x  2 ankur ankur     4096 Nov 26 07:13 security
drwxr-xr-x  3 ankur ankur     4096 Nov 26 06:45 udev

$ ls -al usr/lib/
total 23376
drwxr-xr-x  3 ankur ankur     4096 Nov 26 07:13 .
drwxr-xr-x 10 ankur ankur     4096 Nov 26 07:13 ..
lrwxrwxrwx  1 ankur ankur       15 Nov 26 07:12 libi2c.so.0 ->
libi2c.so.0.1.1
-rwxr-xr-x  1 ankur ankur     5292 Nov 26 07:12 libi2c.so.0.1.1
lrwxrwxrwx  1 ankur ankur       16 Nov 26 06:44 libkmod.so.2 ->
libkmod.so.2.3.3
-rwxr-xr-x  1 ankur ankur    46840 Nov 26 06:44 libkmod.so.2.3.3
-rwxr-xr-x  1 ankur ankur *11930116* Nov 26 06:43 libstdc++.so.6
-rwxr-xr-x  1 ankur ankur *11930116* Nov 26 06:43 libstdc++.so.6.0.25
-rw-r--r--  1 ankur ankur     2388 Nov 26 06:43 libstdc++.so.6.0.25-gdb.py
drwxr-xr-x  3 ankur ankur     4096 Nov 26 07:13 opkg

$ du -h
4.0K    ./sys
4.0K    ./boot
4.0K    ./dev
5.5M    ./sbin
612K    ./bin
4.0K    ./usr/share/dict
4.0K    ./usr/share/man
4.0K    ./usr/share/misc
4.0K    ./usr/share/info
20K    ./usr/share
208K    ./usr/libexec
824K    ./usr/sbin
520K    ./usr/bin
4.0K    ./usr/include
4.0K    ./usr/src
4.0K    ./usr/games
1.1M    ./usr/lib/opkg/alternatives
1.1M    ./usr/lib/opkg

*24M    ./usr/lib26M    ./usr*
4.0K    ./var/local
4.0K    ./var/spool/mail
8.0K    ./var/spool
4.0K    ./var/backups
4.0K    ./var/lib/opkg
4.0K    ./var/lib/misc
4.0K    ./var/lib/urandom
16K    ./var/lib
4.0K    ./var/volatile
4.0K    ./var/cache/opkg
8.0K    ./var/cache/ldconfig
16K    ./var/cache
56K    ./var
4.0K    ./mnt/.psplash
8.0K    ./mnt
4.0K    ./proc
4.0K    ./etc/network/if-up.d
4.0K    ./etc/network/if-down.d
4.0K    ./etc/network/if-post-down.d
8.0K    ./etc/network/if-pre-up.d
28K    ./etc/network
92K    ./etc/pam.d
4.0K    ./etc/rc2.d
140K    ./etc/init.d
4.0K    ./etc/rc6.d
12K    ./etc/skel
4.0K    ./etc/rc0.d
16K    ./etc/udev/rules.d
24K    ./etc/udev
4.0K    ./etc/rcS.d
4.0K    ./etc/security/limits.d
4.0K    ./etc/security/namespace.d
44K    ./etc/security
16K    ./etc/default/volatiles
36K    ./etc/default
4.0K    ./etc/rc1.d
4.0K    ./etc/rc5.d
4.0K    ./etc/rc3.d
8.0K    ./etc/ipk-postinsts
4.0K    ./etc/rc4.d
560K    ./etc
4.0K    ./run
4.0K    ./tmp
104K    ./lib/udev/rules.d
400K    ./lib/udev
172K    ./lib/security
*39M*    *./lib*
4.0K    ./media
4.0K    ./home/root
8.0K    ./home
*71M    .*

Thanks
Ankur

[-- Attachment #2: Type: text/html, Size: 8311 bytes --]

Re: Very large size of libraries in core-image-minimal rootfs

[Ross Burton]

On 25/11/2019 18:37, Ankur Tyagi wrote:
> Hi,
> 
> Based upon "thud" branch, I created core-image-minimal for am335x-evm 
> board and resulting image size is very big(71M)

If disk size is important then a good first step is to use musl instead 
of glibc:

TCLIBC = "musl"

The C++ runtime is also giant, so see if you can not use C++?

Ross

Re: Very large size of libraries in core-image-minimal rootfs

[Ankur Tyagi]

[-- Attachment #1: Type: text/plain, Size: 762 bytes --]

Hi Ross,

Thanks for the suggestion.
But curious why libs are so big now?

Regards
Ankur

On Tue, Nov 26, 2019 at 8:27 AM Ross Burton <ross.burton@intel.com> wrote:

> On 25/11/2019 18:37, Ankur Tyagi wrote:
> > Hi,
> >
> > Based upon "thud" branch, I created core-image-minimal for am335x-evm
> > board and resulting image size is very big(71M)
>
> If disk size is important then a good first step is to use musl instead
> of glibc:
>
> TCLIBC = "musl"
>
> The C++ runtime is also giant, so see if you can not use C++?
>
> Ross
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 1782 bytes --]

Re: Very large size of libraries in core-image-minimal rootfs

[Ross Burton]

On 25/11/2019 19:33, Ankur Tyagi wrote:
> Thanks for the suggestion.
> But curious why libs are so big now?

The same files on my system:

-rwxr-xr-x root/root   1806504 2019-11-04 21:14 ./lib64/libc-2.30.so
-rwxr-xr-x root/root    113296 2019-11-04 21:14 ./lib64/libpthread-2.30.so
-rw-r--r-- root/root    100248 2019-11-04 21:14 ./lib64/libgcc_s.so.1

Your files are a lot larger, so I suspect that you've somehow turned off 
debug stripping.

Ross

Re: Very large size of libraries in core-image-minimal rootfs

[Ankur Tyagi]

[-- Attachment #1: Type: text/plain, Size: 656 bytes --]

That could be it.
How can I make sure debug stripping is enabled?

thanks
Ankur

On Tue, Nov 26, 2019 at 8:49 AM Ross Burton <ross.burton@intel.com> wrote:

> On 25/11/2019 19:33, Ankur Tyagi wrote:
> > Thanks for the suggestion.
> > But curious why libs are so big now?
>
> The same files on my system:
>
> -rwxr-xr-x root/root   1806504 2019-11-04 21:14 ./lib64/libc-2.30.so
> -rwxr-xr-x root/root    113296 2019-11-04 21:14 ./lib64/libpthread-2.30.so
> -rw-r--r-- root/root    100248 2019-11-04 21:14 ./lib64/libgcc_s.so.1
>
> Your files are a lot larger, so I suspect that you've somehow turned off
> debug stripping.
>
> Ross
>

[-- Attachment #2: Type: text/html, Size: 1500 bytes --]

Re: Very large size of libraries in core-image-minimal rootfs

[Adrian Bunk]

On Tue, Nov 26, 2019 at 07:37:48AM +1300, Ankur Tyagi wrote:
> Hi,
> 
> Based upon "thud" branch, I created core-image-minimal for am335x-evm board
> and resulting image size is very big (71M)
> 
> /lib dir is 39M and /usr/lib is 24M.
>...
> -rwxr-xr-x  1 ankur ankur *16537400* Nov 26 06:41 libc.so.6
>...
> -rwxr-xr-x  1 ankur ankur  *6416640* Nov 26 06:41 libm.so.6
>...

These sizes are the expected sizes for the libraries with debug symbols.

Are you setting INHIBIT_PACKAGE_DEBUG_SPLIT or something similar that 
prevents stripping of the target filesystem?

> Thanks
> Ankur

cu
Adrian

Re: Very large size of libraries in core-image-minimal rootfs

[Ankur Tyagi]

[-- Attachment #1: Type: text/plain, Size: 2851 bytes --]

Hi Adrian,

Seems you are correct. My distro is based upon TI's Arago distro which uses
following

# Select external binary prebuilt Arm toolchain
TCMODE = "external-arm"
TCLIBC = "external-arm-toolchain"

And external-arm-toolchain.bb recipe has following
require recipes-core/glibc/glibc-package.inc
INHIBIT_DEFAULT_DEPS = "1"
INHIBIT_PACKAGE_STRIP = "1"
INHIBIT_PACKAGE_DEBUG_SPLIT = "1"

And glibc-package.inc has following
INHIBIT_SYSROOT_STRIP = "1"

I just rebuild by disabling INHIBIT_PACKAGE_STRIP = "1" in
external-arm-toolchain.bb and it reduced image size to 12M

$ ls -al lib/
-rwxr-xr-x  1 ankur ankur  138436 Nov 26 09:08 ld-2.28.so
lrwxrwxrwx  1 ankur ankur      10 Nov 26 09:08 ld-linux-armhf.so.3 ->
ld-2.28.so
-rwxr-xr-x  1 ankur ankur 1246708 Nov 26 09:08 libc.so.6
-rwxr-xr-x  1 ankur ankur    9600 Nov 26 09:08 libdl.so.2
-rw-r--r--  1 ankur ankur     132 Nov 26 09:07 libgcc_s.so
-rw-r--r--  1 ankur ankur  124396 Nov 26 09:07 libgcc_s.so.1
-rwxr-xr-x  1 ankur ankur  451932 Nov 26 09:08 libm.so.6
-rwxr-xr-x  1 ankur ankur   67292 Nov 26 09:08 libnsl.so.1
-rwxr-xr-x  1 ankur ankur   26244 Nov 26 09:08 libnss_compat.so.2
-rwxr-xr-x  1 ankur ankur   17812 Nov 26 09:08 libnss_dns.so.2
-rwxr-xr-x  1 ankur ankur   38368 Nov 26 09:08 libnss_files.so.2
-rwxr-xr-x  1 ankur ankur   13708 Nov 26 09:08 libnss_hesiod.so.2
-rwxr-xr-x  1 ankur ankur   92356 Nov 26 09:08 libpthread.so.0

$ ls -al usr/lib
total 2816
drwxr-xr-x  3 ankur ankur    4096 Nov 26 09:18 .
drwxr-xr-x 10 ankur ankur    4096 Nov 26 09:18 ..
lrwxrwxrwx  1 ankur ankur      15 Nov 26 09:12 libi2c.so.0 ->
libi2c.so.0.1.1
-rwxr-xr-x  1 ankur ankur    5292 Nov 26 09:12 libi2c.so.0.1.1
lrwxrwxrwx  1 ankur ankur      16 Nov 26 09:11 libkmod.so.2 ->
libkmod.so.2.3.3
-rwxr-xr-x  1 ankur ankur   46840 Nov 26 09:11 libkmod.so.2.3.3
-rwxr-xr-x  1 ankur ankur 1404888 Nov 26 09:08 libstdc++.so.6
-rwxr-xr-x  1 ankur ankur 1404888 Nov 26 09:08 libstdc++.so.6.0.25
-rw-r--r--  1 ankur ankur    2388 Nov 26 09:08 libstdc++.so.6.0.25-gdb.py
Thank you so much for help.

Regards
Ankur

On Tue, Nov 26, 2019 at 8:55 AM Adrian Bunk <bunk@stusta.de> wrote:

> On Tue, Nov 26, 2019 at 07:37:48AM +1300, Ankur Tyagi wrote:
> > Hi,
> >
> > Based upon "thud" branch, I created core-image-minimal for am335x-evm
> board
> > and resulting image size is very big (71M)
> >
> > /lib dir is 39M and /usr/lib is 24M.
> >...
> > -rwxr-xr-x  1 ankur ankur *16537400* Nov 26 06:41 libc.so.6
> >...
> > -rwxr-xr-x  1 ankur ankur  *6416640* Nov 26 06:41 libm.so.6
> >...
>
> These sizes are the expected sizes for the libraries with debug symbols.
>
> Are you setting INHIBIT_PACKAGE_DEBUG_SPLIT or something similar that
> prevents stripping of the target filesystem?
>
> > Thanks
> > Ankur
>
> cu
> Adrian
>

[-- Attachment #2: Type: text/html, Size: 5351 bytes --]

Re: Very large size of libraries in core-image-minimal rootfs

[Ross Burton]

On 25/11/2019 19:54, Ankur Tyagi wrote:
> That could be it.
> How can I make sure debug stripping is enabled?

It's on by default.  I'd look at your configuration to see what you've 
done to turn it off...

Re: Very large size of libraries in core-image-minimal rootfs

[Ross Burton]

On 25/11/2019 20:22, Ankur Tyagi wrote:
> # Select external binary prebuilt Arm toolchain
> TCMODE = "external-arm"
> TCLIBC = "external-arm-toolchain"

FWIW, I don't see what the value in external toolchains actually is. 
You might get better results by simply setting:

TCMODE = "default"
TCLIB = "glibc"

(or TCLIBC="musl" if you want to save even more space)

Ross

Re: Very large size of libraries in core-image-minimal rootfs

[Adrian Bunk]

On Mon, Nov 25, 2019 at 07:26:58PM +0000, Ross Burton wrote:
> On 25/11/2019 18:37, Ankur Tyagi wrote:
> > Hi,
> > 
> > Based upon "thud" branch, I created core-image-minimal for am335x-evm
> > board and resulting image size is very big(71M)
> 
> If disk size is important then a good first step is to use musl instead of
> glibc:
> 
> TCLIBC = "musl"
>...

musl is mainly useful for tiny systems, a 40 MB partition is such a huge
amount of space that the size of the C library no longer matters much.

The size benefit of replacing glibc with musl is around 2 MB.

A partition with an -Os build of Yocto with glibc+systemd (sic)
is around 9 MB, together with bootloader and kernel this all
fits in a 16 MB flash. Use compression for kernel and root
filesystem and everything fits in a 8 MB flash.

> Ross

cu
Adrian

Re: Very large size of libraries in core-image-minimal rootfs

[Ankur Tyagi]

[-- Attachment #1: Type: text/plain, Size: 880 bytes --]

Hi Ross,

I just used arago distro as it is but you are right, once I have stable
builds
with default configuration then I will try your suggestion.

Thanks for your help

Ankur

On Mon, Nov 25, 2019 at 12:58 PM Ross Burton <ross.burton@intel.com> wrote:

> On 25/11/2019 20:22, Ankur Tyagi wrote:
> > # Select external binary prebuilt Arm toolchain
> > TCMODE = "external-arm"
> > TCLIBC = "external-arm-toolchain"
>
> FWIW, I don't see what the value in external toolchains actually is.
> You might get better results by simply setting:
>
> TCMODE = "default"
> TCLIB = "glibc"
>
> (or TCLIBC="musl" if you want to save even more space)
>
> Ross
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 2049 bytes --]

Re: Very large size of libraries in core-image-minimal rootfs

[Ankur Tyagi]

[-- Attachment #1: Type: text/plain, Size: 1313 bytes --]

Hi Adrian,

Thanks for explaining things and helping me out.

Once I have stable builds then will try your suggestion as well.

Ankur

On Mon, Nov 25, 2019 at 1:10 PM Adrian Bunk <bunk@stusta.de> wrote:

> On Mon, Nov 25, 2019 at 07:26:58PM +0000, Ross Burton wrote:
> > On 25/11/2019 18:37, Ankur Tyagi wrote:
> > > Hi,
> > >
> > > Based upon "thud" branch, I created core-image-minimal for am335x-evm
> > > board and resulting image size is very big(71M)
> >
> > If disk size is important then a good first step is to use musl instead
> of
> > glibc:
> >
> > TCLIBC = "musl"
> >...
>
> musl is mainly useful for tiny systems, a 40 MB partition is such a huge
> amount of space that the size of the C library no longer matters much.
>
> The size benefit of replacing glibc with musl is around 2 MB.
>
> A partition with an -Os build of Yocto with glibc+systemd (sic)
> is around 9 MB, together with bootloader and kernel this all
> fits in a 16 MB flash. Use compression for kernel and root
> filesystem and everything fits in a 8 MB flash.
>
> > Ross
>
> cu
> Adrian
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 2437 bytes --]

Re: Very large size of libraries in core-image-minimal rootfs

[Adrian Bunk]

On Mon, Nov 25, 2019 at 10:17:42PM -0800, Ankur Tyagi wrote:
> Hi Adrian,

Hi Ankur,

> Thanks for explaining things and helping me out.
> 
> Once I have stable builds then will try your suggestion as well.

if everything you need fits into your partition and is stable
just be happy.

FULL_OPTIMIZATION_append = " -Os" would be an option to reduce the size 
of the filesystem further, but it makes the code not only smaller but
also slower.

> Ankur

cu
Adrian

Re: Very large size of libraries in core-image-minimal rootfs

[Khem Raj]

On Mon, 2019-11-25 at 23:10 +0200, Adrian Bunk wrote:
> On Mon, Nov 25, 2019 at 07:26:58PM +0000, Ross Burton wrote:
> > On 25/11/2019 18:37, Ankur Tyagi wrote:
> > > Hi,
> > > 
> > > Based upon "thud" branch, I created core-image-minimal for
> > > am335x-evm
> > > board and resulting image size is very big(71M)
> > 
> > If disk size is important then a good first step is to use musl
> > instead of
> > glibc:
> > 
> > TCLIBC = "musl"
> > ...
> 
> musl is mainly useful for tiny systems, a 40 MB partition is such a
> huge
> amount of space that the size of the C library no longer matters
> much.
> 
> The size benefit of replacing glibc with musl is around 2 MB.

I would refrain from labeling as useful for tiny systems. since
static size is one thing, there is DRAM behaviour which is equally
important, how malloc and other resource oriented algorithms behave is
important as well.

secondly, if I have 40MB partition, one might rather want to use extra
space for implementing features in their product than spend it for
enabling the system if they can.

there are many other design considerations where musl might make more
sense

> 
> A partition with an -Os build of Yocto with glibc+systemd (sic)
> is around 9 MB, together with bootloader and kernel this all
> fits in a 16 MB flash. Use compression for kernel and root
> filesystem and everything fits in a 8 MB flash.
> 

-Os is not as well tested option as -O2 is. So we should always
recommend it with grain of salt, speaking from experience there are
bugs that will show up with Os which you wont see with O2 and leave a
bad taste for user

> > Ross
> 
> cu
> Adrian

Re: Very large size of libraries in core-image-minimal rootfs

[Ankur Tyagi]

[-- Attachment #1: Type: text/plain, Size: 2209 bytes --]

Hi Khem,

I have no experience with musl but keen to understand does musl
improves/impacts DRAM performance as compared to glibc?

regards
Ankur

On Wed, 27 Nov 2019, 7:17 a.m. Khem Raj, <raj.khem@gmail.com> wrote:

> On Mon, 2019-11-25 at 23:10 +0200, Adrian Bunk wrote:
> > On Mon, Nov 25, 2019 at 07:26:58PM +0000, Ross Burton wrote:
> > > On 25/11/2019 18:37, Ankur Tyagi wrote:
> > > > Hi,
> > > >
> > > > Based upon "thud" branch, I created core-image-minimal for
> > > > am335x-evm
> > > > board and resulting image size is very big(71M)
> > >
> > > If disk size is important then a good first step is to use musl
> > > instead of
> > > glibc:
> > >
> > > TCLIBC = "musl"
> > > ...
> >
> > musl is mainly useful for tiny systems, a 40 MB partition is such a
> > huge
> > amount of space that the size of the C library no longer matters
> > much.
> >
> > The size benefit of replacing glibc with musl is around 2 MB.
>
> I would refrain from labeling as useful for tiny systems. since
> static size is one thing, there is DRAM behaviour which is equally
> important, how malloc and other resource oriented algorithms behave is
> important as well.
>
> secondly, if I have 40MB partition, one might rather want to use extra
> space for implementing features in their product than spend it for
> enabling the system if they can.
>
> there are many other design considerations where musl might make more
> sense
>
> >
> > A partition with an -Os build of Yocto with glibc+systemd (sic)
> > is around 9 MB, together with bootloader and kernel this all
> > fits in a 16 MB flash. Use compression for kernel and root
> > filesystem and everything fits in a 8 MB flash.
> >
>
> -Os is not as well tested option as -O2 is. So we should always
> recommend it with grain of salt, speaking from experience there are
> bugs that will show up with Os which you wont see with O2 and leave a
> bad taste for user
>
> > > Ross
> >
> > cu
> > Adrian
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 3188 bytes --]

Re: Very large size of libraries in core-image-minimal rootfs

[Khem Raj]

On Wed, 2019-11-27 at 07:27 +1300, Ankur Tyagi wrote:
> Hi Khem,
> 
> I have no experience with musl but keen to understand does musl
> improves/impacts DRAM performance as compared to glibc? 
> 

it really depends on load it runs, so I would not say its better or
worse but for many loads it makes better use of memory.

I would recommend to do some benchmarking on your platform and
application

> regards
> Ankur
> 
> On Wed, 27 Nov 2019, 7:17 a.m. Khem Raj, <raj.khem@gmail.com> wrote:
> > On Mon, 2019-11-25 at 23:10 +0200, Adrian Bunk wrote:
> > > On Mon, Nov 25, 2019 at 07:26:58PM +0000, Ross Burton wrote:
> > > > On 25/11/2019 18:37, Ankur Tyagi wrote:
> > > > > Hi,
> > > > > 
> > > > > Based upon "thud" branch, I created core-image-minimal for
> > > > > am335x-evm
> > > > > board and resulting image size is very big(71M)
> > > > 
> > > > If disk size is important then a good first step is to use musl
> > > > instead of
> > > > glibc:
> > > > 
> > > > TCLIBC = "musl"
> > > > ...
> > > 
> > > musl is mainly useful for tiny systems, a 40 MB partition is such
> > a
> > > huge
> > > amount of space that the size of the C library no longer matters
> > > much.
> > > 
> > > The size benefit of replacing glibc with musl is around 2 MB.
> > 
> > I would refrain from labeling as useful for tiny systems. since
> > static size is one thing, there is DRAM behaviour which is equally
> > important, how malloc and other resource oriented algorithms behave
> > is
> > important as well.
> > 
> > secondly, if I have 40MB partition, one might rather want to use
> > extra
> > space for implementing features in their product than spend it for
> > enabling the system if they can.
> > 
> > there are many other design considerations where musl might make
> > more
> > sense
> > 
> > > 
> > > A partition with an -Os build of Yocto with glibc+systemd (sic)
> > > is around 9 MB, together with bootloader and kernel this all
> > > fits in a 16 MB flash. Use compression for kernel and root
> > > filesystem and everything fits in a 8 MB flash.
> > > 
> > 
> > -Os is not as well tested option as -O2 is. So we should always
> > recommend it with grain of salt, speaking from experience there are
> > bugs that will show up with Os which you wont see with O2 and leave
> > a
> > bad taste for user
> > 
> > > > Ross
> > > 
> > > cu
> > > Adrian
> > 

Re: Very large size of libraries in core-image-minimal rootfs

[Adrian Bunk]

On Tue, Nov 26, 2019 at 10:17:09AM -0800, Khem Raj wrote:
> On Mon, 2019-11-25 at 23:10 +0200, Adrian Bunk wrote:
> > On Mon, Nov 25, 2019 at 07:26:58PM +0000, Ross Burton wrote:
> > > On 25/11/2019 18:37, Ankur Tyagi wrote:
> > > > Hi,
> > > > 
> > > > Based upon "thud" branch, I created core-image-minimal for
> > > > am335x-evm
> > > > board and resulting image size is very big(71M)
> > > 
> > > If disk size is important then a good first step is to use musl
> > > instead of
> > > glibc:
> > > 
> > > TCLIBC = "musl"
> > > ...
> > 
> > musl is mainly useful for tiny systems, a 40 MB partition is such a
> > huge
> > amount of space that the size of the C library no longer matters
> > much.
> > 
> > The size benefit of replacing glibc with musl is around 2 MB.
> 
> I would refrain from labeling as useful for tiny systems. since
> static size is one thing, there is DRAM behaviour which is equally
> important, how malloc and other resource oriented algorithms behave is
> important as well.

It is not a clear picture once you ignore "glibc is old and bloated" 
claims, and I am not aware of any reliable objective general overview in 
which cases which C library (or a 3rd party memory allocator) is better.

> secondly, if I have 40MB partition, one might rather want to use extra
> space for implementing features in their product than spend it for
> enabling the system if they can.

If it ain't broken don't fix it.

For the case in question there was already the point that two copies
of libstdc++ were installed, which is a size penalty on the order of 
magnitude of the glibc/musl difference.

And for such huge partitions -Os should bring more benefits than musl,
so that's a more reasonable first choice for size optimization.

> there are many other design considerations where musl might make more
> sense

Which of these are not covered by linking only your application
with musl?

It is not clear that all this outweights the constant pains of forcing 
people to fix the steady stream of new musl issues for all recipes.

> > A partition with an -Os build of Yocto with glibc+systemd (sic)
> > is around 9 MB, together with bootloader and kernel this all
> > fits in a 16 MB flash. Use compression for kernel and root
> > filesystem and everything fits in a 8 MB flash.
> 
> -Os is not as well tested option as -O2 is. So we should always
> recommend it with grain of salt, speaking from experience there are
> bugs that will show up with Os which you wont see with O2 and leave a
> bad taste for user

Where is the grain of salt when recommending musl?

I would consider -Os less risky than musl, especially considering some
of the OE-only patching for software noone else tries to use with musl.
But neither is a huge risk as long as you stay away from known-broken 
musl cases like systemd or elfutils.

On a more general note it is weird how much effort is spent on getting 
things to work with musl, but noone seems to care that this builds and 
works with -Os.

The combinations that are most relevant in practice are:
- normal: -O2 + glibc + systemd
- tiny: -Os + musl + busybox init

Using musl with -O2 or with systemd is far from the primary usecase
of using musl for size benefits.

The other advantage of -Os is that it is also useful for non-tiny 
systems since the benefits scale with the size of the filesystem.
It is a nice knob to fix kernel or userspace space problems that
does not change the API or ABI.

cu
Adrian

Re: Very large size of libraries in core-image-minimal rootfs

[Khem Raj]

On Tue, Nov 26, 2019 at 11:56 AM Adrian Bunk <bunk@stusta.de> wrote:
>
> On Tue, Nov 26, 2019 at 10:17:09AM -0800, Khem Raj wrote:
> > On Mon, 2019-11-25 at 23:10 +0200, Adrian Bunk wrote:
> > > On Mon, Nov 25, 2019 at 07:26:58PM +0000, Ross Burton wrote:
> > > > On 25/11/2019 18:37, Ankur Tyagi wrote:
> > > > > Hi,
> > > > >
> > > > > Based upon "thud" branch, I created core-image-minimal for
> > > > > am335x-evm
> > > > > board and resulting image size is very big(71M)
> > > >
> > > > If disk size is important then a good first step is to use musl
> > > > instead of
> > > > glibc:
> > > >
> > > > TCLIBC = "musl"
> > > > ...
> > >
> > > musl is mainly useful for tiny systems, a 40 MB partition is such a
> > > huge
> > > amount of space that the size of the C library no longer matters
> > > much.
> > >
> > > The size benefit of replacing glibc with musl is around 2 MB.
> >
> > I would refrain from labeling as useful for tiny systems. since
> > static size is one thing, there is DRAM behaviour which is equally
> > important, how malloc and other resource oriented algorithms behave is
> > important as well.
>
> It is not a clear picture once you ignore "glibc is old and bloated"
> claims, and I am not aware of any reliable objective general overview in
> which cases which C library (or a 3rd party memory allocator) is better.
>

Its a good point,  and there are not much benchmarking information openly
reported, but I am sure people have done such benchmarking and analysis
to decide. However, people do refer to such email threads as this.

> > secondly, if I have 40MB partition, one might rather want to use extra
> > space for implementing features in their product than spend it for
> > enabling the system if they can.
>
> If it ain't broken don't fix it.

I dont know if you understood my point above. Its not about being broken or not.

>
> For the case in question there was already the point that two copies
> of libstdc++ were installed, which is a size penalty on the order of
> magnitude of the glibc/musl difference.
>
> And for such huge partitions -Os should bring more benefits than musl,
> so that's a more reasonable first choice for size optimization.
>

As i have said, it does not come free of cost. I can see upfront the cost might
seem to be small.

> > there are many other design considerations where musl might make more
> > sense
>
> Which of these are not covered by linking only your application
> with musl?
>
> It is not clear that all this outweights the constant pains of forcing
> people to fix the steady stream of new musl issues for all recipes.
>
For a moment if you do not assume glibc == linux then
Most of times issues are in application assuming wrong things from system.
in cases where issues are

> > > A partition with an -Os build of Yocto with glibc+systemd (sic)
> > > is around 9 MB, together with bootloader and kernel this all
> > > fits in a 16 MB flash. Use compression for kernel and root
> > > filesystem and everything fits in a 8 MB flash.
> >
> > -Os is not as well tested option as -O2 is. So we should always
> > recommend it with grain of salt, speaking from experience there are
> > bugs that will show up with Os which you wont see with O2 and leave a
> > bad taste for user
>
> Where is the grain of salt when recommending musl?
>

Again, I do not understand, but putting all points in front of user is
best to do
instead of dissing one over another approach, users know their problems
best

> I would consider -Os less risky than musl, especially considering some
> of the OE-only patching for software noone else tries to use with musl.
> But neither is a huge risk as long as you stay away from known-broken
> musl cases like systemd or elfutils.
>

thats more likely the case.

> On a more general note it is weird how much effort is spent on getting
> things to work with musl, but noone seems to care that this builds and
> works with -Os.

I wondered that too, why that is the case. Perhaps in opensource people like
options.

>
> The combinations that are most relevant in practice are:
> - normal: -O2 + glibc + systemd
> - tiny: -Os + musl + busybox init
>

again its two of many combinations  with OE.

> Using musl with -O2 or with systemd is far from the primary usecase
> of using musl for size benefits.
>
> The other advantage of -Os is that it is also useful for non-tiny
> systems since the benefits scale with the size of the filesystem.
> It is a nice knob to fix kernel or userspace space problems that
> does not change the API or ABI.

perhaps, however, reducing code size is a design decision more than
compiler switch problem. Throwing any code at the compiler and expecting
it to optimize for a given vector is never optimal.

>
> cu
> Adrian

Re: Very large size of libraries in core-image-minimal rootfs

[Adrian Bunk]

On Tue, Nov 26, 2019 at 02:57:43PM -0800, Khem Raj wrote:
> On Tue, Nov 26, 2019 at 11:56 AM Adrian Bunk <bunk@stusta.de> wrote:
>...
> > > there are many other design considerations where musl might make more
> > > sense
> >
> > Which of these are not covered by linking only your application
> > with musl?
> >
> > It is not clear that all this outweights the constant pains of forcing
> > people to fix the steady stream of new musl issues for all recipes.
> >
> For a moment if you do not assume glibc == linux then
> Most of times issues are in application assuming wrong things from system.
> in cases where issues are

In general I consider portability a positive thing, but it is also 
legitimate when an upstream makes the decision to use everything
provided by glibc.

But this is not really relevant when discussing what kind of effort 
should be required from people contributing to OE - many build failures 
and work are a price OE pays for musl compatibility, and it can be 
questioned whether it is really worth spending scarce resources on that.

The effort is smaller and the benefits are clearer when supporting musl
only in a limited amount of recipes relevant for tiny systems.

>...
> > The combinations that are most relevant in practice are:
> > - normal: -O2 + glibc + systemd
> > - tiny: -Os + musl + busybox init
> 
> again its two of many combinations  with OE.

There are combinations that match common usecases, and there are 
combinations that don't match common usecases.

Development and testing effort is best spent on combinations that
match common usecases.

poky-tiny building with -O2 is weird.

> > Using musl with -O2 or with systemd is far from the primary usecase
> > of using musl for size benefits.
> >
> > The other advantage of -Os is that it is also useful for non-tiny
> > systems since the benefits scale with the size of the filesystem.
> > It is a nice knob to fix kernel or userspace space problems that
> > does not change the API or ABI.
> 
> perhaps, however, reducing code size is a design decision more than
> compiler switch problem. Throwing any code at the compiler and expecting
> it to optimize for a given vector is never optimal.

The compiler switch is part of the design decision, and on a tiny system 
you do often not have the luxury of picking only some size 
optimizations.

Tiny is when you have to squeeze bootloader, kernel and userspace 
together on 4 MB flash.

cu
Adrian

Re: Very large size of libraries in core-image-minimal rootfs

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 3579 bytes --]

On Tue, Nov 26, 2019 at 10:37 PM Adrian Bunk <bunk@stusta.de> wrote:

> On Tue, Nov 26, 2019 at 02:57:43PM -0800, Khem Raj wrote:
> > On Tue, Nov 26, 2019 at 11:56 AM Adrian Bunk <bunk@stusta.de> wrote:
> >...
> > > > there are many other design considerations where musl might make more
> > > > sense
> > >
> > > Which of these are not covered by linking only your application
> > > with musl?
> > >
> > > It is not clear that all this outweights the constant pains of forcing
> > > people to fix the steady stream of new musl issues for all recipes.
> > >
> > For a moment if you do not assume glibc == linux then
> > Most of times issues are in application assuming wrong things from
> system.
> > in cases where issues are
>
> In general I consider portability a positive thing, but it is also
> legitimate when an upstream makes the decision to use everything
> provided by glibc.
>
> But this is not really relevant when discussing what kind of effort
> should be required from people contributing to OE - many build failures
> and work are a price OE pays for musl compatibility, and it can be
> questioned whether it is really worth spending scarce resources on that.
>
> The effort is smaller and the benefits are clearer when supporting musl
> only in a limited amount of recipes relevant for tiny systems.
>
> >...
> > > The combinations that are most relevant in practice are:
> > > - normal: -O2 + glibc + systemd
> > > - tiny: -Os + musl + busybox init
> >
> > again its two of many combinations  with OE.
>
> There are combinations that match common usecases, and there are
> combinations that don't match common usecases.
>
> Development and testing effort is best spent on combinations that
> match common usecases.


Keeping rat holing and derailing conversation in mind this will be my last
reply to this thread

It’s an opensource project and if contributors stop contributing to musl
effort it will die on its own we do t have to install a kill switch. OE
philosophy is to be a distro builder not a distro that’s key differentiator
and all of these arguments are very distro centric mindset. We do ask for
keeping builds clean and help with servicing the part of project that
interests a person, so pitch in what interests and hopefully there will be
enough of us around to support OE



>
> poky-tiny building with -O2 is weird.
>
> > > Using musl with -O2 or with systemd is far from the primary usecase
> > > of using musl for size benefits.
> > >
> > > The other advantage of -Os is that it is also useful for non-tiny
> > > systems since the benefits scale with the size of the filesystem.
> > > It is a nice knob to fix kernel or userspace space problems that
> > > does not change the API or ABI.
> >
> > perhaps, however, reducing code size is a design decision more than
> > compiler switch problem. Throwing any code at the compiler and expecting
> > it to optimize for a given vector is never optimal.
>
> The compiler switch is part of the design decision, and on a tiny system
> you do often not have the luxury of picking only some size
> optimizations.


And not many upstream packages are built and tested regularly with Os leave
aside writing size friendly code in their dev streams so perhaps someone
should help them provide help testing with Os things will get better


>
> Tiny is when you have to squeeze bootloader, kernel and userspace
> together on 4 MB flash


Why 4 and not 2 or 16 fact is it depends on usecase


>
> cu
> Adrian
>

[-- Attachment #2: Type: text/html, Size: 4799 bytes --]