ppc64le vmlinuz is huge when building with BTF

ppc64le vmlinuz is huge when building with BTF

[Dominique Martinet]

Hi,

coming from alpine: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12563

alice noticed the kernel packages got quite bigger, in particular for
ppc64le I've confirmed that the vmlinuz file size jump when building
with BTF:
currently released package with BTF:
https://dl-cdn.alpinelinux.org/alpine/edge/main/ppc64le/linux-lts-6.1.33-r0.apk
272M	boot/vmlinuz-lts

test build without BTF:
https://gitlab.alpinelinux.org/martinetd/aports/-/jobs/1049335
44M	boot/vmlinuz-lts


Is that a known issue?
We'll probably just turn off BTF for the ppc64le build for now, but it
might be worth checking.


While I have your attention, even the x86_64 package grew much bigger
than I thought it would, the installed modules directory go from 90MB to
108MB gzipped); it's a 18% increase (including kernel: 103->122MB) which
is more than what I'd expect out of BTF.
Most users don't care about BTF so it'd be great if they could be built
and installed separately (debug package all over again..) or limiting
the growth a bit more if possible.
I haven't tried yet but at this point ikheaders is probably worth
considering instead..
Perhaps we're missing some stripping option or something?


Thanks!
-- 
Dominique Martinet | Asmadeus

Re: ppc64le vmlinuz is huge when building with BTF

[Jiri Olsa]

On Thu, Jun 15, 2023 at 12:32:24PM +0900, Dominique Martinet wrote:
> Hi,
> 
> coming from alpine: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12563

hi,
it's probably burried somewhere in that discussion, but do you have
kernel version (or commit) where that increase happened?

also link for used config would be great

thanks,
jirka

> 
> alice noticed the kernel packages got quite bigger, in particular for
> ppc64le I've confirmed that the vmlinuz file size jump when building
> with BTF:
> currently released package with BTF:
> https://dl-cdn.alpinelinux.org/alpine/edge/main/ppc64le/linux-lts-6.1.33-r0.apk
> 272M	boot/vmlinuz-lts
> 
> test build without BTF:
> https://gitlab.alpinelinux.org/martinetd/aports/-/jobs/1049335
> 44M	boot/vmlinuz-lts
> 
> 
> Is that a known issue?
> We'll probably just turn off BTF for the ppc64le build for now, but it
> might be worth checking.
> 
> 
> While I have your attention, even the x86_64 package grew much bigger
> than I thought it would, the installed modules directory go from 90MB to
> 108MB gzipped); it's a 18% increase (including kernel: 103->122MB) which
> is more than what I'd expect out of BTF.
> Most users don't care about BTF so it'd be great if they could be built
> and installed separately (debug package all over again..) or limiting
> the growth a bit more if possible.
> I haven't tried yet but at this point ikheaders is probably worth
> considering instead..
> Perhaps we're missing some stripping option or something?
> 
> 
> Thanks!
> -- 
> Dominique Martinet | Asmadeus
> 

Re: ppc64le vmlinuz is huge when building with BTF

[Dominique Martinet]

Jiri Olsa wrote on Thu, Jun 15, 2023 at 10:39:18AM +0200:
> > coming from alpine: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12563
> 
> it's probably burried somewhere in that discussion, but do you have
> kernel version (or commit) where that increase happened?

Unfortunately not -- we've just tried on the 6.1.33 that's the current
alpine lts kernel, but I cannot say since when this started because
we've just enabled BTF recently.

Alpine also doesn't seem to keep old versions of apk files on its
mirrors so while it probably has been happening since it got enabled I
don't know how to check, and the commit enabling BTF has been done
without a MR so there's no test log that'd allow seeing the package size
either :/

> also link for used config would be great

alpine has two configs which both exhibit the issue (raw file link on
commit before removing BTF):
https://gitlab.alpinelinux.org/alpine/aports/-/raw/749ee7117e1437b7ab3ef2590f7f2e3558fda3ef/main/linux-lts/virt.ppc64le.config
Size difference for linux-virt: 219 MiB -> 47 MiB
https://gitlab.alpinelinux.org/alpine/aports/-/raw/749ee7117e1437b7ab3ef2590f7f2e3558fda3ef/main/linux-lts/lts.ppc64le.config
Size difference for linux-lts: 306 MiB -> 74 MiB


We just disabled BTF for this arch for now, I honestly won't have time
to dig further short term as I'm not that involved in this issue and
day job is busy right now.

Thanks,
-- 
Dominique

Re: ppc64le vmlinuz is huge when building with BTF

[Alan Maguire]

On 15/06/2023 12:52, Dominique Martinet wrote:
> Jiri Olsa wrote on Thu, Jun 15, 2023 at 10:39:18AM +0200:
>>> coming from alpine: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12563
>>
>> it's probably burried somewhere in that discussion, but do you have
>> kernel version (or commit) where that increase happened?
> 
> Unfortunately not -- we've just tried on the 6.1.33 that's the current
> alpine lts kernel, but I cannot say since when this started because
> we've just enabled BTF recently.
> 
> Alpine also doesn't seem to keep old versions of apk files on its
> mirrors so while it probably has been happening since it got enabled I
> don't know how to check, and the commit enabling BTF has been done
> without a MR so there's no test log that'd allow seeing the package size
> either :/
> 
>> also link for used config would be great
> 
> alpine has two configs which both exhibit the issue (raw file link on
> commit before removing BTF):
> https://gitlab.alpinelinux.org/alpine/aports/-/raw/749ee7117e1437b7ab3ef2590f7f2e3558fda3ef/main/linux-lts/virt.ppc64le.config
> Size difference for linux-virt: 219 MiB -> 47 MiB
> https://gitlab.alpinelinux.org/alpine/aports/-/raw/749ee7117e1437b7ab3ef2590f7f2e3558fda3ef/main/linux-lts/lts.ppc64le.config
> Size difference for linux-lts: 306 MiB -> 74 MiB
>

I took a look (gunzip'ed and tar-extracted the apk to get the
vmlinuz-lts file) and the BTF looks reasonable size-wise:

$ objdump -h vmlinuz-lts
...

 13 .BTF          003d64cc  c000000001400844  0000000001400844  01410844
 2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 14 .BTF_ids      00000248  c0000000017d6d10  00000000017d6d10  017e6d10
 2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA

So BTF is ~4MB if I'm reading the above right.

However the problem I suspect is this:

 51 .debug_info   0a488b55  0000000000000000  0000000000000000  026f8d20
 2**0
                  CONTENTS, READONLY, DEBUGGING
 52 .debug_abbrev 004d66f0  0000000000000000  0000000000000000  0cb81875
 2**0
                  CONTENTS, READONLY, DEBUGGING
 53 .debug_line   019192f6  0000000000000000  0000000000000000  0d057f65
 2**0
                  CONTENTS, READONLY, DEBUGGING
 54 .debug_frame  002d4fe0  0000000000000000  0000000000000000  0e971260
 2**3
                  CONTENTS, READONLY, DEBUGGING
 55 .debug_str    00390822  0000000000000000  0000000000000000  0ec46240
 2**0
                  CONTENTS, READONLY, DEBUGGING
 56 .debug_line_str 00000445  0000000000000000  0000000000000000
0efd6a62  2**0
                  CONTENTS, READONLY, DEBUGGING
 57 .debug_loclists 0176a32c  0000000000000000  0000000000000000
0efd6ea7  2**0
                  CONTENTS, READONLY, DEBUGGING
 58 .debug_rnglists 00382d7a  0000000000000000  0000000000000000
107411d3  2**0
                  CONTENTS, READONLY, DEBUGGING

The debug info hasn't been stripped, so I suspect the packaging spec
file or equivalent - in perhaps trying to preserve the .BTF section -
is preserving debug info too. DWARF needs to be there at BTF
generation time in vmlinux but is usually stripped for non-debug
packages.

FYI we're aiming to make BTF module-loadable via CONFIG_DEBUG_INFO_BTF=m
in the future, I'm hoping to get an RFC patch out for that soon once
other BTF-related issues are sorted. Hope this helps

Alan

Re: ppc64le vmlinuz is huge when building with BTF

[Eduard Zingerman]

On Thu, 2023-06-15 at 12:32 +0900, Dominique Martinet wrote:
> Hi,
> 
> coming from alpine: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12563
> 
> alice noticed the kernel packages got quite bigger, in particular for
> ppc64le I've confirmed that the vmlinuz file size jump when building
> with BTF:
> currently released package with BTF:
> https://dl-cdn.alpinelinux.org/alpine/edge/main/ppc64le/linux-lts-6.1.33-r0.apk
> 272M	boot/vmlinuz-lts

Hi Dominique,

I've just checked the linked apk and it looks like DWARF sections are
not stripped and take most of the space, e.g.:

$ llvm-objdump --headers boot/vmlinuz-lts | grep \.debug \
  | gawk '{ sum += strtonum("0x"$3); } END { print sum; }' | numfmt --to iec
228M

Compare this to BTF sections size:

$ llvm-objdump --headers boot/vmlinuz-lts | grep BTF \
  | gawk '{ sum += strtonum("0x"$3); } END { print sum; }' | numfmt --to iec
3,9M

> 
> test build without BTF:
> https://gitlab.alpinelinux.org/martinetd/aports/-/jobs/1049335
> 44M	boot/vmlinuz-lts
> 
> 
> Is that a known issue?
> We'll probably just turn off BTF for the ppc64le build for now, but it
> might be worth checking.
> 
> 
> While I have your attention, even the x86_64 package grew much bigger
> than I thought it would, the installed modules directory go from 90MB to
> 108MB gzipped); it's a 18% increase (including kernel: 103->122MB) which
> is more than what I'd expect out of BTF.
> Most users don't care about BTF so it'd be great if they could be built
> and installed separately (debug package all over again..) or limiting
> the growth a bit more if possible.
> I haven't tried yet but at this point ikheaders is probably worth
> considering instead..
> Perhaps we're missing some stripping option or something?
> 
> 
> Thanks!

Re: ppc64le vmlinuz is huge when building with BTF

[Eduard Zingerman]

On Thu, 2023-06-15 at 18:15 +0300, Eduard Zingerman wrote:
> On Thu, 2023-06-15 at 12:32 +0900, Dominique Martinet wrote:
> > Hi,
> > 
> > coming from alpine: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12563
> > 
> > alice noticed the kernel packages got quite bigger, in particular for
> > ppc64le I've confirmed that the vmlinuz file size jump when building
> > with BTF:
> > currently released package with BTF:
> > https://dl-cdn.alpinelinux.org/alpine/edge/main/ppc64le/linux-lts-6.1.33-r0.apk
> > 272M	boot/vmlinuz-lts
> 
> Hi Dominique,
> 
> I've just checked the linked apk and it looks like DWARF sections are
> not stripped and take most of the space, e.g.:
> 
> $ llvm-objdump --headers boot/vmlinuz-lts | grep \.debug \
>   | gawk '{ sum += strtonum("0x"$3); } END { print sum; }' | numfmt --to iec
> 228M
> 
> Compare this to BTF sections size:
> 
> $ llvm-objdump --headers boot/vmlinuz-lts | grep BTF \
>   | gawk '{ sum += strtonum("0x"$3); } END { print sum; }' | numfmt --to iec
> 3,9M

Heh, Alan answered while I was looking into it, sorry for the spam.

> 
> > 
> > test build without BTF:
> > https://gitlab.alpinelinux.org/martinetd/aports/-/jobs/1049335
> > 44M	boot/vmlinuz-lts
> > 
> > 
> > Is that a known issue?
> > We'll probably just turn off BTF for the ppc64le build for now, but it
> > might be worth checking.
> > 
> > 
> > While I have your attention, even the x86_64 package grew much bigger
> > than I thought it would, the installed modules directory go from 90MB to
> > 108MB gzipped); it's a 18% increase (including kernel: 103->122MB) which
> > is more than what I'd expect out of BTF.
> > Most users don't care about BTF so it'd be great if they could be built
> > and installed separately (debug package all over again..) or limiting
> > the growth a bit more if possible.
> > I haven't tried yet but at this point ikheaders is probably worth
> > considering instead..
> > Perhaps we're missing some stripping option or something?
> > 
> > 
> > Thanks!
> 

Re: ppc64le vmlinuz is huge when building with BTF

[Dominique Martinet]

Alan Maguire wrote on Thu, Jun 15, 2023 at 03:31:49PM +0100:
> However the problem I suspect is this:
> 
>  51 .debug_info   0a488b55  0000000000000000  0000000000000000  026f8d20
>  2**0
>                   CONTENTS, READONLY, DEBUGGING
> [...]
> 
> The debug info hasn't been stripped, so I suspect the packaging spec
> file or equivalent - in perhaps trying to preserve the .BTF section -
> is preserving debug info too. DWARF needs to be there at BTF
> generation time in vmlinux but is usually stripped for non-debug
> packages.

Thanks Alan and Eduard!
I guess I should have checked that first, it helps.

We're not stripping anything in vmlinuz for other archs -- the linker
script already should be including only the bare minimum to decompress
itself (+compressed useful bits), so I guess it's a Kbuild issue for the
arch.
We can add a strip but I unfortunately have no way of testing ppc build,
I'll ask around the build linux-kbuild and linuxppc-dev lists if that's
expected; it shouldn't be that bad now that's figured out.


> FYI we're aiming to make BTF module-loadable via CONFIG_DEBUG_INFO_BTF=m
> in the future, I'm hoping to get an RFC patch out for that soon once
> other BTF-related issues are sorted. Hope this helps

Oh, that's interesting -- I assume that'll only change the 'built-in'
BTF info? Or will that also split BTF info in other modules as
e.g. modfoo_btf.ko?
For x86_64 the size increase of vmlinuz itself is rather acceptable
(<2MB), but the sheer amount of modules (the -lts package has over 3k
modules...) means that even a small size increase for each module ends
up taking proportionally a high amount of space (+20MB from 90MB), so
being able to package separately would be appreciated (alpine likes
splitting optional features in subpackages)
Packaging-wise I'm not sure it'd make sense to keep the overhead in
other modules and just split the 'main' btf infos.

Otoh even if it doesn't help with packaging, having a smaller vmlinuz
means faster boot and lower kernel memory footprint (I think *2?) for
people who don't need it, so I think it's a good idea and we'll probably
enable it once it becomes available. Thanks for the heads up!

-- 
Dominique Martinet | Asmadeus

Re: ppc64le vmlinuz is huge when building with BTF

[Andrii Nakryiko]

On Thu, Jun 15, 2023 at 1:35 PM Dominique Martinet
<asmadeus@codewreck.org> wrote:
>
>
> Alan Maguire wrote on Thu, Jun 15, 2023 at 03:31:49PM +0100:
> > However the problem I suspect is this:
> >
> >  51 .debug_info   0a488b55  0000000000000000  0000000000000000  026f8d20
> >  2**0
> >                   CONTENTS, READONLY, DEBUGGING
> > [...]
> >
> > The debug info hasn't been stripped, so I suspect the packaging spec
> > file or equivalent - in perhaps trying to preserve the .BTF section -
> > is preserving debug info too. DWARF needs to be there at BTF
> > generation time in vmlinux but is usually stripped for non-debug
> > packages.
>
> Thanks Alan and Eduard!
> I guess I should have checked that first, it helps.
>
> We're not stripping anything in vmlinuz for other archs -- the linker
> script already should be including only the bare minimum to decompress
> itself (+compressed useful bits), so I guess it's a Kbuild issue for the
> arch.
> We can add a strip but I unfortunately have no way of testing ppc build,
> I'll ask around the build linux-kbuild and linuxppc-dev lists if that's
> expected; it shouldn't be that bad now that's figured out.
>
>
> > FYI we're aiming to make BTF module-loadable via CONFIG_DEBUG_INFO_BTF=m
> > in the future, I'm hoping to get an RFC patch out for that soon once
> > other BTF-related issues are sorted. Hope this helps
>

Besides this, in the past it was requested a few times to make
CONFIG_DEBUG_INFO_BTF=y not imply CONFIG_DEBUG_INFO_DWARF. That is,
make it possible to get BTF without .dwarf* sections being left in the
vmlinux image.

Of course, implementation-wise, DWARF would still have to be generated
at runtime, but would be automatically stripped out after BTF
generation, unless CONFIG_DEBUG_INFO_DWARF* is set.

I think it would help in this case as well. So if anyone has free
cycles, it would be awesome to make this possible.


> Oh, that's interesting -- I assume that'll only change the 'built-in'
> BTF info? Or will that also split BTF info in other modules as
> e.g. modfoo_btf.ko?
> For x86_64 the size increase of vmlinuz itself is rather acceptable
> (<2MB), but the sheer amount of modules (the -lts package has over 3k
> modules...) means that even a small size increase for each module ends
> up taking proportionally a high amount of space (+20MB from 90MB), so
> being able to package separately would be appreciated (alpine likes
> splitting optional features in subpackages)
> Packaging-wise I'm not sure it'd make sense to keep the overhead in
> other modules and just split the 'main' btf infos.
>
> Otoh even if it doesn't help with packaging, having a smaller vmlinuz
> means faster boot and lower kernel memory footprint (I think *2?) for
> people who don't need it, so I think it's a good idea and we'll probably
> enable it once it becomes available. Thanks for the heads up!
>
> --
> Dominique Martinet | Asmadeus
>

Re: ppc64le vmlinuz is huge when building with BTF

[Naveen N Rao]

[Cc linuxppc-dev]

Dominique Martinet wrote:
> 
> Alan Maguire wrote on Thu, Jun 15, 2023 at 03:31:49PM +0100:
>> However the problem I suspect is this:
>> 
>>  51 .debug_info   0a488b55  0000000000000000  0000000000000000  026f8d20
>>  2**0
>>                   CONTENTS, READONLY, DEBUGGING
>> [...]
>> 
>> The debug info hasn't been stripped, so I suspect the packaging spec
>> file or equivalent - in perhaps trying to preserve the .BTF section -
>> is preserving debug info too. DWARF needs to be there at BTF
>> generation time in vmlinux but is usually stripped for non-debug
>> packages.
> 
> Thanks Alan and Eduard!
> I guess I should have checked that first, it helps.
> 
> We're not stripping anything in vmlinuz for other archs -- the linker
> script already should be including only the bare minimum to decompress
> itself (+compressed useful bits), so I guess it's a Kbuild issue for the
> arch.

For a related discussion, see:
http://lore.kernel.org/CAK18DXZKs2PNmLndeGYqkPxmrrBR=6ca3bhyYCj=GhyA7dHfAQ@mail.gmail.com

> We can add a strip but I unfortunately have no way of testing ppc build,
> I'll ask around the build linux-kbuild and linuxppc-dev lists if that's
> expected; it shouldn't be that bad now that's figured out.

Stripping vmlinux would indeed be the way to go. As mentioned in the 
above link, fedora also packages a strip'ed vmlinux for ppc64le:
https://src.fedoraproject.org/rpms/kernel/blob/4af17bffde7a1eca9ab164e5de0e391c277998a4/f/kernel.spec#_1797


- Naveen

Re: ppc64le vmlinuz is huge when building with BTF

[Dominique Martinet]

Naveen N Rao wrote on Fri, Jun 16, 2023 at 04:28:53PM +0530:
> > We're not stripping anything in vmlinuz for other archs -- the linker
> > script already should be including only the bare minimum to decompress
> > itself (+compressed useful bits), so I guess it's a Kbuild issue for the
> > arch.
> 
> For a related discussion, see:
> http://lore.kernel.org/CAK18DXZKs2PNmLndeGYqkPxmrrBR=6ca3bhyYCj=GhyA7dHfAQ@mail.gmail.com

Thanks, I didn't know that ppc64le boots straight into vmlinux, as 'make
install' somehow installs something called 'vmlinuz-lts' (-lts coming
out of localversion afaiu, but vmlinuz would come from the build
scripts) ; this is somewhat confusing as vmlinuz on other archs is a
compressed/pre-processed binary so I'd expect it to at least be
stripped...

> > We can add a strip but I unfortunately have no way of testing ppc build,
> > I'll ask around the build linux-kbuild and linuxppc-dev lists if that's
> > expected; it shouldn't be that bad now that's figured out.
> 
> Stripping vmlinux would indeed be the way to go. As mentioned in the above
> link, fedora also packages a strip'ed vmlinux for ppc64le:
> https://src.fedoraproject.org/rpms/kernel/blob/4af17bffde7a1eca9ab164e5de0e391c277998a4/f/kernel.spec#_1797

It feels somewhat wrong to add a strip just for ppc64le after make
install, but I guess we probably ought to do the same...
I don't have any hardware to test booting the result though, I'll submit
an update and ask for someone to test when it's done.
(bit busy but that doesn't take long, will do that tomorrow morning
before I forget)

Thanks!
-- 
Dominique Martinet | Asmadeus

Re: ppc64le vmlinuz is huge when building with BTF

[Naveen N Rao]

Dominique Martinet wrote:
> Naveen N Rao wrote on Fri, Jun 16, 2023 at 04:28:53PM +0530:
>> > We're not stripping anything in vmlinuz for other archs -- the linker
>> > script already should be including only the bare minimum to decompress
>> > itself (+compressed useful bits), so I guess it's a Kbuild issue for the
>> > arch.
>> 
>> For a related discussion, see:
>> http://lore.kernel.org/CAK18DXZKs2PNmLndeGYqkPxmrrBR=6ca3bhyYCj=GhyA7dHfAQ@mail.gmail.com
> 
> Thanks, I didn't know that ppc64le boots straight into vmlinux, as 'make
> install' somehow installs something called 'vmlinuz-lts' (-lts coming
> out of localversion afaiu, but vmlinuz would come from the build
> scripts) ; this is somewhat confusing as vmlinuz on other archs is a
> compressed/pre-processed binary so I'd expect it to at least be
> stripped...

As far as I can tell, kernel's install script doesn't give out that 
name, so 'vmlinuz' is likely coming from the distro's 
/[s]bin/installkernel script. It probably needs an override to retain 
'vmlinux'. 

> 
>> > We can add a strip but I unfortunately have no way of testing ppc build,
>> > I'll ask around the build linux-kbuild and linuxppc-dev lists if that's
>> > expected; it shouldn't be that bad now that's figured out.
>> 
>> Stripping vmlinux would indeed be the way to go. As mentioned in the above
>> link, fedora also packages a strip'ed vmlinux for ppc64le:
>> https://src.fedoraproject.org/rpms/kernel/blob/4af17bffde7a1eca9ab164e5de0e391c277998a4/f/kernel.spec#_1797
> 
> It feels somewhat wrong to add a strip just for ppc64le after make
> install, but I guess we probably ought to do the same...
> I don't have any hardware to test booting the result though, I'll submit
> an update and ask for someone to test when it's done.
> (bit busy but that doesn't take long, will do that tomorrow morning
> before I forget)

Thanks! You're right that it's likely just powerpc that is different 
here. It sure would be nice if we can iron out issues with our zImage.


- Naveen