[Buildroot] [PATCH 1/1] linux: add compressions for appended initramfs

[Buildroot] [PATCH 1/1] linux: add compressions for appended initramfs

[Hanspeter Portner]

This changes take effect in the case of BR2_TARGET_ROOTFS_INITRAMFS=y
and one of BR2_LINUX_KERNEL_{GZIP,LZ4,LZMA,LZO,XZ}=y.

Instead of using the kernel's default compression (gzip) for appended initramfs,
this changes set initramfs compression explicitely to the same compression as
the kernel itself, which seems a sensible thing to do, imho.

Apart from these kernel config parameters ...

* CONFIG_KERNEL_GZIP
* CONFIG_KERNEL_LZ4
* CONFIG_KERNEL_LZMA
* CONFIG_KERNEL_LZO
* CONFIG_KERNEL_XZ
* CONFIG_KERNEL_ZSTD
* CONFIG_KERNEL_UNCOMPRESSED

... we now also set these new kernel config parameters, as without those,
the appended initramfs seems to always be compressed with gzip:

* CONFIG_INITRAMFS_COMPRESSION_GZIP
* CONFIG_INITRAMFS_COMPRESSION_LZ4
* CONFIG_INITRAMFS_COMPRESSION_LZMA
* CONFIG_INITRAMFS_COMPRESSION_LZO
* CONFIG_INITRAMFS_COMPRESSION_XZ
* CONFIG_INITRAMFS_COMPRESSION_NONE

The former are automatically set depending on whether one of the
following buildroot config parameters is set analogously to how it is done
for CONFIG_KERNEL_{GZIP,LZ4,LZMA,LZO,XZ,ZSTD,UNCOMPRESSED}:

* BR2_LINUX_KERNEL_GZIP
* BR2_LINUX_KERNEL_LZ4
* BR2_LINUX_KERNEL_LZMA
* BR2_LINUX_KERNEL_LZO
* BR2_LINUX_KERNEL_XZ
* BR2_LINUX_KERNEL_UNCOMPRESSED

Signed-off-by: Hanspeter Portner <dev@open-music-kontrollers.ch>
---
 linux/linux.mk | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/linux/linux.mk b/linux/linux.mk
index 61fdc0c76c..f192341a32 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -106,6 +106,15 @@ LINUX_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_XZ) += CONFIG_KERNEL_XZ
 LINUX_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_ZSTD) += CONFIG_KERNEL_ZSTD
 LINUX_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_UNCOMPRESSED) += CONFIG_KERNEL_UNCOMPRESSED
 
+INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_GZIP) += CONFIG_INITRAMFS_COMPRESSION_GZIP
+INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_LZ4) += CONFIG_INITRAMFS_COMPRESSION_LZ4
+INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_LZMA) += CONFIG_INITRAMFS_COMPRESSION_LZMA
+INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_LZO) += CONFIG_INITRAMFS_COMPRESSION_LZO
+INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_XZ) += CONFIG_INITRAMFS_COMPRESSION_XZ
+# zstd compression seems not supported, yet
+#INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_ZSTD) += CONFIG_INITRAMFS_COMPRESSION_ZSTD
+INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_UNCOMPRESSED) += CONFIG_INITRAMFS_COMPRESSION_NONE
+
 ifeq ($(BR2_LINUX_KERNEL_NEEDS_HOST_OPENSSL),y)
 LINUX_DEPENDENCIES += host-openssl
 endif
@@ -352,6 +361,10 @@ define LINUX_KCONFIG_FIXUP_CMDS
 		$(call KCONFIG_SET_OPT,CONFIG_INITRAMFS_SOURCE,"$${BR_BINARIES_DIR}/rootfs.cpio")
 		$(call KCONFIG_SET_OPT,CONFIG_INITRAMFS_ROOT_UID,0)
 		$(call KCONFIG_SET_OPT,CONFIG_INITRAMFS_ROOT_GID,0))
+		$(call KCONFIG_ENABLE_OPT,$(strip $(INITRAMFS_COMPRESSION_OPT_y)))
+		$(foreach opt, $(INITRAMFS_COMPRESSION_OPT_),
+			$(call KCONFIG_DISABLE_OPT,$(opt))
+		)
 	$(if $(BR2_ROOTFS_DEVICE_CREATION_STATIC),,
 		$(call KCONFIG_ENABLE_OPT,CONFIG_DEVTMPFS)
 		$(call KCONFIG_ENABLE_OPT,CONFIG_DEVTMPFS_MOUNT))
-- 
2.33.0

_______________________________________________
buildroot mailing list
buildroot@lists.buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] linux: add compressions for appended initramfs

[Yann E. MORIN]

Hanspeter, All,

On 2021-09-17 15:18 +0200, Hanspeter Portner spake thusly:
> This changes take effect in the case of BR2_TARGET_ROOTFS_INITRAMFS=y
> and one of BR2_LINUX_KERNEL_{GZIP,LZ4,LZMA,LZO,XZ}=y.
> 
> Instead of using the kernel's default compression (gzip) for appended initramfs,
> this changes set initramfs compression explicitely to the same compression as
> the kernel itself, which seems a sensible thing to do, imho.

Hmmm... But isn't the initranfs compressed as part of the kernel
already? See help for CONFIG_INITRAMFS_COMPRESSION_NONE:

    Do not compress the built-in initramfs at all. This may sound
    wasteful in space, but, you should be aware that the built-in
    initramfs will be compressed at a later stage anyways along
    with the rest of the kernel, on those architectures that support
    this. However, not compressing the initramfs may lead to slightly
    higher memory consumption during a short time at boot, while both
    the cpio image and the unpacked filesystem image will be present
    in memory simultaneously

Additionally, compressing the initramfs separately, means that it will
be copmopressed twice at build time, thus increasing the build time (but
who cares? ;-) ), but also means it will be decompressed twice, thus
increasing the boot time, every time the device boots.

So, forcibly setting the compression to that of the kernel seems totally
superfluous, _as kernel image size matters_; indeed, simply disabling
initramfs compression (actually, cpio compression) would seem like the
only correct thing to do.

The only reason when we would want to actually compress the initramfs,
as hinted in the help text I quoted above, is for machines that are
tight on RAM, so tight indeed that they can't easily fit the whole
uncompressed cpio and the extracted initramfs at the same time at boot
(the uncompressed cpio is eventually garbage-collected once the
initramfs is fully populated).

Nowadays, that probably means machine with less than a very few MiB of
RAM, something around 4MiB. Those kind of machines do exist, but they
are often better served without using an initramfs (i.e. a root on a
flash or something), or are going to require a very tiny initramfs
anyway.

So, I wonder if we should not simply disable compression of the
initramfs altogether...

Could look into that, and look at actual numbers for the resulting
kernel image when the initramfs is compressed or not?

Regards,
Yann E. MORIN.

> Apart from these kernel config parameters ...
> 
> * CONFIG_KERNEL_GZIP
> * CONFIG_KERNEL_LZ4
> * CONFIG_KERNEL_LZMA
> * CONFIG_KERNEL_LZO
> * CONFIG_KERNEL_XZ
> * CONFIG_KERNEL_ZSTD
> * CONFIG_KERNEL_UNCOMPRESSED
> 
> ... we now also set these new kernel config parameters, as without those,
> the appended initramfs seems to always be compressed with gzip:
> 
> * CONFIG_INITRAMFS_COMPRESSION_GZIP
> * CONFIG_INITRAMFS_COMPRESSION_LZ4
> * CONFIG_INITRAMFS_COMPRESSION_LZMA
> * CONFIG_INITRAMFS_COMPRESSION_LZO
> * CONFIG_INITRAMFS_COMPRESSION_XZ
> * CONFIG_INITRAMFS_COMPRESSION_NONE
> 
> The former are automatically set depending on whether one of the
> following buildroot config parameters is set analogously to how it is done
> for CONFIG_KERNEL_{GZIP,LZ4,LZMA,LZO,XZ,ZSTD,UNCOMPRESSED}:
> 
> * BR2_LINUX_KERNEL_GZIP
> * BR2_LINUX_KERNEL_LZ4
> * BR2_LINUX_KERNEL_LZMA
> * BR2_LINUX_KERNEL_LZO
> * BR2_LINUX_KERNEL_XZ
> * BR2_LINUX_KERNEL_UNCOMPRESSED
> 
> Signed-off-by: Hanspeter Portner <dev@open-music-kontrollers.ch>
> ---
>  linux/linux.mk | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/linux/linux.mk b/linux/linux.mk
> index 61fdc0c76c..f192341a32 100644
> --- a/linux/linux.mk
> +++ b/linux/linux.mk
> @@ -106,6 +106,15 @@ LINUX_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_XZ) += CONFIG_KERNEL_XZ
>  LINUX_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_ZSTD) += CONFIG_KERNEL_ZSTD
>  LINUX_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_UNCOMPRESSED) += CONFIG_KERNEL_UNCOMPRESSED
>  
> +INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_GZIP) += CONFIG_INITRAMFS_COMPRESSION_GZIP
> +INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_LZ4) += CONFIG_INITRAMFS_COMPRESSION_LZ4
> +INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_LZMA) += CONFIG_INITRAMFS_COMPRESSION_LZMA
> +INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_LZO) += CONFIG_INITRAMFS_COMPRESSION_LZO
> +INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_XZ) += CONFIG_INITRAMFS_COMPRESSION_XZ
> +# zstd compression seems not supported, yet
> +#INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_ZSTD) += CONFIG_INITRAMFS_COMPRESSION_ZSTD
> +INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_UNCOMPRESSED) += CONFIG_INITRAMFS_COMPRESSION_NONE
> +
>  ifeq ($(BR2_LINUX_KERNEL_NEEDS_HOST_OPENSSL),y)
>  LINUX_DEPENDENCIES += host-openssl
>  endif
> @@ -352,6 +361,10 @@ define LINUX_KCONFIG_FIXUP_CMDS
>  		$(call KCONFIG_SET_OPT,CONFIG_INITRAMFS_SOURCE,"$${BR_BINARIES_DIR}/rootfs.cpio")
>  		$(call KCONFIG_SET_OPT,CONFIG_INITRAMFS_ROOT_UID,0)
>  		$(call KCONFIG_SET_OPT,CONFIG_INITRAMFS_ROOT_GID,0))
> +		$(call KCONFIG_ENABLE_OPT,$(strip $(INITRAMFS_COMPRESSION_OPT_y)))
> +		$(foreach opt, $(INITRAMFS_COMPRESSION_OPT_),
> +			$(call KCONFIG_DISABLE_OPT,$(opt))
> +		)
>  	$(if $(BR2_ROOTFS_DEVICE_CREATION_STATIC),,
>  		$(call KCONFIG_ENABLE_OPT,CONFIG_DEVTMPFS)
>  		$(call KCONFIG_ENABLE_OPT,CONFIG_DEVTMPFS_MOUNT))
> -- 
> 2.33.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@lists.buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@lists.buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] linux: add compressions for appended initramfs

[Arnout Vandecappelle]

On 18/09/2021 09:33, Yann E. MORIN wrote:
> Hanspeter, All,
> 
> On 2021-09-17 15:18 +0200, Hanspeter Portner spake thusly:
>> This changes take effect in the case of BR2_TARGET_ROOTFS_INITRAMFS=y
>> and one of BR2_LINUX_KERNEL_{GZIP,LZ4,LZMA,LZO,XZ}=y.
>>
>> Instead of using the kernel's default compression (gzip) for appended initramfs,
>> this changes set initramfs compression explicitely to the same compression as
>> the kernel itself, which seems a sensible thing to do, imho.
> 
> Hmmm... But isn't the initranfs compressed as part of the kernel
> already? See help for CONFIG_INITRAMFS_COMPRESSION_NONE:
> 
>      Do not compress the built-in initramfs at all. This may sound
>      wasteful in space, but, you should be aware that the built-in
>      initramfs will be compressed at a later stage anyways along
>      with the rest of the kernel, on those architectures that support
>      this. However, not compressing the initramfs may lead to slightly
>      higher memory consumption during a short time at boot, while both
>      the cpio image and the unpacked filesystem image will be present
>      in memory simultaneously
> 
> Additionally, compressing the initramfs separately, means that it will
> be copmopressed twice at build time, thus increasing the build time (but
> who cares? ;-) ), but also means it will be decompressed twice, thus
> increasing the boot time, every time the device boots.
> 
> So, forcibly setting the compression to that of the kernel seems totally
> superfluous, _as kernel image size matters_; indeed, simply disabling
> initramfs compression (actually, cpio compression) would seem like the
> only correct thing to do.
> 
> The only reason when we would want to actually compress the initramfs,
> as hinted in the help text I quoted above, is for machines that are
> tight on RAM, so tight indeed that they can't easily fit the whole
> uncompressed cpio and the extracted initramfs at the same time at boot
> (the uncompressed cpio is eventually garbage-collected once the
> initramfs is fully populated).
> 
> Nowadays, that probably means machine with less than a very few MiB of
> RAM, something around 4MiB. Those kind of machines do exist, but they
> are often better served without using an initramfs (i.e. a root on a
> flash or something), or are going to require a very tiny initramfs
> anyway.
> 
> So, I wonder if we should not simply disable compression of the
> initramfs altogether...

  The current situation allows the user to choose through the kernel config, so 
sounds like the optimal approach. Unfortunately, the kernel apparently does 
default to gzip instead of uncompressed, so there's something to be said for 
forcing it to uncompressed...

  Anyway, setting it to the same as the kernel compression is useless I think. 
Therefore, I marked this patch as Rejected.

  Regards,
  Arnout

> 
> Could look into that, and look at actual numbers for the resulting
> kernel image when the initramfs is compressed or not?
> 
> Regards,
> Yann E. MORIN.
> 
>> Apart from these kernel config parameters ...
>>
>> * CONFIG_KERNEL_GZIP
>> * CONFIG_KERNEL_LZ4
>> * CONFIG_KERNEL_LZMA
>> * CONFIG_KERNEL_LZO
>> * CONFIG_KERNEL_XZ
>> * CONFIG_KERNEL_ZSTD
>> * CONFIG_KERNEL_UNCOMPRESSED
>>
>> ... we now also set these new kernel config parameters, as without those,
>> the appended initramfs seems to always be compressed with gzip:
>>
>> * CONFIG_INITRAMFS_COMPRESSION_GZIP
>> * CONFIG_INITRAMFS_COMPRESSION_LZ4
>> * CONFIG_INITRAMFS_COMPRESSION_LZMA
>> * CONFIG_INITRAMFS_COMPRESSION_LZO
>> * CONFIG_INITRAMFS_COMPRESSION_XZ
>> * CONFIG_INITRAMFS_COMPRESSION_NONE
>>
>> The former are automatically set depending on whether one of the
>> following buildroot config parameters is set analogously to how it is done
>> for CONFIG_KERNEL_{GZIP,LZ4,LZMA,LZO,XZ,ZSTD,UNCOMPRESSED}:
>>
>> * BR2_LINUX_KERNEL_GZIP
>> * BR2_LINUX_KERNEL_LZ4
>> * BR2_LINUX_KERNEL_LZMA
>> * BR2_LINUX_KERNEL_LZO
>> * BR2_LINUX_KERNEL_XZ
>> * BR2_LINUX_KERNEL_UNCOMPRESSED
>>
>> Signed-off-by: Hanspeter Portner <dev@open-music-kontrollers.ch>
>> ---
>>   linux/linux.mk | 13 +++++++++++++
>>   1 file changed, 13 insertions(+)
>>
>> diff --git a/linux/linux.mk b/linux/linux.mk
>> index 61fdc0c76c..f192341a32 100644
>> --- a/linux/linux.mk
>> +++ b/linux/linux.mk
>> @@ -106,6 +106,15 @@ LINUX_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_XZ) += CONFIG_KERNEL_XZ
>>   LINUX_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_ZSTD) += CONFIG_KERNEL_ZSTD
>>   LINUX_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_UNCOMPRESSED) += CONFIG_KERNEL_UNCOMPRESSED
>>   
>> +INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_GZIP) += CONFIG_INITRAMFS_COMPRESSION_GZIP
>> +INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_LZ4) += CONFIG_INITRAMFS_COMPRESSION_LZ4
>> +INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_LZMA) += CONFIG_INITRAMFS_COMPRESSION_LZMA
>> +INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_LZO) += CONFIG_INITRAMFS_COMPRESSION_LZO
>> +INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_XZ) += CONFIG_INITRAMFS_COMPRESSION_XZ
>> +# zstd compression seems not supported, yet
>> +#INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_ZSTD) += CONFIG_INITRAMFS_COMPRESSION_ZSTD
>> +INITRAMFS_COMPRESSION_OPT_$(BR2_LINUX_KERNEL_UNCOMPRESSED) += CONFIG_INITRAMFS_COMPRESSION_NONE
>> +
>>   ifeq ($(BR2_LINUX_KERNEL_NEEDS_HOST_OPENSSL),y)
>>   LINUX_DEPENDENCIES += host-openssl
>>   endif
>> @@ -352,6 +361,10 @@ define LINUX_KCONFIG_FIXUP_CMDS
>>   		$(call KCONFIG_SET_OPT,CONFIG_INITRAMFS_SOURCE,"$${BR_BINARIES_DIR}/rootfs.cpio")
>>   		$(call KCONFIG_SET_OPT,CONFIG_INITRAMFS_ROOT_UID,0)
>>   		$(call KCONFIG_SET_OPT,CONFIG_INITRAMFS_ROOT_GID,0))
>> +		$(call KCONFIG_ENABLE_OPT,$(strip $(INITRAMFS_COMPRESSION_OPT_y)))
>> +		$(foreach opt, $(INITRAMFS_COMPRESSION_OPT_),
>> +			$(call KCONFIG_DISABLE_OPT,$(opt))
>> +		)
>>   	$(if $(BR2_ROOTFS_DEVICE_CREATION_STATIC),,
>>   		$(call KCONFIG_ENABLE_OPT,CONFIG_DEVTMPFS)
>>   		$(call KCONFIG_ENABLE_OPT,CONFIG_DEVTMPFS_MOUNT))
>> -- 
>> 2.33.0
>>
>> _______________________________________________
>> buildroot mailing list
>> buildroot@lists.buildroot.org
>> https://lists.buildroot.org/mailman/listinfo/buildroot
> 
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot