[PATCH] initramfs: allow again choice of the embedded compression algorithm

[PATCH] initramfs: allow again choice of the embedded compression algorithm

[klondike]

[-- Attachment #1: Type: text/plain, Size: 6615 bytes --]

Choosing the appropriate option (for example on my tests, choosing
CONFIG_INITRAMFS_COMPRESSION_NONE when compressing the using XZ) results in
up to 500KiB differences (9MiB to 8.5MiB) in the kernel size as the
dictionary
will not get polluted with uncomprensible data and may reuse kernel data
too.

Despite embedding an uncompressed initramfs, a user may want to allow for a
compressed extra initramfs to be passed using the rd system, for example to
boot a recovery system. Commit 9ba4bcb645898d562498ea66a0df958ef0e7a68c
("initramfs: read CONFIG_RD_ variables for initramfs compression") broke
that behavior by making the choice based on CONFIG_RD_* instead of adding
CONFIG_INITRAMFS_COMPRESSION_LZ4. Saddly, CONFIG_RD_* is also used to
choose the supported RD compression algorithms by the kernel and a user may
want to suppport more than one.

This patch also reverses 3e4e0f0a8756dade3023d1f47d50fbced7749788
("initramfs:
remove "compression mode" choice") restoring back the "compression mode"
choice and includes the CONFIG_INITRAMFS_COMPRESSION_LZ4 option which was
never added.

As a result the following options are added or readed affecting the embedded
initramfs compression:
INITRAMFS_COMPRESSION_NONE Do no compression
INITRAMFS_COMPRESSION_GZIP Compress using gzip
INITRAMFS_COMPRESSION_BZIP2 Compress using bzip2
INITRAMFS_COMPRESSION_LZMA Compress using lzma
INITRAMFS_COMPRESSION_XZ Compress using xz
INITRAMFS_COMPRESSION_LZO Compress using lzo
INITRAMFS_COMPRESSION_LZ4 Compress using lz4

These depend on the corresponding CONFIG_RD_* option (except NONE which
has no
dependencies).


Signed-off-by: Francisco Blas Izquierdo Riera (klondike)
<klondike@klondike.es>
Cc: P J P <ppandit@redhat.com>
Cc: Paul Bolle <pebolle@tiscali.nl>
---
This patch is applied against Torvald's linux tree

Note: scripts/checkpatch.pl complains about style:
"WARNING: please write a paragraph that describes the config symbol fully"
this seems to be a false positive.

 usr/Kconfig  | 85
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 usr/Makefile | 13 +++++++------
 2 files changed, 92 insertions(+), 6 deletions(-)
diff --git a/usr/Kconfig b/usr/Kconfig
index 2d4c77e..09de7b9 100644
--- a/usr/Kconfig
+++ b/usr/Kconfig
@@ -98,3 +98,88 @@ config RD_LZ4
     help
       Support loading of a LZ4 encoded initial ramdisk or cpio buffer
       If unsure, say N.
+
+choice
+    prompt "Built-in initramfs compression mode" if INITRAMFS_SOURCE!=""
+    help
+      This option decides by which algorithm the builtin initramfs
+      will be compressed.  Several compression algorithms are
+      available, which differ in efficiency, compression and
+      decompression speed.  Compression speed is only relevant
+      when building a kernel.  Decompression speed is relevant at
+      each boot.
+
+      If you have any problems with bzip2 or LZMA compressed
+      initramfs, mail me (Alain Knaff) <alain@knaff.lu>.
+
+      High compression options are mostly useful for users who are
+      low on RAM, since it reduces the memory consumption during
+      boot.
+
+      If in doubt, select 'gzip'
+
+config INITRAMFS_COMPRESSION_NONE
+    bool "None"
+    help
+      Do not compress the built-in initramfs at all. This may
+      sound wasteful in space, but, you should be aware that the
+      built-in initramfs will be compressed at a later stage
+      anyways along with the rest of the kernel, on those
+      architectures that support this.
+      However, not compressing the initramfs may lead to slightly
+      higher memory consumption during a short time at boot, while
+      both the cpio image and the unpacked filesystem image will
+      be present in memory simultaneously
+
+config INITRAMFS_COMPRESSION_GZIP
+    bool "Gzip"
+    depends on RD_GZIP
+    help
+      The old and tried gzip compression. It provides a good balance
+      between compression ratio and decompression speed.
+
+config INITRAMFS_COMPRESSION_BZIP2
+    bool "Bzip2"
+    depends on RD_BZIP2
+    help
+      Its compression ratio and speed is intermediate.
+      Decompression speed is slowest among the choices.  The initramfs
+      size is about 10% smaller with bzip2, in comparison to gzip.
+      Bzip2 uses a large amount of memory. For modern kernels you
+      will need at least 8MB RAM or more for booting.
+
+config INITRAMFS_COMPRESSION_LZMA
+    bool "LZMA"
+    depends on RD_LZMA
+    help
+      This algorithm's compression ratio is best.
+      Decompression speed is between the other choices.
+      Compression is slowest. The initramfs size is about 33%
+      smaller with LZMA in comparison to gzip.
+
+config INITRAMFS_COMPRESSION_XZ
+    bool "XZ"
+    depends on RD_XZ
+    help
+      XZ uses the LZMA2 algorithm. The initramfs size is about 30%
+      smaller with XZ in comparison to gzip. Decompression speed
+      is better than that of bzip2 but worse than gzip and LZO.
+      Compression is slow.
+
+config INITRAMFS_COMPRESSION_LZO
+    bool "LZO"
+    depends on RD_LZO
+    help
+      Its compression ratio is the second poorest among the choices. The
+      kernel size is about 10% bigger than gzip; however its decompression
+      speed is the second fastest.
+
+config INITRAMFS_COMPRESSION_LZ4
+    bool "LZ4"
+    depends on RD_LZ4
+    help
+      Its compression ratio is the poorest among the choices. The kernel
+      size is about 15% bigger than gzip; however its decompression speed
+      is the fastest.
+
+endchoice
diff --git a/usr/Makefile b/usr/Makefile
index e767f01..4031e23 100644
--- a/usr/Makefile
+++ b/usr/Makefile
@@ -5,24 +5,25 @@
 klibcdirs:;
 PHONY += klibcdirs
 
+#If unset it means we use CONFIG_INITRAMFS_COMPRESSION_NONE
 
 # Bzip2
-suffix_$(CONFIG_RD_BZIP2)  = .bz2
+suffix_$(CONFIG_INITRAMFS_COMPRESSION_BZIP2)  = .bz2
 
 # Lzma
-suffix_$(CONFIG_RD_LZMA)   = .lzma
+suffix_$(CONFIG_INITRAMFS_COMPRESSION_LZMA)   = .lzma
 
 # XZ
-suffix_$(CONFIG_RD_XZ)     = .xz
+suffix_$(CONFIG_INITRAMFS_COMPRESSION_XZ)     = .xz
 
 # Lzo
-suffix_$(CONFIG_RD_LZO)    = .lzo
+suffix_$(CONFIG_INITRAMFS_COMPRESSION_LZO)    = .lzo
 
 # Lz4
-suffix_$(CONFIG_RD_LZ4)    = .lz4
+suffix_$(CONFIG_INITRAMFS_COMPRESSION_LZ4)    = .lz4
 
 # Gzip
-suffix_$(CONFIG_RD_GZIP)   = .gz
+suffix_$(CONFIG_INITRAMFS_COMPRESSION_GZIP)   = .gz
 
 AFLAGS_initramfs_data.o +=
-DINITRAMFS_IMAGE="usr/initramfs_data.cpio$(suffix_y)"
 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 246 bytes --]

Re: [PATCH] initramfs: allow again choice of the embedded compression algorithm

[P J P]

   Hello Klondike,

+-- On Thu, 25 Sep 2014, klondike wrote --+
| Despite embedding an uncompressed initramfs, a user may want to allow for a
| compressed extra initramfs to be passed using the rd system, for example to
| boot a recovery system. Commit 9ba4bcb645898d562498ea66a0df958ef0e7a68c
| ("initramfs: read CONFIG_RD_ variables for initramfs compression") broke
| that behavior by making the choice based on CONFIG_RD_* instead of adding
| CONFIG_INITRAMFS_COMPRESSION_LZ4. Saddly, CONFIG_RD_* is also used to
| choose the supported RD compression algorithms by the kernel and a user may
| want to suppport more than one.

  Ie. on embedded systems they use CONFIG_INITRAMFS_COMPRESSION_* options to 
compress ram disk and on other platforms they use CONFIG_RD_* options?
 
| As a result the following options are added or readed affecting the embedded
| initramfs compression:
| INITRAMFS_COMPRESSION_NONE Do no compression
| INITRAMFS_COMPRESSION_GZIP Compress using gzip
| INITRAMFS_COMPRESSION_BZIP2 Compress using bzip2
| INITRAMFS_COMPRESSION_LZMA Compress using lzma
| INITRAMFS_COMPRESSION_XZ Compress using xz
| INITRAMFS_COMPRESSION_LZO Compress using lzo
| INITRAMFS_COMPRESSION_LZ4 Compress using lz4

  I haven't tested the patch yet, but does it preserve the CONFIG_RD_* 
options' functionality?

Would unifying the two options into one be a good idea?
--
 - P J P
47AF CE69 3A90 54AA 9045 1053 DD13 3D32 FE5B 041F

Re: [PATCH] initramfs: allow again choice of the embedded compression algorithm

[klondike]

El 29/09/14 10:08, P J P escribió:
>    Hello Klondike,
Hi!
> +-- On Thu, 25 Sep 2014, klondike wrote --+
> | Despite embedding an uncompressed initramfs, a user may want to allow for a
> | compressed extra initramfs to be passed using the rd system, for example to
> | boot a recovery system. Commit 9ba4bcb645898d562498ea66a0df958ef0e7a68c
> | ("initramfs: read CONFIG_RD_ variables for initramfs compression") broke
> | that behavior by making the choice based on CONFIG_RD_* instead of adding
> | CONFIG_INITRAMFS_COMPRESSION_LZ4. Saddly, CONFIG_RD_* is also used to
> | choose the supported RD compression algorithms by the kernel and a user may
> | want to suppport more than one.
>
>   Ie. on embedded systems they use CONFIG_INITRAMFS_COMPRESSION_* options to 
> compress ram disk and on other platforms they use CONFIG_RD_* options?
No, instead CONFIG_RD_* defines the compression formats supported by the
kernel when parsing an initramfs (either embedded into the kernel or
passed by the bootloader) whilst one of CONFIG_INITRAMFS_COMPRESSION_*
is used to choose the one that will be used on the initramfs embedded
into the kernel.

Just to make sure we are using the same terms, a system using a kernel
with an embedded intramfs isn't necessarily an embedded system, for
example I use such on my Gentoo systems which involve both servers and
Desktops.
> | As a result the following options are added or readed affecting the embedded
> | initramfs compression:
> | INITRAMFS_COMPRESSION_NONE Do no compression
> | INITRAMFS_COMPRESSION_GZIP Compress using gzip
> | INITRAMFS_COMPRESSION_BZIP2 Compress using bzip2
> | INITRAMFS_COMPRESSION_LZMA Compress using lzma
> | INITRAMFS_COMPRESSION_XZ Compress using xz
> | INITRAMFS_COMPRESSION_LZO Compress using lzo
> | INITRAMFS_COMPRESSION_LZ4 Compress using lz4
>
>   I haven't tested the patch yet, but does it preserve the CONFIG_RD_* 
> options' functionality?
Yes, except choosing the compression algorithm for the embedded
initramfs (which will default to gzip). For older kernels this means the
user will now experience the intended behaviour, for newer ones it means
that users using expert settings and an embedded initramfs will be asked
for the compression algorithm to be used (with the list of those
available depending on the CONFIG_RD_* options choosen), whilst normal
users will default to gzip compression.
> Would unifying the two options into one be a good idea?
I'm not a Kconfig wizard (more like a wizzard) so I'm not sure if such a
thing is possible. Basically we need a way to choose compression methods
supported (more to enforce the appropriate dependencies according to my
understanding of the code) and a way to choose the particular one used
for the embedded initramfs, as basing this choice in the selected method
may not always be a good idea (in my setups using a non compressed
embedded initramfs works best but external initramfs used for example to
do recovery may be compressed using xz). Of course on an embedded system
with serious memory constrains it may make more sense for example to
compress the embedded initramfs to reduce memory usage during boot. So
if you have suggestions on how to do this with a single option please
share them :)

klondike

[PATCH v2 1/2] initramfs: allow again choice of the embedded compression algorithm

[klondike]

[-- Attachment #1: Type: text/plain, Size: 1688 bytes --]

Select builtin initram compression algorithm on KConfig instead of Makefile

This patch moves the current builtin initram compression algorithm selection
from the Makefile into the INITRAMFS_COMPRESSION variable. This makes
deciding
algorithm precedence easier and would allow for overrides if new algorithms
want to be tested.

Signed-off-by: Francisco Blas Izquierdo Riera (klondike)
<klondike@klondike.es>
Cc: P J P <ppandit@redhat.com>
Cc: Paul Bolle <pebolle@tiscali.nl>
Cc: Andrew Morton <akpm@linux-foundation.org>
diff --git a/usr/Kconfig b/usr/Kconfig
index 572dcf7..bf8e8f1 100644
--- a/usr/Kconfig
+++ b/usr/Kconfig
@@ -98,3 +98,13 @@ config RD_LZ4
     help
       Support loading of a LZ4 encoded initial ramdisk or cpio buffer
       If unsure, say N.
+
+config INITRAMFS_COMPRESSION
+    string
+    default ".gz"   if RD_GZIP
+    default ".lz4"  if RD_LZ4
+    default ".lzo"  if RD_LZO
+    default ".xz"   if RD_XZ
+    default ".lzma" if RD_LZMA
+    default ".bz2"  if RD_BZIP2
+    default ""
diff --git a/usr/Makefile b/usr/Makefile
index e767f01..17a5132 100644
--- a/usr/Makefile
+++ b/usr/Makefile
@@ -5,25 +5,7 @@
 klibcdirs:;
 PHONY += klibcdirs
 
-
-# Bzip2
-suffix_$(CONFIG_RD_BZIP2)  = .bz2
-
-# Lzma
-suffix_$(CONFIG_RD_LZMA)   = .lzma
-
-# XZ
-suffix_$(CONFIG_RD_XZ)     = .xz
-
-# Lzo
-suffix_$(CONFIG_RD_LZO)    = .lzo
-
-# Lz4
-suffix_$(CONFIG_RD_LZ4)    = .lz4
-
-# Gzip
-suffix_$(CONFIG_RD_GZIP)   = .gz
-
+suffix_y = $(CONFIG_INITRAMFS_COMPRESSION)
 AFLAGS_initramfs_data.o +=
-DINITRAMFS_IMAGE="usr/initramfs_data.cpio$(suffix_y)"
 
 # Generate builtin.o based on initramfs_data.o



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[PATCH v2 2/2] initramfs: allow again choice of the embedded compression algorithm

[klondike]

[-- Attachment #1: Type: text/plain, Size: 7691 bytes --]

Choosing the appropriate compression option when using an embeded initramfs
can result in significant size differences in the resulting data.

This is caused by avoiding double compression of the initramfs contents.
For example on my tests, choosing CONFIG_INITRAMFS_COMPRESSION_NONE when
compressing the kernel using XZ) results in up to 500KiB differences
(9MiB to
 8.5MiB) in the kernel size as the dictionary will not get polluted with
uncomprensible data and may reuse kernel data too.

Despite embedding an uncompressed initramfs, a user may want to allow for a
compressed extra initramfs to be passed using the rd system, for example to
boot a recovery system. Commit 9ba4bcb645898d562498ea66a0df958ef0e7a68c
("initramfs: read CONFIG_RD_ variables for initramfs compression") broke
that behavior by making the choice based on CONFIG_RD_* instead of adding
CONFIG_INITRAMFS_COMPRESSION_LZ4. Saddly, CONFIG_RD_* is also used to
choose the supported RD compression algorithms by the kernel and a user may
want to suppport more than one.

This patch also reverses 3e4e0f0a8756dade3023d1f47d50fbced7749788
("initramfs: remove "compression mode" choice") restoring back the
"compression mode" choice and includes the CONFIG_INITRAMFS_COMPRESSION_LZ4
option which was never added.

As a result the following options are added or readed affecting the embedded
initramfs compression:
INITRAMFS_COMPRESSION_NONE Do no compression
INITRAMFS_COMPRESSION_GZIP Compress using gzip
INITRAMFS_COMPRESSION_BZIP2 Compress using bzip2
INITRAMFS_COMPRESSION_LZMA Compress using lzma
INITRAMFS_COMPRESSION_XZ Compress using xz
INITRAMFS_COMPRESSION_LZO Compress using lzo
INITRAMFS_COMPRESSION_LZ4 Compress using lz4

These depend on the corresponding CONFIG_RD_* option being set (except NONE
which has no dependencies).

This patch depends on the previous one (the previous version didn't) to
simplify the way in which the algorithm is chosen and keep backwards
compatibility with the behaviour introduced by commit
 9ba4bcb645898d562498ea66a0df958ef0e7a68c

Signed-off-by: Francisco Blas Izquierdo Riera (klondike)
<klondike@klondike.es>
Cc: P J P <ppandit@redhat.com>
Cc: Paul Bolle <pebolle@tiscali.nl>
Cc: Andrew Morton <akpm@linux-foundation.org>
diff --git a/usr/Kconfig b/usr/Kconfig
index bf8e8f1..6278f13 100644
--- a/usr/Kconfig
+++ b/usr/Kconfig
@@ -99,8 +99,125 @@ config RD_LZ4
       Support loading of a LZ4 encoded initial ramdisk or cpio buffer
       If unsure, say N.
 
+choice
+    prompt "Built-in initramfs compression mode"
+    depends on INITRAMFS_SOURCE!=""
+    optional
+    help
+      This option allows you to decide by which algorithm the builtin
+      initramfs will be compressed.  Several compression algorithms are
+      available, which differ in efficiency, compression and
+      decompression speed.  Compression speed is only relevant
+      when building a kernel.  Decompression speed is relevant at
+      each boot. Also the memory usage during decompression may become
+      relevant on memory constrained systems. This is usually based on the
+      dictionary size of the algorithm with algorithms like XZ and LZMA
+      featuring large dictionary sizes.
+
+      High compression options are mostly useful for users who are
+      low on RAM, since it reduces the memory consumption during
+      boot.
+
+      Keep in mind that your build system needs to provide the appropriate
+      compression tool to compress the generated initram cpio file for
+      embedding.
+
+      If in doubt, select 'None'
+
+config INITRAMFS_COMPRESSION_NONE
+    bool "None"
+    help
+      Do not compress the built-in initramfs at all. This may sound
wasteful
+      in space, but, you should be aware that the built-in initramfs
will be
+      compressed at a later stage anyways along with the rest of the
kernel,
+      on those architectures that support this. However, not
compressing the
+      initramfs may lead to slightly higher memory consumption during a
+      short time at boot, while both the cpio image and the unpacked
+      filesystem image will be present in memory simultaneously
+
+config INITRAMFS_COMPRESSION_GZIP
+    bool "Gzip"
+    depends on RD_GZIP
+    help
+      Use the old and well tested gzip compression algorithm. Gzip provides
+      a good balance between compression ratio and decompression speed and
+      has a reasonable compression speed. It is also more likely to be
+      supported by your build system as the gzip tool is present by default
+      on most distros.
+
+config INITRAMFS_COMPRESSION_BZIP2
+    bool "Bzip2"
+    depends on RD_BZIP2
+    help
+      It's compression ratio and speed is intermediate. Decompression speed
+      is slowest among the choices. The initramfs size is about 10% smaller
+      with bzip2, in comparison to gzip. Bzip2 uses a large amount of
+      memory. For modern kernels you will need at least 8MB RAM or more for
+      booting.
+
+      If you choose this, keep in mind that you need to have the bzip2 tool
+      available to be able to compress the initram.
+
+config INITRAMFS_COMPRESSION_LZMA
+    bool "LZMA"
+    depends on RD_LZMA
+    help
+      This algorithm's compression ratio is best but has a large dictionary
+      size which might cause issues in memory constrained systems.
+      Decompression speed is between the other choices. Compression is
+      slowest. The initramfs size is about 33% smaller with LZMA in
+      comparison to gzip.
+
+      If you choose this, keep in mind that you may need to install the xz
+      or lzma tools to be able to compress the initram.
+
+config INITRAMFS_COMPRESSION_XZ
+    bool "XZ"
+    depends on RD_XZ
+    help
+      XZ uses the LZMA2 algorithm and has a large dictionary which may
cause
+      problems on memory constrained systems. The initramfs size is about
+      30% smaller with XZ in comparison to gzip. Decompression speed is
+      better than that of bzip2 but worse than gzip and LZO. Compression is
+      slow.
+
+      If you choose this, keep in mind that you may need to install the xz
+      tool to be able to compress the initram.
+
+config INITRAMFS_COMPRESSION_LZO
+    bool "LZO"
+    depends on RD_LZO
+    help
+      It's compression ratio is the second poorest amongst the choices. The
+      kernel size is about 10% bigger than gzip. Despite that, it's
+      decompression speed is the second fastest and it's compression speed
+      is quite fast too.
+
+      If you choose this, keep in mind that you may need to install the
lzop
+      tool to be able to compress the initram.
+
+config INITRAMFS_COMPRESSION_LZ4
+    bool "LZ4"
+    depends on RD_LZ4
+    help
+      It's compression ratio is the poorest amongst the choices. The kernel
+      size is about 15% bigger than gzip; however its decompression speed
+      is the fastest.
+
+      If you choose this, keep in mind that most distros don't provide lz4
+      by default which could cause a build failure.
+
+endchoice
+
 config INITRAMFS_COMPRESSION
     string
+    default ""      if INITRAMFS_COMPRESSION_NONE
+    default ".gz"   if INITRAMFS_COMPRESSION_GZIP
+    default ".bz2"  if INITRAMFS_COMPRESSION_BZIP2
+    default ".lzma" if INITRAMFS_COMPRESSION_LZMA
+    default ".xz"   if INITRAMFS_COMPRESSION_XZ
+    default ".lzo"  if INITRAMFS_COMPRESSION_LZO
+    default ".lz4"  if INITRAMFS_COMPRESSION_LZ4
     default ".gz"   if RD_GZIP
     default ".lz4"  if RD_LZ4
     default ".lzo"  if RD_LZO



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[PATCH v3 1/2] initramfs: Select builtin initram compression algorithm on KConfig instead of Makefile

[klondike]

This patch moves the current builtin initram compression algorithm selection
from the Makefile into the INITRAMFS_COMPRESSION variable. This makes deciding
algorithm precedence easier and would allow for overrides if new algorithms
want to be tested.

Signed-off-by: Francisco Blas Izquierdo Riera (klondike) <klondike@klondike.es>
Cc: P J P <ppandit@redhat.com>
Cc: Paul Bolle <pebolle@tiscali.nl>
Cc: Andrew Morton <akpm@linux-foundation.org>
---
I'm sorry for the noise. I'm resending this as a new version because
Thunderbird likes to word wrap things it shouldn't be word wrapping.
diff --git a/usr/Kconfig b/usr/Kconfig
index 572dcf7..bf8e8f1 100644
--- a/usr/Kconfig
+++ b/usr/Kconfig
@@ -98,3 +98,13 @@ config RD_LZ4
 	help
 	  Support loading of a LZ4 encoded initial ramdisk or cpio buffer
 	  If unsure, say N.
+
+config INITRAMFS_COMPRESSION
+	string
+	default ".gz"   if RD_GZIP
+	default ".lz4"  if RD_LZ4
+	default ".lzo"  if RD_LZO
+	default ".xz"   if RD_XZ
+	default ".lzma" if RD_LZMA
+	default ".bz2"  if RD_BZIP2
+	default ""
diff --git a/usr/Makefile b/usr/Makefile
index e767f01..17a5132 100644
--- a/usr/Makefile
+++ b/usr/Makefile
@@ -5,25 +5,7 @@
 klibcdirs:;
 PHONY += klibcdirs
 
-
-# Bzip2
-suffix_$(CONFIG_RD_BZIP2)  = .bz2
-
-# Lzma
-suffix_$(CONFIG_RD_LZMA)   = .lzma
-
-# XZ
-suffix_$(CONFIG_RD_XZ)     = .xz
-
-# Lzo
-suffix_$(CONFIG_RD_LZO)    = .lzo
-
-# Lz4
-suffix_$(CONFIG_RD_LZ4)    = .lz4
-
-# Gzip
-suffix_$(CONFIG_RD_GZIP)   = .gz
-
+suffix_y = $(CONFIG_INITRAMFS_COMPRESSION)
 AFLAGS_initramfs_data.o += -DINITRAMFS_IMAGE="usr/initramfs_data.cpio$(suffix_y)"
 
 # Generate builtin.o based on initramfs_data.o

[PATCH v3 2/2] initramfs: Allow again choice of the embedded initram compression algorithm

[klondike]

Choosing the appropriate compression option when using an embeded initramfs
can result in significant size differences in the resulting data.

This is caused by avoiding double compression of the initramfs contents.
For example on my tests, choosing CONFIG_INITRAMFS_COMPRESSION_NONE when
compressing the kernel using XZ) results in up to 500KiB differences (9MiB to
 8.5MiB) in the kernel size as the dictionary will not get polluted with
uncomprensible data and may reuse kernel data too.

Despite embedding an uncompressed initramfs, a user may want to allow for a
compressed extra initramfs to be passed using the rd system, for example to
boot a recovery system. Commit 9ba4bcb645898d562498ea66a0df958ef0e7a68c
("initramfs: read CONFIG_RD_ variables for initramfs compression") broke
that behavior by making the choice based on CONFIG_RD_* instead of adding
CONFIG_INITRAMFS_COMPRESSION_LZ4. Saddly, CONFIG_RD_* is also used to
choose the supported RD compression algorithms by the kernel and a user may
want to suppport more than one.

This patch also reverses 3e4e0f0a8756dade3023d1f47d50fbced7749788
("initramfs: remove "compression mode" choice") restoring back the
"compression mode" choice and includes the CONFIG_INITRAMFS_COMPRESSION_LZ4
option which was never added.

As a result the following options are added or readed affecting the embedded
initramfs compression:
INITRAMFS_COMPRESSION_NONE Do no compression
INITRAMFS_COMPRESSION_GZIP Compress using gzip
INITRAMFS_COMPRESSION_BZIP2 Compress using bzip2
INITRAMFS_COMPRESSION_LZMA Compress using lzma
INITRAMFS_COMPRESSION_XZ Compress using xz
INITRAMFS_COMPRESSION_LZO Compress using lzo
INITRAMFS_COMPRESSION_LZ4 Compress using lz4

These depend on the corresponding CONFIG_RD_* option being set (except NONE
which has no dependencies).

This patch depends on the previous one (the previous version didn't) to
simplify the way in which the algorithm is chosen and keep backwards
compatibility with the behaviour introduced by commit
 9ba4bcb645898d562498ea66a0df958ef0e7a68c

Signed-off-by: Francisco Blas Izquierdo Riera (klondike) <klondike@klondike.es>
Cc: P J P <ppandit@redhat.com>
Cc: Paul Bolle <pebolle@tiscali.nl>
Cc: Andrew Morton <akpm@linux-foundation.org>
---
I'm sorry for the noise. I'm resending this as a new version because
Thunderbird likes to word wrap things it shouldn't be word wrapping.
diff --git a/usr/Kconfig b/usr/Kconfig
index bf8e8f1..6278f13 100644
--- a/usr/Kconfig
+++ b/usr/Kconfig
@@ -99,8 +99,125 @@ config RD_LZ4
 	  Support loading of a LZ4 encoded initial ramdisk or cpio buffer
 	  If unsure, say N.
 
+choice
+	prompt "Built-in initramfs compression mode"
+	depends on INITRAMFS_SOURCE!=""
+	optional
+	help
+	  This option allows you to decide by which algorithm the builtin
+	  initramfs will be compressed.  Several compression algorithms are
+	  available, which differ in efficiency, compression and
+	  decompression speed.  Compression speed is only relevant
+	  when building a kernel.  Decompression speed is relevant at
+	  each boot. Also the memory usage during decompression may become
+	  relevant on memory constrained systems. This is usually based on the
+	  dictionary size of the algorithm with algorithms like XZ and LZMA
+	  featuring large dictionary sizes.
+
+	  High compression options are mostly useful for users who are
+	  low on RAM, since it reduces the memory consumption during
+	  boot.
+
+	  Keep in mind that your build system needs to provide the appropriate
+	  compression tool to compress the generated initram cpio file for
+	  embedding.
+
+	  If in doubt, select 'None'
+
+config INITRAMFS_COMPRESSION_NONE
+	bool "None"
+	help
+	  Do not compress the built-in initramfs at all. This may sound wasteful
+	  in space, but, you should be aware that the built-in initramfs will be
+	  compressed at a later stage anyways along with the rest of the kernel,
+	  on those architectures that support this. However, not compressing the
+	  initramfs may lead to slightly higher memory consumption during a
+	  short time at boot, while both the cpio image and the unpacked
+	  filesystem image will be present in memory simultaneously
+
+config INITRAMFS_COMPRESSION_GZIP
+	bool "Gzip"
+	depends on RD_GZIP
+	help
+	  Use the old and well tested gzip compression algorithm. Gzip provides
+	  a good balance between compression ratio and decompression speed and
+	  has a reasonable compression speed. It is also more likely to be
+	  supported by your build system as the gzip tool is present by default
+	  on most distros.
+
+config INITRAMFS_COMPRESSION_BZIP2
+	bool "Bzip2"
+	depends on RD_BZIP2
+	help
+	  It's compression ratio and speed is intermediate. Decompression speed
+	  is slowest among the choices. The initramfs size is about 10% smaller
+	  with bzip2, in comparison to gzip. Bzip2 uses a large amount of
+	  memory. For modern kernels you will need at least 8MB RAM or more for
+	  booting.
+
+	  If you choose this, keep in mind that you need to have the bzip2 tool
+	  available to be able to compress the initram.
+
+config INITRAMFS_COMPRESSION_LZMA
+	bool "LZMA"
+	depends on RD_LZMA
+	help
+	  This algorithm's compression ratio is best but has a large dictionary
+	  size which might cause issues in memory constrained systems.
+	  Decompression speed is between the other choices. Compression is
+	  slowest. The initramfs size is about 33% smaller with LZMA in
+	  comparison to gzip.
+
+	  If you choose this, keep in mind that you may need to install the xz
+	  or lzma tools to be able to compress the initram.
+
+config INITRAMFS_COMPRESSION_XZ
+	bool "XZ"
+	depends on RD_XZ
+	help
+	  XZ uses the LZMA2 algorithm and has a large dictionary which may cause
+	  problems on memory constrained systems. The initramfs size is about
+	  30% smaller with XZ in comparison to gzip. Decompression speed is
+	  better than that of bzip2 but worse than gzip and LZO. Compression is
+	  slow.
+
+	  If you choose this, keep in mind that you may need to install the xz
+	  tool to be able to compress the initram.
+
+config INITRAMFS_COMPRESSION_LZO
+	bool "LZO"
+	depends on RD_LZO
+	help
+	  It's compression ratio is the second poorest amongst the choices. The
+	  kernel size is about 10% bigger than gzip. Despite that, it's
+	  decompression speed is the second fastest and it's compression speed
+	  is quite fast too.
+
+	  If you choose this, keep in mind that you may need to install the lzop
+	  tool to be able to compress the initram.
+
+config INITRAMFS_COMPRESSION_LZ4
+	bool "LZ4"
+	depends on RD_LZ4
+	help
+	  It's compression ratio is the poorest amongst the choices. The kernel
+	  size is about 15% bigger than gzip; however its decompression speed
+	  is the fastest.
+
+	  If you choose this, keep in mind that most distros don't provide lz4
+	  by default which could cause a build failure.
+
+endchoice
+
 config INITRAMFS_COMPRESSION
 	string
+	default ""      if INITRAMFS_COMPRESSION_NONE
+	default ".gz"   if INITRAMFS_COMPRESSION_GZIP
+	default ".bz2"  if INITRAMFS_COMPRESSION_BZIP2
+	default ".lzma" if INITRAMFS_COMPRESSION_LZMA
+	default ".xz"   if INITRAMFS_COMPRESSION_XZ
+	default ".lzo"  if INITRAMFS_COMPRESSION_LZO
+	default ".lz4"  if INITRAMFS_COMPRESSION_LZ4
 	default ".gz"   if RD_GZIP
 	default ".lz4"  if RD_LZ4
 	default ".lzo"  if RD_LZO

Re: [PATCH v3 2/2] initramfs: Allow again choice of the embedded initram compression algorithm

[Andrew Morton]

On Tue, 27 Sep 2016 22:32:59 +0200 klondike <klondike@xiscosoft.net> wrote:

> Choosing the appropriate compression option when using an embeded initramfs
> can result in significant size differences in the resulting data.
> 
> This is caused by avoiding double compression of the initramfs contents.
> For example on my tests, choosing CONFIG_INITRAMFS_COMPRESSION_NONE when
> compressing the kernel using XZ) results in up to 500KiB differences (9MiB to
>  8.5MiB) in the kernel size as the dictionary will not get polluted with
> uncomprensible data and may reuse kernel data too.
> 
> Despite embedding an uncompressed initramfs, a user may want to allow for a
> compressed extra initramfs to be passed using the rd system, for example to
> boot a recovery system. Commit 9ba4bcb645898d562498ea66a0df958ef0e7a68c
> ("initramfs: read CONFIG_RD_ variables for initramfs compression") broke
> that behavior by making the choice based on CONFIG_RD_* instead of adding
> CONFIG_INITRAMFS_COMPRESSION_LZ4. Saddly, CONFIG_RD_* is also used to
> choose the supported RD compression algorithms by the kernel and a user may
> want to suppport more than one.
> 
> This patch also reverses 3e4e0f0a8756dade3023d1f47d50fbced7749788
> ("initramfs: remove "compression mode" choice") restoring back the
> "compression mode" choice and includes the CONFIG_INITRAMFS_COMPRESSION_LZ4
> option which was never added.
> 
> As a result the following options are added or readed affecting the embedded
> initramfs compression:
> INITRAMFS_COMPRESSION_NONE Do no compression
> INITRAMFS_COMPRESSION_GZIP Compress using gzip
> INITRAMFS_COMPRESSION_BZIP2 Compress using bzip2
> INITRAMFS_COMPRESSION_LZMA Compress using lzma
> INITRAMFS_COMPRESSION_XZ Compress using xz
> INITRAMFS_COMPRESSION_LZO Compress using lzo
> INITRAMFS_COMPRESSION_LZ4 Compress using lz4
> 
> These depend on the corresponding CONFIG_RD_* option being set (except NONE
> which has no dependencies).

As you sent them, these patches would be merged with

	From: klondike <klondike@xiscosoft.net>
	Signed-off-by: Francisco Blas Izquierdo Riera (klondike) <klondike@klondike.es>

Which is strange.  Different email addresses and "klondike" isn't a
real name.

So I'll rewrite the From: address to match the SOB address.

You can do this yourself by including an explicit From: line as the
first line of the changelog text.

Re: [PATCH v3 2/2] initramfs: Allow again choice of the embedded initram compression algorithm

[Francisco Blas Izquierdo Riera (klondike)]

[-- Attachment #1: Type: text/plain, Size: 2916 bytes --]

Hello Andrew!

El 21/10/16 a las 23:21, Andrew Morton escribió:
> On Tue, 27 Sep 2016 22:32:59 +0200 klondike <klondike@xiscosoft.net> wrote:
>
>> Choosing the appropriate compression option when using an embeded initramfs
>> can result in significant size differences in the resulting data.
>>
>> This is caused by avoiding double compression of the initramfs contents.
>> For example on my tests, choosing CONFIG_INITRAMFS_COMPRESSION_NONE when
>> compressing the kernel using XZ) results in up to 500KiB differences (9MiB to
>>  8.5MiB) in the kernel size as the dictionary will not get polluted with
>> uncomprensible data and may reuse kernel data too.
>>
>> Despite embedding an uncompressed initramfs, a user may want to allow for a
>> compressed extra initramfs to be passed using the rd system, for example to
>> boot a recovery system. Commit 9ba4bcb645898d562498ea66a0df958ef0e7a68c
>> ("initramfs: read CONFIG_RD_ variables for initramfs compression") broke
>> that behavior by making the choice based on CONFIG_RD_* instead of adding
>> CONFIG_INITRAMFS_COMPRESSION_LZ4. Saddly, CONFIG_RD_* is also used to
>> choose the supported RD compression algorithms by the kernel and a user may
>> want to suppport more than one.
>>
>> This patch also reverses 3e4e0f0a8756dade3023d1f47d50fbced7749788
>> ("initramfs: remove "compression mode" choice") restoring back the
>> "compression mode" choice and includes the CONFIG_INITRAMFS_COMPRESSION_LZ4
>> option which was never added.
>>
>> As a result the following options are added or readed affecting the embedded
>> initramfs compression:
>> INITRAMFS_COMPRESSION_NONE Do no compression
>> INITRAMFS_COMPRESSION_GZIP Compress using gzip
>> INITRAMFS_COMPRESSION_BZIP2 Compress using bzip2
>> INITRAMFS_COMPRESSION_LZMA Compress using lzma
>> INITRAMFS_COMPRESSION_XZ Compress using xz
>> INITRAMFS_COMPRESSION_LZO Compress using lzo
>> INITRAMFS_COMPRESSION_LZ4 Compress using lz4
>>
>> These depend on the corresponding CONFIG_RD_* option being set (except NONE
>> which has no dependencies).
> As you sent them, these patches would be merged with
>
> 	From: klondike <klondike@xiscosoft.net>
> 	Signed-off-by: Francisco Blas Izquierdo Riera (klondike) <klondike@klondike.es>
>
> Which is strange.  Different email addresses and "klondike" isn't a
> real name.
Ugh that was my mistake, the e-mail should have been sent from my
klondike@klondike.es address. I guess at some point during sending I
messed it up. Sorry for that.
> So I'll rewrite the From: address to match the SOB address.
That's perfect, thanks :)
> You can do this yourself by including an explicit From: line as the
> first line of the changelog text.
Thanks! I'm still trying to learn how the kernel patching process works
so the tip is very appreciated.I'll be careful and use the From line
next time :)

Sincerely,
Klondike


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [v3, 2/2] initramfs: Allow again choice of the embedded initram compression algorithm

[Florian Fainelli]

Hi Francisco, Nicholas,

Nicholas already fixed part of this commit, but there is more breakage,
see below:

On 09/27/2016 01:32 PM, klondike wrote:
> Choosing the appropriate compression option when using an embeded initramfs
> can result in significant size differences in the resulting data.
> 
> This is caused by avoiding double compression of the initramfs contents.
> For example on my tests, choosing CONFIG_INITRAMFS_COMPRESSION_NONE when
> compressing the kernel using XZ) results in up to 500KiB differences (9MiB to
>  8.5MiB) in the kernel size as the dictionary will not get polluted with
> uncomprensible data and may reuse kernel data too.
> 
> Despite embedding an uncompressed initramfs, a user may want to allow for a
> compressed extra initramfs to be passed using the rd system, for example to
> boot a recovery system. Commit 9ba4bcb645898d562498ea66a0df958ef0e7a68c
> ("initramfs: read CONFIG_RD_ variables for initramfs compression") broke
> that behavior by making the choice based on CONFIG_RD_* instead of adding
> CONFIG_INITRAMFS_COMPRESSION_LZ4. Saddly, CONFIG_RD_* is also used to
> choose the supported RD compression algorithms by the kernel and a user may
> want to suppport more than one.
> 
> This patch also reverses 3e4e0f0a8756dade3023d1f47d50fbced7749788
> ("initramfs: remove "compression mode" choice") restoring back the
> "compression mode" choice and includes the CONFIG_INITRAMFS_COMPRESSION_LZ4
> option which was never added.
> 
> As a result the following options are added or readed affecting the embedded
> initramfs compression:
> INITRAMFS_COMPRESSION_NONE Do no compression
> INITRAMFS_COMPRESSION_GZIP Compress using gzip
> INITRAMFS_COMPRESSION_BZIP2 Compress using bzip2
> INITRAMFS_COMPRESSION_LZMA Compress using lzma
> INITRAMFS_COMPRESSION_XZ Compress using xz
> INITRAMFS_COMPRESSION_LZO Compress using lzo
> INITRAMFS_COMPRESSION_LZ4 Compress using lz4
> 
> These depend on the corresponding CONFIG_RD_* option being set (except NONE
> which has no dependencies).
> 
> This patch depends on the previous one (the previous version didn't) to
> simplify the way in which the algorithm is chosen and keep backwards
> compatibility with the behaviour introduced by commit
>  9ba4bcb645898d562498ea66a0df958ef0e7a68c
> 
> Signed-off-by: Francisco Blas Izquierdo Riera (klondike) <klondike@klondike.es>
> Cc: P J P <ppandit@redhat.com>
> Cc: Paul Bolle <pebolle@tiscali.nl>
> Cc: Andrew Morton <akpm@linux-foundation.org>

Running a bisection against usr/ was not particularly convincing but
here is basically what I am observing which used to work just fine
before as of v4.9 and since I tracked it to this particular
commit/patch. Here is what my build system does:

- kernel is initially configured not to have an initramfs included
- build the user space root file system
- re-configure the kernel to have an initramfs included
(CONFIG_INITRAMFS_SOURCE="/path/to/romfs") and set relevant
CONFIG_INITRAMFS options, in my case, no compression
- kernel is re-built with these options -> kernel+initramfs image is copied
- kernel is re-built again without these options -> kernel image is copied

Now suppose you make changes to your root filesystem, like add/remove
applications, initramfs_data.cpio is now a stale file and go through the
same process again:

- build the kernel without an initramfs
- user space (re)build
- build the kernel with an initramfs

Building a kernel without an initramfs means setting this option:

CONFIG_INITRAMFS_SOURCE=""

whereas building a kernel with an initramfs means setting these options:

CONFIG_INITRAMFS_SOURCE="/home/fainelli/work/uclinux-rootfs/romfs
/home/fainelli/work/uclinux-rootfs/misc/initramfs.dev"
CONFIG_INITRAMFS_ROOT_UID=1000
CONFIG_INITRAMFS_ROOT_GID=1000
CONFIG_INITRAMFS_COMPRESSION_NONE=y
# CONFIG_INITRAMFS_COMPRESSION_GZIP is not set
# CONFIG_INITRAMFS_COMPRESSION_BZIP2 is not set
# CONFIG_INITRAMFS_COMPRESSION_LZMA is not set
# CONFIG_INITRAMFS_COMPRESSION_XZ is not set
# CONFIG_INITRAMFS_COMPRESSION_LZO is not set
# CONFIG_INITRAMFS_COMPRESSION_LZ4 is not set
CONFIG_INITRAMFS_COMPRESSION=""

Commit db2aa7fd15e857891cefbada8348c8d938c7a2bc ("initramfs: allow again
choice of the embedded initram compression algorithm") appears
problematic because CONFIG_INITRAMFS_COMPRESSION which is used to
determine the initramfs cpio extension/compression is a string, and due
to how Kconfig works, it will evaluate, in order, how to assign it.
Setting CONFIG_INITRAMFS_COMPRESSION_NONE with
CONFIG_INITRAMFS_SOURCE="" cannot possibly work (because of the depends
on INITRAMFS_SOURCE!="") yet we still manage to get it assigned to
something: ".gz" because CONFIG_RD_GZIP=y is set in my kernel, even when
there is no initramfs being built.

So we basically end-up generating two initramfs_data.cpio* files, one
without extension, and one with .gz. This causes usr/Makefile to track
usr/initramfs_data.cpio.gz, and not usr/initramfs_data.cpio, that is
also problematic after 9e3596b0c6539e28546ff7c72a06576627068353
("kbuild: initramfs cleanup, set target from Kconfig") because we used
to track everything in $(targets) before that.

The end result is that the kernel with an initramfs clearly does not
contain what we expect it to, it has a stale initramfs_data.cpio file
built into.

So these two specific commits have just rendered the whole thing
unworkable for me, because of how string config options and how they get
assigned they default values work.

So right now, we are completely at the mercy of whichever was the first
CONFIG_RD_* compression option set and the whole process which I
described where we try to build two kernels one after the other is just
badly broken...

Happy to test any patch, otherwise I am just considering making hiding
INITRAMFS_COMPRESSION depends on INITRAMFS_SOURCE!= to make results more
predictable.

Thoughts?

> ---
> I'm sorry for the noise. I'm resending this as a new version because
> Thunderbird likes to word wrap things it shouldn't be word wrapping.
> 
> diff --git a/usr/Kconfig b/usr/Kconfig
> index bf8e8f1..6278f13 100644
> --- a/usr/Kconfig
> +++ b/usr/Kconfig
> @@ -99,8 +99,125 @@ config RD_LZ4
>  	  Support loading of a LZ4 encoded initial ramdisk or cpio buffer
>  	  If unsure, say N.
>  
> +choice
> +	prompt "Built-in initramfs compression mode"
> +	depends on INITRAMFS_SOURCE!=""
> +	optional
> +	help
> +	  This option allows you to decide by which algorithm the builtin
> +	  initramfs will be compressed.  Several compression algorithms are
> +	  available, which differ in efficiency, compression and
> +	  decompression speed.  Compression speed is only relevant
> +	  when building a kernel.  Decompression speed is relevant at
> +	  each boot. Also the memory usage during decompression may become
> +	  relevant on memory constrained systems. This is usually based on the
> +	  dictionary size of the algorithm with algorithms like XZ and LZMA
> +	  featuring large dictionary sizes.
> +
> +	  High compression options are mostly useful for users who are
> +	  low on RAM, since it reduces the memory consumption during
> +	  boot.
> +
> +	  Keep in mind that your build system needs to provide the appropriate
> +	  compression tool to compress the generated initram cpio file for
> +	  embedding.
> +
> +	  If in doubt, select 'None'
> +
> +config INITRAMFS_COMPRESSION_NONE
> +	bool "None"
> +	help
> +	  Do not compress the built-in initramfs at all. This may sound wasteful
> +	  in space, but, you should be aware that the built-in initramfs will be
> +	  compressed at a later stage anyways along with the rest of the kernel,
> +	  on those architectures that support this. However, not compressing the
> +	  initramfs may lead to slightly higher memory consumption during a
> +	  short time at boot, while both the cpio image and the unpacked
> +	  filesystem image will be present in memory simultaneously
> +
> +config INITRAMFS_COMPRESSION_GZIP
> +	bool "Gzip"
> +	depends on RD_GZIP
> +	help
> +	  Use the old and well tested gzip compression algorithm. Gzip provides
> +	  a good balance between compression ratio and decompression speed and
> +	  has a reasonable compression speed. It is also more likely to be
> +	  supported by your build system as the gzip tool is present by default
> +	  on most distros.
> +
> +config INITRAMFS_COMPRESSION_BZIP2
> +	bool "Bzip2"
> +	depends on RD_BZIP2
> +	help
> +	  It's compression ratio and speed is intermediate. Decompression speed
> +	  is slowest among the choices. The initramfs size is about 10% smaller
> +	  with bzip2, in comparison to gzip. Bzip2 uses a large amount of
> +	  memory. For modern kernels you will need at least 8MB RAM or more for
> +	  booting.
> +
> +	  If you choose this, keep in mind that you need to have the bzip2 tool
> +	  available to be able to compress the initram.
> +
> +config INITRAMFS_COMPRESSION_LZMA
> +	bool "LZMA"
> +	depends on RD_LZMA
> +	help
> +	  This algorithm's compression ratio is best but has a large dictionary
> +	  size which might cause issues in memory constrained systems.
> +	  Decompression speed is between the other choices. Compression is
> +	  slowest. The initramfs size is about 33% smaller with LZMA in
> +	  comparison to gzip.
> +
> +	  If you choose this, keep in mind that you may need to install the xz
> +	  or lzma tools to be able to compress the initram.
> +
> +config INITRAMFS_COMPRESSION_XZ
> +	bool "XZ"
> +	depends on RD_XZ
> +	help
> +	  XZ uses the LZMA2 algorithm and has a large dictionary which may cause
> +	  problems on memory constrained systems. The initramfs size is about
> +	  30% smaller with XZ in comparison to gzip. Decompression speed is
> +	  better than that of bzip2 but worse than gzip and LZO. Compression is
> +	  slow.
> +
> +	  If you choose this, keep in mind that you may need to install the xz
> +	  tool to be able to compress the initram.
> +
> +config INITRAMFS_COMPRESSION_LZO
> +	bool "LZO"
> +	depends on RD_LZO
> +	help
> +	  It's compression ratio is the second poorest amongst the choices. The
> +	  kernel size is about 10% bigger than gzip. Despite that, it's
> +	  decompression speed is the second fastest and it's compression speed
> +	  is quite fast too.
> +
> +	  If you choose this, keep in mind that you may need to install the lzop
> +	  tool to be able to compress the initram.
> +
> +config INITRAMFS_COMPRESSION_LZ4
> +	bool "LZ4"
> +	depends on RD_LZ4
> +	help
> +	  It's compression ratio is the poorest amongst the choices. The kernel
> +	  size is about 15% bigger than gzip; however its decompression speed
> +	  is the fastest.
> +
> +	  If you choose this, keep in mind that most distros don't provide lz4
> +	  by default which could cause a build failure.
> +
> +endchoice
> +
>  config INITRAMFS_COMPRESSION
>  	string
> +	default ""      if INITRAMFS_COMPRESSION_NONE
> +	default ".gz"   if INITRAMFS_COMPRESSION_GZIP
> +	default ".bz2"  if INITRAMFS_COMPRESSION_BZIP2
> +	default ".lzma" if INITRAMFS_COMPRESSION_LZMA
> +	default ".xz"   if INITRAMFS_COMPRESSION_XZ
> +	default ".lzo"  if INITRAMFS_COMPRESSION_LZO
> +	default ".lz4"  if INITRAMFS_COMPRESSION_LZ4
>  	default ".gz"   if RD_GZIP
>  	default ".lz4"  if RD_LZ4
>  	default ".lzo"  if RD_LZO
> 

-- 
Florian

Re: [v3, 2/2] initramfs: Allow again choice of the embedded initram compression algorithm

[Florian Fainelli]

On 05/20/2017 03:30 PM, Florian Fainelli wrote:
> Hi Francisco, Nicholas,
> 
> Nicholas already fixed part of this commit, but there is more breakage,
> see below:
> 
> On 09/27/2016 01:32 PM, klondike wrote:
>> Choosing the appropriate compression option when using an embeded initramfs
>> can result in significant size differences in the resulting data.
>>
>> This is caused by avoiding double compression of the initramfs contents.
>> For example on my tests, choosing CONFIG_INITRAMFS_COMPRESSION_NONE when
>> compressing the kernel using XZ) results in up to 500KiB differences (9MiB to
>>  8.5MiB) in the kernel size as the dictionary will not get polluted with
>> uncomprensible data and may reuse kernel data too.
>>
>> Despite embedding an uncompressed initramfs, a user may want to allow for a
>> compressed extra initramfs to be passed using the rd system, for example to
>> boot a recovery system. Commit 9ba4bcb645898d562498ea66a0df958ef0e7a68c
>> ("initramfs: read CONFIG_RD_ variables for initramfs compression") broke
>> that behavior by making the choice based on CONFIG_RD_* instead of adding
>> CONFIG_INITRAMFS_COMPRESSION_LZ4. Saddly, CONFIG_RD_* is also used to
>> choose the supported RD compression algorithms by the kernel and a user may
>> want to suppport more than one.
>>
>> This patch also reverses 3e4e0f0a8756dade3023d1f47d50fbced7749788
>> ("initramfs: remove "compression mode" choice") restoring back the
>> "compression mode" choice and includes the CONFIG_INITRAMFS_COMPRESSION_LZ4
>> option which was never added.
>>
>> As a result the following options are added or readed affecting the embedded
>> initramfs compression:
>> INITRAMFS_COMPRESSION_NONE Do no compression
>> INITRAMFS_COMPRESSION_GZIP Compress using gzip
>> INITRAMFS_COMPRESSION_BZIP2 Compress using bzip2
>> INITRAMFS_COMPRESSION_LZMA Compress using lzma
>> INITRAMFS_COMPRESSION_XZ Compress using xz
>> INITRAMFS_COMPRESSION_LZO Compress using lzo
>> INITRAMFS_COMPRESSION_LZ4 Compress using lz4
>>
>> These depend on the corresponding CONFIG_RD_* option being set (except NONE
>> which has no dependencies).
>>
>> This patch depends on the previous one (the previous version didn't) to
>> simplify the way in which the algorithm is chosen and keep backwards
>> compatibility with the behaviour introduced by commit
>>  9ba4bcb645898d562498ea66a0df958ef0e7a68c
>>
>> Signed-off-by: Francisco Blas Izquierdo Riera (klondike) <klondike@klondike.es>
>> Cc: P J P <ppandit@redhat.com>
>> Cc: Paul Bolle <pebolle@tiscali.nl>
>> Cc: Andrew Morton <akpm@linux-foundation.org>
> 
> Running a bisection against usr/ was not particularly convincing but
> here is basically what I am observing which used to work just fine
> before as of v4.9 and since I tracked it to this particular
> commit/patch. Here is what my build system does:
> 
> - kernel is initially configured not to have an initramfs included
> - build the user space root file system
> - re-configure the kernel to have an initramfs included
> (CONFIG_INITRAMFS_SOURCE="/path/to/romfs") and set relevant
> CONFIG_INITRAMFS options, in my case, no compression
> - kernel is re-built with these options -> kernel+initramfs image is copied
> - kernel is re-built again without these options -> kernel image is copied
> 
> Now suppose you make changes to your root filesystem, like add/remove
> applications, initramfs_data.cpio is now a stale file and go through the
> same process again:
> 
> - build the kernel without an initramfs
> - user space (re)build
> - build the kernel with an initramfs
> 
> Building a kernel without an initramfs means setting this option:
> 
> CONFIG_INITRAMFS_SOURCE=""
> 
> whereas building a kernel with an initramfs means setting these options:
> 
> CONFIG_INITRAMFS_SOURCE="/home/fainelli/work/uclinux-rootfs/romfs
> /home/fainelli/work/uclinux-rootfs/misc/initramfs.dev"
> CONFIG_INITRAMFS_ROOT_UID=1000
> CONFIG_INITRAMFS_ROOT_GID=1000
> CONFIG_INITRAMFS_COMPRESSION_NONE=y
> # CONFIG_INITRAMFS_COMPRESSION_GZIP is not set
> # CONFIG_INITRAMFS_COMPRESSION_BZIP2 is not set
> # CONFIG_INITRAMFS_COMPRESSION_LZMA is not set
> # CONFIG_INITRAMFS_COMPRESSION_XZ is not set
> # CONFIG_INITRAMFS_COMPRESSION_LZO is not set
> # CONFIG_INITRAMFS_COMPRESSION_LZ4 is not set
> CONFIG_INITRAMFS_COMPRESSION=""
> 
> Commit db2aa7fd15e857891cefbada8348c8d938c7a2bc ("initramfs: allow again
> choice of the embedded initram compression algorithm") appears
> problematic because CONFIG_INITRAMFS_COMPRESSION which is used to
> determine the initramfs cpio extension/compression is a string, and due
> to how Kconfig works, it will evaluate, in order, how to assign it.
> Setting CONFIG_INITRAMFS_COMPRESSION_NONE with
> CONFIG_INITRAMFS_SOURCE="" cannot possibly work (because of the depends
> on INITRAMFS_SOURCE!="") yet we still manage to get it assigned to
> something: ".gz" because CONFIG_RD_GZIP=y is set in my kernel, even when
> there is no initramfs being built.
> 
> So we basically end-up generating two initramfs_data.cpio* files, one
> without extension, and one with .gz. This causes usr/Makefile to track
> usr/initramfs_data.cpio.gz, and not usr/initramfs_data.cpio, that is
> also problematic after 9e3596b0c6539e28546ff7c72a06576627068353
> ("kbuild: initramfs cleanup, set target from Kconfig") because we used
> to track everything in $(targets) before that.
> 
> The end result is that the kernel with an initramfs clearly does not
> contain what we expect it to, it has a stale initramfs_data.cpio file
> built into.
> 
> So these two specific commits have just rendered the whole thing
> unworkable for me, because of how string config options and how they get
> assigned they default values work.
> 
> So right now, we are completely at the mercy of whichever was the first
> CONFIG_RD_* compression option set and the whole process which I
> described where we try to build two kernels one after the other is just
> badly broken...
> 
> Happy to test any patch, otherwise I am just considering making hiding
> INITRAMFS_COMPRESSION depends on INITRAMFS_SOURCE!= to make results more
> predictable.

OK, so here is what I just tested and this puts me back in business, I
don't think it makes sense to expose CONFIG_INITRAMFS_COMPRESSION if
initramfs is purposely disabled anyway, will submit a formal fix later
tonight:

diff --git a/usr/Kconfig b/usr/Kconfig
index c0c48507e44e..ad0543e21760 100644
--- a/usr/Kconfig
+++ b/usr/Kconfig
@@ -220,6 +220,7 @@ config INITRAMFS_COMPRESSION_LZ4
 endchoice

 config INITRAMFS_COMPRESSION
+       depends on INITRAMFS_SOURCE!=""
        string
        default ""      if INITRAMFS_COMPRESSION_NONE
        default ".gz"   if INITRAMFS_COMPRESSION_GZIP


> 
> Thoughts?
> 
>> ---
>> I'm sorry for the noise. I'm resending this as a new version because
>> Thunderbird likes to word wrap things it shouldn't be word wrapping.
>>
>> diff --git a/usr/Kconfig b/usr/Kconfig
>> index bf8e8f1..6278f13 100644
>> --- a/usr/Kconfig
>> +++ b/usr/Kconfig
>> @@ -99,8 +99,125 @@ config RD_LZ4
>>  	  Support loading of a LZ4 encoded initial ramdisk or cpio buffer
>>  	  If unsure, say N.
>>  
>> +choice
>> +	prompt "Built-in initramfs compression mode"
>> +	depends on INITRAMFS_SOURCE!=""
>> +	optional
>> +	help
>> +	  This option allows you to decide by which algorithm the builtin
>> +	  initramfs will be compressed.  Several compression algorithms are
>> +	  available, which differ in efficiency, compression and
>> +	  decompression speed.  Compression speed is only relevant
>> +	  when building a kernel.  Decompression speed is relevant at
>> +	  each boot. Also the memory usage during decompression may become
>> +	  relevant on memory constrained systems. This is usually based on the
>> +	  dictionary size of the algorithm with algorithms like XZ and LZMA
>> +	  featuring large dictionary sizes.
>> +
>> +	  High compression options are mostly useful for users who are
>> +	  low on RAM, since it reduces the memory consumption during
>> +	  boot.
>> +
>> +	  Keep in mind that your build system needs to provide the appropriate
>> +	  compression tool to compress the generated initram cpio file for
>> +	  embedding.
>> +
>> +	  If in doubt, select 'None'
>> +
>> +config INITRAMFS_COMPRESSION_NONE
>> +	bool "None"
>> +	help
>> +	  Do not compress the built-in initramfs at all. This may sound wasteful
>> +	  in space, but, you should be aware that the built-in initramfs will be
>> +	  compressed at a later stage anyways along with the rest of the kernel,
>> +	  on those architectures that support this. However, not compressing the
>> +	  initramfs may lead to slightly higher memory consumption during a
>> +	  short time at boot, while both the cpio image and the unpacked
>> +	  filesystem image will be present in memory simultaneously
>> +
>> +config INITRAMFS_COMPRESSION_GZIP
>> +	bool "Gzip"
>> +	depends on RD_GZIP
>> +	help
>> +	  Use the old and well tested gzip compression algorithm. Gzip provides
>> +	  a good balance between compression ratio and decompression speed and
>> +	  has a reasonable compression speed. It is also more likely to be
>> +	  supported by your build system as the gzip tool is present by default
>> +	  on most distros.
>> +
>> +config INITRAMFS_COMPRESSION_BZIP2
>> +	bool "Bzip2"
>> +	depends on RD_BZIP2
>> +	help
>> +	  It's compression ratio and speed is intermediate. Decompression speed
>> +	  is slowest among the choices. The initramfs size is about 10% smaller
>> +	  with bzip2, in comparison to gzip. Bzip2 uses a large amount of
>> +	  memory. For modern kernels you will need at least 8MB RAM or more for
>> +	  booting.
>> +
>> +	  If you choose this, keep in mind that you need to have the bzip2 tool
>> +	  available to be able to compress the initram.
>> +
>> +config INITRAMFS_COMPRESSION_LZMA
>> +	bool "LZMA"
>> +	depends on RD_LZMA
>> +	help
>> +	  This algorithm's compression ratio is best but has a large dictionary
>> +	  size which might cause issues in memory constrained systems.
>> +	  Decompression speed is between the other choices. Compression is
>> +	  slowest. The initramfs size is about 33% smaller with LZMA in
>> +	  comparison to gzip.
>> +
>> +	  If you choose this, keep in mind that you may need to install the xz
>> +	  or lzma tools to be able to compress the initram.
>> +
>> +config INITRAMFS_COMPRESSION_XZ
>> +	bool "XZ"
>> +	depends on RD_XZ
>> +	help
>> +	  XZ uses the LZMA2 algorithm and has a large dictionary which may cause
>> +	  problems on memory constrained systems. The initramfs size is about
>> +	  30% smaller with XZ in comparison to gzip. Decompression speed is
>> +	  better than that of bzip2 but worse than gzip and LZO. Compression is
>> +	  slow.
>> +
>> +	  If you choose this, keep in mind that you may need to install the xz
>> +	  tool to be able to compress the initram.
>> +
>> +config INITRAMFS_COMPRESSION_LZO
>> +	bool "LZO"
>> +	depends on RD_LZO
>> +	help
>> +	  It's compression ratio is the second poorest amongst the choices. The
>> +	  kernel size is about 10% bigger than gzip. Despite that, it's
>> +	  decompression speed is the second fastest and it's compression speed
>> +	  is quite fast too.
>> +
>> +	  If you choose this, keep in mind that you may need to install the lzop
>> +	  tool to be able to compress the initram.
>> +
>> +config INITRAMFS_COMPRESSION_LZ4
>> +	bool "LZ4"
>> +	depends on RD_LZ4
>> +	help
>> +	  It's compression ratio is the poorest amongst the choices. The kernel
>> +	  size is about 15% bigger than gzip; however its decompression speed
>> +	  is the fastest.
>> +
>> +	  If you choose this, keep in mind that most distros don't provide lz4
>> +	  by default which could cause a build failure.
>> +
>> +endchoice
>> +
>>  config INITRAMFS_COMPRESSION
>>  	string
>> +	default ""      if INITRAMFS_COMPRESSION_NONE
>> +	default ".gz"   if INITRAMFS_COMPRESSION_GZIP
>> +	default ".bz2"  if INITRAMFS_COMPRESSION_BZIP2
>> +	default ".lzma" if INITRAMFS_COMPRESSION_LZMA
>> +	default ".xz"   if INITRAMFS_COMPRESSION_XZ
>> +	default ".lzo"  if INITRAMFS_COMPRESSION_LZO
>> +	default ".lz4"  if INITRAMFS_COMPRESSION_LZ4
>>  	default ".gz"   if RD_GZIP
>>  	default ".lz4"  if RD_LZ4
>>  	default ".lzo"  if RD_LZO
>>
> 

-- 
Florian