[PATCH 2/2] Export initial ramdisk compression config

[PATCH 2/2] Export initial ramdisk compression config

[P J P]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1137 bytes --]

   Hello,

While building the 3.11 kernel recently, I bumped into this issue.

Menuconfig allows one to choose compression format of an initial ramdisk 
image. But this selection does not result in creation of a duly compressed 
initramfs image. Because - $ make install - does not pass on the selected 
compression option to dracut(8) tool, which generates the initramfs file. This 
leads to a situation wherein the initramfs file is gzip(1) compressed, but the 
kernel knows only to decompress the user's chosen compression format.

The attached patch herein, attempts to fix this issue by exporting the 
compression configuration option to an environment variable of a sub-shell 
spawned by Make(1) with the invocation of installkernel(8) during the 
installation process.

The dracut(8) & new-kernel-pkg(8) tools need to be patched to recognise and 
read such an environment variable and use the selected compression algorithm 
while creating the initial ramdisk image files.

Could someone please review this one too?

Thank you.
--
Prasad J Pandit / Red Hat Security Response Team
DB7A 84C5 D3F9 7CD1 B5EB  C939 D048 7860 3655 602B

[-- Attachment #2: Type: TEXT/PLAIN, Size: 2507 bytes --]

From 88878c7dfb04e69b75f1518fe809b917095a33b8 Mon Sep 17 00:00:00 2001
From: P J P <prasad@redhat.com>
Date: Sun, 15 Sep 2013 13:37:43 +0530
Subject: [PATCH 2/2] Export initial ramdisk compression config option

Make menuconfig allows one to choose compression format of an
initial ramdisk image. But this choice does not result in duly
compressed ramdisk image. Because - $ make install - does not
pass on the selected compression choice to the dracut(8) tool,
which creates the initramfs file. dracut(8) generates the image
with the default compression, ie. gzip(1).

This patch exports the selected compression option to a sub-shell
environment, so that it could be used by dracut(8) tool to generate
appropriately compressed initramfs images.

There isn't a straight forward way to pass on options to dracut(8)
via positional parameters. Because it is indirectly invoked at the
end of a $ make install sequence.

 # make install
   -> arch/$arch/boot/Makefile
    -> arch/$arch/boot/install.sh
     -> /sbing/installkernel ...
      -> /sbin/new-kernel-pkg ...
       -> /sbin/dracut ...

Signed-off-by: P J P <prasad@redhat.com>

diff --git a/arch/x86/Makefile b/arch/x86/Makefile
index 41250fb..9f4cdf1d 100644
--- a/arch/x86/Makefile
+++ b/arch/x86/Makefile
@@ -11,6 +11,23 @@ else
         KBUILD_DEFCONFIG := $(ARCH)_defconfig
 endif
 
+# Select initial ramdisk compression format, default is gzip(1).
+# This shall be used by the dracut(8) tool while creating an initramfs image.
+#
+INITRD_COMPRESS=gzip
+ifeq ($(CONFIG_RD_BZIP2), y)
+        INITRD_COMPRESS=bzip2
+else ifeq ($(CONFIG_RD_LZMA), y)
+        INITRD_COMPRESS=lzma
+else ifeq ($(CONFIG_RD_XZ), y)
+        INITRD_COMPRESS=xz
+else ifeq ($(CONFIG_RD_LZO), y)
+        INITRD_COMPRESS=lzo
+else ifeq ($(CONFIG_RD_LZ4), y)
+        INITRD_COMPRESS=lz4
+endif
+export INITRD_COMPRESS
+
 # BITS is used as extension for files which are available in a 32 bit
 # and a 64 bit version to simplify shared Makefiles.
 # e.g.: obj-y += foo_$(BITS).o
diff --git a/init/do_mounts_rd.c b/init/do_mounts_rd.c
index 76faec1..b3d6c38 100644
--- a/init/do_mounts_rd.c
+++ b/init/do_mounts_rd.c
@@ -57,6 +57,11 @@ static int __init crd_load(int in_fd, int out_fd, decompress_fn deco);
  *	cramfs
  *	squashfs
  *	gzip
+ *	bzip2
+ *	lzma
+ *	xz
+ *	lzo
+ *	lz4
  */
 static int __init
 identify_ramdisk_image(int fd, int start_block, decompress_fn *decompressor)
-- 
1.8.3.1

Re: [PATCH 2/2] Export initial ramdisk compression config

[Andrew Morton]

On Sun, 15 Sep 2013 14:49:05 +0530 (IST) P J P <ppandit@redhat.com> wrote:

> Make menuconfig allows one to choose compression format of an
> initial ramdisk image. But this choice does not result in duly
> compressed ramdisk image. Because - $ make install - does not
> pass on the selected compression choice to the dracut(8) tool,
> which creates the initramfs file. dracut(8) generates the image
> with the default compression, ie. gzip(1).
> 
> This patch exports the selected compression option to a sub-shell
> environment, so that it could be used by dracut(8) tool to generate
> appropriately compressed initramfs images.
> 
> There isn't a straight forward way to pass on options to dracut(8)
> via positional parameters. Because it is indirectly invoked at the
> end of a $ make install sequence.
> 
>  # make install
>    -> arch/$arch/boot/Makefile
>     -> arch/$arch/boot/install.sh
>      -> /sbing/installkernel ...
>       -> /sbin/new-kernel-pkg ...
>        -> /sbin/dracut ...
> 

It's a bit confusing whether all this appiles to initrd, to initramfs
or to both.  Can you please clarify all this and be sure that it's all
consistent?

> --- a/arch/x86/Makefile
> +++ b/arch/x86/Makefile
> @@ -11,6 +11,23 @@ else
>          KBUILD_DEFCONFIG := $(ARCH)_defconfig
>  endif
>  
> +# Select initial ramdisk compression format, default is gzip(1).
> +# This shall be used by the dracut(8) tool while creating an initramfs image.
> +#
> +INITRD_COMPRESS=gzip
> +ifeq ($(CONFIG_RD_BZIP2), y)
> +        INITRD_COMPRESS=bzip2
> +else ifeq ($(CONFIG_RD_LZMA), y)
> +        INITRD_COMPRESS=lzma
> +else ifeq ($(CONFIG_RD_XZ), y)
> +        INITRD_COMPRESS=xz
> +else ifeq ($(CONFIG_RD_LZO), y)
> +        INITRD_COMPRESS=lzo
> +else ifeq ($(CONFIG_RD_LZ4), y)
> +        INITRD_COMPRESS=lz4
> +endif
> +export INITRD_COMPRESS

A few things...

- Why is it specific to x86?  Other architcetures use initramfs?

- People add new compression schemes fairly regularly.  We should add
  a code comment somewhere in a place where such people are sure to
  find it.  That comment should explain what's going on, remind them to
  update this environment varlaible and should explain to them the
  process by which they get dracut(8) updated if needed.

- Can we avoid having to update dracut each time a new compression
  scheme is added?  I assume your dracut changes will just exec
  "$INITRD_COMPRESS -d", so as long as any new decompression
  application uses the expected -d argument in the expected way, it
  should work seamlessly?

Re: [PATCH 2/2] Export initial ramdisk compression config

[P J P]

   Hello Andrew,

Thank you so much for reviewing these patches.

+-- On Mon, 23 Sep 2013, Andrew Morton wrote --+
| It's a bit confusing whether all this appiles to initrd, to initramfs
| or to both.  Can you please clarify all this and be sure that it's all
| consistent?

  IIUC, we no longer use initrd block device images. Instead we use initramfs 
which is a cpio(1) archive image. Both mkinitrd(8) & dracut(8) are invoked 
from /sbin/new-kernel-pkg tool. And mkinitrd(8) is nothing but a wrapper 
around dracut(8) these days.
===
  $ man mkinitrd
       mkinitrd - is a compat wrapper, which calls dracut to generate an
       initramfs

  $ less /sbin/new-kernel-pkg     # my local patched version
...
doMkinitrd() {
    if [ -n "$dracut" ]; then
+       if [ -n "$INITRD_COMPRESS" ]; then
+           dracutcompress=--$INITRD_COMPRESS
+       fi
+       tool="dracut $dracuthostonly -f $dracutcompress $initrdfile $version"
    else
        tool="mkinitrd --allow-missing -f $initrdfile $version"
    fi
    [ -n "$verbose" ] && echo "creating initrd: $tool"
    $tool
    rc=$?
    if [ $rc != 0 ]; then
        echo "mkinitrd failed" >&2
        exit 1
    fi
}
===

Because mkinitrd(8) does not support passing of compression argument to 
dracut(8), it must use the default compression, ie. gzip(1).


| A few things...
| - Why is it specific to x86?  Other architcetures use initramfs?

  No, it is not specific to x86, most likely all architecturess' Makefile 
would need similar patch. But I haven't looked into those yet.
 
| - People add new compression schemes fairly regularly.  We should add
|   a code comment somewhere in a place where such people are sure to
|   find it.  That comment should explain what's going on, remind them to
|   update this environment varlaible and should explain to them the
|   process by which they get dracut(8) updated if needed.

  True. Maybe user/Kconfig could have a note about updating INITRD_COMPRESS 
environment variable through a relevant $arch/Makefile ? 

Looking at the dracut(8) tool, it seems it can easily support any number of 
compression tools, for it already has options --gzip, --bzip2, --xz & --lzma.
Adding --lzo, --lz4 etc won't be much difficult.

| - Can we avoid having to update dracut each time a new compression
|   scheme is added?  I assume your dracut changes will just exec
|   "$INITRD_COMPRESS -d", so as long as any new decompression
|   application uses the expected -d argument in the expected way, it
|   should work seamlessly?

  Yes, dracut(8) has | --compress=$INITRD_COMPRESS | option which could be 
used to pass compression program to dracut(8). Not sure if dracut(8) expects 
an absolute path or just the program name. But to use this option, we'll need 
to patch /sbin/new-kernel-pkg tool.

Alternatively, dracut(8) could be patched to recognise and read 
$INITRD_COMPRESS variable and internally it can define precedence order 
between environment variable, command-line options and default fall-back 
option.


Thank you.
--
Prasad J Pandit / Red Hat Security Response Team
DB7A 84C5 D3F9 7CD1 B5EB  C939 D048 7860 3655 602B

Re: [PATCH 2/2] Export initial ramdisk compression config

[Rob Landley]

On 09/24/2013 02:41:54 PM, P J P wrote:
>    Hello Andrew,
> 
> Thank you so much for reviewing these patches.
> 
> +-- On Mon, 23 Sep 2013, Andrew Morton wrote --+
> | It's a bit confusing whether all this appiles to initrd, to  
> initramfs
> | or to both.  Can you please clarify all this and be sure that it's  
> all
> | consistent?
> 
>   IIUC, we no longer use initrd block device images. Instead we use  
> initramfs
> which is a cpio(1) archive image. Both mkinitrd(8) & dracut(8) are  
> invoked
> from /sbin/new-kernel-pkg tool. And mkinitrd(8) is nothing but a  
> wrapper
> around dracut(8) these days.

$ find . -name "Makefile*" | xargs grep new-kernel-pkg
$ grep -r new-kernel-pkg scripts
$

Ah, so it's an out of tree bespoke Red Hat tool. No wonder I couldn't  
find it.

> Looking at the dracut(8) tool, it seems it can easily support any  
> number of
> compression tools, for it already has options --gzip, --bzip2, --xz &  
> --lzma.
> Adding --lzo, --lz4 etc won't be much difficult.

You're reimplemented the posix "pax" command?

> | - Can we avoid having to update dracut each time a new compression
> |   scheme is added?  I assume your dracut changes will just exec
> |   "$INITRD_COMPRESS -d", so as long as any new decompression
> |   application uses the expected -d argument in the expected way, it
> |   should work seamlessly?
> 
>   Yes, dracut(8) has | --compress=$INITRD_COMPRESS | option which  
> could be
> used to pass compression program to dracut(8).

$ grep -r INITRD_COMPRESS linux
$

Is this what you're currently doing, or the change you're requesting in  
the kernel?

> Not sure if dracut(8) expects
> an absolute path or just the program name. But to use this option,  
> we'll need
> to patch /sbin/new-kernel-pkg tool.
> 
> Alternatively, dracut(8) could be patched to recognise and read
> $INITRD_COMPRESS variable and internally it can define precedence  
> order
> between environment variable, command-line options and default  
> fall-back
> option.

How does your code get called?

Rob

Re: [PATCH 2/2] Export initial ramdisk compression config

[P J P]

+-- On Wed, 25 Sep 2013, Rob Landley wrote --+
| Ah, so it's an out of tree bespoke Red Hat tool. No wonder I couldn't find it.

  It is not Red Hat tool.

| You're reimplemented the posix "pax" command?

  Ummn, not sure. Didn't see anything about 'pax'.
 
| Is this what you're currently doing, or the change you're requesting in the 
| kernel?

  Yes, I'm proposing to export $INITRD_COMPRESS variable so that it can be 
used by dracut(8).

| How does your code get called?

  The patch's commit log explains it.


Thank you.
--
Prasad J Pandit / Red Hat Security Response Team
DB7A 84C5 D3F9 7CD1 B5EB  C939 D048 7860 3655 602B

Re: [PATCH 2/2] Export initial ramdisk compression config

[P J P]

  Hello Andrew,

I was wondering if you had a chance to review this patch further? Should I 
send similar patches for other architectures too? As in you aren't waiting on 
me for that, are you?

Thank you.
--
Prasad J Pandit / Red Hat Security Response Team
DB7A 84C5 D3F9 7CD1 B5EB  C939 D048 7860 3655 602B

Re: [PATCH 2/2] Export initial ramdisk compression config

[P J P]

   Hello Andrew,

Just to check, did you have a chance to review it further? Are you waiting on 
me for something?
--
Prasad J Pandit / Red Hat Security Response Team
DB7A 84C5 D3F9 7CD1 B5EB  C939 D048 7860 3655 602B

Re: [PATCH 2/2] Export initial ramdisk compression config

[Andrew Morton]

On Wed, 25 Sep 2013 01:11:54 +0530 (IST) P J P <ppandit@redhat.com> wrote:

> | A few things...
> | - Why is it specific to x86?  Other architcetures use initramfs?
> 
>   No, it is not specific to x86, most likely all architecturess' Makefile 
> would need similar patch. But I haven't looked into those yet.

It would be better to make the change in one place, rather than for
each architecture.  That would appear to involve moving a hunk from
arch/x86/Makefile into init/Makefile, or perhaps ./Makefile.

That way it can still be tested on x86, and it is very reasonable to
assume that it will work as intended on other architectures.

Re: [PATCH 2/2] Export initial ramdisk compression config

[P J P]

  Hello Andrew,

+-- On Wed, 9 Oct 2013, Andrew Morton wrote --+
| It would be better to make the change in one place, rather than for each 
| architecture.  That would appear to involve moving a hunk from 
| arch/x86/Makefile into init/Makefile, or perhaps ./Makefile.
 
  Right, I was trying to figure the same. I'll do the changes and send an 
updated patch.

Thank you so much for reviewing it and the update.

Thank you.
--
Prasad J Pandit / Red Hat Security Response Team

Re: [PATCH 2/2] Export initial ramdisk compression config

[P J P]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 488 bytes --]

  Hi,

+-- On Wed, 9 Oct 2013, Andrew Morton wrote --+
| It would be better to make the change in one place, rather than for
| each architecture.  That would appear to involve moving a hunk from
| arch/x86/Makefile into init/Makefile, or perhaps ./Makefile.

  Please find attached herein, an updated patch which adds the export snippet 
to the top level ./Makefile. I tried init/Makefile too, but it does not seem 
to work.

Thank you!
--
Prasad J Pandit / Red Hat Security Response Team

[-- Attachment #2: Type: TEXT/PLAIN, Size: 1910 bytes --]

From 2bb603f666371289ee660be5cf7a6f87740edd8e Mon Sep 17 00:00:00 2001
From: P J P <prasad@redhat.com>
Date: Thu, 10 Oct 2013 19:53:24 +0530
Subject: Export initial ramdisk compression config option

Make menuconfig allows one to choose compression format of an
initial ramdisk image. But this choice does not result in duly
compressed ramdisk image. Because - $ make install - does not
pass on the selected compression choice to the dracut(8) tool,
which creates the initramfs file. dracut(8) generates the image
with the default compression, ie. gzip(1).

This patch exports the selected compression option to a sub-shell
environment, so that it could be used by dracut(8) tool to generate
appropriately compressed initramfs images.

There isn't a straight forward way to pass on options to dracut(8)
via positional parameters. Because it is indirectly invoked at the
end of a $ make install sequence.

 # make install
   -> arch/$arch/boot/Makefile
    -> arch/$arch/boot/install.sh
     -> /sbing/installkernel ...
      -> /sbin/new-kernel-pkg ...
       -> /sbin/dracut ...

diff --git a/Makefile b/Makefile
index 8d0668f..5c7f8aa 100644
--- a/Makefile
+++ b/Makefile
@@ -720,6 +720,22 @@ mod_strip_cmd = true
 endif # INSTALL_MOD_STRIP
 export mod_strip_cmd
 
+# Select initial ramdisk compression format, default is gzip(1).
+# This shall be used by the dracut(8) tool while creating an initramfs image.
+#
+INITRD_COMPRESS=gzip
+ifeq ($(CONFIG_RD_BZIP2), y)
+        INITRD_COMPRESS=bzip2
+else ifeq ($(CONFIG_RD_LZMA), y)
+        INITRD_COMPRESS=lzma
+else ifeq ($(CONFIG_RD_XZ), y)
+        INITRD_COMPRESS=xz
+else ifeq ($(CONFIG_RD_LZO), y)
+        INITRD_COMPRESS=lzo
+else ifeq ($(CONFIG_RD_LZ4), y)
+        INITRD_COMPRESS=lz4
+endif
+export INITRD_COMPRESS
 
 ifdef CONFIG_MODULE_SIG_ALL
 MODSECKEY = ./signing_key.priv
-- 
1.8.3.1

Re: [PATCH 2/2] Export initial ramdisk compression config

[P J P]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 437 bytes --]

   Hello Andrew,
+-- On Wed, 9 Oct 2013, Andrew Morton wrote --+
| It would be better to make the change in one place, rather than for
| each architecture.  That would appear to involve moving a hunk from
| arch/x86/Makefile into init/Makefile, or perhaps ./Makefile.

  Please see this revised version, I missed to include init/do_mounts_rd.c 
changes in the previous one.

Thank you.
--
Prasad J Pandit / Red Hat Security Response Team

[-- Attachment #2: Type: TEXT/PLAIN, Size: 2330 bytes --]

From 14dccee98f12f2dbca224510bcfedd1a397ed177 Mon Sep 17 00:00:00 2001
From: P J P <prasad@redhat.com>
Date: Thu, 10 Oct 2013 20:37:48 +0530
Subject: Export initial ramdisk compression config option

Make menuconfig allows one to choose compression format of an
initial ramdisk image. But this choice does not result in duly
compressed ramdisk image. Because - $ make install - does not
pass on the selected compression choice to the dracut(8) tool,
which creates the initramfs file. dracut(8) generates the image
with the default compression, ie. gzip(1).

This patch exports the selected compression option to a sub-shell
environment, so that it could be used by dracut(8) tool to generate
appropriately compressed initramfs images.

There isn't a straight forward way to pass on options to dracut(8)
via positional parameters. Because it is indirectly invoked at the
end of a $ make install sequence.

 # make install
   -> arch/$arch/boot/Makefile
    -> arch/$arch/boot/install.sh
     -> /sbing/installkernel ...
      -> /sbin/new-kernel-pkg ...
       -> /sbin/dracut ...

diff --git a/Makefile b/Makefile
index 8d0668f..5c7f8aa 100644
--- a/Makefile
+++ b/Makefile
@@ -720,6 +720,22 @@ mod_strip_cmd = true
 endif # INSTALL_MOD_STRIP
 export mod_strip_cmd
 
+# Select initial ramdisk compression format, default is gzip(1).
+# This shall be used by the dracut(8) tool while creating an initramfs image.
+#
+INITRD_COMPRESS=gzip
+ifeq ($(CONFIG_RD_BZIP2), y)
+        INITRD_COMPRESS=bzip2
+else ifeq ($(CONFIG_RD_LZMA), y)
+        INITRD_COMPRESS=lzma
+else ifeq ($(CONFIG_RD_XZ), y)
+        INITRD_COMPRESS=xz
+else ifeq ($(CONFIG_RD_LZO), y)
+        INITRD_COMPRESS=lzo
+else ifeq ($(CONFIG_RD_LZ4), y)
+        INITRD_COMPRESS=lz4
+endif
+export INITRD_COMPRESS
 
 ifdef CONFIG_MODULE_SIG_ALL
 MODSECKEY = ./signing_key.priv
diff --git a/init/do_mounts_rd.c b/init/do_mounts_rd.c
index bdaa345..9ee3427 100644
--- a/init/do_mounts_rd.c
+++ b/init/do_mounts_rd.c
@@ -57,6 +57,11 @@ static int __init crd_load(int in_fd, int out_fd, decompress_fn deco);
  *	cramfs
  *	squashfs
  *	gzip
+ *	bzip2
+ *	lzma
+ *	xz
+ *	lzo
+ *	lz4
  */
 static int __init
 identify_ramdisk_image(int fd, int start_block, decompress_fn *decompressor)
-- 
1.8.3.1