[PATCHv2 0/3] handle module compression at install

[PATCHv2 0/3] handle module compression at install

[Bertrand Jacquin]

Hi,

Here is patchset based on commit 480cadc (v3.17-rc1-22-g480cadc) that
aims to handle kernel modules compression while running make modules_install.

Please Cc me, as I'm not subscribed to LKML.

Changes since v1:

 - Drop -9 argument to gzip and xz since Andi Kleen and Rusty Russell
   prove no gain is made using them.

Bertrand Jacquin (3):
  modinst: wrap long lines in order to enhance cmd_modules_install
  kbuild: handle module compression while running 'make
    modules_install'.
  modsign: lookup lines ending in .ko in .mod files

 Makefile                 | 15 +++++++++++++++
 init/Kconfig             | 43 +++++++++++++++++++++++++++++++++++++++++++
 scripts/Makefile.modinst |  7 ++++++-
 scripts/Makefile.modsign |  2 +-
 4 files changed, 65 insertions(+), 2 deletions(-)

-- 
2.0.4

[PATCH 1/3] modinst: wrap long lines in order to enhance cmd_modules_install

[Bertrand Jacquin]

Note: shouldn't we use 'install -D $(2)/$@ $@' instead of mkdir
and cp ?

Reviewed-by: Willy Tarreau <w@1wt.eu>
Signed-off-by: Bertrand Jacquin <beber@meleeweb.net>
---
 scripts/Makefile.modinst | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index 95ec7b3..aa911b5 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -18,7 +18,11 @@ __modinst: $(modules)
 
 # Don't stop modules_install if we can't sign external modules.
 quiet_cmd_modules_install = INSTALL $@
-      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) $(2)/$(notdir $@) ; $(mod_sign_cmd) $(2)/$(notdir $@) $(patsubst %,|| true,$(KBUILD_EXTMOD))
+      cmd_modules_install = \
+    mkdir -p $(2) ; \
+    cp $@ $(2) ; \
+    $(mod_strip_cmd) $(2)/$(notdir $@) ; \
+    $(mod_sign_cmd) $(2)/$(notdir $@) $(patsubst %,|| true,$(KBUILD_EXTMOD))
 
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
-- 
2.0.4

[PATCH 2/3] kbuild: handle module compression while running 'make modules_install'.

[Bertrand Jacquin]

Since module-init-tools (gzip) and kmod (gzip and xz) support compressed
modules, it could be useful to include a support for compressing modules
right after having them installed. Doing this in kbuild instead of per
distro can permit to make this kind of usage more generic.

This patch add a Kconfig entry to "Enable loadable module support" menu
and let you choose to compress using gzip (default) or xz.

Both gzip and xz does not used any extra -[1-9] option since Andi Kleen
and Rusty Russell prove no gain is made using them. gzip is called with -n
argument to avoid storing original filename inside compressed file, that
way we can save some more bytes.

On a v3.16 kernel, 'make allmodconfig' generated 4680 modules for a
total of 378MB (no strip, no sign, no compress), the following table
shows observed disk space gain based on the allmodconfig .config :

       |           time                |
       +-------------+-----------------+
       | manual .ko  |       make      | size | percent
       | compression | modules_install |      | gain
       +-------------+-----------------+------+--------
  -    |             |     18.61s      | 378M |
  GZIP |   3m16s     |     3m37s       | 102M | 73.41%
  XZ   |   5m22s     |     5m39s       |  77M | 79.83%

The gain for restricted environnement seems to be interesting while
uncompress can be time consuming but happens only while loading a module,
that is generally done only once.

This is fully compatible with signed modules while the signed module is
compressed. module-init-tools or kmod handles decompression
and provide to other layer the uncompressed but signed payload.

Reviewed-by: Willy Tarreau <w@1wt.eu>
Signed-off-by: Bertrand Jacquin <beber@meleeweb.net>
---
 Makefile                 | 15 +++++++++++++++
 init/Kconfig             | 43 +++++++++++++++++++++++++++++++++++++++++++
 scripts/Makefile.modinst |  3 ++-
 3 files changed, 60 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index e432442..300ff99 100644
--- a/Makefile
+++ b/Makefile
@@ -842,6 +842,21 @@ mod_strip_cmd = true
 endif # INSTALL_MOD_STRIP
 export mod_strip_cmd
 
+# CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed
+# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP
+# or CONFIG_MODULE_COMPRESS_XZ.
+
+mod_compress_cmd = true
+ifdef CONFIG_MODULE_COMPRESS
+  ifdef CONFIG_MODULE_COMPRESS_GZIP
+    mod_compress_cmd = gzip -n
+  endif # CONFIG_MODULE_COMPRESS_GZIP
+  ifdef CONFIG_MODULE_COMPRESS_XZ
+    mod_compress_cmd = xz
+  endif # CONFIG_MODULE_COMPRESS_XZ
+endif # CONFIG_MODULE_COMPRESS
+export mod_compress_cmd
+
 # Select initial ramdisk compression format, default is gzip(1).
 # This shall be used by the dracut(8) tool while creating an initramfs image.
 #
diff --git a/init/Kconfig b/init/Kconfig
index e84c642..4980925 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1906,6 +1906,49 @@ config MODULE_SIG_HASH
 	default "sha384" if MODULE_SIG_SHA384
 	default "sha512" if MODULE_SIG_SHA512
 
+config MODULE_COMPRESS
+	bool "Compress modules on installation"
+	depends on MODULES
+	help
+	  This option compresses the kernel modules when 'make
+	  modules_install' is run.
+
+	  The modules will be compressed either using gzip or xz depend on the
+	  choice made in "Compression algorithm".
+
+	  module-init-tools has support for gzip format while kmod handle gzip
+	  and xz compressed modules.
+
+	  When a kernel module is installed from outside of the main kernel
+	  source and uses the Kbuild system for installing modules then that
+	  kernel module will also be compressed when it is installed.
+
+	  This option provides little benefit when the modules are to be used inside
+	  an initrd or initramfs, it generally is more efficient to compress the whole
+	  initrd or initramfs instead.
+
+	  This is fully compatible with signed modules while the signed module is
+	  compressed. module-init-tools or kmod handles decompression and provide to
+	  other layer the uncompressed but signed payload.
+
+choice
+	prompt "Compression algorithm"
+	depends on MODULE_COMPRESS
+	default MODULE_COMPRESS_GZIP
+	help
+	  This determines which sort of compression will be used during
+	  'make modules_install'.
+
+	  GZIP (default) and XZ are supported.
+
+config MODULE_COMPRESS_GZIP
+	bool "GZIP"
+
+config MODULE_COMPRESS_XZ
+	bool "XZ"
+
+endchoice
+
 endif # MODULES
 
 config INIT_ALL_POSSIBLE
diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index aa911b5..e48a4e9 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -22,7 +22,8 @@ quiet_cmd_modules_install = INSTALL $@
     mkdir -p $(2) ; \
     cp $@ $(2) ; \
     $(mod_strip_cmd) $(2)/$(notdir $@) ; \
-    $(mod_sign_cmd) $(2)/$(notdir $@) $(patsubst %,|| true,$(KBUILD_EXTMOD))
+    $(mod_sign_cmd) $(2)/$(notdir $@) $(patsubst %,|| true,$(KBUILD_EXTMOD)) ; \
+    $(mod_compress_cmd) $(2)/$(notdir $@)
 
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
-- 
2.0.4

[PATCH 3/3] modsign: lookup lines ending in .ko in .mod files

[Bertrand Jacquin]

This does the same as commit ef591a5 (scripts/Makefile.modpost: error
in finding modules from .mod files), but for scripts/Makefile.modsign

Maybe we should also apply to Makefile.modsign and Makefile.modinst
the change applied to Makefile.modpost by commit ea4054a (modpost:
handle huge numbers of modules) ?

Reviewed-by: Willy Tarreau <w@1wt.eu>
Signed-off-by: Bertrand Jacquin <beber@meleeweb.net>
---
 scripts/Makefile.modsign | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/Makefile.modsign b/scripts/Makefile.modsign
index abfda62..b6ac708 100644
--- a/scripts/Makefile.modsign
+++ b/scripts/Makefile.modsign
@@ -7,7 +7,7 @@ __modsign:
 
 include scripts/Kbuild.include
 
-__modules := $(sort $(shell grep -h '\.ko' /dev/null $(wildcard $(MODVERDIR)/*.mod)))
+__modules := $(sort $(shell grep -h '\.ko$$' /dev/null $(wildcard $(MODVERDIR)/*.mod)))
 modules := $(patsubst %.o,%.ko,$(wildcard $(__modules:.ko=.o)))
 
 PHONY += $(modules)
-- 
2.0.4

Re: [PATCH 1/3] modinst: wrap long lines in order to enhance cmd_modules_install

[Jim Davis]

On Tue, Aug 19, 2014 at 11:57 AM, Bertrand Jacquin <beber@meleeweb.net> wrote:
> Note: shouldn't we use 'install -D $(2)/$@ $@' instead of mkdir
> and cp ?

Will an install with -D in that sense always be available?  That's not
a flag in the original (BSD) program, and I did find one SPARC system
-- admittedly an ancient one -- where the install program had a -D
option but it did something else.  install isn't listed as a required
program in Documentation/Changes, for that matter.

Re: [PATCH 3/3] modsign: lookup lines ending in .ko in .mod files

[Rusty Russell]

Bertrand Jacquin <beber@meleeweb.net> writes:
> This does the same as commit ef591a5 (scripts/Makefile.modpost: error
> in finding modules from .mod files), but for scripts/Makefile.modsign
>
> Maybe we should also apply to Makefile.modsign and Makefile.modinst
> the change applied to Makefile.modpost by commit ea4054a (modpost:
> handle huge numbers of modules) ?

Probably, yes.

Applied!
Rusty.

>
> Reviewed-by: Willy Tarreau <w@1wt.eu>
> Signed-off-by: Bertrand Jacquin <beber@meleeweb.net>
> ---
>  scripts/Makefile.modsign | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/Makefile.modsign b/scripts/Makefile.modsign
> index abfda62..b6ac708 100644
> --- a/scripts/Makefile.modsign
> +++ b/scripts/Makefile.modsign
> @@ -7,7 +7,7 @@ __modsign:
>  
>  include scripts/Kbuild.include
>  
> -__modules := $(sort $(shell grep -h '\.ko' /dev/null $(wildcard $(MODVERDIR)/*.mod)))
> +__modules := $(sort $(shell grep -h '\.ko$$' /dev/null $(wildcard $(MODVERDIR)/*.mod)))
>  modules := $(patsubst %.o,%.ko,$(wildcard $(__modules:.ko=.o)))
>  
>  PHONY += $(modules)
> -- 
> 2.0.4

Re: [PATCH 1/3] modinst: wrap long lines in order to enhance cmd_modules_install

[Rusty Russell]

Jim Davis <jim.epost@gmail.com> writes:
> On Tue, Aug 19, 2014 at 11:57 AM, Bertrand Jacquin <beber@meleeweb.net> wrote:
>> Note: shouldn't we use 'install -D $(2)/$@ $@' instead of mkdir
>> and cp ?
>
> Will an install with -D in that sense always be available?  That's not
> a flag in the original (BSD) program, and I did find one SPARC system
> -- admittedly an ancient one -- where the install program had a -D
> option but it did something else.  install isn't listed as a required
> program in Documentation/Changes, for that matter.

Yes, we don't count on install.  And ISTR that install -D would try to
change perms on directories in the path, which can fail if you don't own
them and aren't root...

Cheers,
Rusty.

Re: [PATCH 2/3] kbuild: handle module compression while running 'make modules_install'.

[Rusty Russell]

Bertrand Jacquin <beber@meleeweb.net> writes:

> Since module-init-tools (gzip) and kmod (gzip and xz) support compressed
> modules, it could be useful to include a support for compressing modules
> right after having them installed. Doing this in kbuild instead of per
> distro can permit to make this kind of usage more generic.
>
> This patch add a Kconfig entry to "Enable loadable module support" menu
> and let you choose to compress using gzip (default) or xz.
>
> Both gzip and xz does not used any extra -[1-9] option since Andi Kleen
> and Rusty Russell prove no gain is made using them. gzip is called with -n
> argument to avoid storing original filename inside compressed file, that
> way we can save some more bytes.
>
> On a v3.16 kernel, 'make allmodconfig' generated 4680 modules for a
> total of 378MB (no strip, no sign, no compress), the following table
> shows observed disk space gain based on the allmodconfig .config :
>
>        |           time                |
>        +-------------+-----------------+
>        | manual .ko  |       make      | size | percent
>        | compression | modules_install |      | gain
>        +-------------+-----------------+------+--------
>   -    |             |     18.61s      | 378M |
>   GZIP |   3m16s     |     3m37s       | 102M | 73.41%
>   XZ   |   5m22s     |     5m39s       |  77M | 79.83%
>
> The gain for restricted environnement seems to be interesting while
> uncompress can be time consuming but happens only while loading a module,
> that is generally done only once.
>
> This is fully compatible with signed modules while the signed module is
> compressed. module-init-tools or kmod handles decompression
> and provide to other layer the uncompressed but signed payload.
>
> Reviewed-by: Willy Tarreau <w@1wt.eu>
> Signed-off-by: Bertrand Jacquin <beber@meleeweb.net>

Thanks, applied these two as well.  They'll go in *next* merge window
(ie. 3.18).

Cheers.
Rusty.

Re: [PATCH 2/3] kbuild: handle module compression while running 'make modules_install'.

[Willy Tarreau]

Hi,

On Sun, Oct 12, 2014 at 11:50:59AM -0400, Andev wrote:
> Hello Bertrand,
> 
> Does this need any user space support? Cos currently on a debian
> testing box(powerpc) enabling this options causes a boot hang while
> mounting the root file system.

I suspect that your module-ini-tools/kmod are built without support for
compressed modules. One method to verify this is to run it against ldd :

$ ldd /sbin/modprobe
        linux-vdso.so.1 (0x00007fff918ed000)
        libkmod.so.2 => /lib64/libkmod.so.2 (0x00007ff9317f1000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff9315d5000)
        libc.so.6 => /lib64/libc.so.6 (0x00007ff931214000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007ff930ff2000)
        libz.so.1 => /usr/lib64/libz.so.1 (0x00007ff930ddd000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff931a06000)

As you can see, mine depends on liblzma and libz so it was clearly built
with support for both of them.

Regards,
Willy

Re: [PATCH 2/3] kbuild: handle module compression while running 'make modules_install'.

[Andev]

On Sun, Oct 12, 2014 at 12:40 PM, Willy Tarreau <w@1wt.eu> wrote:
> Hi,
>
> On Sun, Oct 12, 2014 at 11:50:59AM -0400, Andev wrote:
>> Hello Bertrand,
>>
>> Does this need any user space support? Cos currently on a debian
>> testing box(powerpc) enabling this options causes a boot hang while
>> mounting the root file system.
>
> I suspect that your module-ini-tools/kmod are built without support for
> compressed modules. One method to verify this is to run it against ldd :
>
> $ ldd /sbin/modprobe
>         linux-vdso.so.1 (0x00007fff918ed000)
>         libkmod.so.2 => /lib64/libkmod.so.2 (0x00007ff9317f1000)
>         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff9315d5000)
>         libc.so.6 => /lib64/libc.so.6 (0x00007ff931214000)
>         liblzma.so.5 => /lib64/liblzma.so.5 (0x00007ff930ff2000)
>         libz.so.1 => /usr/lib64/libz.so.1 (0x00007ff930ddd000)
>         /lib64/ld-linux-x86-64.so.2 (0x00007ff931a06000)
>
> As you can see, mine depends on liblzma and libz so it was clearly built
> with support for both of them.
>

Yes, that indeed seems to be the problem. On my machine:

$ ldd /sbin/modprobe
linux-vdso32.so.1 (0x00100000)
libc.so.6 => /lib/powerpc-linux-gnu/libc.so.6 (0x6fe17000)
/lib/ld.so.1 (0x20229000)

Thanks!
-- 
Andev