[PATCH 0/3] handle module compression at install

[PATCH 0/3] handle module compression at install

[Bertrand Jacquin]

Hi,

Here is patchset based on tag v3.17-rc1 that aims to handle kernel
modules compression while running make modules_install.

Details about this operation are present in patch 2/3.

Please Cc me, as I'm not subscribed to LKML.

Bertrand Jacquin (3):
  modinst: wrap long lines in order to enhance cmd_modules_install
  kbuild: handle module compression while running 'make
    modules_install'.
  modsign: lookup lines ending in .ko in .mod files

 Makefile                 | 15 +++++++++++++++
 init/Kconfig             | 43 +++++++++++++++++++++++++++++++++++++++++++
 scripts/Makefile.modinst |  7 ++++++-
 scripts/Makefile.modsign |  2 +-
 4 files changed, 65 insertions(+), 2 deletions(-)

-- 
2.0.4

[PATCH 1/3] modinst: wrap long lines in order to enhance cmd_modules_install

[Bertrand Jacquin]

Note: shouldn't we use 'install -D $(2)/$@ $@' instead of mkdir
and cp ?

Reviewed-by: Willy Tarreau <w@1wt.eu>
Signed-off-by: Bertrand Jacquin <beber@meleeweb.net>
---
 scripts/Makefile.modinst | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index 95ec7b3..aa911b5 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -18,7 +18,11 @@ __modinst: $(modules)
 
 # Don't stop modules_install if we can't sign external modules.
 quiet_cmd_modules_install = INSTALL $@
-      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) $(2)/$(notdir $@) ; $(mod_sign_cmd) $(2)/$(notdir $@) $(patsubst %,|| true,$(KBUILD_EXTMOD))
+      cmd_modules_install = \
+    mkdir -p $(2) ; \
+    cp $@ $(2) ; \
+    $(mod_strip_cmd) $(2)/$(notdir $@) ; \
+    $(mod_sign_cmd) $(2)/$(notdir $@) $(patsubst %,|| true,$(KBUILD_EXTMOD))
 
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
-- 
2.0.4

[PATCH 2/3] kbuild: handle module compression while running 'make modules_install'.

[Bertrand Jacquin]

Since module-init-tools (gzip) and kmod (gzip and xz) support compressed
modules, it could be useful to include a support for compressing modules
right after having them installed. Doing this in kbuild instead of per
distro can permit to make this kind of usage more generic.

This patch add a Kconfig entry to "Enable loadable module support" menu
and let you choose to compress using gzip (default) or xz.

Both gzip and xz used option -9 to get the maximum compression ratio. gzip is
called with -n argument to avoid storing original filename inside compressed
file, that way we can save some more bytes.

On a v3.16 kernel, 'make allmodconfig' generated 4680 modules for a
total of 378MB (no strip, no sign, no compress), the following table
shows observed disk space gain based on the allmodconfig .config :

       |           time                |
       +-------------+-----------------+
       | manual .ko  |       make      | size | percent
       | compression | modules_install |      | gain
       +-------------+-----------------+------+--------
  -    |             |     18.61s      | 378M |
  GZIP |   3m16s     |     3m37s       | 102M | 73.41%
  XZ   |   5m22s     |     5m39s       |  77M | 79.83%

The gain for restricted environnement seems to be interesting while
uncompress can be time consuming but happens only while loading a module,
that is generally done only once.

This is fully compatible with signed modules while the signed module is
compressed. module-init-tools or kmod handles decompression
and provide to other layer the uncompressed but signed payload.

Reviewed-by: Willy Tarreau <w@1wt.eu>
Signed-off-by: Bertrand Jacquin <beber@meleeweb.net>
---
 Makefile                 | 15 +++++++++++++++
 init/Kconfig             | 43 +++++++++++++++++++++++++++++++++++++++++++
 scripts/Makefile.modinst |  3 ++-
 3 files changed, 60 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index e432442..ba7873e 100644
--- a/Makefile
+++ b/Makefile
@@ -842,6 +842,21 @@ mod_strip_cmd = true
 endif # INSTALL_MOD_STRIP
 export mod_strip_cmd
 
+# CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed
+# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP
+# or CONFIG_MODULE_COMPRESS_XZ.
+
+mod_compress_cmd = true
+ifdef CONFIG_MODULE_COMPRESS
+  ifdef CONFIG_MODULE_COMPRESS_GZIP
+    mod_compress_cmd = gzip -n -9
+  endif # CONFIG_MODULE_COMPRESS_GZIP
+  ifdef CONFIG_MODULE_COMPRESS_XZ
+    mod_compress_cmd = xz -9
+  endif # CONFIG_MODULE_COMPRESS_XZ
+endif # CONFIG_MODULE_COMPRESS
+export mod_compress_cmd
+
 # Select initial ramdisk compression format, default is gzip(1).
 # This shall be used by the dracut(8) tool while creating an initramfs image.
 #
diff --git a/init/Kconfig b/init/Kconfig
index e84c642..4980925 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1906,6 +1906,49 @@ config MODULE_SIG_HASH
 	default "sha384" if MODULE_SIG_SHA384
 	default "sha512" if MODULE_SIG_SHA512
 
+config MODULE_COMPRESS
+	bool "Compress modules on installation"
+	depends on MODULES
+	help
+	  This option compresses the kernel modules when 'make
+	  modules_install' is run.
+
+	  The modules will be compressed either using gzip or xz depend on the
+	  choice made in "Compression algorithm".
+
+	  module-init-tools has support for gzip format while kmod handle gzip
+	  and xz compressed modules.
+
+	  When a kernel module is installed from outside of the main kernel
+	  source and uses the Kbuild system for installing modules then that
+	  kernel module will also be compressed when it is installed.
+
+	  This option provides little benefit when the modules are to be used inside
+	  an initrd or initramfs, it generally is more efficient to compress the whole
+	  initrd or initramfs instead.
+
+	  This is fully compatible with signed modules while the signed module is
+	  compressed. module-init-tools or kmod handles decompression and provide to
+	  other layer the uncompressed but signed payload.
+
+choice
+	prompt "Compression algorithm"
+	depends on MODULE_COMPRESS
+	default MODULE_COMPRESS_GZIP
+	help
+	  This determines which sort of compression will be used during
+	  'make modules_install'.
+
+	  GZIP (default) and XZ are supported.
+
+config MODULE_COMPRESS_GZIP
+	bool "GZIP"
+
+config MODULE_COMPRESS_XZ
+	bool "XZ"
+
+endchoice
+
 endif # MODULES
 
 config INIT_ALL_POSSIBLE
diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index aa911b5..e48a4e9 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -22,7 +22,8 @@ quiet_cmd_modules_install = INSTALL $@
     mkdir -p $(2) ; \
     cp $@ $(2) ; \
     $(mod_strip_cmd) $(2)/$(notdir $@) ; \
-    $(mod_sign_cmd) $(2)/$(notdir $@) $(patsubst %,|| true,$(KBUILD_EXTMOD))
+    $(mod_sign_cmd) $(2)/$(notdir $@) $(patsubst %,|| true,$(KBUILD_EXTMOD)) ; \
+    $(mod_compress_cmd) $(2)/$(notdir $@)
 
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
-- 
2.0.4

[PATCH 3/3] modsign: lookup lines ending in .ko in .mod files

[Bertrand Jacquin]

This does the same as commit ef591a5 (scripts/Makefile.modpost: error
in finding modules from .mod files), but for scripts/Makefile.modsign

Maybe we should also apply to Makefile.modsign and Makefile.modinst
the change applied to Makefile.modpost by commit ea4054a (modpost:
handle huge numbers of modules) ?

Reviewed-by: Willy Tarreau <w@1wt.eu>
Signed-off-by: Bertrand Jacquin <beber@meleeweb.net>
---
 scripts/Makefile.modsign | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/Makefile.modsign b/scripts/Makefile.modsign
index abfda62..b6ac708 100644
--- a/scripts/Makefile.modsign
+++ b/scripts/Makefile.modsign
@@ -7,7 +7,7 @@ __modsign:
 
 include scripts/Kbuild.include
 
-__modules := $(sort $(shell grep -h '\.ko' /dev/null $(wildcard $(MODVERDIR)/*.mod)))
+__modules := $(sort $(shell grep -h '\.ko$$' /dev/null $(wildcard $(MODVERDIR)/*.mod)))
 modules := $(patsubst %.o,%.ko,$(wildcard $(__modules:.ko=.o)))
 
 PHONY += $(modules)
-- 
2.0.4

Re: [PATCH 3/3] modsign: lookup lines ending in .ko in .mod files

[Rusty Russell]

Bertrand Jacquin <beber@meleeweb.net> writes:
> This does the same as commit ef591a5 (scripts/Makefile.modpost: error
> in finding modules from .mod files), but for scripts/Makefile.modsign
>
> Maybe we should also apply to Makefile.modsign and Makefile.modinst
> the change applied to Makefile.modpost by commit ea4054a (modpost:
> handle huge numbers of modules) ?

Good question.  David Woodhose cc'd...

Cheers,
Rusty.

> Reviewed-by: Willy Tarreau <w@1wt.eu>
> Signed-off-by: Bertrand Jacquin <beber@meleeweb.net>
> ---
>  scripts/Makefile.modsign | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/Makefile.modsign b/scripts/Makefile.modsign
> index abfda62..b6ac708 100644
> --- a/scripts/Makefile.modsign
> +++ b/scripts/Makefile.modsign
> @@ -7,7 +7,7 @@ __modsign:
>  
>  include scripts/Kbuild.include
>  
> -__modules := $(sort $(shell grep -h '\.ko' /dev/null $(wildcard $(MODVERDIR)/*.mod)))
> +__modules := $(sort $(shell grep -h '\.ko$$' /dev/null $(wildcard $(MODVERDIR)/*.mod)))
>  modules := $(patsubst %.o,%.ko,$(wildcard $(__modules:.ko=.o)))
>  
>  PHONY += $(modules)
> -- 
> 2.0.4

Re: [PATCH 2/3] kbuild: handle module compression while running 'make modules_install'.

[Andi Kleen]

Bertrand Jacquin <beber@meleeweb.net> writes:
>  
> +# CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed
> +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP
> +# or CONFIG_MODULE_COMPRESS_XZ.
> +
> +mod_compress_cmd = true
> +ifdef CONFIG_MODULE_COMPRESS
> +  ifdef CONFIG_MODULE_COMPRESS_GZIP
> +    mod_compress_cmd = gzip -n -9

Please don't use gzip -9. The gain is very small, but it can make
the time to compress much longer. Just use plain gzip.

I suspect it doesn't make sense either for xz.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only

Re: [PATCH 2/3] kbuild: handle module compression while running 'make modules_install'.

[Bertrand Jacquin]

Hi,

D'ar meurzh 19 a viz Eost 2014 e 15 eur 17, « Andi Kleen » he deus skrivet :
> Bertrand Jacquin <beber@meleeweb.net> writes:
> >  
> > +# CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed
> > +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP
> > +# or CONFIG_MODULE_COMPRESS_XZ.
> > +
> > +mod_compress_cmd = true
> > +ifdef CONFIG_MODULE_COMPRESS
> > +  ifdef CONFIG_MODULE_COMPRESS_GZIP
> > +    mod_compress_cmd = gzip -n -9
> 
> Please don't use gzip -9. The gain is very small, but it can make
> the time to compress much longer. Just use plain gzip.
> 
> I suspect it doesn't make sense either for xz.

I understand your concern about this, but does is make sense to not use
-9 when then following files use gzip or other compression tools with -9 :

  arch/blackfin/Makefile:14:GZFLAGS          := -9

  arch/powerpc/boot/wrapper:297:        gzip -n -f -9 "$vmz.$$"
  arch/powerpc/boot/wrapper:422:    gzip -n -f -9 "$ofile"
  arch/powerpc/boot/wrapper:469:    gzip -n --force -9 --stdout "$ofile.bin" > "$odir/otheros.bld"

  arch/arm/Makefile:23:GZFLAGS            :=-9
  arch/arm64/Makefile:16:GZFLAGS          :=-9

  arch/m68k/Makefile:138: gzip -9c vmlinux.tmp >vmlinux.gz
  arch/m68k/Makefile:141: gzip -9c vmlinux >vmlinux.gz

  arch/mips/lasat/image/Makefile:46:      gzip -cf -9 $< > $@

  arch/parisc/Makefile:121:       @gzip -cf -9 $< > $@

  scripts/package/mkspec:115:echo 'bzip2 -9 vmlinux'
  scripts/package/builddeb:20:    gzip -9 "$pdir/usr/share/doc/$pname/changelog.Debian"
  scripts/package/Makefile:127:   -f -9 $(perf-tar).tar)

  scripts/gen_initramfs_list.sh:229:compr="gzip -n -9 -f"
  scripts/gen_initramfs_list.sh:245:                && compr="gzip -n -9 -f"
  scripts/gen_initramfs_list.sh:248:                && compr="bzip2 -9 -f"
  scripts/gen_initramfs_list.sh:251:                && compr="lzma -9 -f"
  scripts/gen_initramfs_list.sh:257:                && compr="lzop -9 -f"
  scripts/gen_initramfs_list.sh:260:                && compr="lz4 -l -9 -f"

  scripts/Makefile.lib:241:cmd_gzip = (cat $(filter-out FORCE,$^) | gzip -n -f -9 > $@) || \
  scripts/Makefile.lib:311:       bzip2 -9 && $(call size_append, $(filter-out FORCE,$^))) > $@ || \
  scripts/Makefile.lib:319:       lzma -9 && $(call size_append, $(filter-out FORCE,$^))) > $@ || \
  scripts/Makefile.lib:324:       lzop -9 && $(call size_append, $(filter-out FORCE,$^))) > $@ || \

-- 
Beber

Re: [PATCH 2/3] kbuild: handle module compression while running 'make modules_install'.

[Rusty Russell]

Andi Kleen <andi@firstfloor.org> writes:
> Bertrand Jacquin <beber@meleeweb.net> writes:
>>  
>> +# CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed
>> +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP
>> +# or CONFIG_MODULE_COMPRESS_XZ.
>> +
>> +mod_compress_cmd = true
>> +ifdef CONFIG_MODULE_COMPRESS
>> +  ifdef CONFIG_MODULE_COMPRESS_GZIP
>> +    mod_compress_cmd = gzip -n -9
>
> Please don't use gzip -9. The gain is very small, but it can make
> the time to compress much longer. Just use plain gzip.

Interesting:

$ time find * -name '*.ko' | xargs -n 1 gzip
real	0m9.404s
user	0m7.283s
sys	0m1.667s
$ du -c `find * -name '*.ko.gz'` | tail -n1
56412	total

time find * -name '*.ko' | xargs -n 1 gzip -9
real	1m1.968s
user	0m58.850s
sys	0m1.786s
$ du -c `find * -name '*.ko.gz'` | tail -n1
55764	total

$ time find * -name '*.ko' | xargs -n 1 xz
real	1m23.396s
user	1m13.893s
sys	0m9.134s
$ du -c `find * -name '*.ko.xz` | tail -n1
46868	total

$ time find * -name '*.ko' | xargs -n 1 xz -9
real	1m47.202s
user	1m25.962s
sys	0m21.298s
$ du -c `find * -name '*.ko.xz` | tail -n1
46868	total

So, Andi is right.  Please skip -9 for both options: it makes little
(.gz) or no (.xz) difference and it slows things down.

Cheers,
Rusty.

Re: [PATCH 2/3] kbuild: handle module compression while running 'make modules_install'.

[Bertrand Jacquin]

Hi,

D'ar meurzh 19 a viz Eost 2014 e 19 eur 55, « Rusty Russell » he deus skrivet :
> Andi Kleen <andi@firstfloor.org> writes:
> > Bertrand Jacquin <beber@meleeweb.net> writes:
> >>  
> >> +# CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed
> >> +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP
> >> +# or CONFIG_MODULE_COMPRESS_XZ.
> >> +
> >> +mod_compress_cmd = true
> >> +ifdef CONFIG_MODULE_COMPRESS
> >> +  ifdef CONFIG_MODULE_COMPRESS_GZIP
> >> +    mod_compress_cmd = gzip -n -9
> >
> > Please don't use gzip -9. The gain is very small, but it can make
> > the time to compress much longer. Just use plain gzip.
> 
> Interesting:
> 
> $ time find * -name '*.ko' | xargs -n 1 gzip
> real	0m9.404s
> user	0m7.283s
> sys	0m1.667s
> $ du -c `find * -name '*.ko.gz'` | tail -n1
> 56412	total
> 
> time find * -name '*.ko' | xargs -n 1 gzip -9
> real	1m1.968s
> user	0m58.850s
> sys	0m1.786s
> $ du -c `find * -name '*.ko.gz'` | tail -n1
> 55764	total
> 
> $ time find * -name '*.ko' | xargs -n 1 xz
> real	1m23.396s
> user	1m13.893s
> sys	0m9.134s
> $ du -c `find * -name '*.ko.xz` | tail -n1
> 46868	total
> 
> $ time find * -name '*.ko' | xargs -n 1 xz -9
> real	1m47.202s
> user	1m25.962s
> sys	0m21.298s
> $ du -c `find * -name '*.ko.xz` | tail -n1
> 46868	total
> 
> So, Andi is right.  Please skip -9 for both options: it makes little
> (.gz) or no (.xz) difference and it slows things down.

Thank you Andi and Rusty for the feedback. I'm current updating patches
to drop -9 usage.

-- 
Beber

Re: [PATCH 2/3] kbuild: handle module compression while running 'make modules_install'.

[Andi Kleen]

> I understand your concern about this, but does is make sense to not use
> -9 when then following files use gzip or other compression tools with -9 :

Just because someone else makes mistakes doesn't mean that you should
make them too.

-Andi

Re: [PATCH 2/3] kbuild: handle module compression while running 'make modules_install'.

[Rusty Russell]

Andi Kleen <andi@firstfloor.org> writes:
>> I understand your concern about this, but does is make sense to not use
>> -9 when then following files use gzip or other compression tools with -9 :
>
> Just because someone else makes mistakes doesn't mean that you should
> make them too.

To be fair, you didn't provide hard numbers: *I* was surprised by the
results when I tested it.

Cheers,
Rusty.

Re: [PATCH 3/3] modsign: lookup lines ending in .ko in .mod files

[Rusty Russell]

Bertrand Jacquin <beber@meleeweb.net> writes:
> This does the same as commit ef591a5 (scripts/Makefile.modpost: error
> in finding modules from .mod files), but for scripts/Makefile.modsign
>
> Maybe we should also apply to Makefile.modsign and Makefile.modinst
> the change applied to Makefile.modpost by commit ea4054a (modpost:
> handle huge numbers of modules) ?

Probably, yes.

Applied!
Rusty.

>
> Reviewed-by: Willy Tarreau <w@1wt.eu>
> Signed-off-by: Bertrand Jacquin <beber@meleeweb.net>
> ---
>  scripts/Makefile.modsign | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/Makefile.modsign b/scripts/Makefile.modsign
> index abfda62..b6ac708 100644
> --- a/scripts/Makefile.modsign
> +++ b/scripts/Makefile.modsign
> @@ -7,7 +7,7 @@ __modsign:
>  
>  include scripts/Kbuild.include
>  
> -__modules := $(sort $(shell grep -h '\.ko' /dev/null $(wildcard $(MODVERDIR)/*.mod)))
> +__modules := $(sort $(shell grep -h '\.ko$$' /dev/null $(wildcard $(MODVERDIR)/*.mod)))
>  modules := $(patsubst %.o,%.ko,$(wildcard $(__modules:.ko=.o)))
>  
>  PHONY += $(modules)
> -- 
> 2.0.4

[PATCH 3/3] modsign: lookup lines ending in .ko in .mod files

[Bertrand Jacquin]

This does the same as commit ef591a5 (scripts/Makefile.modpost: error
in finding modules from .mod files), but for scripts/Makefile.modsign

Maybe we should also apply to Makefile.modsign and Makefile.modinst
the change applied to Makefile.modpost by commit ea4054a (modpost:
handle huge numbers of modules) ?

Reviewed-by: Willy Tarreau <w@1wt.eu>
Signed-off-by: Bertrand Jacquin <beber@meleeweb.net>
---
 scripts/Makefile.modsign | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/Makefile.modsign b/scripts/Makefile.modsign
index abfda62..b6ac708 100644
--- a/scripts/Makefile.modsign
+++ b/scripts/Makefile.modsign
@@ -7,7 +7,7 @@ __modsign:
 
 include scripts/Kbuild.include
 
-__modules := $(sort $(shell grep -h '\.ko' /dev/null $(wildcard $(MODVERDIR)/*.mod)))
+__modules := $(sort $(shell grep -h '\.ko$$' /dev/null $(wildcard $(MODVERDIR)/*.mod)))
 modules := $(patsubst %.o,%.ko,$(wildcard $(__modules:.ko=.o)))
 
 PHONY += $(modules)
-- 
2.0.4