* [PATCHv2 0/3] handle module compression at install
@ 2014-08-19 18:57 Bertrand Jacquin
2014-08-19 18:57 ` [PATCH 1/3] modinst: wrap long lines in order to enhance cmd_modules_install Bertrand Jacquin
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: Bertrand Jacquin @ 2014-08-19 18:57 UTC (permalink / raw)
To: Rusty Russell, linux-kbuild; +Cc: Bertrand Jacquin, linux-kernel, Willy Tarreau
Hi,
Here is patchset based on commit 480cadc (v3.17-rc1-22-g480cadc) that
aims to handle kernel modules compression while running make modules_install.
Please Cc me, as I'm not subscribed to LKML.
Changes since v1:
- Drop -9 argument to gzip and xz since Andi Kleen and Rusty Russell
prove no gain is made using them.
Bertrand Jacquin (3):
modinst: wrap long lines in order to enhance cmd_modules_install
kbuild: handle module compression while running 'make
modules_install'.
modsign: lookup lines ending in .ko in .mod files
Makefile | 15 +++++++++++++++
init/Kconfig | 43 +++++++++++++++++++++++++++++++++++++++++++
scripts/Makefile.modinst | 7 ++++++-
scripts/Makefile.modsign | 2 +-
4 files changed, 65 insertions(+), 2 deletions(-)
--
2.0.4
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH 1/3] modinst: wrap long lines in order to enhance cmd_modules_install 2014-08-19 18:57 [PATCHv2 0/3] handle module compression at install Bertrand Jacquin @ 2014-08-19 18:57 ` Bertrand Jacquin 2014-08-19 22:02 ` Jim Davis 2014-08-19 18:57 ` [PATCH 2/3] kbuild: handle module compression while running 'make modules_install' Bertrand Jacquin 2014-08-19 18:57 ` [PATCH 3/3] modsign: lookup lines ending in .ko in .mod files Bertrand Jacquin 2 siblings, 1 reply; 10+ messages in thread From: Bertrand Jacquin @ 2014-08-19 18:57 UTC (permalink / raw) To: Rusty Russell, linux-kbuild; +Cc: Bertrand Jacquin, linux-kernel, Willy Tarreau Note: shouldn't we use 'install -D $(2)/$@ $@' instead of mkdir and cp ? Reviewed-by: Willy Tarreau <w@1wt.eu> Signed-off-by: Bertrand Jacquin <beber@meleeweb.net> --- scripts/Makefile.modinst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst index 95ec7b3..aa911b5 100644 --- a/scripts/Makefile.modinst +++ b/scripts/Makefile.modinst @@ -18,7 +18,11 @@ __modinst: $(modules) # Don't stop modules_install if we can't sign external modules. quiet_cmd_modules_install = INSTALL $@ - cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) $(2)/$(notdir $@) ; $(mod_sign_cmd) $(2)/$(notdir $@) $(patsubst %,|| true,$(KBUILD_EXTMOD)) + cmd_modules_install = \ + mkdir -p $(2) ; \ + cp $@ $(2) ; \ + $(mod_strip_cmd) $(2)/$(notdir $@) ; \ + $(mod_sign_cmd) $(2)/$(notdir $@) $(patsubst %,|| true,$(KBUILD_EXTMOD)) # Modules built outside the kernel source tree go into extra by default INSTALL_MOD_DIR ?= extra -- 2.0.4 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH 1/3] modinst: wrap long lines in order to enhance cmd_modules_install 2014-08-19 18:57 ` [PATCH 1/3] modinst: wrap long lines in order to enhance cmd_modules_install Bertrand Jacquin @ 2014-08-19 22:02 ` Jim Davis 2014-08-20 16:08 ` Rusty Russell 0 siblings, 1 reply; 10+ messages in thread From: Jim Davis @ 2014-08-19 22:02 UTC (permalink / raw) To: Bertrand Jacquin; +Cc: Rusty Russell, linux-kbuild, linux-kernel, Willy Tarreau On Tue, Aug 19, 2014 at 11:57 AM, Bertrand Jacquin <beber@meleeweb.net> wrote: > Note: shouldn't we use 'install -D $(2)/$@ $@' instead of mkdir > and cp ? Will an install with -D in that sense always be available? That's not a flag in the original (BSD) program, and I did find one SPARC system -- admittedly an ancient one -- where the install program had a -D option but it did something else. install isn't listed as a required program in Documentation/Changes, for that matter. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 1/3] modinst: wrap long lines in order to enhance cmd_modules_install 2014-08-19 22:02 ` Jim Davis @ 2014-08-20 16:08 ` Rusty Russell 0 siblings, 0 replies; 10+ messages in thread From: Rusty Russell @ 2014-08-20 16:08 UTC (permalink / raw) To: Jim Davis, Bertrand Jacquin; +Cc: linux-kbuild, linux-kernel, Willy Tarreau Jim Davis <jim.epost@gmail.com> writes: > On Tue, Aug 19, 2014 at 11:57 AM, Bertrand Jacquin <beber@meleeweb.net> wrote: >> Note: shouldn't we use 'install -D $(2)/$@ $@' instead of mkdir >> and cp ? > > Will an install with -D in that sense always be available? That's not > a flag in the original (BSD) program, and I did find one SPARC system > -- admittedly an ancient one -- where the install program had a -D > option but it did something else. install isn't listed as a required > program in Documentation/Changes, for that matter. Yes, we don't count on install. And ISTR that install -D would try to change perms on directories in the path, which can fail if you don't own them and aren't root... Cheers, Rusty. ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 2/3] kbuild: handle module compression while running 'make modules_install'. 2014-08-19 18:57 [PATCHv2 0/3] handle module compression at install Bertrand Jacquin 2014-08-19 18:57 ` [PATCH 1/3] modinst: wrap long lines in order to enhance cmd_modules_install Bertrand Jacquin @ 2014-08-19 18:57 ` Bertrand Jacquin 2014-08-20 16:09 ` Rusty Russell [not found] ` <CAJUuVQ7eEn8r2mAQKmydTDvDgSXYKjUH15VPEQfOVPJxNtU9ag@mail.gmail.com> 2014-08-19 18:57 ` [PATCH 3/3] modsign: lookup lines ending in .ko in .mod files Bertrand Jacquin 2 siblings, 2 replies; 10+ messages in thread From: Bertrand Jacquin @ 2014-08-19 18:57 UTC (permalink / raw) To: Rusty Russell, linux-kbuild; +Cc: Bertrand Jacquin, linux-kernel, Willy Tarreau Since module-init-tools (gzip) and kmod (gzip and xz) support compressed modules, it could be useful to include a support for compressing modules right after having them installed. Doing this in kbuild instead of per distro can permit to make this kind of usage more generic. This patch add a Kconfig entry to "Enable loadable module support" menu and let you choose to compress using gzip (default) or xz. Both gzip and xz does not used any extra -[1-9] option since Andi Kleen and Rusty Russell prove no gain is made using them. gzip is called with -n argument to avoid storing original filename inside compressed file, that way we can save some more bytes. On a v3.16 kernel, 'make allmodconfig' generated 4680 modules for a total of 378MB (no strip, no sign, no compress), the following table shows observed disk space gain based on the allmodconfig .config : | time | +-------------+-----------------+ | manual .ko | make | size | percent | compression | modules_install | | gain +-------------+-----------------+------+-------- - | | 18.61s | 378M | GZIP | 3m16s | 3m37s | 102M | 73.41% XZ | 5m22s | 5m39s | 77M | 79.83% The gain for restricted environnement seems to be interesting while uncompress can be time consuming but happens only while loading a module, that is generally done only once. This is fully compatible with signed modules while the signed module is compressed. module-init-tools or kmod handles decompression and provide to other layer the uncompressed but signed payload. Reviewed-by: Willy Tarreau <w@1wt.eu> Signed-off-by: Bertrand Jacquin <beber@meleeweb.net> --- Makefile | 15 +++++++++++++++ init/Kconfig | 43 +++++++++++++++++++++++++++++++++++++++++++ scripts/Makefile.modinst | 3 ++- 3 files changed, 60 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index e432442..300ff99 100644 --- a/Makefile +++ b/Makefile @@ -842,6 +842,21 @@ mod_strip_cmd = true endif # INSTALL_MOD_STRIP export mod_strip_cmd +# CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP +# or CONFIG_MODULE_COMPRESS_XZ. + +mod_compress_cmd = true +ifdef CONFIG_MODULE_COMPRESS + ifdef CONFIG_MODULE_COMPRESS_GZIP + mod_compress_cmd = gzip -n + endif # CONFIG_MODULE_COMPRESS_GZIP + ifdef CONFIG_MODULE_COMPRESS_XZ + mod_compress_cmd = xz + endif # CONFIG_MODULE_COMPRESS_XZ +endif # CONFIG_MODULE_COMPRESS +export mod_compress_cmd + # Select initial ramdisk compression format, default is gzip(1). # This shall be used by the dracut(8) tool while creating an initramfs image. # diff --git a/init/Kconfig b/init/Kconfig index e84c642..4980925 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -1906,6 +1906,49 @@ config MODULE_SIG_HASH default "sha384" if MODULE_SIG_SHA384 default "sha512" if MODULE_SIG_SHA512 +config MODULE_COMPRESS + bool "Compress modules on installation" + depends on MODULES + help + This option compresses the kernel modules when 'make + modules_install' is run. + + The modules will be compressed either using gzip or xz depend on the + choice made in "Compression algorithm". + + module-init-tools has support for gzip format while kmod handle gzip + and xz compressed modules. + + When a kernel module is installed from outside of the main kernel + source and uses the Kbuild system for installing modules then that + kernel module will also be compressed when it is installed. + + This option provides little benefit when the modules are to be used inside + an initrd or initramfs, it generally is more efficient to compress the whole + initrd or initramfs instead. + + This is fully compatible with signed modules while the signed module is + compressed. module-init-tools or kmod handles decompression and provide to + other layer the uncompressed but signed payload. + +choice + prompt "Compression algorithm" + depends on MODULE_COMPRESS + default MODULE_COMPRESS_GZIP + help + This determines which sort of compression will be used during + 'make modules_install'. + + GZIP (default) and XZ are supported. + +config MODULE_COMPRESS_GZIP + bool "GZIP" + +config MODULE_COMPRESS_XZ + bool "XZ" + +endchoice + endif # MODULES config INIT_ALL_POSSIBLE diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst index aa911b5..e48a4e9 100644 --- a/scripts/Makefile.modinst +++ b/scripts/Makefile.modinst @@ -22,7 +22,8 @@ quiet_cmd_modules_install = INSTALL $@ mkdir -p $(2) ; \ cp $@ $(2) ; \ $(mod_strip_cmd) $(2)/$(notdir $@) ; \ - $(mod_sign_cmd) $(2)/$(notdir $@) $(patsubst %,|| true,$(KBUILD_EXTMOD)) + $(mod_sign_cmd) $(2)/$(notdir $@) $(patsubst %,|| true,$(KBUILD_EXTMOD)) ; \ + $(mod_compress_cmd) $(2)/$(notdir $@) # Modules built outside the kernel source tree go into extra by default INSTALL_MOD_DIR ?= extra -- 2.0.4 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH 2/3] kbuild: handle module compression while running 'make modules_install'. 2014-08-19 18:57 ` [PATCH 2/3] kbuild: handle module compression while running 'make modules_install' Bertrand Jacquin @ 2014-08-20 16:09 ` Rusty Russell [not found] ` <CAJUuVQ7eEn8r2mAQKmydTDvDgSXYKjUH15VPEQfOVPJxNtU9ag@mail.gmail.com> 1 sibling, 0 replies; 10+ messages in thread From: Rusty Russell @ 2014-08-20 16:09 UTC (permalink / raw) To: Bertrand Jacquin, linux-kbuild; +Cc: linux-kernel, Willy Tarreau Bertrand Jacquin <beber@meleeweb.net> writes: > Since module-init-tools (gzip) and kmod (gzip and xz) support compressed > modules, it could be useful to include a support for compressing modules > right after having them installed. Doing this in kbuild instead of per > distro can permit to make this kind of usage more generic. > > This patch add a Kconfig entry to "Enable loadable module support" menu > and let you choose to compress using gzip (default) or xz. > > Both gzip and xz does not used any extra -[1-9] option since Andi Kleen > and Rusty Russell prove no gain is made using them. gzip is called with -n > argument to avoid storing original filename inside compressed file, that > way we can save some more bytes. > > On a v3.16 kernel, 'make allmodconfig' generated 4680 modules for a > total of 378MB (no strip, no sign, no compress), the following table > shows observed disk space gain based on the allmodconfig .config : > > | time | > +-------------+-----------------+ > | manual .ko | make | size | percent > | compression | modules_install | | gain > +-------------+-----------------+------+-------- > - | | 18.61s | 378M | > GZIP | 3m16s | 3m37s | 102M | 73.41% > XZ | 5m22s | 5m39s | 77M | 79.83% > > The gain for restricted environnement seems to be interesting while > uncompress can be time consuming but happens only while loading a module, > that is generally done only once. > > This is fully compatible with signed modules while the signed module is > compressed. module-init-tools or kmod handles decompression > and provide to other layer the uncompressed but signed payload. > > Reviewed-by: Willy Tarreau <w@1wt.eu> > Signed-off-by: Bertrand Jacquin <beber@meleeweb.net> Thanks, applied these two as well. They'll go in *next* merge window (ie. 3.18). Cheers. Rusty. ^ permalink raw reply [flat|nested] 10+ messages in thread
[parent not found: <CAJUuVQ7eEn8r2mAQKmydTDvDgSXYKjUH15VPEQfOVPJxNtU9ag@mail.gmail.com>]
* Re: [PATCH 2/3] kbuild: handle module compression while running 'make modules_install'. [not found] ` <CAJUuVQ7eEn8r2mAQKmydTDvDgSXYKjUH15VPEQfOVPJxNtU9ag@mail.gmail.com> @ 2014-10-12 16:40 ` Willy Tarreau 2014-10-12 17:06 ` Andev 0 siblings, 1 reply; 10+ messages in thread From: Willy Tarreau @ 2014-10-12 16:40 UTC (permalink / raw) To: Andev; +Cc: Bertrand Jacquin, Rusty Russell, linux-kbuild, LKML Hi, On Sun, Oct 12, 2014 at 11:50:59AM -0400, Andev wrote: > Hello Bertrand, > > Does this need any user space support? Cos currently on a debian > testing box(powerpc) enabling this options causes a boot hang while > mounting the root file system. I suspect that your module-ini-tools/kmod are built without support for compressed modules. One method to verify this is to run it against ldd : $ ldd /sbin/modprobe linux-vdso.so.1 (0x00007fff918ed000) libkmod.so.2 => /lib64/libkmod.so.2 (0x00007ff9317f1000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff9315d5000) libc.so.6 => /lib64/libc.so.6 (0x00007ff931214000) liblzma.so.5 => /lib64/liblzma.so.5 (0x00007ff930ff2000) libz.so.1 => /usr/lib64/libz.so.1 (0x00007ff930ddd000) /lib64/ld-linux-x86-64.so.2 (0x00007ff931a06000) As you can see, mine depends on liblzma and libz so it was clearly built with support for both of them. Regards, Willy ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/3] kbuild: handle module compression while running 'make modules_install'. 2014-10-12 16:40 ` Willy Tarreau @ 2014-10-12 17:06 ` Andev 0 siblings, 0 replies; 10+ messages in thread From: Andev @ 2014-10-12 17:06 UTC (permalink / raw) To: Willy Tarreau; +Cc: Bertrand Jacquin, Rusty Russell, linux-kbuild, LKML On Sun, Oct 12, 2014 at 12:40 PM, Willy Tarreau <w@1wt.eu> wrote: > Hi, > > On Sun, Oct 12, 2014 at 11:50:59AM -0400, Andev wrote: >> Hello Bertrand, >> >> Does this need any user space support? Cos currently on a debian >> testing box(powerpc) enabling this options causes a boot hang while >> mounting the root file system. > > I suspect that your module-ini-tools/kmod are built without support for > compressed modules. One method to verify this is to run it against ldd : > > $ ldd /sbin/modprobe > linux-vdso.so.1 (0x00007fff918ed000) > libkmod.so.2 => /lib64/libkmod.so.2 (0x00007ff9317f1000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff9315d5000) > libc.so.6 => /lib64/libc.so.6 (0x00007ff931214000) > liblzma.so.5 => /lib64/liblzma.so.5 (0x00007ff930ff2000) > libz.so.1 => /usr/lib64/libz.so.1 (0x00007ff930ddd000) > /lib64/ld-linux-x86-64.so.2 (0x00007ff931a06000) > > As you can see, mine depends on liblzma and libz so it was clearly built > with support for both of them. > Yes, that indeed seems to be the problem. On my machine: $ ldd /sbin/modprobe linux-vdso32.so.1 (0x00100000) libc.so.6 => /lib/powerpc-linux-gnu/libc.so.6 (0x6fe17000) /lib/ld.so.1 (0x20229000) Thanks! -- Andev ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 3/3] modsign: lookup lines ending in .ko in .mod files 2014-08-19 18:57 [PATCHv2 0/3] handle module compression at install Bertrand Jacquin 2014-08-19 18:57 ` [PATCH 1/3] modinst: wrap long lines in order to enhance cmd_modules_install Bertrand Jacquin 2014-08-19 18:57 ` [PATCH 2/3] kbuild: handle module compression while running 'make modules_install' Bertrand Jacquin @ 2014-08-19 18:57 ` Bertrand Jacquin 2014-08-20 16:06 ` Rusty Russell 2 siblings, 1 reply; 10+ messages in thread From: Bertrand Jacquin @ 2014-08-19 18:57 UTC (permalink / raw) To: Rusty Russell, linux-kbuild; +Cc: Bertrand Jacquin, linux-kernel, Willy Tarreau This does the same as commit ef591a5 (scripts/Makefile.modpost: error in finding modules from .mod files), but for scripts/Makefile.modsign Maybe we should also apply to Makefile.modsign and Makefile.modinst the change applied to Makefile.modpost by commit ea4054a (modpost: handle huge numbers of modules) ? Reviewed-by: Willy Tarreau <w@1wt.eu> Signed-off-by: Bertrand Jacquin <beber@meleeweb.net> --- scripts/Makefile.modsign | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/Makefile.modsign b/scripts/Makefile.modsign index abfda62..b6ac708 100644 --- a/scripts/Makefile.modsign +++ b/scripts/Makefile.modsign @@ -7,7 +7,7 @@ __modsign: include scripts/Kbuild.include -__modules := $(sort $(shell grep -h '\.ko' /dev/null $(wildcard $(MODVERDIR)/*.mod))) +__modules := $(sort $(shell grep -h '\.ko$$' /dev/null $(wildcard $(MODVERDIR)/*.mod))) modules := $(patsubst %.o,%.ko,$(wildcard $(__modules:.ko=.o))) PHONY += $(modules) -- 2.0.4 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH 3/3] modsign: lookup lines ending in .ko in .mod files 2014-08-19 18:57 ` [PATCH 3/3] modsign: lookup lines ending in .ko in .mod files Bertrand Jacquin @ 2014-08-20 16:06 ` Rusty Russell 0 siblings, 0 replies; 10+ messages in thread From: Rusty Russell @ 2014-08-20 16:06 UTC (permalink / raw) To: Bertrand Jacquin, linux-kbuild; +Cc: linux-kernel, Willy Tarreau Bertrand Jacquin <beber@meleeweb.net> writes: > This does the same as commit ef591a5 (scripts/Makefile.modpost: error > in finding modules from .mod files), but for scripts/Makefile.modsign > > Maybe we should also apply to Makefile.modsign and Makefile.modinst > the change applied to Makefile.modpost by commit ea4054a (modpost: > handle huge numbers of modules) ? Probably, yes. Applied! Rusty. > > Reviewed-by: Willy Tarreau <w@1wt.eu> > Signed-off-by: Bertrand Jacquin <beber@meleeweb.net> > --- > scripts/Makefile.modsign | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/scripts/Makefile.modsign b/scripts/Makefile.modsign > index abfda62..b6ac708 100644 > --- a/scripts/Makefile.modsign > +++ b/scripts/Makefile.modsign > @@ -7,7 +7,7 @@ __modsign: > > include scripts/Kbuild.include > > -__modules := $(sort $(shell grep -h '\.ko' /dev/null $(wildcard $(MODVERDIR)/*.mod))) > +__modules := $(sort $(shell grep -h '\.ko$$' /dev/null $(wildcard $(MODVERDIR)/*.mod))) > modules := $(patsubst %.o,%.ko,$(wildcard $(__modules:.ko=.o))) > > PHONY += $(modules) > -- > 2.0.4 ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2014-10-12 17:06 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-08-19 18:57 [PATCHv2 0/3] handle module compression at install Bertrand Jacquin
2014-08-19 18:57 ` [PATCH 1/3] modinst: wrap long lines in order to enhance cmd_modules_install Bertrand Jacquin
2014-08-19 22:02 ` Jim Davis
2014-08-20 16:08 ` Rusty Russell
2014-08-19 18:57 ` [PATCH 2/3] kbuild: handle module compression while running 'make modules_install' Bertrand Jacquin
2014-08-20 16:09 ` Rusty Russell
[not found] ` <CAJUuVQ7eEn8r2mAQKmydTDvDgSXYKjUH15VPEQfOVPJxNtU9ag@mail.gmail.com>
2014-10-12 16:40 ` Willy Tarreau
2014-10-12 17:06 ` Andev
2014-08-19 18:57 ` [PATCH 3/3] modsign: lookup lines ending in .ko in .mod files Bertrand Jacquin
2014-08-20 16:06 ` Rusty Russell
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).