[PATCH 1/2] kbuild: add target to install gzipped modules

[PATCH 1/2] kbuild: add target to install gzipped modules

[Lucas De Marchi]

Add target in Makefile to compress the module after it's installed.
Module-init-tools and libkmod can handle gzipped modules.

This is not much useful for distributions because the package will gzip
the modules and call depmod in a install rule. However for those
compiling the kernel on their own and debugging module loading, it's
useful so depmod doesn't have to be called twice and we don't have to
manually compress the modules.

Signed-off-by: Lucas De Marchi <lucas.demarchi@profusion.mobi>
---
 Makefile                 |   10 ++++++++--
 scripts/Makefile.modinst |    8 ++++++++
 2 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index ea51081..baddcfe 100644
--- a/Makefile
+++ b/Makefile
@@ -1104,8 +1104,9 @@ PHONY += modules_prepare
 modules_prepare: prepare scripts
 
 # Target to install modules
-PHONY += modules_install
-modules_install: _modinst_ _modinst_post
+PHONY += modules_install modules_install_gz
+modules_install: _modinst_bare _modinst_post
+modules_install_gz: _modinst_gz _modinst_post
 
 PHONY += _modinst_
 _modinst_:
@@ -1119,8 +1120,13 @@ _modinst_:
 	fi
 	@cp -f $(objtree)/modules.order $(MODLIB)/
 	@cp -f $(objtree)/modules.builtin $(MODLIB)/
+
+_modinst_bare: _modinst_
 	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modinst
 
+_modinst_gz: _modinst_
+	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modinst compression=gz
+
 # This depmod is only for convenience to give the initial
 # boot a modules.dep even before / is mounted read-write.  However the
 # boot script depmod is the master version.
diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index efa5d94..442f953 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -12,6 +12,8 @@ include scripts/Kbuild.include
 __modules := $(sort $(shell grep -h '\.ko' /dev/null $(wildcard $(MODVERDIR)/*.mod)))
 modules := $(patsubst %.o,%.ko,$(wildcard $(__modules:.ko=.o)))
 
+compression ?= none
+
 PHONY += $(modules)
 __modinst: $(modules)
 	@:
@@ -19,6 +21,9 @@ __modinst: $(modules)
 quiet_cmd_modules_install = INSTALL $@
       cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) $(2)/$(notdir $@)
 
+quiet_cmd_modules_compress_gz = GZ      $@.gz
+      cmd_modules_compress_gz = gzip $(2)/$(notdir $@)
+
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
 ext-mod-dir = $(INSTALL_MOD_DIR)$(subst $(patsubst %/,%,$(KBUILD_EXTMOD)),,$(@D))
@@ -27,6 +32,9 @@ modinst_dir = $(if $(KBUILD_EXTMOD),$(ext-mod-dir),kernel/$(@D))
 
 $(modules):
 	$(call cmd,modules_install,$(MODLIB)/$(modinst_dir))
+ifeq ($(compression),gz)
+	$(call cmd,modules_compress_gz,$(MODLIB)/$(modinst_dir))
+endif
 
 
 # Declare the contents of the .PHONY variable as phony.  We keep that
-- 
1.7.8.1

[PATCH 2/2] kbuild: add target to install xz modules

[Lucas De Marchi]

Add target in Makefile to compress the module with XZ  after it's
installed. kmod can handle modules compressed with xz and it's replacing
module-init-tools.

Signed-off-by: Lucas De Marchi <lucas.demarchi@profusion.mobi>
---
 Makefile                 |    6 +++++-
 scripts/Makefile.modinst |    7 +++++++
 2 files changed, 12 insertions(+), 1 deletions(-)

diff --git a/Makefile b/Makefile
index baddcfe..f06f886 100644
--- a/Makefile
+++ b/Makefile
@@ -1104,9 +1104,10 @@ PHONY += modules_prepare
 modules_prepare: prepare scripts
 
 # Target to install modules
-PHONY += modules_install modules_install_gz
+PHONY += modules_install modules_install_gz modules_install_xz
 modules_install: _modinst_bare _modinst_post
 modules_install_gz: _modinst_gz _modinst_post
+modules_install_xz: _modinst_xz _modinst_post
 
 PHONY += _modinst_
 _modinst_:
@@ -1127,6 +1128,9 @@ _modinst_bare: _modinst_
 _modinst_gz: _modinst_
 	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modinst compression=gz
 
+_modinst_xz: _modinst_
+	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modinst compression=xz
+
 # This depmod is only for convenience to give the initial
 # boot a modules.dep even before / is mounted read-write.  However the
 # boot script depmod is the master version.
diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index 442f953..297c546 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -24,6 +24,9 @@ quiet_cmd_modules_install = INSTALL $@
 quiet_cmd_modules_compress_gz = GZ      $@.gz
       cmd_modules_compress_gz = gzip $(2)/$(notdir $@)
 
+quiet_cmd_modules_compress_xz = XZ      $@.xz
+      cmd_modules_compress_xz = xz $(2)/$(notdir $@)
+
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
 ext-mod-dir = $(INSTALL_MOD_DIR)$(subst $(patsubst %/,%,$(KBUILD_EXTMOD)),,$(@D))
@@ -34,6 +37,10 @@ $(modules):
 	$(call cmd,modules_install,$(MODLIB)/$(modinst_dir))
 ifeq ($(compression),gz)
 	$(call cmd,modules_compress_gz,$(MODLIB)/$(modinst_dir))
+else
+ifeq ($(compression),xz)
+	$(call cmd,modules_compress_xz,$(MODLIB)/$(modinst_dir))
+endif
 endif
 
 
-- 
1.7.8.1

Re: [PATCH 1/2] kbuild: add target to install gzipped modules

[Michal Marek]

Dne 29.12.2011 16:50, Lucas De Marchi napsal(a):
> Add target in Makefile to compress the module after it's installed.
> Module-init-tools and libkmod can handle gzipped modules.

I am not convinced that this is a needed feature. Compressing elf files
means that depmod and modinfo need to read the whole compressed file
from disk and unpack it, while only a couple of bytes need to be read.
Those concerned about disk space either compile only the few needed
modules and/or use some compressed filesystem, which allows for random
access.


> This is not much useful for distributions because the package will gzip
> the modules and call depmod in a install rule. However for those
> compiling the kernel on their own and debugging module loading, it's
> useful so depmod doesn't have to be called twice and we don't have to
> manually compress the modules.

I understand that you need *.ko.gz support in kmod for the sake of
feature parity and that such patch would help you with debugging. But I
doubt there is use for it apart of developing kmod. Wouldn't an external
script like this do the same job for you?

#!/bin/sh
make "$@" modules_install
rel="$(make -s "$@" kernelrelease)"
find "/lib/modules/$rel" -name '*.ko' -exec gzip '{}' ';'
depmod "$rel"

Michal

Re: [PATCH 1/2] kbuild: add target to install gzipped modules

[Lucas De Marchi]

2011/12/29 Michal Marek <mmarek@suse.cz>:
> Dne 29.12.2011 16:50, Lucas De Marchi napsal(a):
>> Add target in Makefile to compress the module after it's installed.
>> Module-init-tools and libkmod can handle gzipped modules.
>
> I am not convinced that this is a needed feature. Compressing elf files
> means that depmod and modinfo need to read the whole compressed file
> from disk and unpack it, while only a couple of bytes need to be read.
> Those concerned about disk space either compile only the few needed
> modules and/or use some compressed filesystem, which allows for random
> access.
>
>
>> This is not much useful for distributions because the package will gzip
>> the modules and call depmod in a install rule. However for those
>> compiling the kernel on their own and debugging module loading, it's
>> useful so depmod doesn't have to be called twice and we don't have to
>> manually compress the modules.
>
> I understand that you need *.ko.gz support in kmod for the sake of
> feature parity and that such patch would help you with debugging. But I
> doubt there is use for it apart of developing kmod. Wouldn't an external
> script like this do the same job for you?
>
> #!/bin/sh
> make "$@" modules_install
> rel="$(make -s "$@" kernelrelease)"
> find "/lib/modules/$rel" -name '*.ko' -exec gzip '{}' ';'
> depmod "$rel"

Sure, but it has the same problem of running depmod twice. I was
thinking that compressed modules might be useful for others, not
developing kmod. At least I know Archlinux uses gzipped modules by
default and probably gentoo guys, too.


Lucas De Marchi

Re: [PATCH 1/2] kbuild: add target to install gzipped modules

[Michal Marek]

Dne 29.12.2011 17:55, Lucas De Marchi napsal(a):
> Sure, but it has the same problem of running depmod twice. I was
> thinking that compressed modules might be useful for others, not
> developing kmod. At least I know Archlinux uses gzipped modules by
> default and probably gentoo guys, too.

And what's the benefit to users of Archlinux / Gentoo? Do they also
compress libraries in /usr/lib? :)

Michal

Re: [PATCH 1/2] kbuild: add target to install gzipped modules

[Dave Jones]

On Thu, Dec 29, 2011 at 01:50:18PM -0200, Lucas De Marchi wrote:
 > Add target in Makefile to compress the module after it's installed.
 > Module-init-tools and libkmod can handle gzipped modules.
 > 
 > This is not much useful for distributions because the package will gzip
 > the modules and call depmod in a install rule. 

It might actually be a worthwhile thing for distributions.

For a Fedora kernel, gzipping modules saves around 80MB of diskspace per
installed kernel.  That the RPM is compressed is irrelevant, the on-disk
footprint is more interesting, given that the bulk of the modules installed
will never even be loaded.

	Dave

Re: [PATCH 1/2] kbuild: add target to install gzipped modules

[Lucas De Marchi]

2011/12/29 Michal Marek <mmarek@suse.cz>:
> Dne 29.12.2011 16:50, Lucas De Marchi napsal(a):
>> Add target in Makefile to compress the module after it's installed.
>> Module-init-tools and libkmod can handle gzipped modules.
>
> I am not convinced that this is a needed feature. Compressing elf files
> means that depmod and modinfo need to read the whole compressed file
> from disk and unpack it, while only a couple of bytes need to be read.

How many times do you need to call depmod / modinfo?

The operation that is used most of the time is to load modules. The
reason for having modules gzipped is the same one as for kernel image:
fast bootup on slow disks and less disk space used per installed
kernel.


Lucas De Marchi

Re: [PATCH 1/2] kbuild: add target to install gzipped modules

[Michal Marek]

On 29.12.2011 18:21, Dave Jones wrote:
> On Thu, Dec 29, 2011 at 01:50:18PM -0200, Lucas De Marchi wrote:
>  > Add target in Makefile to compress the module after it's installed.
>  > Module-init-tools and libkmod can handle gzipped modules.
>  > 
>  > This is not much useful for distributions because the package will gzip
>  > the modules and call depmod in a install rule. 
> 
> It might actually be a worthwhile thing for distributions.
> 
> For a Fedora kernel, gzipping modules saves around 80MB of diskspace per
> installed kernel.  That the RPM is compressed is irrelevant, the on-disk
> footprint is more interesting, given that the bulk of the modules installed
> will never even be loaded.

But it kills performance of the tools.

Michal

Re: [PATCH 1/2] kbuild: add target to install gzipped modules

[Michal Marek]

On 29.12.2011 18:35, Lucas De Marchi wrote:
> The operation that is used most of the time is to load modules. The
> reason for having modules gzipped is the same one as for kernel image:

The reason for having the kernel compressed is that the x86 bootloader
reads the kernel (and initrd) using bios calls. That's not the case for
modules.

Michal

Re: [PATCH 1/2] kbuild: add target to install gzipped modules

[Lucas De Marchi]

On Fri, Dec 30, 2011 at 8:30 AM, Michal Marek <mmarek@suse.cz> wrote:
> On 29.12.2011 18:35, Lucas De Marchi wrote:
>> The operation that is used most of the time is to load modules. The
>> reason for having modules gzipped is the same one as for kernel image:
>
> The reason for having the kernel compressed is that the x86 bootloader
> reads the kernel (and initrd) using bios calls. That's not the case for
> modules.

It could be when it was added, but it's not anymore.


Really, I wouldn't like the discussion to be whether having compressed
modules is good or not, but rather if it's good to have a install rule
that does all the things the right way and only once, given people
*are using* compressed modules.

Lucas De Marchi

Re: [PATCH 1/2] kbuild: add target to install gzipped modules

[Dave Jones]

On Fri, Dec 30, 2011 at 11:23:58AM +0100, Michal Marek wrote:
 > On 29.12.2011 18:21, Dave Jones wrote:
 > > On Thu, Dec 29, 2011 at 01:50:18PM -0200, Lucas De Marchi wrote:
 > >  > Add target in Makefile to compress the module after it's installed.
 > >  > Module-init-tools and libkmod can handle gzipped modules.
 > >  > 
 > >  > This is not much useful for distributions because the package will gzip
 > >  > the modules and call depmod in a install rule. 
 > > 
 > > It might actually be a worthwhile thing for distributions.
 > > 
 > > For a Fedora kernel, gzipping modules saves around 80MB of diskspace per
 > > installed kernel.  That the RPM is compressed is irrelevant, the on-disk
 > > footprint is more interesting, given that the bulk of the modules installed
 > > will never even be loaded.
 > 
 > But it kills performance of the tools.

How often do you run depmod ?

If modprobe is a bottleneck, you have other problems.

	Dave

Re: [PATCH 1/2] kbuild: add target to install gzipped modules

[Lucas De Marchi]

On Fri, Dec 30, 2011 at 1:37 PM, Dave Jones <davej@redhat.com> wrote:
> On Fri, Dec 30, 2011 at 11:23:58AM +0100, Michal Marek wrote:
>  > On 29.12.2011 18:21, Dave Jones wrote:
>  > > On Thu, Dec 29, 2011 at 01:50:18PM -0200, Lucas De Marchi wrote:
>  > >  > Add target in Makefile to compress the module after it's installed.
>  > >  > Module-init-tools and libkmod can handle gzipped modules.
>  > >  >
>  > >  > This is not much useful for distributions because the package will gzip
>  > >  > the modules and call depmod in a install rule.
>  > >
>  > > It might actually be a worthwhile thing for distributions.
>  > >
>  > > For a Fedora kernel, gzipping modules saves around 80MB of diskspace per
>  > > installed kernel.  That the RPM is compressed is irrelevant, the on-disk
>  > > footprint is more interesting, given that the bulk of the modules installed
>  > > will never even be loaded.
>  >
>  > But it kills performance of the tools.
>
> How often do you run depmod ?
>
> If modprobe is a bottleneck, you have other problems.

modprobe is not a problem because it needs to read the entire blob to
insert regardless. The only affected tools would be depmod and
modinfo, but if this is really a problem, we could change the loader
of gz and xz files to stop when the ".modinfo" section was read for
the modinfo command.

As I said, I don't want to debate here whether it's good or not to
have compressed modules. It would be nice though to cook up some
numbers on different setups with the different compression methods
(none, gz, xz).

Lucas De Marchi