[PATCH] Compress kernel modules on installation.

[PATCH] Compress kernel modules on installation.

[Steve Brokenshire]

Hi,

(I've sent this to the linux-kbuild and linux-kernel lists as this
patch modifies the Makefile.modinst file. I also don't subscribe to the 
linux-kbuild and linux-kernel mailing lists so can I have any replies
CC'ed to me please)

This patch allows kernel modules to be compressed when 'make
modules_install' is run after being copied to
the /lib/module/<version>/<...> directory which is useful if you have
module-init-tools installed with --enable-zlib. This patch adds an
option (MODULE_COMPRESS) to the kernel configuration file (specifically
init/Kconfig) so that the kernel modules will compressed if
MODULE_COMPRESS is set.

When MODULE_COMPRESS is set the output of 'make modules_install' will
go as the following:

INSTALL  drivers/fs/xfs/xfs.ko
COMPRESS drivers/fs/xfs/xfs.ko
INSTALL  drivers/fs/fat/fat.ko
COMPRESS drivers/fs/fat/fat.ko
...

I've tested my patch on kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
2.6.19 and they compile, install and compress into the respective
module directories without any errors.

I've also tested this with the uvcvideo (linux-uvc) kernel module (from
the SVN branch and with kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
2.6.19) as that uses Kbuild properly when installing the module and
after installing the uvcvideo the module is then compressed.
Unfortunately, I couldn't find any other kernel modules which used the
Kbuild system for installing their kernel modules. :(

I've included include/config/auto.conf in Makefile.modinst so that it
can check if MODULE_COMPRESS is set when installing the kernel modules.

Unfortunately when I ran mkinitrd (CentOS version) to create the initrd
image to go with the kernel, I get errors saying that certain kernel
modules don't exist despite the fact they do actually exist. When I
pass --allow-missing to mkinitrd, it seems to go fine but when booting
up with the system with the new initrd image I get error messages
saying that the modules don't exist.

A workaround is to compile the modules, don't have MODULE_COMPRESS set 
in .config, install the modules, run mkinitrd and copy it to /boot, set 
MODULE_COMPRESS in .config and then install the modules again but
compressed.

That's about it really. The only showstopper I feel is mkinitrd not
working properly with the compressed kernel modules.

Thanks,
Steve

---

Signed-off-by: Steve Brokenshire <sbrokenshire@xestia.co.uk>

diff -u -r -up a/init/Kconfig b/init/Kconfig
--- a/init/Kconfig	2008-02-02 16:34:29.000000000 +0000
+++ b/init/Kconfig	2008-02-03 09:55:52.000000000 +0000
@@ -725,6 +725,30 @@ config MODULE_FORCE_UNLOAD
 	  rmmod).  This is mainly for kernel developers and desperate users.
 	  If unsure, say N.
 
+config MODULE_COMPRESS
+	bool "Compress kernel modules on installation"
+	depends on MODULES
+	help
+	  This option compresses the kernel modules when 'make
+	  modules_install' is run.
+
+	  The modules will be compressed into the gzip (GNU zip) format
+	  and will use less space than an uncompressed kernel module would.
+
+	  When a kernel module is installed from outside of the main kernel
+	  source and uses the Kbuild system for installing modules then that
+	  kernel module will also be compressed when it is installed.
+
+	  When running mkinitrd you will find that an error message
+	  appears saying that it cannot find a certain kernel module.
+	  As a workaround, unset CONFIG_MODULE_COMPRESS, build the modules
+	  and install them, run mkinitrd and create the initrd image, place
+	  the initrd image in the correct place for booting, set
+	  CONFIG_MODULE_COMPRESS and then install the modules again.
+
+	  This options requires the module-init-tools package to be 
+	  configured with --enable-zlib.
+
 config MODVERSIONS
 	bool "Module versioning support"
 	depends on MODULES
diff -u -r -up a/scripts/Makefile.modinst b/scripts/Makefile.modinst
--- a/scripts/Makefile.modinst	2008-02-02 16:34:33.000000000 +0000
+++ b/scripts/Makefile.modinst	2008-02-02 17:21:46.000000000 +0000
@@ -5,6 +5,7 @@
 PHONY := __modinst
 __modinst:
 
+include include/config/auto.conf
 include scripts/Kbuild.include
 
 #
@@ -16,8 +17,15 @@ PHONY += $(modules)
 __modinst: $(modules)
 	@:
 
-quiet_cmd_modules_install = INSTALL $@
-      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) $(2)/$(notdir $@)
+quiet_cmd_modules_install = INSTALL  $@
+      cmd_modules_install = mkdir -p $(2); \
+				cp $@ $(2) ; \
+				$(mod_strip_cmd) $(2)/$(notdir $@)
+
+quiet_cmd_modules_compress = COMPRESS $@
+      cmd_modules_compress = gzip --best -c $(2)/`basename $@` \
+				 > $(2)/`basename $@`.gz; \
+				rm $(2)/`basename $@`
 
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
@@ -27,7 +35,8 @@ modinst_dir = $(if $(KBUILD_EXTMOD),$(ex
 
 $(modules):
 	$(call cmd,modules_install,$(MODLIB)/$(modinst_dir))
-
+	$(if $(CONFIG_MODULE_COMPRESS), \
+		$(call cmd,modules_compress,$(MODLIB)/$(modinst_dir)))
 
 # Declare the contents of the .PHONY variable as phony.  We keep that
 # information in a variable se we can use it in if_changed and friends.

Re: [PATCH] Compress kernel modules on installation.

[Oleg Verych]

On Mon, Feb 25, 2008 at 10:42 PM, Steve Brokenshire:
> Hi,
>
>  (I've sent this to the linux-kbuild and linux-kernel lists as this
>  patch modifies the Makefile.modinst file. I also don't subscribe to the
>  linux-kbuild and linux-kernel mailing lists so can I have any replies
>  CC'ed to me please)

And what if i like bzip2 (yea sometimes better) or 7zip (better if !EMBEDDED)
or whatever?

It's pure user/distro question.
_______

Re: [PATCH] Compress kernel modules on installation.

[Willy Tarreau]

On Mon, Feb 25, 2008 at 11:17:23PM +0100, Oleg Verych wrote:
> On Mon, Feb 25, 2008 at 10:42 PM, Steve Brokenshire:
> > Hi,
> >
> >  (I've sent this to the linux-kbuild and linux-kernel lists as this
> >  patch modifies the Makefile.modinst file. I also don't subscribe to the
> >  linux-kbuild and linux-kernel mailing lists so can I have any replies
> >  CC'ed to me please)
> 
> And what if i like bzip2 (yea sometimes better) or 7zip (better if !EMBEDDED)
> or whatever?
> 
> It's pure user/distro question.

not exactly, as only gzip is supported by module-init-tools. However,
if you come up with a patch to implement lzma or equivalent for
module-init-tools, it may be useful.

Willy

Re: [PATCH] Compress kernel modules on installation.

[Willy Tarreau]

On Mon, Feb 25, 2008 at 09:42:09PM +0000, Steve Brokenshire wrote:
> Hi,
> 
> (I've sent this to the linux-kbuild and linux-kernel lists as this
> patch modifies the Makefile.modinst file. I also don't subscribe to the 
> linux-kbuild and linux-kernel mailing lists so can I have any replies
> CC'ed to me please)
> 
> This patch allows kernel modules to be compressed when 'make
> modules_install' is run after being copied to
> the /lib/module/<version>/<...> directory which is useful if you have
> module-init-tools installed with --enable-zlib. This patch adds an
> option (MODULE_COMPRESS) to the kernel configuration file (specifically
> init/Kconfig) so that the kernel modules will compressed if
> MODULE_COMPRESS is set.
> 
> When MODULE_COMPRESS is set the output of 'make modules_install' will
> go as the following:
> 
> INSTALL  drivers/fs/xfs/xfs.ko
> COMPRESS drivers/fs/xfs/xfs.ko
> INSTALL  drivers/fs/fat/fat.ko
> COMPRESS drivers/fs/fat/fat.ko
> ...
> 
> I've tested my patch on kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> 2.6.19 and they compile, install and compress into the respective
> module directories without any errors.
> 
> I've also tested this with the uvcvideo (linux-uvc) kernel module (from
> the SVN branch and with kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> 2.6.19) as that uses Kbuild properly when installing the module and
> after installing the uvcvideo the module is then compressed.
> Unfortunately, I couldn't find any other kernel modules which used the
> Kbuild system for installing their kernel modules. :(
> 
> I've included include/config/auto.conf in Makefile.modinst so that it
> can check if MODULE_COMPRESS is set when installing the kernel modules.
> 
> Unfortunately when I ran mkinitrd (CentOS version) to create the initrd
> image to go with the kernel, I get errors saying that certain kernel
> modules don't exist despite the fact they do actually exist. When I
> pass --allow-missing to mkinitrd, it seems to go fine but when booting
> up with the system with the new initrd image I get error messages
> saying that the modules don't exist.
> 
> A workaround is to compile the modules, don't have MODULE_COMPRESS set 
> in .config, install the modules, run mkinitrd and copy it to /boot, set 
> MODULE_COMPRESS in .config and then install the modules again but
> compressed.
> 
> That's about it really. The only showstopper I feel is mkinitrd not
> working properly with the compressed kernel modules.

Have you tried keeping the module names intact (.ko, not .ko.gz) ?
It's what I was doing with modutils in 2.4 and what I'm still doing
with module-init-tools in 2.6. While I don't particularly use mkinitrd,
I think that keeping the name intact is preferable and should help.

Regards,
Willy

Re: [PATCH] Compress kernel modules on installation.

[Oleg Verych]

On Mon, Feb 25, 2008 at 11:19 PM, Willy Tarreau:
>
> On Mon, Feb 25, 2008 at 11:17:23PM +0100, Oleg Verych wrote:
>  > On Mon, Feb 25, 2008 at 10:42 PM, Steve Brokenshire:
>  > > Hi,
>  > >
>  > >  (I've sent this to the linux-kbuild and linux-kernel lists as this
>  > >  patch modifies the Makefile.modinst file. I also don't subscribe to the
>  > >  linux-kbuild and linux-kernel mailing lists so can I have any replies
>  > >  CC'ed to me please)
>  >
>  > And what if i like bzip2 (yea sometimes better) or 7zip (better if !EMBEDDED)
>  > or whatever?
>  >
>  > It's pure user/distro question.
>
>  not exactly, as only gzip is supported by module-init-tools. However,
>  if you come up with a patch to implement lzma or equivalent for
>  module-init-tools, it may be useful.

Oh, come on! It's userspace, i have scripts managing existence/compression
on per-file basis on my comp. If most distros just drop thousands of useless
(sometimes compressed) stuff to your harddisk, it's not kernel's or
modules-init-tools'
problem.

Also having compressed over compressed stuff, like on initramfs booting image
with such modules (if by defaults) may cause bigger file, more CPU overhead.

It is userspace and all that policy stuff, etc., etc.

Want to make a minidistro on lkml? Well, i see klibc working fine already.
_______

Re: [PATCH] Compress kernel modules on installation.

[Willy Tarreau]

On Mon, Feb 25, 2008 at 11:32:44PM +0100, Oleg Verych wrote:
> On Mon, Feb 25, 2008 at 11:19 PM, Willy Tarreau:
> >
> > On Mon, Feb 25, 2008 at 11:17:23PM +0100, Oleg Verych wrote:
> >  > On Mon, Feb 25, 2008 at 10:42 PM, Steve Brokenshire:
> >  > > Hi,
> >  > >
> >  > >  (I've sent this to the linux-kbuild and linux-kernel lists as this
> >  > >  patch modifies the Makefile.modinst file. I also don't subscribe to the
> >  > >  linux-kbuild and linux-kernel mailing lists so can I have any replies
> >  > >  CC'ed to me please)
> >  >
> >  > And what if i like bzip2 (yea sometimes better) or 7zip (better if !EMBEDDED)
> >  > or whatever?
> >  >
> >  > It's pure user/distro question.
> >
> >  not exactly, as only gzip is supported by module-init-tools. However,
> >  if you come up with a patch to implement lzma or equivalent for
> >  module-init-tools, it may be useful.
> 
> Oh, come on! It's userspace,

and your point is ... ?

> i have scripts managing existence/compression
> on per-file basis on my comp.

I'm definitely happy for you.

> If most distros just drop thousands of useless
> (sometimes compressed) stuff to your harddisk, it's not kernel's or
> modules-init-tools' problem.

who said there was a problem ?

> Also having compressed over compressed stuff, like on initramfs booting image
> with such modules (if by defaults) may cause bigger file, more CPU overhead.

Already known and obvious, thanks. But that's unrelated to the initial post.

> It is userspace and all that policy stuff, etc., etc.

A makefile is userspace too. I still fail to see your point.
 
> Want to make a minidistro on lkml? Well, i see klibc working fine already.

Huh? drinking before posting does no good it seems... :-)

Steve proposed a simple patch to ease modules compression during the
build process. Obviously it's userspace (as all the build process,
mind you). And there's nothing dirty nor funny there. Maybe you just
don't understand what it's useful for, but in this case it simply means
you don't need this feature :-)

Willy

Re: [PATCH] Compress kernel modules on installation.

[Adrian Bunk]

On Mon, Feb 25, 2008 at 11:21:38PM +0100, Willy Tarreau wrote:
> On Mon, Feb 25, 2008 at 09:42:09PM +0000, Steve Brokenshire wrote:
> > Hi,
> > 
> > (I've sent this to the linux-kbuild and linux-kernel lists as this
> > patch modifies the Makefile.modinst file. I also don't subscribe to the 
> > linux-kbuild and linux-kernel mailing lists so can I have any replies
> > CC'ed to me please)
> > 
> > This patch allows kernel modules to be compressed when 'make
> > modules_install' is run after being copied to
> > the /lib/module/<version>/<...> directory which is useful if you have
> > module-init-tools installed with --enable-zlib. This patch adds an
> > option (MODULE_COMPRESS) to the kernel configuration file (specifically
> > init/Kconfig) so that the kernel modules will compressed if
> > MODULE_COMPRESS is set.
> > 
> > When MODULE_COMPRESS is set the output of 'make modules_install' will
> > go as the following:
> > 
> > INSTALL  drivers/fs/xfs/xfs.ko
> > COMPRESS drivers/fs/xfs/xfs.ko
> > INSTALL  drivers/fs/fat/fat.ko
> > COMPRESS drivers/fs/fat/fat.ko
> > ...
> > 
> > I've tested my patch on kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> > 2.6.19 and they compile, install and compress into the respective
> > module directories without any errors.
> > 
> > I've also tested this with the uvcvideo (linux-uvc) kernel module (from
> > the SVN branch and with kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> > 2.6.19) as that uses Kbuild properly when installing the module and
> > after installing the uvcvideo the module is then compressed.
> > Unfortunately, I couldn't find any other kernel modules which used the
> > Kbuild system for installing their kernel modules. :(
> > 
> > I've included include/config/auto.conf in Makefile.modinst so that it
> > can check if MODULE_COMPRESS is set when installing the kernel modules.
> > 
> > Unfortunately when I ran mkinitrd (CentOS version) to create the initrd
> > image to go with the kernel, I get errors saying that certain kernel
> > modules don't exist despite the fact they do actually exist. When I
> > pass --allow-missing to mkinitrd, it seems to go fine but when booting
> > up with the system with the new initrd image I get error messages
> > saying that the modules don't exist.
> > 
> > A workaround is to compile the modules, don't have MODULE_COMPRESS set 
> > in .config, install the modules, run mkinitrd and copy it to /boot, set 
> > MODULE_COMPRESS in .config and then install the modules again but
> > compressed.
> > 
> > That's about it really. The only showstopper I feel is mkinitrd not
> > working properly with the compressed kernel modules.
> 
> Have you tried keeping the module names intact (.ko, not .ko.gz) ?
> It's what I was doing with modutils in 2.4 and what I'm still doing
> with module-init-tools in 2.6. While I don't particularly use mkinitrd,
> I think that keeping the name intact is preferable and should help.

How would you see if, and if yes with what program, a module was 
compressed if the name is kept intact?

I'd assume stuff like mkinitrd simply also needs to gain knowledge about 
compressed modules.

> Regards,
> Willy

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] Compress kernel modules on installation.

[Willy Tarreau]

On Tue, Feb 26, 2008 at 11:14:55AM +0200, Adrian Bunk wrote:
> On Mon, Feb 25, 2008 at 11:21:38PM +0100, Willy Tarreau wrote:
> > On Mon, Feb 25, 2008 at 09:42:09PM +0000, Steve Brokenshire wrote:
> > > Hi,
> > > 
> > > (I've sent this to the linux-kbuild and linux-kernel lists as this
> > > patch modifies the Makefile.modinst file. I also don't subscribe to the 
> > > linux-kbuild and linux-kernel mailing lists so can I have any replies
> > > CC'ed to me please)
> > > 
> > > This patch allows kernel modules to be compressed when 'make
> > > modules_install' is run after being copied to
> > > the /lib/module/<version>/<...> directory which is useful if you have
> > > module-init-tools installed with --enable-zlib. This patch adds an
> > > option (MODULE_COMPRESS) to the kernel configuration file (specifically
> > > init/Kconfig) so that the kernel modules will compressed if
> > > MODULE_COMPRESS is set.
> > > 
> > > When MODULE_COMPRESS is set the output of 'make modules_install' will
> > > go as the following:
> > > 
> > > INSTALL  drivers/fs/xfs/xfs.ko
> > > COMPRESS drivers/fs/xfs/xfs.ko
> > > INSTALL  drivers/fs/fat/fat.ko
> > > COMPRESS drivers/fs/fat/fat.ko
> > > ...
> > > 
> > > I've tested my patch on kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> > > 2.6.19 and they compile, install and compress into the respective
> > > module directories without any errors.
> > > 
> > > I've also tested this with the uvcvideo (linux-uvc) kernel module (from
> > > the SVN branch and with kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> > > 2.6.19) as that uses Kbuild properly when installing the module and
> > > after installing the uvcvideo the module is then compressed.
> > > Unfortunately, I couldn't find any other kernel modules which used the
> > > Kbuild system for installing their kernel modules. :(
> > > 
> > > I've included include/config/auto.conf in Makefile.modinst so that it
> > > can check if MODULE_COMPRESS is set when installing the kernel modules.
> > > 
> > > Unfortunately when I ran mkinitrd (CentOS version) to create the initrd
> > > image to go with the kernel, I get errors saying that certain kernel
> > > modules don't exist despite the fact they do actually exist. When I
> > > pass --allow-missing to mkinitrd, it seems to go fine but when booting
> > > up with the system with the new initrd image I get error messages
> > > saying that the modules don't exist.
> > > 
> > > A workaround is to compile the modules, don't have MODULE_COMPRESS set 
> > > in .config, install the modules, run mkinitrd and copy it to /boot, set 
> > > MODULE_COMPRESS in .config and then install the modules again but
> > > compressed.
> > > 
> > > That's about it really. The only showstopper I feel is mkinitrd not
> > > working properly with the compressed kernel modules.
> > 
> > Have you tried keeping the module names intact (.ko, not .ko.gz) ?
> > It's what I was doing with modutils in 2.4 and what I'm still doing
> > with module-init-tools in 2.6. While I don't particularly use mkinitrd,
> > I think that keeping the name intact is preferable and should help.
> 
> How would you see if, and if yes with what program, a module was 
> compressed if the name is kept intact?

depmod/modinfo/insmod/modprobe already know it. And quite honnestly,
I don't know about any other program which really needs to process
those files once installed. Well, maybe ksymoops, but I'd have to
check, as I don't recall having ever been annoyed with this.

> I'd assume stuff like mkinitrd simply also needs to gain knowledge about 
> compressed modules.

I'm not that sure. At most it might want to uncompress them to gain some
space, but someone who uses compressed modules for later use in an initrd
is probably missing something.

Regards,
Willy

Re: [PATCH] Compress kernel modules on installation.

[Sam Ravnborg]

On Mon, Feb 25, 2008 at 09:42:09PM +0000, Steve Brokenshire wrote:
> Hi,
> 
> (I've sent this to the linux-kbuild and linux-kernel lists as this
> patch modifies the Makefile.modinst file. I also don't subscribe to the 
> linux-kbuild and linux-kernel mailing lists so can I have any replies
> CC'ed to me please)
> 
> This patch allows kernel modules to be compressed when 'make
> modules_install' is run after being copied to
> the /lib/module/<version>/<...> directory which is useful if you have
> module-init-tools installed with --enable-zlib. This patch adds an
> option (MODULE_COMPRESS) to the kernel configuration file (specifically
> init/Kconfig) so that the kernel modules will compressed if
> MODULE_COMPRESS is set.
> 
> When MODULE_COMPRESS is set the output of 'make modules_install' will
> go as the following:
> 
> INSTALL  drivers/fs/xfs/xfs.ko
> COMPRESS drivers/fs/xfs/xfs.ko
> INSTALL  drivers/fs/fat/fat.ko
> COMPRESS drivers/fs/fat/fat.ko
> ...
> 
> I've tested my patch on kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> 2.6.19 and they compile, install and compress into the respective
> module directories without any errors.
> 
> I've also tested this with the uvcvideo (linux-uvc) kernel module (from
> the SVN branch and with kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> 2.6.19) as that uses Kbuild properly when installing the module and
> after installing the uvcvideo the module is then compressed.
> Unfortunately, I couldn't find any other kernel modules which used the
> Kbuild system for installing their kernel modules. :(
> 
> I've included include/config/auto.conf in Makefile.modinst so that it
> can check if MODULE_COMPRESS is set when installing the kernel modules.

Should we mention this possibility in Documentation/kbuild/modules.txt also?

Otherwise from a quick scan it looks OK.
As module-init-tols are our userspae counterpart we should align the
functionality with that tool.
I assume this gives us a simple choice of compression method and naming.

	Sam

Re: [PATCH] Compress kernel modules on installation.

[Adrian Bunk]

On Tue, Feb 26, 2008 at 11:22:03AM +0100, Willy Tarreau wrote:
> On Tue, Feb 26, 2008 at 11:14:55AM +0200, Adrian Bunk wrote:
> > On Mon, Feb 25, 2008 at 11:21:38PM +0100, Willy Tarreau wrote:
>...
> > > Have you tried keeping the module names intact (.ko, not .ko.gz) ?
> > > It's what I was doing with modutils in 2.4 and what I'm still doing
> > > with module-init-tools in 2.6. While I don't particularly use mkinitrd,
> > > I think that keeping the name intact is preferable and should help.
> > 
> > How would you see if, and if yes with what program, a module was 
> > compressed if the name is kept intact?
> 
> depmod/modinfo/insmod/modprobe already know it. And quite honnestly,
> I don't know about any other program which really needs to process
> those files once installed. Well, maybe ksymoops, but I'd have to
> check, as I don't recall having ever been annoyed with this.
>...

depmod/modinfo/insmod/modprobe know only if you compile 
module-init-tools with zlib support.

And what about the busybox versions?

A different name would e.g.:
- easily allow proper error handling if the userspace modules program 
  doesn't support the compression used
- better scale to support additional compressions
- give the user a hint what is happening and what might be the problem
  when anythig goes wrong

> Regards,
> Willy

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] Compress kernel modules on installation.

[Willy Tarreau]

On Tue, Feb 26, 2008 at 02:30:16PM +0200, Adrian Bunk wrote:
> On Tue, Feb 26, 2008 at 11:22:03AM +0100, Willy Tarreau wrote:
> > On Tue, Feb 26, 2008 at 11:14:55AM +0200, Adrian Bunk wrote:
> > > On Mon, Feb 25, 2008 at 11:21:38PM +0100, Willy Tarreau wrote:
> >...
> > > > Have you tried keeping the module names intact (.ko, not .ko.gz) ?
> > > > It's what I was doing with modutils in 2.4 and what I'm still doing
> > > > with module-init-tools in 2.6. While I don't particularly use mkinitrd,
> > > > I think that keeping the name intact is preferable and should help.
> > > 
> > > How would you see if, and if yes with what program, a module was 
> > > compressed if the name is kept intact?
> > 
> > depmod/modinfo/insmod/modprobe already know it. And quite honnestly,
> > I don't know about any other program which really needs to process
> > those files once installed. Well, maybe ksymoops, but I'd have to
> > check, as I don't recall having ever been annoyed with this.
> >...
> 
> depmod/modinfo/insmod/modprobe know only if you compile 
> module-init-tools with zlib support.
> 
> And what about the busybox versions?
> 
> A different name would e.g.:
> - easily allow proper error handling if the userspace modules program 
>   doesn't support the compression used
> - better scale to support additional compressions
> - give the user a hint what is happening and what might be the problem
>   when anythig goes wrong

I agree, but right now module.dep references existing files with their
real names. Maybe something should define exactly what it should contain
(eg: module_name.ko even if .ko.gz is used) so that all tools relying on
it do not stop after noticing that the file referenced there does not
exist.

regards,
Willy

Re: [PATCH] Compress kernel modules on installation.

[Steve Brokenshire]

On Tue, 26 Feb 2008 12:28:40 +0100
Sam Ravnborg <sam@ravnborg.org> wrote:

> On Mon, Feb 25, 2008 at 09:42:09PM +0000, Steve Brokenshire wrote:
> > Hi,
> > 
> > (I've sent this to the linux-kbuild and linux-kernel lists as this
> > patch modifies the Makefile.modinst file. I also don't subscribe to the 
> > linux-kbuild and linux-kernel mailing lists so can I have any replies
> > CC'ed to me please)
> > 
> > This patch allows kernel modules to be compressed when 'make
> > modules_install' is run after being copied to
> > the /lib/module/<version>/<...> directory which is useful if you have
> > module-init-tools installed with --enable-zlib. This patch adds an
> > option (MODULE_COMPRESS) to the kernel configuration file (specifically
> > init/Kconfig) so that the kernel modules will compressed if
> > MODULE_COMPRESS is set.
> > 
> > When MODULE_COMPRESS is set the output of 'make modules_install' will
> > go as the following:
> > 
> > INSTALL  drivers/fs/xfs/xfs.ko
> > COMPRESS drivers/fs/xfs/xfs.ko
> > INSTALL  drivers/fs/fat/fat.ko
> > COMPRESS drivers/fs/fat/fat.ko
> > ...
> > 
> > I've tested my patch on kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> > 2.6.19 and they compile, install and compress into the respective
> > module directories without any errors.
> > 
> > I've also tested this with the uvcvideo (linux-uvc) kernel module (from
> > the SVN branch and with kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> > 2.6.19) as that uses Kbuild properly when installing the module and
> > after installing the uvcvideo the module is then compressed.
> > Unfortunately, I couldn't find any other kernel modules which used the
> > Kbuild system for installing their kernel modules. :(
> > 
> > I've included include/config/auto.conf in Makefile.modinst so that it
> > can check if MODULE_COMPRESS is set when installing the kernel modules.
> 
> Should we mention this possibility in Documentation/kbuild/modules.txt also?
> 
> Otherwise from a quick scan it looks OK.
> As module-init-tols are our userspae counterpart we should align the
> functionality with that tool.
> I assume this gives us a simple choice of compression method and naming.
> 
> 	Sam

(Sorry for my lazy replying...)

I think it should be mentioned in Documentation/kbuild/modules.txt at
Chapter 2.2. The following should be added to the modules_install
target part:

If MODULES_COMPRESS is set when the modules_install target is run then
the module is compressed after it has been copied to 
/lib/modules/<kernel-version>. Compressed modules require 
module-init-tools installed with --zlib-enabled.

Does that sound right?

Anyway, as it currently is with the patch, MODULE_COMPRESS isn't as scalable as it perhaps could be and I've been thinking I should try and make a better version of the patch which is scalable (So depending on how module-init-tools is updated in the future each option would also give which version of module-init-tools is required to use that compression method in the init/KConfig file)

Steve

Re: [PATCH] Compress kernel modules on installation.

[Jan Engelhardt]

On Tuesday 2008-02-26 12:28, Sam Ravnborg wrote:
>> 
>> (I've sent this to the linux-kbuild and linux-kernel lists as this
>> patch modifies the Makefile.modinst file. I also don't subscribe to the 
>> linux-kbuild and linux-kernel mailing lists so can I have any replies
>> CC'ed to me please)
>> 
>> This patch allows kernel modules to be compressed when 'make
>> modules_install' is run after being copied to
>> the /lib/module/<version>/<...> directory which is useful if you have
>> module-init-tools installed with --enable-zlib. This patch adds an
>> option (MODULE_COMPRESS) to the kernel configuration file (specifically
>> init/Kconfig) so that the kernel modules will compressed if
>> MODULE_COMPRESS is set.

I recently started compressing my kernel modules and that saved me
at least 70 MB of disk space on mostlyallmodconfig.
(And no, the argument of disks being cheap is not so true with
CF or SSD.)
Distro is lazy and wants to wait for upstream to have it,
so is there any chance of getting this proposal in?


Jan

Re: [PATCH] Compress kernel modules on installation.

[Sam Ravnborg]

On Fri, Dec 26, 2008 at 08:30:19PM +0100, Jan Engelhardt wrote:
> 
> On Tuesday 2008-02-26 12:28, Sam Ravnborg wrote:
> >> 
> >> (I've sent this to the linux-kbuild and linux-kernel lists as this
> >> patch modifies the Makefile.modinst file. I also don't subscribe to the 
> >> linux-kbuild and linux-kernel mailing lists so can I have any replies
> >> CC'ed to me please)
> >> 
> >> This patch allows kernel modules to be compressed when 'make
> >> modules_install' is run after being copied to
> >> the /lib/module/<version>/<...> directory which is useful if you have
> >> module-init-tools installed with --enable-zlib. This patch adds an
> >> option (MODULE_COMPRESS) to the kernel configuration file (specifically
> >> init/Kconfig) so that the kernel modules will compressed if
> >> MODULE_COMPRESS is set.
> 
> I recently started compressing my kernel modules and that saved me
> at least 70 MB of disk space on mostlyallmodconfig.
> (And no, the argument of disks being cheap is not so true with
> CF or SSD.)
> Distro is lazy and wants to wait for upstream to have it,
> so is there any chance of getting this proposal in?

Steve said he wanted to try to make the solution more
scalable so I am awaiting a new patch.

	Sam

Re: [PATCH] Compress kernel modules on installation.

[Jan Engelhardt]

On Friday 2008-12-26 20:48, Sam Ravnborg wrote:
>> >> 
>> >> This patch allows kernel modules to be compressed when 'make
>> >> modules_install' is run after being copied to
>> >> the /lib/module/<version>/<...> directory which is useful if you have
>> >> module-init-tools installed with --enable-zlib. This patch adds an
>> >> option (MODULE_COMPRESS) to the kernel configuration file (specifically
>> >> init/Kconfig) so that the kernel modules will compressed if
>> >> MODULE_COMPRESS is set.
>> 
>> I recently started compressing my kernel modules and that saved me
>> at least 70 MB of disk space on mostlyallmodconfig.
>> (And no, the argument of disks being cheap is not so true with
>> CF or SSD.)
>> Distro is lazy and wants to wait for upstream to have it,
>> so is there any chance of getting this proposal in?
>
>Steve said he wanted to try to make the solution more
>scalable so I am awaiting a new patch.

Hm, all I needed was this patch. It might fire up some people,
but it's got all the scalability I could think of..


commit b4a3e1c610c99d4e8b543b97fd722076c6f7c5dd
Author: Jan Engelhardt <jengelh@medozas.de>
Date:   Wed Dec 10 20:39:21 2008 +0100

build: install modules compressed
---
 scripts/Makefile.modinst |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index efa5d94..c3421a1 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -17,7 +17,7 @@ __modinst: $(modules)
 	@:
 
 quiet_cmd_modules_install = INSTALL $@
-      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) $(2)/$(notdir $@)
+      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) $(2)/$(notdir $@); gzip -9f $(2)/$(notdir $@)
 
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra

Re: [PATCH] Compress kernel modules on installation.

[Sam Ravnborg]

On Fri, Dec 26, 2008 at 08:50:34PM +0100, Jan Engelhardt wrote:
> 
> On Friday 2008-12-26 20:48, Sam Ravnborg wrote:
> >> >> 
> >> >> This patch allows kernel modules to be compressed when 'make
> >> >> modules_install' is run after being copied to
> >> >> the /lib/module/<version>/<...> directory which is useful if you have
> >> >> module-init-tools installed with --enable-zlib. This patch adds an
> >> >> option (MODULE_COMPRESS) to the kernel configuration file (specifically
> >> >> init/Kconfig) so that the kernel modules will compressed if
> >> >> MODULE_COMPRESS is set.
> >> 
> >> I recently started compressing my kernel modules and that saved me
> >> at least 70 MB of disk space on mostlyallmodconfig.
> >> (And no, the argument of disks being cheap is not so true with
> >> CF or SSD.)
> >> Distro is lazy and wants to wait for upstream to have it,
> >> so is there any chance of getting this proposal in?
> >
> >Steve said he wanted to try to make the solution more
> >scalable so I am awaiting a new patch.
> 
> Hm, all I needed was this patch. It might fire up some people,
> but it's got all the scalability I could think of..

Jan - there is obviously no way I could apply this patch
so late in the cycle. 
The original patch that made this a CONFIG option is
then much better as we avoid forcing new and untested
behaviour on the users.

We all know that compressing the modules are simple.
And unless someone comes up with *very* good arguments
then we should just use gzip with default parameters.

If we go for the "keep the .ko extension but compress"
then someone needs to answer the obvious questions:
- will this break on a typical distribution
- will this break busybox users

	Sam

Re: [PATCH] Compress kernel modules on installation.

[Jan Engelhardt]

On Friday 2008-12-26 23:57, Sam Ravnborg wrote:
>>>
>>>Steve said [in Feb 2008] he wanted to try to make the solution
>>>more scalable so I am awaiting a new patch.
>> 
>>Hm, all I needed was this patch. It might fire up some people,
>>but it's got all the scalability I could think of..
>
>Jan - there is obviously no way I could apply this patch
>so late in the cycle. 

2.6.29 just started, did not it. Even if not, just queue it for
the next.

>The original patch that made this a CONFIG option is
>then much better as we avoid forcing new and untested
>behaviour on the users.
>
>We all know that compressing the modules are simple.
>And unless someone comes up with *very* good arguments
>then we should just use gzip with default parameters.

Besides the -9 flag, where would there be nondefault
parameters?

>If we go for the "keep the .ko extension but compress"
>then someone needs to answer the obvious questions:
>
>- will this break on a typical distribution

No; module-init-tools already uses gzopen even on
uncompressed files.

>- will this break busybox users

I did not see any gzip support in there, so the
answer is likely "yes, as usual".

Re: [PATCH] Compress kernel modules on installation.

[Sam Ravnborg]

On Fri, Dec 26, 2008 at 08:50:34PM +0100, Jan Engelhardt wrote:
> 
> On Friday 2008-12-26 20:48, Sam Ravnborg wrote:
> >> >> 
> >> >> This patch allows kernel modules to be compressed when 'make
> >> >> modules_install' is run after being copied to
> >> >> the /lib/module/<version>/<...> directory which is useful if you have
> >> >> module-init-tools installed with --enable-zlib. This patch adds an
> >> >> option (MODULE_COMPRESS) to the kernel configuration file (specifically
> >> >> init/Kconfig) so that the kernel modules will compressed if
> >> >> MODULE_COMPRESS is set.
> >> 
> >> I recently started compressing my kernel modules and that saved me
> >> at least 70 MB of disk space on mostlyallmodconfig.
> >> (And no, the argument of disks being cheap is not so true with
> >> CF or SSD.)
> >> Distro is lazy and wants to wait for upstream to have it,
> >> so is there any chance of getting this proposal in?
> >
> >Steve said he wanted to try to make the solution more
> >scalable so I am awaiting a new patch.
> 
> Hm, all I needed was this patch. It might fire up some people,
> but it's got all the scalability I could think of..

Can I ask you to redo this patch.
I want a way to swith off compression, but it should default be enable.
So if KBUILD_MODULE_NOCOMPRESS is undefined we will compress the modules
when we install them.

Something like:

      cmd_modules_install =                                               \
         mkdir -p $(2); cp $@ $(2) ;                                      \
         $(mod_strip_cmd) $(2)/$(notdir $@)                               \
         $(if $(KBUILD_MODULES_NOCOMPRESS),, ;gzip -9f $(2)/$(notdir $@))

And add documentation to Documentation/kbuild/kbuild.txt too...

Thanks,
	Sam

Re: [PATCH] Compress kernel modules on installation.

[Arjan van de Ven]

On Sun, 18 Jan 2009 22:04:16 +0100
Sam Ravnborg <sam@ravnborg.org> wrote:

> On Fri, Dec 26, 2008 at 08:50:34PM +0100, Jan Engelhardt wrote:
> > 
> > On Friday 2008-12-26 20:48, Sam Ravnborg wrote:
> > >> >> 
> > >> >> This patch allows kernel modules to be compressed when 'make
> > >> >> modules_install' is run after being copied to
> > >> >> the /lib/module/<version>/<...> directory which is useful if
> > >> >> you have module-init-tools installed with --enable-zlib. This
> > >> >> patch adds an option (MODULE_COMPRESS) to the kernel
> > >> >> configuration file (specifically init/Kconfig) so that the
> > >> >> kernel modules will compressed if MODULE_COMPRESS is set.
> > >> 
> > >> I recently started compressing my kernel modules and that saved
> > >> me at least 70 MB of disk space on mostlyallmodconfig.
> > >> (And no, the argument of disks being cheap is not so true with
> > >> CF or SSD.)
> > >> Distro is lazy and wants to wait for upstream to have it,
> > >> so is there any chance of getting this proposal in?
> > >
> > >Steve said he wanted to try to make the solution more
> > >scalable so I am awaiting a new patch.
> > 
> > Hm, all I needed was this patch. It might fire up some people,
> > but it's got all the scalability I could think of..
>  
> Can I ask you to redo this patch.
> I want a way to swith off compression, but it should default be
> enable. So if KBUILD_MODULE_NOCOMPRESS is undefined we will compress
> the modules when we install them.
> 

especially since the decompression is an option for the module tools,
it should be an option for the kernel... 
since it does add a dependency for the module tools (and one that you
might not need; for example I disabled it just to remove a build
requires)


-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: [PATCH] Compress kernel modules on installation.

[Michael Tokarev]

Arjan van de Ven wrote:
> On Sun, 18 Jan 2009 22:04:16 +0100
> Sam Ravnborg <sam@ravnborg.org> wrote:
[]
>> Can I ask you to redo this patch.
>> I want a way to swith off compression, but it should default be
>> enable. So if KBUILD_MODULE_NOCOMPRESS is undefined we will compress
>> the modules when we install them.
> 
> especially since the decompression is an option for the module tools,
> it should be an option for the kernel... 
> since it does add a dependency for the module tools (and one that you
> might not need; for example I disabled it just to remove a build
> requires)

By the way, I for one use busybox for initramfs stuff, since a long
time ago.  And it (still?) does not support decompression of kernel
modules, as of the latest (1.13) version...  I don't think I'm alone
using busybox this way, and it looks like decompression support
should be added to it first, and next to the kernel.. ;)

/mjt

Re: [PATCH] Compress kernel modules on installation.

[Jan Engelhardt]

On Sunday 2009-01-18 22:15, Michael Tokarev wrote:
>Arjan van de Ven wrote:
>> On Sun, 18 Jan 2009 22:04:16 +0100
>> Sam Ravnborg <sam@ravnborg.org> wrote:
>[]
>>> Can I ask you to redo this patch.
>>> I want a way to swith off compression, but it should default be
>>> enable. So if KBUILD_MODULE_NOCOMPRESS is undefined we will compress
>>> the modules when we install them.
>> 
>> especially since the decompression is an option for the module tools,
>> it should be an option for the kernel... 
>> since it does add a dependency for the module tools (and one that you
>> might not need; for example I disabled it just to remove a build
>> requires)
>
>By the way, I for one use busybox for initramfs stuff, since a long
>time ago.  And it (still?) does not support decompression of kernel
>modules, as of the latest (1.13) version...  I don't think I'm alone
>using busybox this way, and it looks like decompression support
>should be added to it first, and next to the kernel.. ;)

If we went by that metric, the kernel would still be in its infancy.

Re: [PATCH] Compress kernel modules on installation.

[Michael Tokarev]

Jan Engelhardt wrote:
>> []it looks like decompression support
>> should be added to it first, and next to the kernel.. ;)
> 
> If we went by that metric, the kernel would still be in its infancy.

Hey, note the smile at the end of my statement! :)

/mjt

Re: [PATCH] Compress kernel modules on installation.

[Jan Engelhardt]

On Sunday 2009-01-18 22:04, Sam Ravnborg wrote:
>
>Can I ask you to redo this patch.
>I want a way to swith off compression, but it should default be enable.
>So if KBUILD_MODULE_NOCOMPRESS is undefined we will compress the modules
>when we install them.
>
>And add documentation to Documentation/kbuild/kbuild.txt too...

changes to previous:
- added choice for compress (none, -1, -2, ..., -9)
- documentation

parent ad2507ab58aa5a3e50b0482889882ff3adf6ef33 (v2.6.29-rc2-24-gad2507a)
commit 3fc20a79496c3c5c8d4b906ea7e2496de448002d
Author: Jan Engelhardt <jengelh@medozas.de>
Date:   Mon Jan 19 21:18:39 2009 +0100

kbuild: install modules gzipped

This saves lots of disk space. Yes we had discussions about gzip -6
having a better compression-per-time ratio, but the point is that
people doing the compression in the first place often do not care
about the amount of compression time as long as the final filesize
will be smaller (that's what LZMA does over bzip2, and what bzip2
does over gzip). The KBUILD_MODULES_COMPRESS make variable can be set
over the command line to tune the compression level to your need,
including disabling it entirely.

Signed-off-by: Jan Engelhardt <jengelh@medozas.de>
---
 Documentation/kbuild/kbuild.txt |   11 +++++++++++
 scripts/Makefile.modinst        |    6 +++++-
 2 files changed, 16 insertions(+), 1 deletions(-)

diff --git a/Documentation/kbuild/kbuild.txt b/Documentation/kbuild/kbuild.txt
index 923f9dd..453ba6c 100644
--- a/Documentation/kbuild/kbuild.txt
+++ b/Documentation/kbuild/kbuild.txt
@@ -125,6 +125,17 @@ KBUILD_EXTRA_SYMBOLS
 For modules use symbols from another modules.
 See more details in modules.txt.
 
+KBUILD_MODULES_COMPRESS
+--------------------------------------------------
+By default, kbuild will compress modules with gzip -9 compression to
+reduce disk space. The KBUILD_MODULES_COMPRESS make parameter can be
+used to set an alternate compression level or disable compression
+entirely.
+Example for using the standard gzip compression level:
+	make KBUILD_MODULES_COMPRESS="-6"
+Disabling compression:
+	make KBUILD_MODULES_COMPRESS=""
+
 ALLSOURCE_ARCHS
 --------------------------------------------------
 For tags/TAGS/cscope targets, you can specify more than one archs
diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index 18b4bf8..e0708b1 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -16,9 +16,13 @@ PHONY += $(modules)
 __modinst: $(modules)
 	@:
 
+KBUILD_MODULES_COMPRESS  ?= -9
 quiet_cmd_modules_install = INSTALL $@
       cmd_modules_install = mkdir -p $(2); \
-			    $(mod_strip_cmd) $@ $(2)/$(notdir $@) || cp $@ $(2)
+			    $(mod_strip_cmd) $@ $(2)/$(notdir $@) || cp $@ $(2); \
+			    $(if ${KBUILD_MODULES_COMPRESS}, \
+			    	gzip -f ${KBUILD_MODULES_COMPRESS} \
+			    	$(2)/$(notdir $@), )
 
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
-- 
# Created with git-export-patch

Re: [PATCH] Compress kernel modules on installation.

[Steve Brokenshire]

On Fri, 26 Dec 2008 20:48:33 +0100
Sam Ravnborg <sam@ravnborg.org> wrote:

> On Fri, Dec 26, 2008 at 08:30:19PM +0100, Jan Engelhardt wrote:
> > 
> > On Tuesday 2008-02-26 12:28, Sam Ravnborg wrote:
> > >> 
> > >> (I've sent this to the linux-kbuild and linux-kernel lists as this
> > >> patch modifies the Makefile.modinst file. I also don't subscribe to the 
> > >> linux-kbuild and linux-kernel mailing lists so can I have any replies
> > >> CC'ed to me please)
> > >> 
> > >> This patch allows kernel modules to be compressed when 'make
> > >> modules_install' is run after being copied to
> > >> the /lib/module/<version>/<...> directory which is useful if you have
> > >> module-init-tools installed with --enable-zlib. This patch adds an
> > >> option (MODULE_COMPRESS) to the kernel configuration file (specifically
> > >> init/Kconfig) so that the kernel modules will compressed if
> > >> MODULE_COMPRESS is set.
> > 
> > I recently started compressing my kernel modules and that saved me
> > at least 70 MB of disk space on mostlyallmodconfig.
> > (And no, the argument of disks being cheap is not so true with
> > CF or SSD.)
> > Distro is lazy and wants to wait for upstream to have it,
> > so is there any chance of getting this proposal in?
> 
> Steve said he wanted to try to make the solution more
> scalable so I am awaiting a new patch.
> 
> 	Sam
> 

Yikes! I do apologise. I have been a little bit busy lately and that my
old PC decided to fail on the 24th December so I've been using that as
an excuse for panic buying, building and installing a 64-bit system but
I do have an updated patch available that is scalable (in that... if
new compression formats are used in the future they can be easily added
by adding text to init/Kconfig and code to Makefile.modinst).

I have noticed the patch from Jan for specifying settings for gzip. I
don't see why something like this can't be added to init/Kconfig...

config MODULE_COMPRESS_OPTIONS
	string "Compression method command line options"
	depends on MODULES
	help
	  This option specifies the command line options to be used for
	  the selected compression method.

	  Please refer to the selected compression method's documentation 
	  on which options should be used.

	  If unsure, leave this option blank.

And perhaps alter for Makefile.modinst (after applying the patch below)...

quiet_cmd_modules_compress_gzip = COMPRESS $@
      cmd_modules_compress_gzip = gzip $(CONFIG_MODULE_COMPRESS_OPTIONS) \
				 -c $(2)/`basename $@` \ 
				 > $(2)/`basename $@`.gz; \
				rm $(2)/`basename $@`

I'll supply the patch for what's been done so far (that and it's gone
3am here...) so the above makes a bit more sense. I should point out
that in the patch there should be an additional blank line after
endchoice for init/Kconfig but adding that blank line with the plus
sign seems to cause the patching process to fail so I had to take it
out and that this patch works with 2.6.27.4 (which is the kernel I am
currently using) with a slight fuzz and I haven't chance to test it
with the latest kernel.

I'll have a go at incorporating what I've said regarding
MODULE_COMPRESS_OPTIONS into another patch using the latest kernel (and
add the standard 'If unsure' message to MODULE_COMPRESS as well).

Steve

Patch Follows...

================================

Signed-off-by: Steve Brokenshire <sbrokenshire@xestia.co.uk>

--- a/scripts/Makefile.modinst	2008-02-11 05:51:11.000000000 +0000
+++ b/scripts/Makefile.modinst	2008-05-29 19:30:51.000000000 +0100
@@ -5,6 +5,7 @@
 PHONY := __modinst
 __modinst:
 
+include include/config/auto.conf
 include scripts/Kbuild.include
 
 #
@@ -16,8 +17,15 @@
 __modinst: $(modules)
 	@:
 
-quiet_cmd_modules_install = INSTALL $@
-      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) $(2)/$(notdir $@)
+quiet_cmd_modules_install = INSTALL  $@
+      cmd_modules_install = mkdir -p $(2); \
+				cp $@ $(2) ; \
+				$(mod_strip_cmd) $(2)/$(notdir $@)
+
+quiet_cmd_modules_compress_gzip = COMPRESS $@
+      cmd_modules_compress_gzip = gzip --best -c $(2)/`basename $@` \
+				 > $(2)/`basename $@`.gz; \
+				rm $(2)/`basename $@`
 
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
@@ -26,8 +34,11 @@
 modinst_dir = $(if $(KBUILD_EXTMOD),$(ext-mod-dir),kernel/$(@D))
 
 $(modules):
+
 	$(call cmd,modules_install,$(MODLIB)/$(modinst_dir))
 
+	$(if $(CONFIG_MODULE_COMPRESS_GZIP), \
+		$(call cmd,modules_compress_gzip,$(MODLIB)/$(modinst_dir)))
 
 # Declare the contents of the .PHONY variable as phony.  We keep that
 # information in a variable se we can use it in if_changed and friends.
--- a/init/Kconfig	2008-02-11 05:51:11.000000000 +0000
+++ b/init/Kconfig	2008-05-29 19:05:34.000000000 +0100
@@ -716,6 +716,45 @@
 	  rmmod).  This is mainly for kernel developers and desperate users.
 	  If unsure, say N.
 
+config MODULE_COMPRESS
+	bool "Compress kernel modules on installation"
+	depends on MODULES
+	help
+	  This option compresses the kernel modules when 'make
+	  modules_install' is run.
+
+	  The modules will be compressed into the selected compression 
+	  format with Gzip being the default compression format.
+
+	  When a kernel module is installed from outside of the main kernel
+	  source and uses the Kbuild system for installing modules then that
+	  kernel module will also be compressed when it is installed.
+
+	  When running mkinitrd you will find that an error message
+	  appears saying that it cannot find a certain kernel module.
+	  As a workaround, unset CONFIG_MODULE_COMPRESS, build the modules
+	  and install them, run mkinitrd and create the initrd image, place
+	  the initrd image in the correct place for booting, set
+	  CONFIG_MODULE_COMPRESS and then install the modules again.
+
+	  This options requires the module-init-tools package to be 
+	  configured with --enable-zlib (if using gzip which is the
+	  default compression method).
+
+choice
+	depends on MODULE_COMPRESS
+	prompt "Kernel module compression method"
+	default MODULE_COMPRESS_GZIP
+
+config MODULE_COMPRESS_GZIP
+	bool "Gzip compression"
+	help
+	  Compresses the kernel modules using the Gzip (GNU zip) 
+	  compression format.
+
+	  This option requires gzip to be installed.
+	  
+endchoice
 config MODVERSIONS
 	bool "Module versioning support"
 	depends on MODULES
--- a/Documentation/kbuild/modules.txt	2008-02-11 05:51:11.000000000 +0000
+++ b/Documentation/kbuild/modules.txt	2008-05-29 19:35:30.000000000 +0100
@@ -96,6 +96,11 @@
 		Installation default is in /lib/modules/<kernel-version>/extra,
 		but may be prefixed with INSTALL_MOD_PATH - see separate
 		chapter.
+		If MODULES_COMPRESS is set when the modules_install target is 
+		run then the module is compressed after it has been 
+		copied to /lib/modules/<kernel-version>. Compressed modules 
+		using the default Gzip compression format will require 
+		module-init-tools installed with --zlib-enabled.
 
 	make -C $KDIR M=`pwd` clean
 		Remove all generated files for the module - the kernel

Re: [PATCH] Compress kernel modules on installation.

[Steve Brokenshire]

On Tue, 20 Jan 2009 03:30:47 +0000
Steve Brokenshire <sbrokenshire@xestia.co.uk> wrote:

> On Fri, 26 Dec 2008 20:48:33 +0100
> Sam Ravnborg <sam@ravnborg.org> wrote:
> 
> > On Fri, Dec 26, 2008 at 08:30:19PM +0100, Jan Engelhardt wrote:
> > > 
> > > On Tuesday 2008-02-26 12:28, Sam Ravnborg wrote:
> > > >> 
> > > >> (I've sent this to the linux-kbuild and linux-kernel lists as this
> > > >> patch modifies the Makefile.modinst file. I also don't subscribe to the 
> > > >> linux-kbuild and linux-kernel mailing lists so can I have any replies
> > > >> CC'ed to me please)
> > > >> 
> > > >> This patch allows kernel modules to be compressed when 'make
> > > >> modules_install' is run after being copied to
> > > >> the /lib/module/<version>/<...> directory which is useful if you have
> > > >> module-init-tools installed with --enable-zlib. This patch adds an
> > > >> option (MODULE_COMPRESS) to the kernel configuration file (specifically
> > > >> init/Kconfig) so that the kernel modules will compressed if
> > > >> MODULE_COMPRESS is set.
> > > 
> > > I recently started compressing my kernel modules and that saved me
> > > at least 70 MB of disk space on mostlyallmodconfig.
> > > (And no, the argument of disks being cheap is not so true with
> > > CF or SSD.)
> > > Distro is lazy and wants to wait for upstream to have it,
> > > so is there any chance of getting this proposal in?
> > 
> > Steve said he wanted to try to make the solution more
> > scalable so I am awaiting a new patch.
> > 
> > 	Sam
> > 
> 
> (Stuff I mentioned earlier has been snipped)

I've done some modifications to the previous patch and that there is
now an option to specify the options for the selected compression
format in init/Kconfig using MODULE_COMPRESS_OPTIONS and also some
minor tidying up.

It didn't go quite as planned from the previous post for altering
Makefile.modinst as gzip doesn't like "" when there is nothing in
MODULE_COMPRESS_OPTIONS or when some options are set with the quotation
marks so when MODULE_COMPRESS_OPTIONS just has "" then nothing will
happen (as there would be no value in MODCOMPOPT) and if there is
something other than "" then strip the quotation marks from the start
and the end then place the result in MODCOMPOPT which is then used by
the selected compression method.

I think I've covered everything...

Steve


Patch follows...

================================
Signed-off-by: Steve Brokenshire <sbrokenshire@xestia.co.uk>

--- a/scripts/Makefile.modinst	2009-01-18 18:45:37.000000000 +0000
+++ b/scripts/Makefile.modinst	2009-01-25 13:35:53.000000000 +0000
@@ -5,6 +5,7 @@
 PHONY := __modinst
 __modinst:
 
+include include/config/auto.conf
 include scripts/Kbuild.include
 
 #
@@ -16,8 +17,21 @@ PHONY += $(modules)
 __modinst: $(modules)
 	@:
 
-quiet_cmd_modules_install = INSTALL $@
-      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) $(2)/$(notdir $@)
+ifeq ($(CONFIG_MODULE_COMPRESS_OPTIONS), "")
+else
+ MODCOMPOPT = $(shell echo -n $(CONFIG_MODULE_COMPRESS_OPTIONS))
+endif
+
+quiet_cmd_modules_install = INSTALL  $@
+      cmd_modules_install = mkdir -p $(2); \
+				cp $@ $(2) ; \
+				$(mod_strip_cmd) $(2)/$(notdir $@)
+
+quiet_cmd_modules_compress_gzip = COMPRESS $@
+      cmd_modules_compress_gzip = gzip $(MODCOMPOPT) -c \
+				 $(2)/`basename $@` \
+				 > $(2)/`basename $@`.gz; \
+				 rm $(2)/`basename $@`
 
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
@@ -26,8 +40,11 @@ ext-mod-dir = $(INSTALL_MOD_DIR)$(subst 
 modinst_dir = $(if $(KBUILD_EXTMOD),$(ext-mod-dir),kernel/$(@D))
 
 $(modules):
+
 	$(call cmd,modules_install,$(MODLIB)/$(modinst_dir))
 
+	$(if $(CONFIG_MODULE_COMPRESS_GZIP), \
+		$(call cmd,modules_compress_gzip,$(MODLIB)/$(modinst_dir)))
 
 # Declare the contents of the .PHONY variable as phony.  We keep that
 # information in a variable se we can use it in if_changed and friends.
--- a/init/Kconfig	2009-01-18 18:45:37.000000000 +0000
+++ b/init/Kconfig	2009-01-25 15:08:39.000000000 +0000
@@ -887,6 +887,62 @@ config MODULE_FORCE_UNLOAD
 	  rmmod).  This is mainly for kernel developers and desperate users.
 	  If unsure, say N.
 
+config MODULE_COMPRESS
+	bool "Compress kernel modules on installation"
+	depends on MODULES
+	help
+	  This option compresses the kernel modules when 'make
+	  modules_install' is run.
+
+	  The modules will be compressed into the selected compression 
+	  format with gzip being the default compression format.
+
+	  When a kernel module is installed from outside of the main kernel
+	  source and uses the Kbuild system for installing modules then that
+	  kernel module will also be compressed when it is installed.
+
+	  When running mkinitrd you will find that an error message
+	  appears saying that it cannot find a certain kernel module.
+	  As a workaround, unset CONFIG_MODULE_COMPRESS, build the modules
+	  and install them, run mkinitrd and create the initrd image, place
+	  the initrd image in the correct place for booting, set
+	  CONFIG_MODULE_COMPRESS and then install the modules again.
+
+	  This option requires the module-init-tools package to be 
+	  configured with --enable-zlib (if using gzip which is the
+	  default compression format).
+
+	  If unsure, say N.
+
+config MODULE_COMPRESS_OPTIONS
+	string "Compression format command line options"
+	depends on MODULE_COMPRESS
+	help
+	  This option specifies the command line options to be used for
+	  the selected compression format.
+
+	  Please refer to the selected compression format's documentation
+	  on which options should be used.
+
+	  If unsure, leave this option blank.
+
+choice
+	prompt "Kernel module compression format"
+	depends on MODULE_COMPRESS
+	default MODULE_COMPRESS_GZIP
+
+config MODULE_COMPRESS_GZIP
+	bool "gzip compression"
+	help
+	  Compresses the kernel modules using the gzip (GNU zip) 
+	  compression format.
+
+	  This option requires gzip to be installed.
+
+	  If unsure, leave this option selected.
+	  
+endchoice
+
 config MODVERSIONS
 	bool "Module versioning support"
 	help
--- a/Documentation/kbuild/modules.txt	2009-01-18 18:45:37.000000000 +0000
+++ b/Documentation/kbuild/modules.txt	2009-01-25 15:10:27.000000000 +0000
@@ -96,6 +96,13 @@ when building an external module.
 		Installation default is in /lib/modules/<kernel-version>/extra,
 		but may be prefixed with INSTALL_MOD_PATH - see separate
 		chapter.
+		If MODULES_COMPRESS is set when the modules_install target is 
+		run then the module is compressed after it has been 
+		copied to /lib/modules/<kernel-version>. Compressed modules 
+		using the default gzip compression format will require 
+		module-init-tools installed with --zlib-enabled.
+		Any options set in MODULE_COMPRESS_OPTIONS will be 
+		passed to the selected compression format.
 
 	make -C $KDIR M=`pwd` clean
 		Remove all generated files for the module - the kernel

Re: [PATCH] Compress kernel modules on installation.

[Jan Engelhardt]

On Sunday 2009-01-25 17:51, Steve Brokenshire wrote:
>+
>+quiet_cmd_modules_install = INSTALL  $@
>+      cmd_modules_install = mkdir -p $(2); \
>+				cp $@ $(2) ; \
>+				$(mod_strip_cmd) $(2)/$(notdir $@)
>+
>+quiet_cmd_modules_compress_gzip = COMPRESS $@
>+      cmd_modules_compress_gzip = gzip $(MODCOMPOPT) -c \
>+				 $(2)/`basename $@` \
>+				 > $(2)/`basename $@`.gz; \
>+				 rm $(2)/`basename $@`

That is *three* extra shell invocations you got there
for something make can compute itself using $(@F) iirc.

Re: [PATCH] Compress kernel modules on installation.

[Steve Brokenshire]

On Sun, 25 Jan 2009 19:02:21 +0100 (CET)
Jan Engelhardt <jengelh@medozas.de> wrote:

> 
> On Sunday 2009-01-25 17:51, Steve Brokenshire wrote:
> >+
> >+quiet_cmd_modules_install = INSTALL  $@
> >+      cmd_modules_install = mkdir -p $(2); \
> >+				cp $@ $(2) ; \
> >+				$(mod_strip_cmd) $(2)/$(notdir $@)
> >+
> >+quiet_cmd_modules_compress_gzip = COMPRESS $@
> >+      cmd_modules_compress_gzip = gzip $(MODCOMPOPT) -c \
> >+				 $(2)/`basename $@` \
> >+				 > $(2)/`basename $@`.gz; \
> >+				 rm $(2)/`basename $@`
> 
> That is *three* extra shell invocations you got there
> for something make can compute itself using $(@F) iirc.

Ahhh. Didn't know that. Thanks. :)

Steve

Full patch (with modifications) follows...

================================
Signed-off-by: Steve Brokenshire <sbrokenshire@xestia.co.uk>

--- a/scripts/Makefile.modinst	2009-01-18 18:45:37.000000000 +0000
+++ b/scripts/Makefile.modinst	2009-01-25 18:39:50.000000000 +0000
@@ -5,6 +5,7 @@
 PHONY := __modinst
 __modinst:
 
+include include/config/auto.conf
 include scripts/Kbuild.include
 
 #
@@ -16,8 +17,21 @@ PHONY += $(modules)
 __modinst: $(modules)
 	@:
 
-quiet_cmd_modules_install = INSTALL $@
-      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) $(2)/$(notdir $@)
+ifeq ($(CONFIG_MODULE_COMPRESS_OPTIONS), "")
+else
+ MODCOMPOPT = $(shell echo -n $(CONFIG_MODULE_COMPRESS_OPTIONS))
+endif
+
+quiet_cmd_modules_install = INSTALL  $@
+      cmd_modules_install = mkdir -p $(2); \
+				cp $@ $(2) ; \
+				$(mod_strip_cmd) $(2)/$(notdir $@)
+
+quiet_cmd_modules_compress_gzip = COMPRESS $@
+      cmd_modules_compress_gzip = gzip $(MODCOMPOPT) -c \
+				 $(2)/$(@F) \
+				 > $(2)/$(@F).gz; \
+				 rm $(2)/$(@F)
 
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
@@ -26,8 +40,11 @@ ext-mod-dir = $(INSTALL_MOD_DIR)$(subst 
 modinst_dir = $(if $(KBUILD_EXTMOD),$(ext-mod-dir),kernel/$(@D))
 
 $(modules):
+
 	$(call cmd,modules_install,$(MODLIB)/$(modinst_dir))
 
+	$(if $(CONFIG_MODULE_COMPRESS_GZIP), \
+		$(call cmd,modules_compress_gzip,$(MODLIB)/$(modinst_dir)))
 
 # Declare the contents of the .PHONY variable as phony.  We keep that
 # information in a variable se we can use it in if_changed and friends.
--- a/init/Kconfig	2009-01-18 18:45:37.000000000 +0000
+++ b/init/Kconfig	2009-01-25 16:07:47.000000000 +0000
@@ -887,6 +887,62 @@ config MODULE_FORCE_UNLOAD
 	  rmmod).  This is mainly for kernel developers and desperate users.
 	  If unsure, say N.
 
+config MODULE_COMPRESS
+	bool "Compress kernel modules on installation"
+	depends on MODULES
+	help
+	  This option compresses the kernel modules when 'make
+	  modules_install' is run.
+
+	  The modules will be compressed into the selected compression 
+	  format with gzip being the default compression format.
+
+	  When a kernel module is installed from outside of the main kernel
+	  source and uses the Kbuild system for installing modules then that
+	  kernel module will also be compressed when it is installed.
+
+	  When running mkinitrd you will find that an error message
+	  appears saying that it cannot find a certain kernel module.
+	  As a workaround, unset CONFIG_MODULE_COMPRESS, build the modules
+	  and install them, run mkinitrd and create the initrd image, place
+	  the initrd image in the correct place for booting, set
+	  CONFIG_MODULE_COMPRESS and then install the modules again.
+
+	  This option requires the module-init-tools package to be 
+	  configured with --enable-zlib (if using gzip which is the
+	  default compression format).
+
+	  If unsure, say N.
+
+config MODULE_COMPRESS_OPTIONS
+	string "Compression format command line options"
+	depends on MODULE_COMPRESS
+	help
+	  This option specifies the command line options to be used for
+	  the selected compression format.
+
+	  Please refer to the selected compression format's documentation
+	  on which options should be used.
+
+	  If unsure, leave this option blank.
+
+choice
+	prompt "Kernel module compression format"
+	depends on MODULE_COMPRESS
+	default MODULE_COMPRESS_GZIP
+
+config MODULE_COMPRESS_GZIP
+	bool "gzip compression"
+	help
+	  Compresses the kernel modules using the gzip (GNU zip) 
+	  compression format.
+
+	  This option requires gzip to be installed.
+
+	  If unsure, leave this option selected.
+	  
+endchoice
+
 config MODVERSIONS
 	bool "Module versioning support"
 	help
--- a/Documentation/kbuild/modules.txt	2009-01-18 18:45:37.000000000 +0000
+++ b/Documentation/kbuild/modules.txt	2009-01-25 16:07:47.000000000 +0000
@@ -96,6 +96,13 @@ when building an external module.
 		Installation default is in /lib/modules/<kernel-version>/extra,
 		but may be prefixed with INSTALL_MOD_PATH - see separate
 		chapter.
+		If MODULES_COMPRESS is set when the modules_install target is 
+		run then the module is compressed after it has been 
+		copied to /lib/modules/<kernel-version>. Compressed modules 
+		using the default gzip compression format will require 
+		module-init-tools installed with --zlib-enabled.
+		Any options set in MODULE_COMPRESS_OPTIONS will be 
+		passed to the selected compression format.
 
 	make -C $KDIR M=`pwd` clean
 		Remove all generated files for the module - the kernel

Re: [PATCH] Compress kernel modules on installation

[Willy Tarreau]

On Fri, Dec 26, 2008 at 02:02:37PM -0800, david@lang.hm wrote:
> On Fri, 26 Dec 2008, Roland wrote:
> 
> >what about some "make modules_install_compressed" instead ?
> >
> >as i have run out of diskspace quite often when installing test kernels, i 
> >think we really need a feature like this.
> >
> >i`d also favour the makefile approach.
> >why another kconfig option?
> >
> >jan`s patch looks clean and simple, but i think it`s a little bit 
> >intrusive...
> 
> by default the main kernel is compressed, why should the modules not be 
> compressed by default as well?

I've always compressed my modules for years, and I can say there is *one*
situation where I uncompress them, it's before making an initrd or initramfs,
because compressing an FS with many compressed files leads to a fairly bigger
image than leaving them uncompressed on the compressed FS.

Willy

Re: [PATCH] Compress kernel modules on installation

[david]

On Sat, 27 Dec 2008, Jan Engelhardt wrote:

> On Friday 2008-12-26 23:28, david@lang.hm wrote:
>>>>> ( see
>>>>> http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/080/8051/8051f1.png
>>>>> )
>>>>
>>>> this varies a lot on the particulars of the data being compressed.
>>>
>>> see my other reply.
>>
>> IIRC the default is 5
>
> default is 6 (man gzip).
>
>> instead of doing the compression when doing the modules_install do it when they
>> are compiled (just like we do for the kernel itself). if you just spent the cpu
>> to build the thing, the cpu to compress it isn't much.
>
> I'd rather do the compression later on - or do you want
> to spend gzipping all day when doing `make foo.ko`
> as a developer? ;-)
>
>>> $ find . -iname "*.gz" | xargs du -cs -B 4096 | grep total
>>> 7086    total
>>> $ echo $[7086*4096]
>>> 29024256
>>> $ echo $[29024256-24782942]
>>> 4241314
>>>
>>> So that's like 4M going off for the block stuff. Given that
>>> the compression actually saved about 50 MB, I think we
>>> can live with the 4M - especially since they have been
>>> there already one way or another.
>>
>> if I'm reading the numbers correctly that's about 16% of the space taken by the
>> final result
>
> Assuming that the x in (filesize % 4096 == x) is randomly distributed
> over 0..4095 for gzip modules, I proclaim that it is similarly
> randomly distributed for noncompressed modules. As such, the overhead
> is always there and gzipping does not make it worse or better.
> So you always have the overhead.
>
> uncompressed:
> $ du as above
> 113360896
> $ byte counted:
> 109270606
>
> differnece: 4.09 MB
>
> You always pay that price for having single files.
> So I dismiss it from the discussion ;-)

the point I was making is that if we are changing things to add new ways 
to get the module contents does it make sense to add another wrinkle that 
would allow you to eliminate the single files

  one pain point/upgrade of tools to take advantage of it instead of doing 
the first step and then coming back and do the second one.

David Lang

Re: [PATCH] Compress kernel modules on installation

[Roland]

> >>> what about some "make modules_install_compressed" instead ?
> >>>
> >>> as i have run out of diskspace quite often when installing test
> >>> kernels, i
> >>> think we really need a feature like this.
> >>>
> >>> i`d also favour the makefile approach.
> >>> why another kconfig option?
> >>>
> >>> jan`s patch looks clean and simple, but i think it`s a little bit
> >>> intrusive...
> >>
> >> Why so? module-ini
> >
> > with "intrusive" i meant, that all modules are now compressed by default
> > and there is no switch to build them uncompressed.
> > so you change a long established default which may not be welcomed by
> > everyone and give no option for conservative people.
> >
> How many people use the option to install an uncompressed kernel?


i don`t know, probable few to none on pc/server hardware, but things may 
look different in embedded sector...


btw, here is some real-world data on compression and block-alignment:


kernel 2.6.27.8

                               (du -sk)      (du -sk --apparent-size)
uncompressed          72140         66554
gzip                          29988         25447
gzip -9                      29892         25374

roland

Re: [PATCH] Compress kernel modules on installation

[Bill Davidsen]

Roland wrote:
>> On Friday 2008-12-26 21:23, Roland wrote:
>>
>>> what about some "make modules_install_compressed" instead ?
>>>
>>> as i have run out of diskspace quite often when installing test 
>>> kernels, i
>>> think we really need a feature like this.
>>>
>>> i`d also favour the makefile approach.
>>> why another kconfig option?
>>>
>>> jan`s patch looks clean and simple, but i think it`s a little bit 
>>> intrusive...
>>
>> Why so? module-ini
> 
> with "intrusive" i meant, that all modules are now compressed by default 
> and there is no switch to build them uncompressed.
> so you change a long established default which may not be welcomed by 
> everyone and give no option for conservative people.
> 
How many people use the option to install an uncompressed kernel?

> we should take care and not break things.
> default uncompressed, compressed implented as an option is a better 
> strategy, imho.
> discuss changing the default at a later time then.
> 

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: [PATCH] Compress kernel modules on installation

[Jan Engelhardt]

On Friday 2008-12-26 23:28, david@lang.hm wrote:
>>>> ( see
>>>> http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/080/8051/8051f1.png
>>>> )
>>>
>>> this varies a lot on the particulars of the data being compressed.
>>
>> see my other reply.
>
> IIRC the default is 5

default is 6 (man gzip).

> instead of doing the compression when doing the modules_install do it when they
> are compiled (just like we do for the kernel itself). if you just spent the cpu
> to build the thing, the cpu to compress it isn't much.

I'd rather do the compression later on - or do you want
to spend gzipping all day when doing `make foo.ko`
as a developer? ;-)

>> $ find . -iname "*.gz" | xargs du -cs -B 4096 | grep total
>> 7086    total
>> $ echo $[7086*4096]
>> 29024256
>> $ echo $[29024256-24782942]
>> 4241314
>>
>> So that's like 4M going off for the block stuff. Given that
>> the compression actually saved about 50 MB, I think we
>> can live with the 4M - especially since they have been
>> there already one way or another.
>
> if I'm reading the numbers correctly that's about 16% of the space taken by the
> final result

Assuming that the x in (filesize % 4096 == x) is randomly distributed
over 0..4095 for gzip modules, I proclaim that it is similarly
randomly distributed for noncompressed modules. As such, the overhead
is always there and gzipping does not make it worse or better.
So you always have the overhead.

uncompressed:
$ du as above
113360896
$ byte counted:
109270606

differnece: 4.09 MB

You always pay that price for having single files.
So I dismiss it from the discussion ;-)

Re: [PATCH] Compress kernel modules on installation

[Roland]

> On Friday 2008-12-26 21:23, Roland wrote:
>
>> what about some "make modules_install_compressed" instead ?
>>
>> as i have run out of diskspace quite often when installing test kernels, 
>> i
>> think we really need a feature like this.
>>
>> i`d also favour the makefile approach.
>> why another kconfig option?
>>
>> jan`s patch looks clean and simple, but i think it`s a little bit 
>> intrusive...
>
> Why so? module-ini

with "intrusive" i meant, that all modules are now compressed by default and 
there is no switch to build them uncompressed.
so you change a long established default which may not be welcomed by 
everyone and give no option for conservative people.

we should take care and not break things.
default uncompressed, compressed implented as an option is a better 
strategy, imho.
discuss changing the default at a later time then.

roland

>>
>> regards
>> roland
>>
>> ps:
>> i`d use gzip without "-9" as this gives very little space savings. it 
>> mostly
>> burns cpu and slows things down too much.
>> ( see
>> http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/080/8051/8051f1.png
>
> Here is one that is tailored for kernel modules:
> http://tinyurl.com/8f2xta
> Looks pretty much like the one from LJ but of course, .ko files
> compress better than LJ :)
> Oh well. 

Re: [PATCH] Compress kernel modules on installation

[david]

On Fri, 26 Dec 2008, Jan Engelhardt wrote:

> On Friday 2008-12-26 23:02, david@lang.hm wrote:
>>> ( see
>>> http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/080/8051/8051f1.png
>>> )
>>
>> this varies a lot on the particulars of the data being compressed.
>
> see my other reply.

IIRC the default is 5, from the graph you posted going to 6 or 7 would be 
a fairly significant win (final size to final size) without eathing _that_ 
much more cpu.

instead of doing the compression when doing the modules_install do it when 
they are compiled (just like we do for the kernel itself). if you just 
spent the cpu to build the thing, the cpu to compress it isn't much.

>> however, I do need to ask what the sizes of the resulting files end up being
>> (on my system the last kernel that used modules was 2.4.31, so I can't just
>> look this up myself). in most cases disks are formatted with 4K blocks, so
>> unless it shrinks across such a boundry it's not actually going to help much.
>
> Hey, don't forget exotic filesystems that pack that together anyway.
> And if I fired up du right, then...
>
> $ find . -iname "*.gz" | xargs du -cs -B 4096 | grep total
> 7086    total
> $ echo $[7086*4096]
> 29024256
> $ echo $[29024256-24782942]
> 4241314
>
> So that's like 4M going off for the block stuff. Given that
> the compression actually saved about 50 MB, I think we
> can live with the 4M - especially since they have been
> there already one way or another.

if I'm reading the numbers correctly that's about 16% of the space taken 
by the final result

>> one other thing to consider would be some ability to load the
>> module out of a tar or cpio bundle, that way you end up saving all
>> the partial blocks from each module as well.
>
> tarfs, it's somewhere around the corner on the fuse pages probably.

watch out for a deadlock here. the more external tools that you need to 
get at the kernel modules, the more likely it is that one of those tools 
will need something that you can't get to without the modules loaded.

David Lang

Re: [PATCH] Compress kernel modules on installation

[david]

On Fri, 26 Dec 2008, Roland wrote:

> what about some "make modules_install_compressed" instead ?
>
> as i have run out of diskspace quite often when installing test kernels, i 
> think we really need a feature like this.
>
> i`d also favour the makefile approach.
> why another kconfig option?
>
> jan`s patch looks clean and simple, but i think it`s a little bit 
> intrusive...

by default the main kernel is compressed, why should the modules not be 
compressed by default as well?

> regards
> roland
>
> ps:
> i`d use gzip without "-9" as this gives very little space savings. it mostly 
> burns cpu and slows things down too much.
> ( see 
> http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/080/8051/8051f1.png 
> )

this varies a lot on the particulars of the data being compressed. I don't 
know what difference it makes on the kernel modules, but I have seen it be 
worth it in some other cases.

however, I do need to ask what the sizes of the resulting files end up 
being (on my system the last kernel that used modules was 2.4.31, so I 
can't just look this up myself). in most cases disks are formatted with 4K 
blocks, so unless it shrinks across such a boundry it's not actually going 
to help much.

one other thing to consider would be some ability to load the module out 
of a tar or cpio bundle, that way you end up saving all the partial blocks 
from each module as well.

David Lang

>
> On Friday 2008-12-26 20:48, Sam Ravnborg wrote:
>> > > >
>> > > > This patch allows kernel modules to be compressed when 'make
>> > > > modules_install' is run after being copied to
>> > > > the /lib/module/<version>/<...> directory which is useful if you > > 
>> > have
>> > > > module-init-tools installed with --enable-zlib. This patch adds an
>> > > > option (MODULE_COMPRESS) to the kernel configuration file > > > 
>> (specifically
>> > > > init/Kconfig) so that the kernel modules will compressed if
>> > > > MODULE_COMPRESS is set.
>> >
>> > I recently started compressing my kernel modules and that saved me
>> > at least 70 MB of disk space on mostlyallmodconfig.
>> > (And no, the argument of disks being cheap is not so true with
>> > CF or SSD.)
>> > Distro is lazy and wants to wait for upstream to have it,
>> > so is there any chance of getting this proposal in?
>> 
>> Steve said he wanted to try to make the solution more
>> scalable so I am awaiting a new patch.
>
> Hm, all I needed was this patch. It might fire up some people,
> but it's got all the scalability I could think of..
>
>
> commit b4a3e1c610c99d4e8b543b97fd722076c6f7c5dd
> Author: Jan Engelhardt <jengelh@medozas.de>
> Date:   Wed Dec 10 20:39:21 2008 +0100
>
> build: install modules compressed
> ---
> scripts/Makefile.modinst |    2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
>
> diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
> index efa5d94..c3421a1 100644
> --- a/scripts/Makefile.modinst
> +++ b/scripts/Makefile.modinst
> @@ -17,7 +17,7 @@ __modinst: $(modules)
> @:
>
> quiet_cmd_modules_install = INSTALL $@
> -      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) \
> $(2)/$(notdir $@) +      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; \
> $(mod_strip_cmd) $(2)/$(notdir $@); gzip -9f $(2)/$(notdir $@)
> # Modules built outside the kernel source tree go into extra by default
> INSTALL_MOD_DIR ?= extra
> --
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: [PATCH] Compress kernel modules on installation

[Jan Engelhardt]

On Friday 2008-12-26 23:02, david@lang.hm wrote:
>> ( see
>> http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/080/8051/8051f1.png
>> )
>
> this varies a lot on the particulars of the data being compressed.

see my other reply.

> however, I do need to ask what the sizes of the resulting files end up being
> (on my system the last kernel that used modules was 2.4.31, so I can't just
> look this up myself). in most cases disks are formatted with 4K blocks, so
> unless it shrinks across such a boundry it's not actually going to help much.

Hey, don't forget exotic filesystems that pack that together anyway.
And if I fired up du right, then...

$ find . -iname "*.gz" | xargs du -cs -B 4096 | grep total
7086    total
$ echo $[7086*4096]
29024256
$ echo $[29024256-24782942]
4241314

So that's like 4M going off for the block stuff. Given that
the compression actually saved about 50 MB, I think we
can live with the 4M - especially since they have been
there already one way or another.

> one other thing to consider would be some ability to load the
> module out of a tar or cpio bundle, that way you end up saving all
> the partial blocks from each module as well.

tarfs, it's somewhere around the corner on the fuse pages probably.

Re: [PATCH] Compress kernel modules on installation

[Jan Engelhardt]

On Friday 2008-12-26 21:23, Roland wrote:

> what about some "make modules_install_compressed" instead ?
>
> as i have run out of diskspace quite often when installing test kernels, i
> think we really need a feature like this.
>
> i`d also favour the makefile approach.
> why another kconfig option?
>
> jan`s patch looks clean and simple, but i think it`s a little bit intrusive...

Why so? module-ini

>
> regards
> roland
>
> ps:
> i`d use gzip without "-9" as this gives very little space savings. it mostly
> burns cpu and slows things down too much.
> ( see
> http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/080/8051/8051f1.png

Here is one that is tailored for kernel modules:
http://tinyurl.com/8f2xta
Looks pretty much like the one from LJ but of course, .ko files
compress better than LJ :)
Oh well.

Re: [PATCH] Compress kernel modules on installation

[Roland]

what about some "make modules_install_compressed" instead ?

as i have run out of diskspace quite often when installing test kernels, i 
think we really need a feature like this.

i`d also favour the makefile approach.
why another kconfig option?

jan`s patch looks clean and simple, but i think it`s a little bit 
intrusive...

regards
roland

ps:
i`d use gzip without "-9" as this gives very little space savings. it mostly 
burns cpu and slows things down too much.
( see 
http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/080/8051/8051f1.png )


On Friday 2008-12-26 20:48, Sam Ravnborg wrote:
> > > >
> > > > This patch allows kernel modules to be compressed when 'make
> > > > modules_install' is run after being copied to
> > > > the /lib/module/<version>/<...> directory which is useful if you 
> > > > have
> > > > module-init-tools installed with --enable-zlib. This patch adds an
> > > > option (MODULE_COMPRESS) to the kernel configuration file 
> > > > (specifically
> > > > init/Kconfig) so that the kernel modules will compressed if
> > > > MODULE_COMPRESS is set.
> >
> > I recently started compressing my kernel modules and that saved me
> > at least 70 MB of disk space on mostlyallmodconfig.
> > (And no, the argument of disks being cheap is not so true with
> > CF or SSD.)
> > Distro is lazy and wants to wait for upstream to have it,
> > so is there any chance of getting this proposal in?
>
> Steve said he wanted to try to make the solution more
> scalable so I am awaiting a new patch.

Hm, all I needed was this patch. It might fire up some people,
but it's got all the scalability I could think of..


commit b4a3e1c610c99d4e8b543b97fd722076c6f7c5dd
Author: Jan Engelhardt <jengelh@medozas.de>
Date:   Wed Dec 10 20:39:21 2008 +0100

build: install modules compressed
---
 scripts/Makefile.modinst |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index efa5d94..c3421a1 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -17,7 +17,7 @@ __modinst: $(modules)
  @:

 quiet_cmd_modules_install = INSTALL $@
-      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) \
$(2)/$(notdir $@) +      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; \
$(mod_strip_cmd) $(2)/$(notdir $@); gzip -9f $(2)/$(notdir $@)
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
--