Uncompressed kernel doesn't build on x86_64

Uncompressed kernel doesn't build on x86_64

[Pavel Roskin]

Hi Christian,

Your recent patch enables uncompressed kernels, but selecting that
option (CONFIG_KERNEL_UNCOMPRESSED) fails on x86_86:

commit 69f0554ec261fd686ac7fa1c598cc9eb27b83a80
Author: Christian Ruppert <christian.ruppert@abilis.com>
Date:   Tue Nov 12 15:11:43 2013 -0800

    init/Kconfig: add option to disable kernel compression

That's how it fails:

  LD      init/built-in.o
  KSYM    .tmp_kallsyms1.o
  KSYM    .tmp_kallsyms2.o
  LD      vmlinux
  SORTEX  vmlinux
  SYSMAP  System.map
  VOFFSET arch/x86/boot/voffset.h
make[4]: *** No rule to make target
  `arch/x86/boot/compressed/vmlinux.bin.', needed by
  `arch/x86/boot/compressed/piggy.S'.  Stop. make[3]: ***
  [arch/x86/boot/compressed/vmlinux] Error 2 make[2]: *** [bzImage]
  Error 2 make[1]: *** [sub-make] Error 2 make: *** [all] Error 2

arch/x86/boot/compressed/Makefile has no provision for uncompressed
files.  There is no definition for suffix-$(CONFIG_KERNEL_UNCOMPRESSED)
so $(suffix-y) evaluates to an empty string and there is no rule to
make vmlinuz.bin.$(suffix-y)

I think the patch should be either reverted or fixed as soon as
possible.

Any suggestions for uncompressed suffix?  I'd go with "copy".  Then the
"compression" could be a symlink.  Alternatively, the "." could be
included in other suffixes, the uncompressed suffix would be empty and
the compression would be an empty statement.

-- 
Regards,
Pavel Roskin

Re: Uncompressed kernel doesn't build on x86_64

[H. Peter Anvin]

On 11/13/2013 08:34 AM, Pavel Roskin wrote:
> 
> Any suggestions for uncompressed suffix?  I'd go with "copy".  Then the
> "compression" could be a symlink.  Alternatively, the "." could be
> included in other suffixes, the uncompressed suffix would be empty and
> the compression would be an empty statement.
> 

The latter would be my preference.

	-hpa

Re: Uncompressed kernel doesn't build on x86_64

[Christian Ruppert]

On Wed, Nov 13, 2013 at 11:34:18AM -0500, Pavel Roskin wrote:
> Hi Christian,
> 
> Your recent patch enables uncompressed kernels, but selecting that
> option (CONFIG_KERNEL_UNCOMPRESSED) fails on x86_86:
> [...]
> arch/x86/boot/compressed/Makefile has no provision for uncompressed
> files.  There is no definition for suffix-$(CONFIG_KERNEL_UNCOMPRESSED)
> so $(suffix-y) evaluates to an empty string and there is no rule to
> make vmlinuz.bin.$(suffix-y)
> 
> I think the patch should be either reverted or fixed as soon as
> possible.

This is an interesting observation, thanks. Although this patch is
rather intended for embedded architectures than x86, it highlights
potential issues with self-decompressing images of all sorts. I have no
way of testing all architectures and so I'm sending two patches in the
follow-up:
1. A conservative version of the original patch which requires
   architectures to explicitly enable uncompresed kernel images if they
   support compression as well and falls back to previous behaviour if
   nothing is specified. I have copied Vineet Gupta (maintainer of the
   ARC platform) since I enabled GZIP and uncompressed kernels for ARC
   by default.
2. A patch to enable uncompressed x86 kernels. As stated above, I don't
   think this makes a lot of sense in itself but it might serve as an
   example for people working on other platforms with self-extracting
   kernels and the nozip not-decompression algorithm might be useful on
   those platforms as well. I only had a single x86-64 machine available
   to test this, however, so some more testing might be required.

> Any suggestions for uncompressed suffix?  I'd go with "copy".  Then the
> "compression" could be a symlink.  Alternatively, the "." could be
> included in other suffixes, the uncompressed suffix would be empty and
> the compression would be an empty statement.

I followed H. Peter's suggestion and chose option 2 in patch 2.

Greetings,
  Christian

[PATCH 1/2] init/Kconfig: add option to disable kernel compression

[Christian Ruppert]

Some ARC users say they can boot faster with without kernel compression.
This probably depends on things like the FLASH chip they use etc.

Until now, kernel compression can only be disabled by removing "select
HAVE_<compression>" lines from the architecture Kconfig.  So add the
Kconfig logic to permit disabling of kernel compression.

Signed-off-by: Christian Ruppert <christian.ruppert@abilis.com>
---
 arch/arc/Kconfig |  2 ++
 init/Kconfig     | 12 +++++++++++-
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/arch/arc/Kconfig b/arch/arc/Kconfig
index 91dbb27..3991f03 100644
--- a/arch/arc/Kconfig
+++ b/arch/arc/Kconfig
@@ -21,6 +21,8 @@ config ARC
 	select HAVE_ARCH_KGDB
 	select HAVE_ARCH_TRACEHOOK
 	select HAVE_IOREMAP_PROT
+	select HAVE_KERNEL_UNCOMPRESSED
+	select HAVE_KERNEL_GZIP
 	select HAVE_KPROBES
 	select HAVE_KRETPROBES
 	select HAVE_MEMBLOCK
diff --git a/init/Kconfig b/init/Kconfig
index 3ecd8a1..b1a6f92 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -97,6 +97,9 @@ config LOCALVERSION_AUTO
 
 	  which is done within the script "scripts/setlocalversion".)
 
+config HAVE_KERNEL_UNCOMPRESSED
+        bool
+
 config HAVE_KERNEL_GZIP
 	bool
 
@@ -118,7 +121,6 @@ config HAVE_KERNEL_LZ4
 choice
 	prompt "Kernel compression mode"
 	default KERNEL_GZIP
-	depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4
 	help
 	  The linux kernel is a kind of self-extracting executable.
 	  Several compression algorithms are available, which differ
@@ -137,6 +139,14 @@ choice
 
 	  If in doubt, select 'gzip'
 
+config KERNEL_UNCOMPRESSED
+	bool "No compression"
+	depends on HAVE_KERNEL_UNCOMPRESSED || ! ( HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4 )
+	help
+	  No compression at all. The kernel is huge but the compression and
+	  decompression times are zero.
+	  This is usually not what you want.
+
 config KERNEL_GZIP
 	bool "Gzip"
 	depends on HAVE_KERNEL_GZIP
-- 
1.8.4

Re: [PATCH 1/2] init/Kconfig: add option to disable kernel compression

[Vineet Gupta]

+CC Sam for Kconfig wisdom

On 11/14/2013 02:08 PM, Christian Ruppert wrote:
> Some ARC users say they can boot faster with without kernel compression.
> This probably depends on things like the FLASH chip they use etc.
> 
> Until now, kernel compression can only be disabled by removing "select
> HAVE_<compression>" lines from the architecture Kconfig.  So add the
> Kconfig logic to permit disabling of kernel compression.
> 
> Signed-off-by: Christian Ruppert <christian.ruppert@abilis.com>
> ---
>  arch/arc/Kconfig |  2 ++
>  init/Kconfig     | 12 +++++++++++-
>  2 files changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arc/Kconfig b/arch/arc/Kconfig
> index 91dbb27..3991f03 100644
> --- a/arch/arc/Kconfig
> +++ b/arch/arc/Kconfig
> @@ -21,6 +21,8 @@ config ARC
>  	select HAVE_ARCH_KGDB
>  	select HAVE_ARCH_TRACEHOOK
>  	select HAVE_IOREMAP_PROT
> +	select HAVE_KERNEL_UNCOMPRESSED
> +	select HAVE_KERNEL_GZIP

Fine.

>  	select HAVE_KPROBES
>  	select HAVE_KRETPROBES
>  	select HAVE_MEMBLOCK
> diff --git a/init/Kconfig b/init/Kconfig
> index 3ecd8a1..b1a6f92 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -97,6 +97,9 @@ config LOCALVERSION_AUTO
>  
>  	  which is done within the script "scripts/setlocalversion".)
>  
> +config HAVE_KERNEL_UNCOMPRESSED
> +        bool
> +

This is good to avoid perturbing other arches.

>  config HAVE_KERNEL_GZIP
>  	bool
>  
> @@ -118,7 +121,6 @@ config HAVE_KERNEL_LZ4
>  choice
>  	prompt "Kernel compression mode"
>  	default KERNEL_GZIP
> -	depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4
>  	help
>  	  The linux kernel is a kind of self-extracting executable.
>  	  Several compression algorithms are available, which differ
> @@ -137,6 +139,14 @@ choice
>  
>  	  If in doubt, select 'gzip'
>  
> +config KERNEL_UNCOMPRESSED
> +	bool "No compression"
> +	depends on HAVE_KERNEL_UNCOMPRESSED || ! ( HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4 )
> +	help
> +	  No compression at all. The kernel is huge but the compression and
> +	  decompression times are zero.
> +	  This is usually not what you want.
> +
>  config KERNEL_GZIP
>  	bool "Gzip"
>  	depends on HAVE_KERNEL_GZIP
> 

How about doing this part slightly differently (simpler IMO).
We add uncompressed as just another category rather than being a special case.

Indicative diff against current mainline code

 choice
        prompt "Kernel compression mode"
        default KERNEL_GZIP
-       depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA ||
HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4
+       depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA ||
HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4 || HAVE_KERNEL_UNCOMPRESSED


+config KERNEL_UNCOMPRESSED
+       bool "No compression"
+       depends on HAVE_KERNEL_UNCOMPRESSED
+

-Vineet

Re: [PATCH 1/2] init/Kconfig: add option to disable kernel compression

[Christian Ruppert]

On Thu, Nov 14, 2013 at 03:51:22PM +0530, Vineet Gupta wrote:
> +CC Sam for Kconfig wisdom
> 
> On 11/14/2013 02:08 PM, Christian Ruppert wrote:
> > Some ARC users say they can boot faster with without kernel compression.
> > This probably depends on things like the FLASH chip they use etc.
> > 
> > Until now, kernel compression can only be disabled by removing "select
> > HAVE_<compression>" lines from the architecture Kconfig.  So add the
> > Kconfig logic to permit disabling of kernel compression.
> > 
> > Signed-off-by: Christian Ruppert <christian.ruppert@abilis.com>
> > ---
> >  arch/arc/Kconfig |  2 ++
> >  init/Kconfig     | 12 +++++++++++-
> >  2 files changed, 13 insertions(+), 1 deletion(-)
> > 
> > diff --git a/arch/arc/Kconfig b/arch/arc/Kconfig
> > index 91dbb27..3991f03 100644
> > --- a/arch/arc/Kconfig
> > +++ b/arch/arc/Kconfig
> > @@ -21,6 +21,8 @@ config ARC
> >  	select HAVE_ARCH_KGDB
> >  	select HAVE_ARCH_TRACEHOOK
> >  	select HAVE_IOREMAP_PROT
> > +	select HAVE_KERNEL_UNCOMPRESSED
> > +	select HAVE_KERNEL_GZIP
> 
> Fine.
> 
> >  	select HAVE_KPROBES
> >  	select HAVE_KRETPROBES
> >  	select HAVE_MEMBLOCK
> > diff --git a/init/Kconfig b/init/Kconfig
> > index 3ecd8a1..b1a6f92 100644
> > --- a/init/Kconfig
> > +++ b/init/Kconfig
> > @@ -97,6 +97,9 @@ config LOCALVERSION_AUTO
> >  
> >  	  which is done within the script "scripts/setlocalversion".)
> >  
> > +config HAVE_KERNEL_UNCOMPRESSED
> > +        bool
> > +
> 
> This is good to avoid perturbing other arches.
> 
> >  config HAVE_KERNEL_GZIP
> >  	bool
> >  
> > @@ -118,7 +121,6 @@ config HAVE_KERNEL_LZ4
> >  choice
> >  	prompt "Kernel compression mode"
> >  	default KERNEL_GZIP
> > -	depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4
> >  	help
> >  	  The linux kernel is a kind of self-extracting executable.
> >  	  Several compression algorithms are available, which differ
> > @@ -137,6 +139,14 @@ choice
> >  
> >  	  If in doubt, select 'gzip'
> >  
> > +config KERNEL_UNCOMPRESSED
> > +	bool "No compression"
> > +	depends on HAVE_KERNEL_UNCOMPRESSED || ! ( HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4 )
> > +	help
> > +	  No compression at all. The kernel is huge but the compression and
> > +	  decompression times are zero.
> > +	  This is usually not what you want.
> > +
> >  config KERNEL_GZIP
> >  	bool "Gzip"
> >  	depends on HAVE_KERNEL_GZIP
> > 
> 
> How about doing this part slightly differently (simpler IMO).
> We add uncompressed as just another category rather than being a special case.
> 
> Indicative diff against current mainline code
> 
>  choice
>         prompt "Kernel compression mode"
>         default KERNEL_GZIP
> -       depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA ||
> HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4
> +       depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA ||
> HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4 || HAVE_KERNEL_UNCOMPRESSED
> 
> 
> +config KERNEL_UNCOMPRESSED
> +       bool "No compression"
> +       depends on HAVE_KERNEL_UNCOMPRESSED
> +

Good Idea. I fixed this and sent the updated patch directly in reply to
H. Peter's revert patch to avoid too much patch ping-pong:
http://www.spinics.net/lists/kernel/msg1636397.html

Greetings,
  Christian

Re: [PATCH 1/2] init/Kconfig: add option to disable kernel compression

[Vineet Gupta]

On 11/15/2013 10:27 PM, Christian Ruppert wrote:
> On Thu, Nov 14, 2013 at 03:51:22PM +0530, Vineet Gupta wrote:
>> +CC Sam for Kconfig wisdom
>>
>> On 11/14/2013 02:08 PM, Christian Ruppert wrote:
>>> Some ARC users say they can boot faster with without kernel compression.
>>> This probably depends on things like the FLASH chip they use etc.
>>>
>>> Until now, kernel compression can only be disabled by removing "select
>>> HAVE_<compression>" lines from the architecture Kconfig.  So add the
>>> Kconfig logic to permit disabling of kernel compression.
>>>
>>> Signed-off-by: Christian Ruppert <christian.ruppert@abilis.com>
>>> ---
>>>  arch/arc/Kconfig |  2 ++
>>>  init/Kconfig     | 12 +++++++++++-
>>>  2 files changed, 13 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/arch/arc/Kconfig b/arch/arc/Kconfig
>>> index 91dbb27..3991f03 100644
>>> --- a/arch/arc/Kconfig
>>> +++ b/arch/arc/Kconfig
>>> @@ -21,6 +21,8 @@ config ARC
>>>  	select HAVE_ARCH_KGDB
>>>  	select HAVE_ARCH_TRACEHOOK
>>>  	select HAVE_IOREMAP_PROT
>>> +	select HAVE_KERNEL_UNCOMPRESSED
>>> +	select HAVE_KERNEL_GZIP
>>
>> Fine.
>>
>>>  	select HAVE_KPROBES
>>>  	select HAVE_KRETPROBES
>>>  	select HAVE_MEMBLOCK
>>> diff --git a/init/Kconfig b/init/Kconfig
>>> index 3ecd8a1..b1a6f92 100644
>>> --- a/init/Kconfig
>>> +++ b/init/Kconfig
>>> @@ -97,6 +97,9 @@ config LOCALVERSION_AUTO
>>>  
>>>  	  which is done within the script "scripts/setlocalversion".)
>>>  
>>> +config HAVE_KERNEL_UNCOMPRESSED
>>> +        bool
>>> +
>>
>> This is good to avoid perturbing other arches.
>>
>>>  config HAVE_KERNEL_GZIP
>>>  	bool
>>>  
>>> @@ -118,7 +121,6 @@ config HAVE_KERNEL_LZ4
>>>  choice
>>>  	prompt "Kernel compression mode"
>>>  	default KERNEL_GZIP
>>> -	depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4
>>>  	help
>>>  	  The linux kernel is a kind of self-extracting executable.
>>>  	  Several compression algorithms are available, which differ
>>> @@ -137,6 +139,14 @@ choice
>>>  
>>>  	  If in doubt, select 'gzip'
>>>  
>>> +config KERNEL_UNCOMPRESSED
>>> +	bool "No compression"
>>> +	depends on HAVE_KERNEL_UNCOMPRESSED || ! ( HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4 )
>>> +	help
>>> +	  No compression at all. The kernel is huge but the compression and
>>> +	  decompression times are zero.
>>> +	  This is usually not what you want.
>>> +
>>>  config KERNEL_GZIP
>>>  	bool "Gzip"
>>>  	depends on HAVE_KERNEL_GZIP
>>>
>>
>> How about doing this part slightly differently (simpler IMO).
>> We add uncompressed as just another category rather than being a special case.
>>
>> Indicative diff against current mainline code
>>
>>  choice
>>         prompt "Kernel compression mode"
>>         default KERNEL_GZIP
>> -       depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA ||
>> HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4
>> +       depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA ||
>> HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4 || HAVE_KERNEL_UNCOMPRESSED
>>
>>
>> +config KERNEL_UNCOMPRESSED
>> +       bool "No compression"
>> +       depends on HAVE_KERNEL_UNCOMPRESSED
>> +
> 
> Good Idea. I fixed this and sent the updated patch directly in reply to
> H. Peter's revert patch to avoid too much patch ping-pong:
> http://www.spinics.net/lists/kernel/msg1636397.html

I didn't see that patch in my mailbox. It needs an ACK from me anyways.

-Vineet

[PATCH 2/2] x86: Add support for uncompressed kernel images

[Christian Ruppert]

Signed-off-by: Christian Ruppert <christian.ruppert@abilis.com>
---
 arch/x86/Kconfig                  |  1 +
 arch/x86/boot/compressed/Makefile | 14 ++++++------
 arch/x86/boot/compressed/misc.c   |  4 ++++
 lib/decompress_copy.c             | 47 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 59 insertions(+), 7 deletions(-)
 create mode 100644 lib/decompress_copy.c

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index f67e839..2a32df9 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -61,6 +61,7 @@ config X86
 	select USER_STACKTRACE_SUPPORT
 	select HAVE_REGS_AND_STACK_ACCESS_API
 	select HAVE_DMA_API_DEBUG
+	select HAVE_KERNEL_UNCOMPRESSED
 	select HAVE_KERNEL_GZIP
 	select HAVE_KERNEL_BZIP2
 	select HAVE_KERNEL_LZMA
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index dcd90df..f65e444 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -67,16 +67,16 @@ $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE
 $(obj)/vmlinux.bin.lz4: $(vmlinux.bin.all-y) FORCE
 	$(call if_changed,lz4)
 
-suffix-$(CONFIG_KERNEL_GZIP)	:= gz
-suffix-$(CONFIG_KERNEL_BZIP2)	:= bz2
-suffix-$(CONFIG_KERNEL_LZMA)	:= lzma
-suffix-$(CONFIG_KERNEL_XZ)	:= xz
-suffix-$(CONFIG_KERNEL_LZO) 	:= lzo
-suffix-$(CONFIG_KERNEL_LZ4) 	:= lz4
+suffix-$(CONFIG_KERNEL_GZIP)	:= .gz
+suffix-$(CONFIG_KERNEL_BZIP2)	:= .bz2
+suffix-$(CONFIG_KERNEL_LZMA)	:= .lzma
+suffix-$(CONFIG_KERNEL_XZ)	:= .xz
+suffix-$(CONFIG_KERNEL_LZO) 	:= .lzo
+suffix-$(CONFIG_KERNEL_LZ4) 	:= .lz4
 
 quiet_cmd_mkpiggy = MKPIGGY $@
       cmd_mkpiggy = $(obj)/mkpiggy $< > $@ || ( rm -f $@ ; false )
 
 targets += piggy.S
-$(obj)/piggy.S: $(obj)/vmlinux.bin.$(suffix-y) $(obj)/mkpiggy FORCE
+$(obj)/piggy.S: $(obj)/vmlinux.bin$(suffix-y) $(obj)/mkpiggy FORCE
 	$(call if_changed,mkpiggy)
diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c
index 434f077..c210314 100644
--- a/arch/x86/boot/compressed/misc.c
+++ b/arch/x86/boot/compressed/misc.c
@@ -149,6 +149,10 @@ static int lines, cols;
 #include "../../../../lib/decompress_unlz4.c"
 #endif
 
+#ifdef CONFIG_KERNEL_UNCOMPRESSED
+#include "../../../../lib/decompress_copy.c"
+#endif
+
 static void scroll(void)
 {
 	int i;
diff --git a/lib/decompress_copy.c b/lib/decompress_copy.c
new file mode 100644
index 0000000..8a41090
--- /dev/null
+++ b/lib/decompress_copy.c
@@ -0,0 +1,47 @@
+#include <linux/decompress/mm.h>
+
+#define NOZIP_BUFSZ (16 * 1024)
+STATIC int INIT nozip(unsigned char *buf, int len,
+			int(*fill)(void*, unsigned int),
+			int(*flush)(void*, unsigned int),
+			unsigned char *outbuf,
+			int *pos,
+			void(*error)(char *x))
+{
+	char *b;
+
+	if (buf)
+		b = buf;
+	else
+		b = malloc(NOZIP_BUFSZ);
+
+	if (!b) {
+		error("Out of memory while allocating buffer");
+		return -1;
+	}
+
+	if (flush) {
+		if (!len)
+			len = fill(b, NOZIP_BUFSZ);
+
+		len = flush(b, len);
+	} else {
+		if (!len)
+			len = fill(outbuf, NOZIP_BUFSZ);
+		else {
+			int i;
+			for (i = 0; i < len; i++)
+				outbuf[i] = b[i];
+		}
+	}
+
+	if (pos)
+		*pos = len;
+
+	if (!buf)
+		free(b);
+
+	return 0;
+}
+
+#define decompress nozip
-- 
1.8.4

Re: [PATCH 2/2] x86: Add support for uncompressed kernel images

[H. Peter Anvin]

So it sounds like we're starting from the beginning - I presume that
means the patch currently in the kernel should be reverted first?

	-hpa

Re: [PATCH 2/2] x86: Add support for uncompressed kernel images

[Christian Ruppert]

On Thu, Nov 14, 2013 at 09:31:44AM -0800, H. Peter Anvin wrote:
> So it sounds like we're starting from the beginning - I presume that
> means the patch currently in the kernel should be reverted first?

Ack

Re: [PATCH 2/2] x86: Add support for uncompressed kernel images

[H. Peter Anvin]

On 11/14/2013 12:38 AM, Christian Ruppert wrote:
> diff --git a/lib/decompress_copy.c b/lib/decompress_copy.c
> new file mode 100644
> index 0000000..8a41090
> --- /dev/null
> +++ b/lib/decompress_copy.c
> @@ -0,0 +1,47 @@
> +#include <linux/decompress/mm.h>
> +
> +#define NOZIP_BUFSZ (16 * 1024)
> +STATIC int INIT nozip(unsigned char *buf, int len,
> +			int(*fill)(void*, unsigned int),
> +			int(*flush)(void*, unsigned int),
> +			unsigned char *outbuf,
> +			int *pos,
> +			void(*error)(char *x))
> +{
> +	char *b;
> +
> +	if (buf)
> +		b = buf;
> +	else
> +		b = malloc(NOZIP_BUFSZ);
> +
> +	if (!b) {
> +		error("Out of memory while allocating buffer");
> +		return -1;
> +	}
> +
> +	if (flush) {
> +		if (!len)
> +			len = fill(b, NOZIP_BUFSZ);
> +
> +		len = flush(b, len);
> +	} else {
> +		if (!len)
> +			len = fill(outbuf, NOZIP_BUFSZ);
> +		else {
> +			int i;
> +			for (i = 0; i < len; i++)
> +				outbuf[i] = b[i];

Clever way of doing memcpy().  I know some other compression
implementations open-code memmove(), but using it for the entire data
set in the most common case is atrocious.  If we need to get a default
implementation of memmove() we should just do that.

	-hpa

Re: [PATCH 2/2] x86: Add support for uncompressed kernel images

[Christian Ruppert]

On Thu, Nov 14, 2013 at 09:45:12AM -0800, H. Peter Anvin wrote:
> On 11/14/2013 12:38 AM, Christian Ruppert wrote:
> > diff --git a/lib/decompress_copy.c b/lib/decompress_copy.c
> > new file mode 100644
> > index 0000000..8a41090
> > --- /dev/null
> > +++ b/lib/decompress_copy.c
> > @@ -0,0 +1,47 @@
> > +#include <linux/decompress/mm.h>
> > +
> > +#define NOZIP_BUFSZ (16 * 1024)
> > +STATIC int INIT nozip(unsigned char *buf, int len,
> > +			int(*fill)(void*, unsigned int),
> > +			int(*flush)(void*, unsigned int),
> > +			unsigned char *outbuf,
> > +			int *pos,
> > +			void(*error)(char *x))
> > +{
> > +	char *b;
> > +
> > +	if (buf)
> > +		b = buf;
> > +	else
> > +		b = malloc(NOZIP_BUFSZ);
> > +
> > +	if (!b) {
> > +		error("Out of memory while allocating buffer");
> > +		return -1;
> > +	}
> > +
> > +	if (flush) {
> > +		if (!len)
> > +			len = fill(b, NOZIP_BUFSZ);
> > +
> > +		len = flush(b, len);
> > +	} else {
> > +		if (!len)
> > +			len = fill(outbuf, NOZIP_BUFSZ);
> > +		else {
> > +			int i;
> > +			for (i = 0; i < len; i++)
> > +				outbuf[i] = b[i];
> 
> Clever way of doing memcpy().  I know some other compression
> implementations open-code memmove(), but using it for the entire data
> set in the most common case is atrocious.  If we need to get a default
> implementation of memmove() we should just do that.

Actually, the x86 architecture already contains a basic implementation
of memcpy in its self-decompressor. Correct me if I'm wrong but I guess
memmove is not required since I don't expect the output of a generic
decompression function to overlap with its input.

We can simply use that function if we agree that
. either nozip is x86 specific
. or every architecture using this code must bring along its own memcpy
  implementation in its self-decompressor.

Do you think that would be OK?

Greetings,
  Christian

Re: [PATCH 2/2] x86: Add support for uncompressed kernel images

[H. Peter Anvin]

On 11/15/2013 01:49 AM, Christian Ruppert wrote:
> 
> Actually, the x86 architecture already contains a basic implementation
> of memcpy in its self-decompressor. Correct me if I'm wrong but I guess
> memmove is not required since I don't expect the output of a generic
> decompression function to overlap with its input.
> 
> We can simply use that function if we agree that
> . either nozip is x86 specific
> . or every architecture using this code must bring along its own memcpy
>   implementation in its self-decompressor.
> 
> Do you think that would be OK?
> 

I think all of them have memcpy().

	-hpa

Re: Uncompressed kernel doesn't build on x86_64

[Austin S Hemmelgarn]

On 2013-11-14 03:32, Christian Ruppert wrote:
> 2. A patch to enable uncompressed x86 kernels. As stated above, I don't
>    think this makes a lot of sense in itself but it might serve as an
>    example for people working on other platforms with self-extracting
>    kernels and the nozip not-decompression algorithm might be useful on
>    those platforms as well. I only had a single x86-64 machine available
>    to test this, however, so some more testing might be required.

I disagree with the argument that an uncompressed x86 kernel doesn't
make sense, If you have a very fast boot device, then it is fully
conceivable that an uncompressed kernel could boot faster than a
compressed one.  I have seen a very large number of systems where the
LZO compression boots at least twice as fast as gzip or bz2 (because the
disks are fast enough that a few megabytes of size difference make much
less of an impact than a slow decompressor).

Re: Uncompressed kernel doesn't build on x86_64

[Pavel Roskin]

On Thu, 14 Nov 2013 09:13:44 -0500
Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:

> On 2013-11-14 03:32, Christian Ruppert wrote:
> > 2. A patch to enable uncompressed x86 kernels. As stated above, I
> > don't think this makes a lot of sense in itself but it might serve
> > as an example for people working on other platforms with
> > self-extracting kernels and the nozip not-decompression algorithm
> > might be useful on those platforms as well. I only had a single
> > x86-64 machine available to test this, however, so some more
> > testing might be required.
> 
> I disagree with the argument that an uncompressed x86 kernel doesn't
> make sense, If you have a very fast boot device, then it is fully
> conceivable that an uncompressed kernel could boot faster than a
> compressed one.  I have seen a very large number of systems where the
> LZO compression boots at least twice as fast as gzip or bz2 (because
> the disks are fast enough that a few megabytes of size difference
> make much less of an impact than a slow decompressor).

I concur, the uncompressed kernel certainly makes sense in some cases.

If the kernel runs in an emulator, decompression could be slow.

If the kernel runs in a virtual machine, the kernel would need to be
decompressed separately for every virtual machine.  Reading from the
disk would be done only once if several virtual machines are started in
a short period of time.

-- 
Regards,
Pavel Roskin

Re: [PATCH] [FIX] init/Kconfig: fix option to disable kernel compression

[Christian Ruppert]

On Sat, Nov 16, 2013 at 03:12:56PM +0530, Vineet Gupta wrote:
> On 11/15/2013 10:21 PM, Christian Ruppert wrote:
> > Some architectures with self-decompressing kernel images did not compile
> > with commit 69f0554ec261fd686ac7fa1c598cc9eb27b83a80 because they don't
> > provide a non-decompression mechanism for uncompressed kernels.
> > 
> > Rectify this problem by allowing uncompressed kernels only for architectures
> > which explicitly state they support them.
> > 
> > Signed-off-by: Christian Ruppert <christian.ruppert@abilis.com>
> Acked-by: Vineet Gupta <vgupta@synopsys.com>

Looks like the revert was applied to main line (and not this fix). In
the follow-up the complete (fixed) patch set rebased on the latest main
line: First patch for Kconfig/ARC and as requested a cleaned up second
patch for x86. I could only test the second patch on one physical
machine and one VM so some more testing might be required on x86.

Greetings,
  Christian

[PATCH 2/2] x86: Add support for uncompressed kernel images

[Christian Ruppert]

Signed-off-by: Christian Ruppert <christian.ruppert@abilis.com>
---
 arch/x86/Kconfig                  |    1 +
 arch/x86/boot/compressed/Makefile |   14 ++++++------
 arch/x86/boot/compressed/misc.c   |    4 +++
 lib/decompress_copy.c             |   44 +++++++++++++++++++++++++++++++++++++
 4 files changed, 56 insertions(+), 7 deletions(-)
 create mode 100644 lib/decompress_copy.c

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index e903c71..009349f 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -62,6 +62,7 @@ config X86
 	select USER_STACKTRACE_SUPPORT
 	select HAVE_REGS_AND_STACK_ACCESS_API
 	select HAVE_DMA_API_DEBUG
+	select HAVE_KERNEL_UNCOMPRESSED
 	select HAVE_KERNEL_GZIP
 	select HAVE_KERNEL_BZIP2
 	select HAVE_KERNEL_LZMA
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index dcd90df..f65e444 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -67,16 +67,16 @@ $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE
 $(obj)/vmlinux.bin.lz4: $(vmlinux.bin.all-y) FORCE
 	$(call if_changed,lz4)
 
-suffix-$(CONFIG_KERNEL_GZIP)	:= gz
-suffix-$(CONFIG_KERNEL_BZIP2)	:= bz2
-suffix-$(CONFIG_KERNEL_LZMA)	:= lzma
-suffix-$(CONFIG_KERNEL_XZ)	:= xz
-suffix-$(CONFIG_KERNEL_LZO) 	:= lzo
-suffix-$(CONFIG_KERNEL_LZ4) 	:= lz4
+suffix-$(CONFIG_KERNEL_GZIP)	:= .gz
+suffix-$(CONFIG_KERNEL_BZIP2)	:= .bz2
+suffix-$(CONFIG_KERNEL_LZMA)	:= .lzma
+suffix-$(CONFIG_KERNEL_XZ)	:= .xz
+suffix-$(CONFIG_KERNEL_LZO) 	:= .lzo
+suffix-$(CONFIG_KERNEL_LZ4) 	:= .lz4
 
 quiet_cmd_mkpiggy = MKPIGGY $@
       cmd_mkpiggy = $(obj)/mkpiggy $< > $@ || ( rm -f $@ ; false )
 
 targets += piggy.S
-$(obj)/piggy.S: $(obj)/vmlinux.bin.$(suffix-y) $(obj)/mkpiggy FORCE
+$(obj)/piggy.S: $(obj)/vmlinux.bin$(suffix-y) $(obj)/mkpiggy FORCE
 	$(call if_changed,mkpiggy)
diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c
index 434f077..c210314 100644
--- a/arch/x86/boot/compressed/misc.c
+++ b/arch/x86/boot/compressed/misc.c
@@ -149,6 +149,10 @@ static int lines, cols;
 #include "../../../../lib/decompress_unlz4.c"
 #endif
 
+#ifdef CONFIG_KERNEL_UNCOMPRESSED
+#include "../../../../lib/decompress_copy.c"
+#endif
+
 static void scroll(void)
 {
 	int i;
diff --git a/lib/decompress_copy.c b/lib/decompress_copy.c
new file mode 100644
index 0000000..109ef22
--- /dev/null
+++ b/lib/decompress_copy.c
@@ -0,0 +1,44 @@
+#include <linux/decompress/mm.h>
+
+#define NOZIP_BUFSZ (16 * 1024)
+STATIC int INIT nozip(unsigned char *buf, int len,
+			int(*fill)(void*, unsigned int),
+			int(*flush)(void*, unsigned int),
+			unsigned char *outbuf,
+			int *pos,
+			void(*error)(char *x))
+{
+	char *b;
+
+	if (buf)
+		b = buf;
+	else
+		b = malloc(NOZIP_BUFSZ);
+
+	if (!b) {
+		error("Out of memory while allocating buffer");
+		return -1;
+	}
+
+	if (flush) {
+		if (!len)
+			len = fill(b, NOZIP_BUFSZ);
+
+		len = flush(b, len);
+	} else {
+		if (!len)
+			len = fill(outbuf, NOZIP_BUFSZ);
+		else
+			memcpy(outbuf, b, len);
+	}
+
+	if (pos)
+		*pos = len;
+
+	if (!buf)
+		free(b);
+
+	return 0;
+}
+
+#define decompress nozip
-- 
1.7.1

Re: [PATCH 2/2] x86: Add support for uncompressed kernel images

[Borislav Petkov]

On Mon, Nov 18, 2013 at 10:51:51AM +0100, Christian Ruppert wrote:

How about a commit message ... somewhere around here would be nice, for
example :)

> Signed-off-by: Christian Ruppert <christian.ruppert@abilis.com>
> ---
>  arch/x86/Kconfig                  |    1 +
>  arch/x86/boot/compressed/Makefile |   14 ++++++------
>  arch/x86/boot/compressed/misc.c   |    4 +++
>  lib/decompress_copy.c             |   44 +++++++++++++++++++++++++++++++++++++
>  4 files changed, 56 insertions(+), 7 deletions(-)
>  create mode 100644 lib/decompress_copy.c

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--