[PATCH] arm: Add support for ZSTD compressed kernel

[PATCH] arm: Add support for ZSTD compressed kernel

[Peter Geis]

Good Evening,

I had attempted to get this working as well, but have run into
difficulties with both my implementation and yours as well.
My implementation was almost the same as yours, with the exception of
also changing:
@@ -303,7 +305,7 @@ restart: adr r0, LC1

 #ifndef CONFIG_ZBOOT_ROM
  /* malloc space is above the relocated stack (64k max) */
- add r10, sp, #0x10000
+ add r10, sp, #0x30000
 #else
  /*
  * With ZBOOT_ROM the bss/stack is non relocatable,

On QEMU this implementation works fine.
However on bare metal tegra30, I get the following error:

Jumping to kernel at:4861 ms

C:0x80A000C0-0x8112BA40->0x8152C700-0x81C58080
Uncompressing Linux...

ZSTD-compressed dstSize is too small

 -- System halted

The only difference between the bare metal test and the qemu test is
the zImage with appended dtb is packaged in the android boot format
for the bare metal test.
Otherwise it's exactly the same file.

I had to modify the original zstd error message because it grouped
several errors together.
Here is my patch for that:

diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c
index 062617bb0afe..89ac73e900ce 100644
--- a/lib/decompress_unzstd.c
+++ b/lib/decompress_unzstd.c
@@ -103,10 +103,14 @@ static int INIT handle_zstd_error(size_t ret,
void (*error)(char *x))
  error("Input is not in the ZSTD format (wrong magic bytes)");
  break;
  case ZSTD_error_dstSize_tooSmall:
+ error("ZSTD-compressed dstSize is too small");
+ break;
  case ZSTD_error_corruption_detected:
- case ZSTD_error_checksum_wrong:
  error("ZSTD-compressed data is corrupt");
  break;
+ case ZSTD_error_checksum_wrong:
+ error("ZSTD-compressed data checksum is wrong");
+ break;
  default:
  error("ZSTD-compressed data is probably corrupt");
  break;

Very Respectfully,
Peter Geis

Re: [PATCH] arm: Add support for ZSTD compressed kernel

[Russell King - ARM Linux admin]

On Thu, Aug 06, 2020 at 05:46:50PM -0400, Peter Geis wrote:
> Good Evening,
> 
> I had attempted to get this working as well, but have run into
> difficulties with both my implementation and yours as well.
> My implementation was almost the same as yours, with the exception of
> also changing:
> @@ -303,7 +305,7 @@ restart: adr r0, LC1
> 
>  #ifndef CONFIG_ZBOOT_ROM
>   /* malloc space is above the relocated stack (64k max) */
> - add r10, sp, #0x10000
> + add r10, sp, #0x30000
>  #else
>   /*
>   * With ZBOOT_ROM the bss/stack is non relocatable,
> 
> On QEMU this implementation works fine.
> However on bare metal tegra30, I get the following error:
> 
> Jumping to kernel at:4861 ms
> 
> C:0x80A000C0-0x8112BA40->0x8152C700-0x81C58080
> Uncompressing Linux...
> 
> ZSTD-compressed dstSize is too small
> 
>  -- System halted
> 
> The only difference between the bare metal test and the qemu test is
> the zImage with appended dtb is packaged in the android boot format
> for the bare metal test.
> Otherwise it's exactly the same file.

So it's relocating the compressed kernel and decompressor from
0x80A000C0-0x8112BA40 to 0x8152C700-0x81C58080 and then failing.
Does the QEMU version also do similar?

On the off-hand, I'm not sure why it should fail.  I assume that
you've tried the other decompressors and they work fine on the
same setups?

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: [PATCH] arm: Add support for ZSTD compressed kernel

[Peter Geis]

On Thu, Aug 6, 2020 at 6:22 PM Russell King - ARM Linux admin
<linux@armlinux.org.uk> wrote:
>
> On Thu, Aug 06, 2020 at 05:46:50PM -0400, Peter Geis wrote:
> > Good Evening,
> >
> > I had attempted to get this working as well, but have run into
> > difficulties with both my implementation and yours as well.
> > My implementation was almost the same as yours, with the exception of
> > also changing:
> > @@ -303,7 +305,7 @@ restart: adr r0, LC1
> >
> >  #ifndef CONFIG_ZBOOT_ROM
> >   /* malloc space is above the relocated stack (64k max) */
> > - add r10, sp, #0x10000
> > + add r10, sp, #0x30000
> >  #else
> >   /*
> >   * With ZBOOT_ROM the bss/stack is non relocatable,
> >
> > On QEMU this implementation works fine.
> > However on bare metal tegra30, I get the following error:
> >
> > Jumping to kernel at:4861 ms
> >
> > C:0x80A000C0-0x8112BA40->0x8152C700-0x81C58080
> > Uncompressing Linux...
> >
> > ZSTD-compressed dstSize is too small
> >
> >  -- System halted
> >
> > The only difference between the bare metal test and the qemu test is
> > the zImage with appended dtb is packaged in the android boot format
> > for the bare metal test.
> > Otherwise it's exactly the same file.
>
> So it's relocating the compressed kernel and decompressor from
> 0x80A000C0-0x8112BA40 to 0x8152C700-0x81C58080 and then failing.
> Does the QEMU version also do similar?

Here is the output from QEMU, note boot doesn't work because this
image isn't for QEMU:

C:0x400100C0-0x4073B1E0->0x4152C600-0x41C57720
Uncompressing Linux... done, booting the kernel.

>
> On the off-hand, I'm not sure why it should fail.  I assume that
> you've tried the other decompressors and they work fine on the
> same setups?

Correct, all other compressors work.
ZSTD is handy for arm because size and speed are both important.

>
> --
> RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
> FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: [PATCH] arm: Add support for ZSTD compressed kernel

[Russell King - ARM Linux admin]

On Thu, Aug 06, 2020 at 10:45:42AM +0200, Norbert Lange wrote:
> Am Do., 6. Aug. 2020 um 02:30 Uhr schrieb Russell King - ARM Linux
> admin <linux@armlinux.org.uk>:
> >
> > On Thu, Aug 06, 2020 at 01:05:55AM +0200, Norbert Lange wrote:
> > > diff --git a/arch/arm/boot/compressed/head.S b/arch/arm/boot/compressed/head.S
> > > index 434a16982e34..1af01bfe6638 100644
> > > --- a/arch/arm/boot/compressed/head.S
> > > +++ b/arch/arm/boot/compressed/head.S
> > > @@ -614,7 +614,11 @@ not_relocated:   mov     r0, #0
> > >   */
> > >               mov     r0, r4
> > >               mov     r1, sp                  @ malloc space above stack
> > > +#if defined(CONFIG_KERNEL_ZSTD)
> > > +             add     r2, sp, #0x30000        @ Context needs ~160K
> >
> > That's going to mess up kexec:
> >
> >         /*
> >          * The zImage length does not include its stack (4k) or its
> >          * malloc space (64k).  Include this.
> >          */
> >         len += 0x11000;
> >
> > I guess we need to add this to the information provided to kexec.
> 
> Ouch, I guess it's rather impossible to load a new ZSTD kernel with a old
> kexec version in that case.
> 
> Some ideas to fix that would be:
> 
> -   Increase the padding to "192K + 4K is enough for everyone" in kexec
> -   Add the required/recommended stack+heap size to zImage so kexec can use it.

This is definitely preferred.  See patches - though this is only the
kernel side.  The first I've had for some time and isn't related to
this issue.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: [PATCH] arm: Add support for ZSTD compressed kernel

[Norbert Lange]

Am Do., 6. Aug. 2020 um 02:30 Uhr schrieb Russell King - ARM Linux
admin <linux@armlinux.org.uk>:
>
> On Thu, Aug 06, 2020 at 01:05:55AM +0200, Norbert Lange wrote:
> > diff --git a/arch/arm/boot/compressed/head.S b/arch/arm/boot/compressed/head.S
> > index 434a16982e34..1af01bfe6638 100644
> > --- a/arch/arm/boot/compressed/head.S
> > +++ b/arch/arm/boot/compressed/head.S
> > @@ -614,7 +614,11 @@ not_relocated:   mov     r0, #0
> >   */
> >               mov     r0, r4
> >               mov     r1, sp                  @ malloc space above stack
> > +#if defined(CONFIG_KERNEL_ZSTD)
> > +             add     r2, sp, #0x30000        @ Context needs ~160K
>
> That's going to mess up kexec:
>
>         /*
>          * The zImage length does not include its stack (4k) or its
>          * malloc space (64k).  Include this.
>          */
>         len += 0x11000;
>
> I guess we need to add this to the information provided to kexec.

Ouch, I guess it's rather impossible to load a new ZSTD kernel with a old
kexec version in that case.

Some ideas to fix that would be:

-   Increase the padding to "192K + 4K is enough for everyone" in kexec
-   Add the required/recommended stack+heap size to zImage so kexec can use it.
-   change boot logic to place heap before/after the *output* range.
    with the various ways of booting arm this will likely cause other
issues in some combinations.
-   try to reduce zstd context size safetly below 64K.
    I believe the entropy tables are computed once and never modified,
so those could be part of the zImage
    (at some loss of compression rate of course). Would need some host
tool to inject those.

Norbert

Re: [PATCH] arm: Add support for ZSTD compressed kernel

[Russell King - ARM Linux admin]

On Thu, Aug 06, 2020 at 01:05:55AM +0200, Norbert Lange wrote:
> diff --git a/arch/arm/boot/compressed/head.S b/arch/arm/boot/compressed/head.S
> index 434a16982e34..1af01bfe6638 100644
> --- a/arch/arm/boot/compressed/head.S
> +++ b/arch/arm/boot/compressed/head.S
> @@ -614,7 +614,11 @@ not_relocated:	mov	r0, #0
>   */
>  		mov	r0, r4
>  		mov	r1, sp			@ malloc space above stack
> +#if defined(CONFIG_KERNEL_ZSTD)
> +		add	r2, sp, #0x30000	@ Context needs ~160K

That's going to mess up kexec:

        /*
         * The zImage length does not include its stack (4k) or its
         * malloc space (64k).  Include this.
         */
        len += 0x11000;

I guess we need to add this to the information provided to kexec.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

[PATCH] arm: Add support for ZSTD compressed kernel

[Norbert Lange]

* Add support for zstd compressed kernel
* Define __DISABLE_EXPORTS in Makefile
* Bump the heap size for zstd.

This replicates the steps taken for x86 support.

Only if ZSTD is selected, heap size for kernel decompression is
increased, this should guarantee no changes in code / sideeffects for
other compressors.

The patch is so far only tested with Qemu, and some 64MB size limit
exists (might be Qemu specific), but this seems not caused by
this patch (upstream 5.8 and GZIP exhibits similar problems).

Signed-off-by: Norbert Lange <nolange79@gmail.com>
---
 arch/arm/Kconfig                      | 1 +
 arch/arm/boot/compressed/Makefile     | 2 ++
 arch/arm/boot/compressed/decompress.c | 4 ++++
 arch/arm/boot/compressed/head.S       | 4 ++++
 4 files changed, 11 insertions(+)

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 2ac74904a3ce..3e853cdc688d 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -93,6 +93,7 @@ config ARM
 	select HAVE_KERNEL_LZMA
 	select HAVE_KERNEL_LZO
 	select HAVE_KERNEL_XZ
+	select HAVE_KERNEL_ZSTD
 	select HAVE_KPROBES if !XIP_KERNEL && !CPU_ENDIAN_BE32 && !CPU_V7M
 	select HAVE_KRETPROBES if HAVE_KPROBES
 	select HAVE_MOD_ARCH_SPECIFIC
diff --git a/arch/arm/boot/compressed/Makefile b/arch/arm/boot/compressed/Makefile
index 00602a6fba04..61229e93ea4a 100644
--- a/arch/arm/boot/compressed/Makefile
+++ b/arch/arm/boot/compressed/Makefile
@@ -75,6 +75,7 @@ compress-$(CONFIG_KERNEL_LZO)  = lzo
 compress-$(CONFIG_KERNEL_LZMA) = lzma
 compress-$(CONFIG_KERNEL_XZ)   = xzkern
 compress-$(CONFIG_KERNEL_LZ4)  = lz4
+compress-$(CONFIG_KERNEL_ZSTD) = zstd22
 
 libfdt_objs := fdt_rw.o fdt_ro.o fdt_wip.o fdt.o
 
@@ -102,6 +103,7 @@ targets       := vmlinux vmlinux.lds piggy_data piggy.o \
 clean-files += piggy_data lib1funcs.S ashldi3.S bswapsdi2.S hyp-stub.S
 
 KBUILD_CFLAGS += -DDISABLE_BRANCH_PROFILING
+KBUILD_CFLAGS += -D__DISABLE_EXPORTS
 
 ifeq ($(CONFIG_FUNCTION_TRACER),y)
 ORIG_CFLAGS := $(KBUILD_CFLAGS)
diff --git a/arch/arm/boot/compressed/decompress.c b/arch/arm/boot/compressed/decompress.c
index aa075d8372ea..04f77491975a 100644
--- a/arch/arm/boot/compressed/decompress.c
+++ b/arch/arm/boot/compressed/decompress.c
@@ -56,6 +56,10 @@ extern char * strchrnul(const char *, int);
 #include "../../../../lib/decompress_unlz4.c"
 #endif
 
+#ifdef CONFIG_KERNEL_ZSTD
+#include "../../../../lib/decompress_unzstd.c"
+#endif
+
 int do_decompress(u8 *input, int len, u8 *output, void (*error)(char *x))
 {
 	return __decompress(input, len, NULL, NULL, output, 0, NULL, error);
diff --git a/arch/arm/boot/compressed/head.S b/arch/arm/boot/compressed/head.S
index 434a16982e34..1af01bfe6638 100644
--- a/arch/arm/boot/compressed/head.S
+++ b/arch/arm/boot/compressed/head.S
@@ -614,7 +614,11 @@ not_relocated:	mov	r0, #0
  */
 		mov	r0, r4
 		mov	r1, sp			@ malloc space above stack
+#if defined(CONFIG_KERNEL_ZSTD)
+		add	r2, sp, #0x30000	@ Context needs ~160K
+#else
 		add	r2, sp, #0x10000	@ 64k max
+#endif
 		mov	r3, r7
 		bl	decompress_kernel
 
-- 
2.27.0