[PATCH v3 0/2] MIPS: Add support for ZSTD v3

[PATCH v3 0/2] MIPS: Add support for ZSTD v3

[Paul Cercueil]

Hi,

This small patchset adds support for self-extracting kernels compressed
with ZSTD.

Nick:
I dropped your Reviewed-by on patch [1/2] since the code changed.

Cheers,
-Paul

Paul Cercueil (2):
  lib: decompress_unzstd: Limit output size
  MIPS: Add support for ZSTD-compressed kernels

 arch/mips/Kconfig                      |  1 +
 arch/mips/boot/compressed/Makefile     |  3 ++-
 arch/mips/boot/compressed/decompress.c |  4 ++++
 arch/mips/boot/compressed/string.c     | 17 +++++++++++++++++
 lib/decompress_unzstd.c                |  7 ++++++-
 5 files changed, 30 insertions(+), 2 deletions(-)

-- 
2.28.0

[PATCH v3 1/2] lib: decompress_unzstd: Limit output size

[Paul Cercueil]

The zstd decompression code, as it is right now, will most likely fail
on 32-bit systems, as the default output buffer size causes the buffer's
end address to overflow.

Address this issue by setting a sane default to the default output size,
with a value that won't overflow the buffer's end address.

Signed-off-by: Paul Cercueil <paul@crapouillou.net>
---

Notes:
    v2: Change limit to 1 GiB
    
    v3: Compute size limit instead of using hardcoded value

 lib/decompress_unzstd.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c
index 0ad2c15479ed..790abc472f5b 100644
--- a/lib/decompress_unzstd.c
+++ b/lib/decompress_unzstd.c
@@ -178,8 +178,13 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len,
 	int err;
 	size_t ret;
 
+	/*
+	 * ZSTD decompression code won't be happy if the buffer size is so big
+	 * that its end address overflows. When the size is not provided, make
+	 * it as big as possible without having the end address overflow.
+	 */
 	if (out_len == 0)
-		out_len = LONG_MAX; /* no limit */
+		out_len = UINTPTR_MAX - (uintptr_t)out_buf;
 
 	if (fill == NULL && flush == NULL)
 		/*
-- 
2.28.0

[PATCH v3 2/2] MIPS: Add support for ZSTD-compressed kernels

[Paul Cercueil]

Add support for self-extracting kernels with a ZSTD compression.

Tested on a kernel for the GCW-Zero, it allows to reduce the size of the
kernel file from 4.1 MiB with gzip to 3.5 MiB with ZSTD, and boots just
as fast.

Compressed kernels are now also compiled with -D__DISABLE_EXPORTS in
order to disable the EXPORT_SYMBOL() macros inside of
lib/zstd/decompress.c.

Signed-off-by: Paul Cercueil <paul@crapouillou.net>
---

Notes:
    v2: Add -D__DISABLE_EXPORTS to CFLAGS and use zstd22
    
    v3: Make memmove __weak, since it is provided for other compression
        schemes

 arch/mips/Kconfig                      |  1 +
 arch/mips/boot/compressed/Makefile     |  3 ++-
 arch/mips/boot/compressed/decompress.c |  4 ++++
 arch/mips/boot/compressed/string.c     | 17 +++++++++++++++++
 4 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index c95fa3a2484c..b9d7c4249dc9 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -1890,6 +1890,7 @@ config SYS_SUPPORTS_ZBOOT
 	select HAVE_KERNEL_LZMA
 	select HAVE_KERNEL_LZO
 	select HAVE_KERNEL_XZ
+	select HAVE_KERNEL_ZSTD
 
 config SYS_SUPPORTS_ZBOOT_UART16550
 	bool
diff --git a/arch/mips/boot/compressed/Makefile b/arch/mips/boot/compressed/Makefile
index 6e56caef69f0..9a9ba77b745e 100644
--- a/arch/mips/boot/compressed/Makefile
+++ b/arch/mips/boot/compressed/Makefile
@@ -22,7 +22,7 @@ KBUILD_CFLAGS := $(filter-out -pg, $(KBUILD_CFLAGS))
 
 KBUILD_CFLAGS := $(filter-out -fstack-protector, $(KBUILD_CFLAGS))
 
-KBUILD_CFLAGS := $(KBUILD_CFLAGS) -D__KERNEL__ \
+KBUILD_CFLAGS := $(KBUILD_CFLAGS) -D__KERNEL__ -D__DISABLE_EXPORTS \
 	-DBOOT_HEAP_SIZE=$(BOOT_HEAP_SIZE) -D"VMLINUX_LOAD_ADDRESS_ULL=$(VMLINUX_LOAD_ADDRESS)ull"
 
 KBUILD_AFLAGS := $(KBUILD_AFLAGS) -D__ASSEMBLY__ \
@@ -70,6 +70,7 @@ tool_$(CONFIG_KERNEL_LZ4)     = lz4
 tool_$(CONFIG_KERNEL_LZMA)    = lzma
 tool_$(CONFIG_KERNEL_LZO)     = lzo
 tool_$(CONFIG_KERNEL_XZ)      = xzkern
+tool_$(CONFIG_KERNEL_ZSTD)    = zstd22
 
 targets += vmlinux.bin.z
 $(obj)/vmlinux.bin.z: $(obj)/vmlinux.bin FORCE
diff --git a/arch/mips/boot/compressed/decompress.c b/arch/mips/boot/compressed/decompress.c
index 88f5d637b1c4..c61c641674e6 100644
--- a/arch/mips/boot/compressed/decompress.c
+++ b/arch/mips/boot/compressed/decompress.c
@@ -72,6 +72,10 @@ void error(char *x)
 #include "../../../../lib/decompress_unxz.c"
 #endif
 
+#ifdef CONFIG_KERNEL_ZSTD
+#include "../../../../lib/decompress_unzstd.c"
+#endif
+
 const unsigned long __stack_chk_guard = 0x000a0dff;
 
 void __stack_chk_fail(void)
diff --git a/arch/mips/boot/compressed/string.c b/arch/mips/boot/compressed/string.c
index 43beecc3587c..0b593b709228 100644
--- a/arch/mips/boot/compressed/string.c
+++ b/arch/mips/boot/compressed/string.c
@@ -5,6 +5,7 @@
  * Very small subset of simple string routines
  */
 
+#include <linux/compiler_attributes.h>
 #include <linux/types.h>
 
 void *memcpy(void *dest, const void *src, size_t n)
@@ -27,3 +28,19 @@ void *memset(void *s, int c, size_t n)
 		ss[i] = c;
 	return s;
 }
+
+void * __weak memmove(void *dest, const void *src, size_t n)
+{
+	unsigned int i;
+	const char *s = src;
+	char *d = dest;
+
+	if ((uintptr_t)dest < (uintptr_t)src) {
+		for (i = 0; i < n; i++)
+			d[i] = s[i];
+	} else {
+		for (i = n; i > 0; i--)
+			d[i - 1] = s[i - 1];
+	}
+	return dest;
+}
-- 
2.28.0

Re: [PATCH v3 1/2] lib: decompress_unzstd: Limit output size

[Nick Terrell]

> On Sep 1, 2020, at 7:26 AM, Paul Cercueil <paul@crapouillou.net> wrote:
> 
> The zstd decompression code, as it is right now, will most likely fail
> on 32-bit systems, as the default output buffer size causes the buffer's
> end address to overflow.
> 
> Address this issue by setting a sane default to the default output size,
> with a value that won't overflow the buffer's end address.
> 
> Signed-off-by: Paul Cercueil <paul@crapouillou.net>
> ---
> 
> Notes:
>    v2: Change limit to 1 GiB
> 
>    v3: Compute size limit instead of using hardcoded value
> 
> lib/decompress_unzstd.c | 7 ++++++-
> 1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c
> index 0ad2c15479ed..790abc472f5b 100644
> --- a/lib/decompress_unzstd.c
> +++ b/lib/decompress_unzstd.c
> @@ -178,8 +178,13 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len,
> 	int err;
> 	size_t ret;
> 
> +	/*
> +	 * ZSTD decompression code won't be happy if the buffer size is so big
> +	 * that its end address overflows. When the size is not provided, make
> +	 * it as big as possible without having the end address overflow.
> +	 */
> 	if (out_len == 0)
> -		out_len = LONG_MAX; /* no limit */
> +		out_len = UINTPTR_MAX - (uintptr_t)out_buf;

Great, that works for me. Thanks for fixing this!

Reviewed-by: Nick Terrell <terrelln@fb.com>

> 	if (fill == NULL && flush == NULL)
> 		/*
> -- 
> 2.28.0
> 

Re: [PATCH v3 0/2] MIPS: Add support for ZSTD v3

[Thomas Bogendoerfer]

On Tue, Sep 01, 2020 at 04:26:49PM +0200, Paul Cercueil wrote:
> Hi,
> 
> This small patchset adds support for self-extracting kernels compressed
> with ZSTD.
> 
> Nick:
> I dropped your Reviewed-by on patch [1/2] since the code changed.
> 
> Cheers,
> -Paul
> 
> Paul Cercueil (2):
>   lib: decompress_unzstd: Limit output size
>   MIPS: Add support for ZSTD-compressed kernels
> 
>  arch/mips/Kconfig                      |  1 +
>  arch/mips/boot/compressed/Makefile     |  3 ++-
>  arch/mips/boot/compressed/decompress.c |  4 ++++
>  arch/mips/boot/compressed/string.c     | 17 +++++++++++++++++
>  lib/decompress_unzstd.c                |  7 ++++++-
>  5 files changed, 30 insertions(+), 2 deletions(-)

series applied to mips-next.

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessarily a
good idea.                                                [ RFC1925, 2.3 ]