* [PATCH v2 1/2] lib: decompress_unzstd: Limit output size
@ 2020-08-25 21:01 Paul Cercueil
2020-08-25 21:01 ` [PATCH v2 2/2] MIPS: Add support for ZSTD-compressed kernels Paul Cercueil
2020-08-25 22:51 ` [PATCH v2 1/2] lib: decompress_unzstd: Limit output size Nick Terrell
0 siblings, 2 replies; 4+ messages in thread
From: Paul Cercueil @ 2020-08-25 21:01 UTC (permalink / raw)
To: Nick Terrell, Thomas Bogendoerfer
Cc: linux-mips, linux-kernel, od, Paul Cercueil
The zstd decompression code, as it is right now, will have internal
values overflow on 32-bit systems when the output size is bigger than
1 GiB.
Until someone smarter than me can figure out how to fix the zstd code
properly, limit the destination buffer size to 1 GiB, which should be
enough for everybody, in order to make it usable on 32-bit systems.
Signed-off-by: Paul Cercueil <paul@crapouillou.net>
Reviewed-by: Nick Terrell <terrelln@fb.com>
---
Notes:
v2: Change limit to 1 GiB
lib/decompress_unzstd.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c
index 0ad2c15479ed..414517baedb0 100644
--- a/lib/decompress_unzstd.c
+++ b/lib/decompress_unzstd.c
@@ -77,6 +77,7 @@
#include <linux/decompress/mm.h>
#include <linux/kernel.h>
+#include <linux/sizes.h>
#include <linux/zstd.h>
/* 128MB is the maximum window size supported by zstd. */
@@ -179,7 +180,7 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len,
size_t ret;
if (out_len == 0)
- out_len = LONG_MAX; /* no limit */
+ out_len = SZ_1G; /* should be big enough, right? */
if (fill == NULL && flush == NULL)
/*
--
2.28.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH v2 2/2] MIPS: Add support for ZSTD-compressed kernels
2020-08-25 21:01 [PATCH v2 1/2] lib: decompress_unzstd: Limit output size Paul Cercueil
@ 2020-08-25 21:01 ` Paul Cercueil
2020-08-26 15:51 ` kernel test robot
2020-08-25 22:51 ` [PATCH v2 1/2] lib: decompress_unzstd: Limit output size Nick Terrell
1 sibling, 1 reply; 4+ messages in thread
From: Paul Cercueil @ 2020-08-25 21:01 UTC (permalink / raw)
To: Nick Terrell, Thomas Bogendoerfer
Cc: linux-mips, linux-kernel, od, Paul Cercueil
Add support for self-extracting kernels with a ZSTD compression.
Tested on a kernel for the GCW-Zero, it allows to reduce the size of the
kernel file from 4.1 MiB with gzip to 3.5 MiB with ZSTD, and boots just
as fast.
Compressed kernels are now also compiled with -D__DISABLE_EXPORTS in
order to disable the EXPORT_SYMBOL() macros inside of
lib/zstd/decompress.c.
Signed-off-by: Paul Cercueil <paul@crapouillou.net>
---
Notes:
v2: Add -D__DISABLE_EXPORTS to CFLAGS and use zstd22
arch/mips/Kconfig | 1 +
arch/mips/boot/compressed/Makefile | 3 ++-
arch/mips/boot/compressed/decompress.c | 4 ++++
arch/mips/boot/compressed/string.c | 16 ++++++++++++++++
4 files changed, 23 insertions(+), 1 deletion(-)
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index c95fa3a2484c..b9d7c4249dc9 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -1890,6 +1890,7 @@ config SYS_SUPPORTS_ZBOOT
select HAVE_KERNEL_LZMA
select HAVE_KERNEL_LZO
select HAVE_KERNEL_XZ
+ select HAVE_KERNEL_ZSTD
config SYS_SUPPORTS_ZBOOT_UART16550
bool
diff --git a/arch/mips/boot/compressed/Makefile b/arch/mips/boot/compressed/Makefile
index 6e56caef69f0..9a9ba77b745e 100644
--- a/arch/mips/boot/compressed/Makefile
+++ b/arch/mips/boot/compressed/Makefile
@@ -22,7 +22,7 @@ KBUILD_CFLAGS := $(filter-out -pg, $(KBUILD_CFLAGS))
KBUILD_CFLAGS := $(filter-out -fstack-protector, $(KBUILD_CFLAGS))
-KBUILD_CFLAGS := $(KBUILD_CFLAGS) -D__KERNEL__ \
+KBUILD_CFLAGS := $(KBUILD_CFLAGS) -D__KERNEL__ -D__DISABLE_EXPORTS \
-DBOOT_HEAP_SIZE=$(BOOT_HEAP_SIZE) -D"VMLINUX_LOAD_ADDRESS_ULL=$(VMLINUX_LOAD_ADDRESS)ull"
KBUILD_AFLAGS := $(KBUILD_AFLAGS) -D__ASSEMBLY__ \
@@ -70,6 +70,7 @@ tool_$(CONFIG_KERNEL_LZ4) = lz4
tool_$(CONFIG_KERNEL_LZMA) = lzma
tool_$(CONFIG_KERNEL_LZO) = lzo
tool_$(CONFIG_KERNEL_XZ) = xzkern
+tool_$(CONFIG_KERNEL_ZSTD) = zstd22
targets += vmlinux.bin.z
$(obj)/vmlinux.bin.z: $(obj)/vmlinux.bin FORCE
diff --git a/arch/mips/boot/compressed/decompress.c b/arch/mips/boot/compressed/decompress.c
index 88f5d637b1c4..c61c641674e6 100644
--- a/arch/mips/boot/compressed/decompress.c
+++ b/arch/mips/boot/compressed/decompress.c
@@ -72,6 +72,10 @@ void error(char *x)
#include "../../../../lib/decompress_unxz.c"
#endif
+#ifdef CONFIG_KERNEL_ZSTD
+#include "../../../../lib/decompress_unzstd.c"
+#endif
+
const unsigned long __stack_chk_guard = 0x000a0dff;
void __stack_chk_fail(void)
diff --git a/arch/mips/boot/compressed/string.c b/arch/mips/boot/compressed/string.c
index 43beecc3587c..ab95722ec0c9 100644
--- a/arch/mips/boot/compressed/string.c
+++ b/arch/mips/boot/compressed/string.c
@@ -27,3 +27,19 @@ void *memset(void *s, int c, size_t n)
ss[i] = c;
return s;
}
+
+void *memmove(void *dest, const void *src, size_t n)
+{
+ unsigned int i;
+ const char *s = src;
+ char *d = dest;
+
+ if ((uintptr_t)dest < (uintptr_t)src) {
+ for (i = 0; i < n; i++)
+ d[i] = s[i];
+ } else {
+ for (i = n; i > 0; i--)
+ d[i - 1] = s[i - 1];
+ }
+ return dest;
+}
--
2.28.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH v2 1/2] lib: decompress_unzstd: Limit output size
2020-08-25 21:01 [PATCH v2 1/2] lib: decompress_unzstd: Limit output size Paul Cercueil
2020-08-25 21:01 ` [PATCH v2 2/2] MIPS: Add support for ZSTD-compressed kernels Paul Cercueil
@ 2020-08-25 22:51 ` Nick Terrell
1 sibling, 0 replies; 4+ messages in thread
From: Nick Terrell @ 2020-08-25 22:51 UTC (permalink / raw)
To: Paul Cercueil
Cc: Thomas Bogendoerfer, linux-mips, Linux Kernel Mailing List, od,
Yann Collet
> On Aug 25, 2020, at 2:01 PM, Paul Cercueil <paul@crapouillou.net> wrote:
>
> The zstd decompression code, as it is right now, will have internal
> values overflow on 32-bit systems when the output size is bigger than
> 1 GiB.
>
> Until someone smarter than me can figure out how to fix the zstd code
> properly, limit the destination buffer size to 1 GiB, which should be
> enough for everybody, in order to make it usable on 32-bit systems.
I was talking with Yann Collet, and we believe that it isn’t the long that
is overflowing, but the pointers. Zstd expects to be given a valid output
size. It generally uses a begin/end pointer with its output buffer. So when
it is given a very large output size in 32-bit mode the end pointer will
overflow the pointer either causing UB, or end pointer < begin pointer,
which breaks zstd.
Zstd will probably never be able to work properly in this way. A better
solution might be to pass MAX_ADDRESS_PTR - OUTPUT_PTR as
the size to the __decompress() call. Or some other size that won’t
overflow the pointer.
Best,
Nick
> Signed-off-by: Paul Cercueil <paul@crapouillou.net>
> Reviewed-by: Nick Terrell <terrelln@fb.com>
> ---
>
> Notes:
> v2: Change limit to 1 GiB
>
> lib/decompress_unzstd.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c
> index 0ad2c15479ed..414517baedb0 100644
> --- a/lib/decompress_unzstd.c
> +++ b/lib/decompress_unzstd.c
> @@ -77,6 +77,7 @@
>
> #include <linux/decompress/mm.h>
> #include <linux/kernel.h>
> +#include <linux/sizes.h>
> #include <linux/zstd.h>
>
> /* 128MB is the maximum window size supported by zstd. */
> @@ -179,7 +180,7 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len,
> size_t ret;
>
> if (out_len == 0)
> - out_len = LONG_MAX; /* no limit */
> + out_len = SZ_1G; /* should be big enough, right? */
>
> if (fill == NULL && flush == NULL)
> /*
> --
> 2.28.0
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v2 2/2] MIPS: Add support for ZSTD-compressed kernels
2020-08-25 21:01 ` [PATCH v2 2/2] MIPS: Add support for ZSTD-compressed kernels Paul Cercueil
@ 2020-08-26 15:51 ` kernel test robot
0 siblings, 0 replies; 4+ messages in thread
From: kernel test robot @ 2020-08-26 15:51 UTC (permalink / raw)
To: Paul Cercueil, Nick Terrell, Thomas Bogendoerfer
Cc: kbuild-all, linux-mips, linux-kernel, od, Paul Cercueil
[-- Attachment #1: Type: text/plain, Size: 1464 bytes --]
Hi Paul,
I love your patch! Yet something to improve:
[auto build test ERROR on linus/master]
[also build test ERROR on v5.9-rc2 next-20200826]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Paul-Cercueil/lib-decompress_unzstd-Limit-output-size/20200826-050247
base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 6a9dc5fd6170d0a41c8a14eb19e63d94bea5705a
config: mips-loongson1c_defconfig (attached as .config)
compiler: mipsel-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=mips
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
mipsel-linux-ld: arch/mips/boot/compressed/string.o: in function `memmove':
>> string.c:(.text+0x50): multiple definition of `memmove'; arch/mips/boot/compressed/decompress.o:decompress.c:(.text+0x11f4): first defined here
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 14804 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-08-26 15:52 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-08-25 21:01 [PATCH v2 1/2] lib: decompress_unzstd: Limit output size Paul Cercueil
2020-08-25 21:01 ` [PATCH v2 2/2] MIPS: Add support for ZSTD-compressed kernels Paul Cercueil
2020-08-26 15:51 ` kernel test robot
2020-08-25 22:51 ` [PATCH v2 1/2] lib: decompress_unzstd: Limit output size Nick Terrell
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).