* [PATCH 1/2] lib: decompress_unzstd: Limit output size
@ 2020-08-21 16:29 Paul Cercueil
2020-08-21 16:29 ` [PATCH 2/2] MIPS: Add support for ZSTD-compressed kernels Paul Cercueil
2020-08-24 20:11 ` [PATCH 1/2] lib: decompress_unzstd: Limit output size Nick Terrell
0 siblings, 2 replies; 8+ messages in thread
From: Paul Cercueil @ 2020-08-21 16:29 UTC (permalink / raw)
To: Nick Terrell, Thomas Bogendoerfer
Cc: linux-mips, linux-kernel, od, Paul Cercueil
The zstd decompression code, as it is right now, will have internal
values overflow on 32-bit systems when the output size is LONG_MAX.
Until someone smarter than me can figure out how to fix the zstd code
properly, limit the destination buffer size to 512 MiB, which should be
enough for everybody, in order to make it usable on 32-bit systems.
Signed-off-by: Paul Cercueil <paul@crapouillou.net>
---
lib/decompress_unzstd.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c
index 0ad2c15479ed..e1c03b1eaa6e 100644
--- a/lib/decompress_unzstd.c
+++ b/lib/decompress_unzstd.c
@@ -77,6 +77,7 @@
#include <linux/decompress/mm.h>
#include <linux/kernel.h>
+#include <linux/sizes.h>
#include <linux/zstd.h>
/* 128MB is the maximum window size supported by zstd. */
@@ -179,7 +180,7 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len,
size_t ret;
if (out_len == 0)
- out_len = LONG_MAX; /* no limit */
+ out_len = SZ_512M; /* should be big enough, right? */
if (fill == NULL && flush == NULL)
/*
--
2.28.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 2/2] MIPS: Add support for ZSTD-compressed kernels
2020-08-21 16:29 [PATCH 1/2] lib: decompress_unzstd: Limit output size Paul Cercueil
@ 2020-08-21 16:29 ` Paul Cercueil
2020-08-24 19:51 ` Nick Terrell
2020-08-24 20:11 ` [PATCH 1/2] lib: decompress_unzstd: Limit output size Nick Terrell
1 sibling, 1 reply; 8+ messages in thread
From: Paul Cercueil @ 2020-08-21 16:29 UTC (permalink / raw)
To: Nick Terrell, Thomas Bogendoerfer
Cc: linux-mips, linux-kernel, od, Paul Cercueil
Add support for self-extracting kernels with a ZSTD compression.
Tested on a kernel for the GCW-Zero, it allows to reduce the size of the
kernel file from 4.1 MiB with gzip to 3.5 MiB with ZSTD, and boots just
as fast.
Signed-off-by: Paul Cercueil <paul@crapouillou.net>
---
arch/mips/Kconfig | 1 +
arch/mips/boot/compressed/Makefile | 1 +
arch/mips/boot/compressed/decompress.c | 4 ++++
arch/mips/boot/compressed/string.c | 16 ++++++++++++++++
4 files changed, 22 insertions(+)
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index c95fa3a2484c..b9d7c4249dc9 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -1890,6 +1890,7 @@ config SYS_SUPPORTS_ZBOOT
select HAVE_KERNEL_LZMA
select HAVE_KERNEL_LZO
select HAVE_KERNEL_XZ
+ select HAVE_KERNEL_ZSTD
config SYS_SUPPORTS_ZBOOT_UART16550
bool
diff --git a/arch/mips/boot/compressed/Makefile b/arch/mips/boot/compressed/Makefile
index 6e56caef69f0..86ddc6fc16f4 100644
--- a/arch/mips/boot/compressed/Makefile
+++ b/arch/mips/boot/compressed/Makefile
@@ -70,6 +70,7 @@ tool_$(CONFIG_KERNEL_LZ4) = lz4
tool_$(CONFIG_KERNEL_LZMA) = lzma
tool_$(CONFIG_KERNEL_LZO) = lzo
tool_$(CONFIG_KERNEL_XZ) = xzkern
+tool_$(CONFIG_KERNEL_ZSTD) = zstd
targets += vmlinux.bin.z
$(obj)/vmlinux.bin.z: $(obj)/vmlinux.bin FORCE
diff --git a/arch/mips/boot/compressed/decompress.c b/arch/mips/boot/compressed/decompress.c
index 88f5d637b1c4..c61c641674e6 100644
--- a/arch/mips/boot/compressed/decompress.c
+++ b/arch/mips/boot/compressed/decompress.c
@@ -72,6 +72,10 @@ void error(char *x)
#include "../../../../lib/decompress_unxz.c"
#endif
+#ifdef CONFIG_KERNEL_ZSTD
+#include "../../../../lib/decompress_unzstd.c"
+#endif
+
const unsigned long __stack_chk_guard = 0x000a0dff;
void __stack_chk_fail(void)
diff --git a/arch/mips/boot/compressed/string.c b/arch/mips/boot/compressed/string.c
index 43beecc3587c..ab95722ec0c9 100644
--- a/arch/mips/boot/compressed/string.c
+++ b/arch/mips/boot/compressed/string.c
@@ -27,3 +27,19 @@ void *memset(void *s, int c, size_t n)
ss[i] = c;
return s;
}
+
+void *memmove(void *dest, const void *src, size_t n)
+{
+ unsigned int i;
+ const char *s = src;
+ char *d = dest;
+
+ if ((uintptr_t)dest < (uintptr_t)src) {
+ for (i = 0; i < n; i++)
+ d[i] = s[i];
+ } else {
+ for (i = n; i > 0; i--)
+ d[i - 1] = s[i - 1];
+ }
+ return dest;
+}
--
2.28.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] MIPS: Add support for ZSTD-compressed kernels
2020-08-21 16:29 ` [PATCH 2/2] MIPS: Add support for ZSTD-compressed kernels Paul Cercueil
@ 2020-08-24 19:51 ` Nick Terrell
2020-08-24 21:02 ` Paul Cercueil
0 siblings, 1 reply; 8+ messages in thread
From: Nick Terrell @ 2020-08-24 19:51 UTC (permalink / raw)
To: Paul Cercueil
Cc: Thomas Bogendoerfer, linux-mips, Linux Kernel Mailing List, od
> On Aug 21, 2020, at 9:29 AM, Paul Cercueil <paul@crapouillou.net> wrote:
>
> Add support for self-extracting kernels with a ZSTD compression.
>
> Tested on a kernel for the GCW-Zero, it allows to reduce the size of the
> kernel file from 4.1 MiB with gzip to 3.5 MiB with ZSTD, and boots just
> as fast.
>
> Signed-off-by: Paul Cercueil <paul@crapouillou.net>
> ---
> arch/mips/Kconfig | 1 +
> arch/mips/boot/compressed/Makefile | 1 +
> arch/mips/boot/compressed/decompress.c | 4 ++++
> arch/mips/boot/compressed/string.c | 16 ++++++++++++++++
> 4 files changed, 22 insertions(+)
>
> diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
> index c95fa3a2484c..b9d7c4249dc9 100644
> --- a/arch/mips/Kconfig
> +++ b/arch/mips/Kconfig
> @@ -1890,6 +1890,7 @@ config SYS_SUPPORTS_ZBOOT
> select HAVE_KERNEL_LZMA
> select HAVE_KERNEL_LZO
> select HAVE_KERNEL_XZ
> + select HAVE_KERNEL_ZSTD
>
> config SYS_SUPPORTS_ZBOOT_UART16550
> bool
> diff --git a/arch/mips/boot/compressed/Makefile b/arch/mips/boot/compressed/Makefile
> index 6e56caef69f0..86ddc6fc16f4 100644
> --- a/arch/mips/boot/compressed/Makefile
> +++ b/arch/mips/boot/compressed/Makefile
> @@ -70,6 +70,7 @@ tool_$(CONFIG_KERNEL_LZ4) = lz4
> tool_$(CONFIG_KERNEL_LZMA) = lzma
> tool_$(CONFIG_KERNEL_LZO) = lzo
> tool_$(CONFIG_KERNEL_XZ) = xzkern
> +tool_$(CONFIG_KERNEL_ZSTD) = zstd
You can use zstd22 here. It will give you slightly better compression
without any additional memory usage. Also, you should add
-D__DISABLE_EXPORTS to the KBUILD_CFLAGS like x86 does [1].
[1] https://github.com/torvalds/linux/blob/master/arch/x86/boot/compressed/Makefile
-Nick
> targets += vmlinux.bin.z
> $(obj)/vmlinux.bin.z: $(obj)/vmlinux.bin FORCE
> diff --git a/arch/mips/boot/compressed/decompress.c b/arch/mips/boot/compressed/decompress.c
> index 88f5d637b1c4..c61c641674e6 100644
> --- a/arch/mips/boot/compressed/decompress.c
> +++ b/arch/mips/boot/compressed/decompress.c
> @@ -72,6 +72,10 @@ void error(char *x)
> #include "../../../../lib/decompress_unxz.c"
> #endif
>
> +#ifdef CONFIG_KERNEL_ZSTD
> +#include "../../../../lib/decompress_unzstd.c"
> +#endif
> +
> const unsigned long __stack_chk_guard = 0x000a0dff;
>
> void __stack_chk_fail(void)
> diff --git a/arch/mips/boot/compressed/string.c b/arch/mips/boot/compressed/string.c
> index 43beecc3587c..ab95722ec0c9 100644
> --- a/arch/mips/boot/compressed/string.c
> +++ b/arch/mips/boot/compressed/string.c
> @@ -27,3 +27,19 @@ void *memset(void *s, int c, size_t n)
> ss[i] = c;
> return s;
> }
> +
> +void *memmove(void *dest, const void *src, size_t n)
> +{
> + unsigned int i;
> + const char *s = src;
> + char *d = dest;
> +
> + if ((uintptr_t)dest < (uintptr_t)src) {
> + for (i = 0; i < n; i++)
> + d[i] = s[i];
> + } else {
> + for (i = n; i > 0; i--)
> + d[i - 1] = s[i - 1];
> + }
> + return dest;
> +}
> --
> 2.28.0
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/2] lib: decompress_unzstd: Limit output size
2020-08-21 16:29 [PATCH 1/2] lib: decompress_unzstd: Limit output size Paul Cercueil
2020-08-21 16:29 ` [PATCH 2/2] MIPS: Add support for ZSTD-compressed kernels Paul Cercueil
@ 2020-08-24 20:11 ` Nick Terrell
2020-08-24 21:05 ` Paul Cercueil
1 sibling, 1 reply; 8+ messages in thread
From: Nick Terrell @ 2020-08-24 20:11 UTC (permalink / raw)
To: Paul Cercueil; +Cc: Thomas Bogendoerfer, linux-mips, linux-kernel, od
> On Aug 21, 2020, at 9:29 AM, Paul Cercueil <paul@crapouillou.net> wrote:
>
> The zstd decompression code, as it is right now, will have internal
> values overflow on 32-bit systems when the output size is LONG_MAX.
>
> Until someone smarter than me can figure out how to fix the zstd code
> properly, limit the destination buffer size to 512 MiB, which should be
> enough for everybody, in order to make it usable on 32-bit systems.
Can you bump the size up to 2GB? I suspect the problem inside of zstd
is an off-by-one error or something similar, so getting closer to the limit
shouldn't be a problem. I’d feel more comfortable with 2GB, since
kernels can get pretty large.
Hmm, zstd shouldn’t be overflowing that value. I’m currently preparing
a patch to updating the version of zstd in the kernel, and using upstream
directly. I will add a test upstream in 32-bit mode to ensure that we don’t
overflow a 32-bit size_t, so this will be fixed after the update.
-Nick
> Signed-off-by: Paul Cercueil <paul@crapouillou.net>
> ---
> lib/decompress_unzstd.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c
> index 0ad2c15479ed..e1c03b1eaa6e 100644
> --- a/lib/decompress_unzstd.c
> +++ b/lib/decompress_unzstd.c
> @@ -77,6 +77,7 @@
>
> #include <linux/decompress/mm.h>
> #include <linux/kernel.h>
> +#include <linux/sizes.h>
> #include <linux/zstd.h>
>
> /* 128MB is the maximum window size supported by zstd. */
> @@ -179,7 +180,7 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len,
> size_t ret;
>
> if (out_len == 0)
> - out_len = LONG_MAX; /* no limit */
> + out_len = SZ_512M; /* should be big enough, right? */
>
> if (fill == NULL && flush == NULL)
> /*
> --
> 2.28.0
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] MIPS: Add support for ZSTD-compressed kernels
2020-08-24 19:51 ` Nick Terrell
@ 2020-08-24 21:02 ` Paul Cercueil
2020-08-24 21:45 ` Nick Terrell
0 siblings, 1 reply; 8+ messages in thread
From: Paul Cercueil @ 2020-08-24 21:02 UTC (permalink / raw)
To: Nick Terrell
Cc: Thomas Bogendoerfer, linux-mips, Linux Kernel Mailing List, od
Hi Nick,
Le lun. 24 août 2020 à 19:51, Nick Terrell <terrelln@fb.com> a écrit
:
>
>
>> On Aug 21, 2020, at 9:29 AM, Paul Cercueil <paul@crapouillou.net>
>> wrote:
>>
>> Add support for self-extracting kernels with a ZSTD compression.
>>
>> Tested on a kernel for the GCW-Zero, it allows to reduce the size
>> of the
>> kernel file from 4.1 MiB with gzip to 3.5 MiB with ZSTD, and boots
>> just
>> as fast.
>>
>> Signed-off-by: Paul Cercueil <paul@crapouillou.net>
>> ---
>> arch/mips/Kconfig | 1 +
>> arch/mips/boot/compressed/Makefile | 1 +
>> arch/mips/boot/compressed/decompress.c | 4 ++++
>> arch/mips/boot/compressed/string.c | 16 ++++++++++++++++
>> 4 files changed, 22 insertions(+)
>>
>> diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
>> index c95fa3a2484c..b9d7c4249dc9 100644
>> --- a/arch/mips/Kconfig
>> +++ b/arch/mips/Kconfig
>> @@ -1890,6 +1890,7 @@ config SYS_SUPPORTS_ZBOOT
>> select HAVE_KERNEL_LZMA
>> select HAVE_KERNEL_LZO
>> select HAVE_KERNEL_XZ
>> + select HAVE_KERNEL_ZSTD
>>
>> config SYS_SUPPORTS_ZBOOT_UART16550
>> bool
>> diff --git a/arch/mips/boot/compressed/Makefile
>> b/arch/mips/boot/compressed/Makefile
>> index 6e56caef69f0..86ddc6fc16f4 100644
>> --- a/arch/mips/boot/compressed/Makefile
>> +++ b/arch/mips/boot/compressed/Makefile
>> @@ -70,6 +70,7 @@ tool_$(CONFIG_KERNEL_LZ4) = lz4
>> tool_$(CONFIG_KERNEL_LZMA) = lzma
>> tool_$(CONFIG_KERNEL_LZO) = lzo
>> tool_$(CONFIG_KERNEL_XZ) = xzkern
>> +tool_$(CONFIG_KERNEL_ZSTD) = zstd
>
> You can use zstd22 here. It will give you slightly better compression
> without any additional memory usage. Also, you should add
> -D__DISABLE_EXPORTS to the KBUILD_CFLAGS like x86 does [1].
Indeed, it's 0.01% smaller :)
What is __DISABLE_EXPORTS for?
-Paul
>
> [1]
> https://github.com/torvalds/linux/blob/master/arch/x86/boot/compressed/Makefile
>
> -Nick
>
>> targets += vmlinux.bin.z
>> $(obj)/vmlinux.bin.z: $(obj)/vmlinux.bin FORCE
>> diff --git a/arch/mips/boot/compressed/decompress.c
>> b/arch/mips/boot/compressed/decompress.c
>> index 88f5d637b1c4..c61c641674e6 100644
>> --- a/arch/mips/boot/compressed/decompress.c
>> +++ b/arch/mips/boot/compressed/decompress.c
>> @@ -72,6 +72,10 @@ void error(char *x)
>> #include "../../../../lib/decompress_unxz.c"
>> #endif
>>
>> +#ifdef CONFIG_KERNEL_ZSTD
>> +#include "../../../../lib/decompress_unzstd.c"
>> +#endif
>> +
>> const unsigned long __stack_chk_guard = 0x000a0dff;
>>
>> void __stack_chk_fail(void)
>> diff --git a/arch/mips/boot/compressed/string.c
>> b/arch/mips/boot/compressed/string.c
>> index 43beecc3587c..ab95722ec0c9 100644
>> --- a/arch/mips/boot/compressed/string.c
>> +++ b/arch/mips/boot/compressed/string.c
>> @@ -27,3 +27,19 @@ void *memset(void *s, int c, size_t n)
>> ss[i] = c;
>> return s;
>> }
>> +
>> +void *memmove(void *dest, const void *src, size_t n)
>> +{
>> + unsigned int i;
>> + const char *s = src;
>> + char *d = dest;
>> +
>> + if ((uintptr_t)dest < (uintptr_t)src) {
>> + for (i = 0; i < n; i++)
>> + d[i] = s[i];
>> + } else {
>> + for (i = n; i > 0; i--)
>> + d[i - 1] = s[i - 1];
>> + }
>> + return dest;
>> +}
>> --
>> 2.28.0
>>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/2] lib: decompress_unzstd: Limit output size
2020-08-24 20:11 ` [PATCH 1/2] lib: decompress_unzstd: Limit output size Nick Terrell
@ 2020-08-24 21:05 ` Paul Cercueil
2020-08-24 21:50 ` Nick Terrell
0 siblings, 1 reply; 8+ messages in thread
From: Paul Cercueil @ 2020-08-24 21:05 UTC (permalink / raw)
To: Nick Terrell; +Cc: Thomas Bogendoerfer, linux-mips, linux-kernel, od
Le lun. 24 août 2020 à 20:11, Nick Terrell <terrelln@fb.com> a écrit
:
>
>
>> On Aug 21, 2020, at 9:29 AM, Paul Cercueil <paul@crapouillou.net>
>> wrote:
>>
>> The zstd decompression code, as it is right now, will have internal
>> values overflow on 32-bit systems when the output size is LONG_MAX.
>>
>> Until someone smarter than me can figure out how to fix the zstd
>> code
>> properly, limit the destination buffer size to 512 MiB, which
>> should be
>> enough for everybody, in order to make it usable on 32-bit systems.
>
> Can you bump the size up to 2GB? I suspect the problem inside of zstd
> is an off-by-one error or something similar, so getting closer to the
> limit
> shouldn't be a problem. I’d feel more comfortable with 2GB, since
> kernels can get pretty large.
SZ_1G is the biggest I can go to get the kernel to boot. With SZ_2G it
won't boot.
> Hmm, zstd shouldn’t be overflowing that value. I’m currently
> preparing
> a patch to updating the version of zstd in the kernel, and using
> upstream
> directly. I will add a test upstream in 32-bit mode to ensure that we
> don’t
> overflow a 32-bit size_t, so this will be fixed after the update.
Great, thanks.
Cheers,
-Paul
>
> -Nick
>
>> Signed-off-by: Paul Cercueil <paul@crapouillou.net>
>> ---
>> lib/decompress_unzstd.c | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c
>> index 0ad2c15479ed..e1c03b1eaa6e 100644
>> --- a/lib/decompress_unzstd.c
>> +++ b/lib/decompress_unzstd.c
>> @@ -77,6 +77,7 @@
>>
>> #include <linux/decompress/mm.h>
>> #include <linux/kernel.h>
>> +#include <linux/sizes.h>
>> #include <linux/zstd.h>
>>
>> /* 128MB is the maximum window size supported by zstd. */
>> @@ -179,7 +180,7 @@ static int INIT __unzstd(unsigned char *in_buf,
>> long in_len,
>> size_t ret;
>>
>> if (out_len == 0)
>> - out_len = LONG_MAX; /* no limit */
>> + out_len = SZ_512M; /* should be big enough, right? */
>>
>> if (fill == NULL && flush == NULL)
>> /*
>> --
>> 2.28.0
>>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] MIPS: Add support for ZSTD-compressed kernels
2020-08-24 21:02 ` Paul Cercueil
@ 2020-08-24 21:45 ` Nick Terrell
0 siblings, 0 replies; 8+ messages in thread
From: Nick Terrell @ 2020-08-24 21:45 UTC (permalink / raw)
To: Paul Cercueil
Cc: Thomas Bogendoerfer, linux-mips, Linux Kernel Mailing List, od
> On Aug 24, 2020, at 2:02 PM, Paul Cercueil <paul@crapouillou.net> wrote:
>
> Hi Nick,
>
> Le lun. 24 août 2020 à 19:51, Nick Terrell <terrelln@fb.com> a écrit :
>>> On Aug 21, 2020, at 9:29 AM, Paul Cercueil <paul@crapouillou.net> wrote:
>>> Add support for self-extracting kernels with a ZSTD compression.
>>> Tested on a kernel for the GCW-Zero, it allows to reduce the size of the
>>> kernel file from 4.1 MiB with gzip to 3.5 MiB with ZSTD, and boots just
>>> as fast.
>>> Signed-off-by: Paul Cercueil <paul@crapouillou.net>
>>> ---
>>> arch/mips/Kconfig | 1 +
>>> arch/mips/boot/compressed/Makefile | 1 +
>>> arch/mips/boot/compressed/decompress.c | 4 ++++
>>> arch/mips/boot/compressed/string.c | 16 ++++++++++++++++
>>> 4 files changed, 22 insertions(+)
>>> diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
>>> index c95fa3a2484c..b9d7c4249dc9 100644
>>> --- a/arch/mips/Kconfig
>>> +++ b/arch/mips/Kconfig
>>> @@ -1890,6 +1890,7 @@ config SYS_SUPPORTS_ZBOOT
>>> select HAVE_KERNEL_LZMA
>>> select HAVE_KERNEL_LZO
>>> select HAVE_KERNEL_XZ
>>> + select HAVE_KERNEL_ZSTD
>>> config SYS_SUPPORTS_ZBOOT_UART16550
>>> bool
>>> diff --git a/arch/mips/boot/compressed/Makefile b/arch/mips/boot/compressed/Makefile
>>> index 6e56caef69f0..86ddc6fc16f4 100644
>>> --- a/arch/mips/boot/compressed/Makefile
>>> +++ b/arch/mips/boot/compressed/Makefile
>>> @@ -70,6 +70,7 @@ tool_$(CONFIG_KERNEL_LZ4) = lz4
>>> tool_$(CONFIG_KERNEL_LZMA) = lzma
>>> tool_$(CONFIG_KERNEL_LZO) = lzo
>>> tool_$(CONFIG_KERNEL_XZ) = xzkern
>>> +tool_$(CONFIG_KERNEL_ZSTD) = zstd
>> You can use zstd22 here. It will give you slightly better compression
>> without any additional memory usage. Also, you should add
>> -D__DISABLE_EXPORTS to the KBUILD_CFLAGS like x86 does [1].
>
> Indeed, it's 0.01% smaller :)
>
> What is __DISABLE_EXPORTS for?
It disables the EXPORT_SYMBOL() macros inside of lib/zstd/decompress.c.
On x86 the kernel won’t boot with these defined. Other decompressors hide
them if the STATIC macro is defined, but zstd uses this method, which was
added somewhat recently.
-Nick
> -Paul
>
>> [1] https://github.com/torvalds/linux/blob/master/arch/x86/boot/compressed/Makefile
>> -Nick
>>> targets += vmlinux.bin.z
>>> $(obj)/vmlinux.bin.z: $(obj)/vmlinux.bin FORCE
>>> diff --git a/arch/mips/boot/compressed/decompress.c b/arch/mips/boot/compressed/decompress.c
>>> index 88f5d637b1c4..c61c641674e6 100644
>>> --- a/arch/mips/boot/compressed/decompress.c
>>> +++ b/arch/mips/boot/compressed/decompress.c
>>> @@ -72,6 +72,10 @@ void error(char *x)
>>> #include "../../../../lib/decompress_unxz.c"
>>> #endif
>>> +#ifdef CONFIG_KERNEL_ZSTD
>>> +#include "../../../../lib/decompress_unzstd.c"
>>> +#endif
>>> +
>>> const unsigned long __stack_chk_guard = 0x000a0dff;
>>> void __stack_chk_fail(void)
>>> diff --git a/arch/mips/boot/compressed/string.c b/arch/mips/boot/compressed/string.c
>>> index 43beecc3587c..ab95722ec0c9 100644
>>> --- a/arch/mips/boot/compressed/string.c
>>> +++ b/arch/mips/boot/compressed/string.c
>>> @@ -27,3 +27,19 @@ void *memset(void *s, int c, size_t n)
>>> ss[i] = c;
>>> return s;
>>> }
>>> +
>>> +void *memmove(void *dest, const void *src, size_t n)
>>> +{
>>> + unsigned int i;
>>> + const char *s = src;
>>> + char *d = dest;
>>> +
>>> + if ((uintptr_t)dest < (uintptr_t)src) {
>>> + for (i = 0; i < n; i++)
>>> + d[i] = s[i];
>>> + } else {
>>> + for (i = n; i > 0; i--)
>>> + d[i - 1] = s[i - 1];
>>> + }
>>> + return dest;
>>> +}
>>> --
>>> 2.28.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/2] lib: decompress_unzstd: Limit output size
2020-08-24 21:05 ` Paul Cercueil
@ 2020-08-24 21:50 ` Nick Terrell
0 siblings, 0 replies; 8+ messages in thread
From: Nick Terrell @ 2020-08-24 21:50 UTC (permalink / raw)
To: Paul Cercueil; +Cc: Thomas Bogendoerfer, linux-mips, linux-kernel, od
> On Aug 24, 2020, at 2:05 PM, Paul Cercueil <paul@crapouillou.net> wrote:
>
>
>
> Le lun. 24 août 2020 à 20:11, Nick Terrell <terrelln@fb.com> a écrit :
>>> On Aug 21, 2020, at 9:29 AM, Paul Cercueil <paul@crapouillou.net> wrote:
>>> The zstd decompression code, as it is right now, will have internal
>>> values overflow on 32-bit systems when the output size is LONG_MAX.
>>> Until someone smarter than me can figure out how to fix the zstd code
>>> properly, limit the destination buffer size to 512 MiB, which should be
>>> enough for everybody, in order to make it usable on 32-bit systems.
>> Can you bump the size up to 2GB? I suspect the problem inside of zstd
>> is an off-by-one error or something similar, so getting closer to the limit
>> shouldn't be a problem. I’d feel more comfortable with 2GB, since
>> kernels can get pretty large.
>
> SZ_1G is the biggest I can go to get the kernel to boot. With SZ_2G it won't boot.
Strange… I don’t quite know what is going on then. Thanks for the fix! You can add:
Reviewed-By: Nick Terrell <terrelln@fb.com>
Best,
Nick
>> Hmm, zstd shouldn’t be overflowing that value. I’m currently preparing
>> a patch to updating the version of zstd in the kernel, and using upstream
>> directly. I will add a test upstream in 32-bit mode to ensure that we don’t
>> overflow a 32-bit size_t, so this will be fixed after the update.
>
> Great, thanks.
>
> Cheers,
> -Paul
>
>> -Nick
>>> Signed-off-by: Paul Cercueil <paul@crapouillou.net>
>>> ---
>>> lib/decompress_unzstd.c | 3 ++-
>>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>> diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c
>>> index 0ad2c15479ed..e1c03b1eaa6e 100644
>>> --- a/lib/decompress_unzstd.c
>>> +++ b/lib/decompress_unzstd.c
>>> @@ -77,6 +77,7 @@
>>> #include <linux/decompress/mm.h>
>>> #include <linux/kernel.h>
>>> +#include <linux/sizes.h>
>>> #include <linux/zstd.h>
>>> /* 128MB is the maximum window size supported by zstd. */
>>> @@ -179,7 +180,7 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len,
>>> size_t ret;
>>> if (out_len == 0)
>>> - out_len = LONG_MAX; /* no limit */
>>> + out_len = SZ_512M; /* should be big enough, right? */
>>> if (fill == NULL && flush == NULL)
>>> /*
>>> --
>>> 2.28.0
>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2020-08-24 21:50 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-08-21 16:29 [PATCH 1/2] lib: decompress_unzstd: Limit output size Paul Cercueil
2020-08-21 16:29 ` [PATCH 2/2] MIPS: Add support for ZSTD-compressed kernels Paul Cercueil
2020-08-24 19:51 ` Nick Terrell
2020-08-24 21:02 ` Paul Cercueil
2020-08-24 21:45 ` Nick Terrell
2020-08-24 20:11 ` [PATCH 1/2] lib: decompress_unzstd: Limit output size Nick Terrell
2020-08-24 21:05 ` Paul Cercueil
2020-08-24 21:50 ` Nick Terrell
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).