* Re: PROBLEM: zstd bzImage decompression fails for some x86_32 config on 5.9-rc1 [not found] <20200928085505.GA22244@shbuild999.sh.intel.com> @ 2020-09-28 9:53 ` Sedat Dilek 2020-09-28 18:02 ` Nick Terrell 1 sibling, 0 replies; 5+ messages in thread From: Sedat Dilek @ 2020-09-28 9:53 UTC (permalink / raw) To: Feng Tang Cc: Nick Terrell, linux-kernel, x86, Ingo Molnar, Masahiro Yamada, linux-kbuild, rong.a.chen, philip.li On Mon, Sep 28, 2020 at 10:55 AM Feng Tang <feng.tang@intel.com> wrote: > > Hi Nick, > > 0day has found some kernel decomprssion failure case since 5.9-rc1 (X86_32 > build), and it could be related with ZSTD code, though initially we bisected > to some other commits. > > The error messages are: > > early console in setup code > Wrong EFI loader signature. > early console in extract_kernel > input_data: 0x046f50b4 > input_len: 0x01ebbeb6 > output: 0x01000000 > output_len: 0x04fc535c > kernel_total_size: 0x055f5000 > needed_size: 0x055f5000 > > Decompressing Linux... > > ZSTD-compressed data is corrupt > > This could be reproduced by compiling the kernel with attached config, > and use QEMU to boot it. > > We suspect it could be related with the kernel size, as we only see > it on big kernel, and some more info are: > > * If we remove a lot of kernel config to build a much smaller kernel, > it will boot fine > > * If we change the zstd algorithm from zstd22 to zstd19, the kernel will > boot fine with below patch > Hi, Recently, Debian has updated initramfs-tools/initramfs-tools-core packages with ZSTD support: [ /usr/sbin/mkinitramfs ] case "${compress}" in gzip) # If we're doing a reproducible build, use gzip -n if [ -n "${SOURCE_DATE_EPOCH}" ]; then compress="gzip -n" # Otherwise, substitute pigz if it's available elif command -v pigz >/dev/null; then compress=pigz fi ;; lz4) compress="lz4 -9 -l" ;; zstd) compress="zstd -q -19 -T0" ;; xz) compress="xz --check=crc32" # If we're not doing a reproducible build, enable multithreading test -z "${SOURCE_DATE_EPOCH}" && compress="$compress --threads=0" ;; bzip2|lzma|lzop) # no parameters needed ;; *) echo "W: Unknown compression command ${compress}" >&2 ;; esac As you can see it uses compression-level 19 for ZSTD. Maybe it is good to change from 22 to 19 in the Linux-kernel sources? Thanks. Regards, - Sedat - > diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile > index 3962f59..8fe71ba 100644 > --- a/arch/x86/boot/compressed/Makefile > +++ b/arch/x86/boot/compressed/Makefile > @@ -147,7 +147,7 @@ $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE > $(obj)/vmlinux.bin.zst: $(vmlinux.bin.all-y) FORCE > - $(call if_changed,zstd22) > + $(call if_changed,zstd) > > > Please let me know if you need more info, and sorry for the late report > as we just tracked down to this point. > > Thanks, > Feng > > > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: PROBLEM: zstd bzImage decompression fails for some x86_32 config on 5.9-rc1 [not found] <20200928085505.GA22244@shbuild999.sh.intel.com> 2020-09-28 9:53 ` PROBLEM: zstd bzImage decompression fails for some x86_32 config on 5.9-rc1 Sedat Dilek @ 2020-09-28 18:02 ` Nick Terrell 2020-09-29 5:15 ` Nick Terrell 1 sibling, 1 reply; 5+ messages in thread From: Nick Terrell @ 2020-09-28 18:02 UTC (permalink / raw) To: Feng Tang Cc: Linux Kernel Mailing List, X86 ML, Sedat Dilek, Ingo Molnar, Masahiro Yamada, linux-kbuild, rong.a.chen, philip.li > On Sep 28, 2020, at 1:55 AM, Feng Tang <feng.tang@intel.com> wrote: > > Hi Nick, > > 0day has found some kernel decomprssion failure case since 5.9-rc1 (X86_32 > build), and it could be related with ZSTD code, though initially we bisected > to some other commits. > > The error messages are: > > early console in setup code > Wrong EFI loader signature. > early console in extract_kernel > input_data: 0x046f50b4 > input_len: 0x01ebbeb6 > output: 0x01000000 > output_len: 0x04fc535c > kernel_total_size: 0x055f5000 > needed_size: 0x055f5000 > > Decompressing Linux... > > ZSTD-compressed data is corrupt > > This could be reproduced by compiling the kernel with attached config, > and use QEMU to boot it. > > We suspect it could be related with the kernel size, as we only see > it on big kernel, and some more info are: > > * If we remove a lot of kernel config to build a much smaller kernel, > it will boot fine > > * If we change the zstd algorithm from zstd22 to zstd19, the kernel will > boot fine with below patch > > diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile > index 3962f59..8fe71ba 100644 > --- a/arch/x86/boot/compressed/Makefile > +++ b/arch/x86/boot/compressed/Makefile > @@ -147,7 +147,7 @@ $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE > $(obj)/vmlinux.bin.zst: $(vmlinux.bin.all-y) FORCE > - $(call if_changed,zstd22) > + $(call if_changed,zstd) > > > Please let me know if you need more info, and sorry for the late report > as we just tracked down to this point. Thanks for the report, I will look into it today. Best, Nick > Thanks, > Feng > > > > <zstd_x86_32.config> ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: PROBLEM: zstd bzImage decompression fails for some x86_32 config on 5.9-rc1 2020-09-28 18:02 ` Nick Terrell @ 2020-09-29 5:15 ` Nick Terrell 2020-09-29 5:47 ` Feng Tang 0 siblings, 1 reply; 5+ messages in thread From: Nick Terrell @ 2020-09-29 5:15 UTC (permalink / raw) To: Feng Tang Cc: Linux Kernel Mailing List, X86 ML, Sedat Dilek, Ingo Molnar, Masahiro Yamada, Linux Kbuild mailing list, rong.a.chen, philip.li, Petr Malat > On Sep 28, 2020, at 11:02 AM, Nick Terrell <terrelln@fb.com> wrote: > > > >> On Sep 28, 2020, at 1:55 AM, Feng Tang <feng.tang@intel.com> wrote: >> >> Hi Nick, >> >> 0day has found some kernel decomprssion failure case since 5.9-rc1 (X86_32 >> build), and it could be related with ZSTD code, though initially we bisected >> to some other commits. >> >> The error messages are: >> >> early console in setup code >> Wrong EFI loader signature. >> early console in extract_kernel >> input_data: 0x046f50b4 >> input_len: 0x01ebbeb6 >> output: 0x01000000 >> output_len: 0x04fc535c >> kernel_total_size: 0x055f5000 >> needed_size: 0x055f5000 >> >> Decompressing Linux... >> >> ZSTD-compressed data is corrupt >> >> This could be reproduced by compiling the kernel with attached config, >> and use QEMU to boot it. >> >> We suspect it could be related with the kernel size, as we only see >> it on big kernel, and some more info are: >> >> * If we remove a lot of kernel config to build a much smaller kernel, >> it will boot fine >> >> * If we change the zstd algorithm from zstd22 to zstd19, the kernel will >> boot fine with below patch >> >> diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile >> index 3962f59..8fe71ba 100644 >> --- a/arch/x86/boot/compressed/Makefile >> +++ b/arch/x86/boot/compressed/Makefile >> @@ -147,7 +147,7 @@ $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE >> $(obj)/vmlinux.bin.zst: $(vmlinux.bin.all-y) FORCE >> - $(call if_changed,zstd22) >> + $(call if_changed,zstd) >> >> >> Please let me know if you need more info, and sorry for the late report >> as we just tracked down to this point. > > Thanks for the report, I will look into it today. CC: Petr Malat I’ve successfully reproduced, and found the issue. It turns out that this patch [0] from Petr Malat fixes the issue. As I mentioned in that thread, his fix corresponds to this upstream commit [1]. Can we get Petr's patch merged into v5.9? This bug only happens when the window size is > 8 MB. A non-kernel workaround would be to compress the kernel level 19 instead of level 22, which uses an 8 MB window size, instead of a 128 MB window size. The reason it only shows up for large kernels, is that the code is only buggy when an offset > 8 MB is used, so a kernel <= 8 MB can't trigger the bug. Best, Nick [0] https://lkml.org/lkml/2020/9/14/94 [1] https://github.com/facebook/zstd/commit/8a5c0c98ae5a7884694589d7a69bc99011add94d > Best, > Nick > >> Thanks, >> Feng >> >> >> >> <zstd_x86_32.config> ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: PROBLEM: zstd bzImage decompression fails for some x86_32 config on 5.9-rc1 2020-09-29 5:15 ` Nick Terrell @ 2020-09-29 5:47 ` Feng Tang 2020-10-03 18:49 ` Sedat Dilek 0 siblings, 1 reply; 5+ messages in thread From: Feng Tang @ 2020-09-29 5:47 UTC (permalink / raw) To: Nick Terrell Cc: Linux Kernel Mailing List, X86 ML, Sedat Dilek, Ingo Molnar, Masahiro Yamada, Linux Kbuild mailing list, rong.a.chen, philip.li, Petr Malat On Tue, Sep 29, 2020 at 05:15:38AM +0000, Nick Terrell wrote: > > > > On Sep 28, 2020, at 11:02 AM, Nick Terrell <terrelln@fb.com> wrote: > > > > > > > >> On Sep 28, 2020, at 1:55 AM, Feng Tang <feng.tang@intel.com> wrote: > >> > >> Hi Nick, > >> > >> 0day has found some kernel decomprssion failure case since 5.9-rc1 (X86_32 > >> build), and it could be related with ZSTD code, though initially we bisected > >> to some other commits. > >> > >> The error messages are: > >> Decompressing Linux... > >> > >> ZSTD-compressed data is corrupt > >> > >> This could be reproduced by compiling the kernel with attached config, > >> and use QEMU to boot it. > >> > >> We suspect it could be related with the kernel size, as we only see > >> it on big kernel, and some more info are: > >> > >> * If we remove a lot of kernel config to build a much smaller kernel, > >> it will boot fine > >> > >> * If we change the zstd algorithm from zstd22 to zstd19, the kernel will > >> boot fine with below patch > >> > >> Please let me know if you need more info, and sorry for the late report > >> as we just tracked down to this point. > > > > Thanks for the report, I will look into it today. > > CC: Petr Malat > > I’ve successfully reproduced, and found the issue. It turns out that this > patch [0] from Petr Malat fixes the issue. As I mentioned in that thread, his > fix corresponds to this upstream commit [1]. Glad to know there is already a fix. > Can we get Petr's patch merged into v5.9? > > This bug only happens when the window size is > 8 MB. A non-kernel workaround > would be to compress the kernel level 19 instead of level 22, which uses an > 8 MB window size, instead of a 128 MB window size. > > The reason it only shows up for large kernels, is that the code is only buggy > when an offset > 8 MB is used, so a kernel <= 8 MB can't trigger the bug. > > Best, > Nick > > [0] https://lkml.org/lkml/2020/9/14/94 With this patch, all the failed cases on my side could boot fine. Tested-by: Feng Tang <feng.tang@intel.com> Thanks, Feng > [1] https://github.com/facebook/zstd/commit/8a5c0c98ae5a7884694589d7a69bc99011add94d ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: PROBLEM: zstd bzImage decompression fails for some x86_32 config on 5.9-rc1 2020-09-29 5:47 ` Feng Tang @ 2020-10-03 18:49 ` Sedat Dilek 0 siblings, 0 replies; 5+ messages in thread From: Sedat Dilek @ 2020-10-03 18:49 UTC (permalink / raw) To: Feng Tang Cc: Nick Terrell, Linux Kernel Mailing List, X86 ML, Ingo Molnar, Masahiro Yamada, Linux Kbuild mailing list, rong.a.chen, philip.li, Petr Malat On Tue, Sep 29, 2020 at 7:47 AM Feng Tang <feng.tang@intel.com> wrote: > > On Tue, Sep 29, 2020 at 05:15:38AM +0000, Nick Terrell wrote: > > > > > > > On Sep 28, 2020, at 11:02 AM, Nick Terrell <terrelln@fb.com> wrote: > > > > > > > > > > > >> On Sep 28, 2020, at 1:55 AM, Feng Tang <feng.tang@intel.com> wrote: > > >> > > >> Hi Nick, > > >> > > >> 0day has found some kernel decomprssion failure case since 5.9-rc1 (X86_32 > > >> build), and it could be related with ZSTD code, though initially we bisected > > >> to some other commits. > > >> > > >> The error messages are: > > >> Decompressing Linux... > > >> > > >> ZSTD-compressed data is corrupt > > >> > > >> This could be reproduced by compiling the kernel with attached config, > > >> and use QEMU to boot it. > > >> > > >> We suspect it could be related with the kernel size, as we only see > > >> it on big kernel, and some more info are: > > >> > > >> * If we remove a lot of kernel config to build a much smaller kernel, > > >> it will boot fine > > >> > > >> * If we change the zstd algorithm from zstd22 to zstd19, the kernel will > > >> boot fine with below patch > > >> > > >> Please let me know if you need more info, and sorry for the late report > > >> as we just tracked down to this point. > > > > > > Thanks for the report, I will look into it today. > > > > CC: Petr Malat > > > > I’ve successfully reproduced, and found the issue. It turns out that this > > patch [0] from Petr Malat fixes the issue. As I mentioned in that thread, his > > fix corresponds to this upstream commit [1]. > > Glad to know there is already a fix. > > > Can we get Petr's patch merged into v5.9? > > > > This bug only happens when the window size is > 8 MB. A non-kernel workaround > > would be to compress the kernel level 19 instead of level 22, which uses an > > 8 MB window size, instead of a 128 MB window size. > > > > The reason it only shows up for large kernels, is that the code is only buggy > > when an offset > 8 MB is used, so a kernel <= 8 MB can't trigger the bug. > > > > Best, > > Nick > > > > [0] https://lkml.org/lkml/2020/9/14/94 > > With this patch, all the failed cases on my side could boot fine. > > Tested-by: Feng Tang <feng.tang@intel.com> > I applied this patch to see if it is OK with x86 64bit - Yes, it is. Feel free to add my: Tested-by: Sedat Dilek <sedat.dilek@gmail.com> - Sedat - > Thanks, > Feng > > > [1] https://github.com/facebook/zstd/commit/8a5c0c98ae5a7884694589d7a69bc99011add94d > > ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-10-03 18:49 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <20200928085505.GA22244@shbuild999.sh.intel.com>
2020-09-28 9:53 ` PROBLEM: zstd bzImage decompression fails for some x86_32 config on 5.9-rc1 Sedat Dilek
2020-09-28 18:02 ` Nick Terrell
2020-09-29 5:15 ` Nick Terrell
2020-09-29 5:47 ` Feng Tang
2020-10-03 18:49 ` Sedat Dilek
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).