Linux-KBuild Archive on lore.kernel.org
 help / color / Atom feed
* Re: PROBLEM: zstd bzImage decompression fails for some x86_32 config on 5.9-rc1
       [not found] <20200928085505.GA22244@shbuild999.sh.intel.com>
@ 2020-09-28  9:53 ` Sedat Dilek
  2020-09-28 18:02 ` Nick Terrell
  1 sibling, 0 replies; 5+ messages in thread
From: Sedat Dilek @ 2020-09-28  9:53 UTC (permalink / raw)
  To: Feng Tang
  Cc: Nick Terrell, linux-kernel, x86, Ingo Molnar, Masahiro Yamada,
	linux-kbuild, rong.a.chen, philip.li

On Mon, Sep 28, 2020 at 10:55 AM Feng Tang <feng.tang@intel.com> wrote:
>
> Hi Nick,
>
> 0day has found some kernel decomprssion failure case since 5.9-rc1 (X86_32
> build), and it could be related with ZSTD code, though initially we bisected
> to some other commits.
>
> The error messages are:
>
>         early console in setup code
>         Wrong EFI loader signature.
>         early console in extract_kernel
>         input_data: 0x046f50b4
>         input_len: 0x01ebbeb6
>         output: 0x01000000
>         output_len: 0x04fc535c
>         kernel_total_size: 0x055f5000
>         needed_size: 0x055f5000
>
>         Decompressing Linux...
>
>         ZSTD-compressed data is corrupt
>
> This could be reproduced by compiling the kernel with attached config,
> and use QEMU to boot it.
>
> We suspect it could be related with the kernel size, as we only see
> it on big kernel, and some more info are:
>
> * If we remove a lot of kernel config to build a much smaller kernel,
>   it will boot fine
>
> * If we change the zstd algorithm from zstd22 to zstd19, the kernel will
>   boot fine with below patch
>

Hi,

Recently, Debian has updated initramfs-tools/initramfs-tools-core packages
with ZSTD support:

[ /usr/sbin/mkinitramfs ]

case "${compress}" in
gzip)   # If we're doing a reproducible build, use gzip -n
       if [ -n "${SOURCE_DATE_EPOCH}" ]; then
               compress="gzip -n"
       # Otherwise, substitute pigz if it's available
       elif command -v pigz >/dev/null; then
               compress=pigz
       fi
       ;;
lz4)    compress="lz4 -9 -l" ;;
zstd)   compress="zstd -q -19 -T0" ;;
xz)     compress="xz --check=crc32"
       # If we're not doing a reproducible build, enable multithreading
       test -z "${SOURCE_DATE_EPOCH}" && compress="$compress --threads=0"
       ;;
bzip2|lzma|lzop)
       # no parameters needed
       ;;
*)      echo "W: Unknown compression command ${compress}" >&2 ;;
esac

As you can see it uses compression-level 19 for ZSTD.
Maybe it is good to change from 22 to 19 in the Linux-kernel sources?

Thanks.

Regards,
- Sedat -

>         diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
>         index 3962f59..8fe71ba 100644
>         --- a/arch/x86/boot/compressed/Makefile
>         +++ b/arch/x86/boot/compressed/Makefile
>         @@ -147,7 +147,7 @@ $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE
>          $(obj)/vmlinux.bin.zst: $(vmlinux.bin.all-y) FORCE
>         -       $(call if_changed,zstd22)
>         +       $(call if_changed,zstd)
>
>
> Please let me know if you need more info, and sorry for the late report
> as we just tracked down to this point.
>
> Thanks,
> Feng
>
>
>

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: PROBLEM: zstd bzImage decompression fails for some x86_32 config on 5.9-rc1
       [not found] <20200928085505.GA22244@shbuild999.sh.intel.com>
  2020-09-28  9:53 ` PROBLEM: zstd bzImage decompression fails for some x86_32 config on 5.9-rc1 Sedat Dilek
@ 2020-09-28 18:02 ` Nick Terrell
  2020-09-29  5:15   ` Nick Terrell
  1 sibling, 1 reply; 5+ messages in thread
From: Nick Terrell @ 2020-09-28 18:02 UTC (permalink / raw)
  To: Feng Tang
  Cc: Linux Kernel Mailing List, X86 ML, Sedat Dilek, Ingo Molnar,
	Masahiro Yamada, linux-kbuild, rong.a.chen, philip.li



> On Sep 28, 2020, at 1:55 AM, Feng Tang <feng.tang@intel.com> wrote:
> 
> Hi Nick,
> 
> 0day has found some kernel decomprssion failure case since 5.9-rc1 (X86_32
> build), and it could be related with ZSTD code, though initially we bisected
> to some other commits.
> 
> The error messages are: 
> 	
> 	early console in setup code
> 	Wrong EFI loader signature.
> 	early console in extract_kernel
> 	input_data: 0x046f50b4
> 	input_len: 0x01ebbeb6
> 	output: 0x01000000
> 	output_len: 0x04fc535c
> 	kernel_total_size: 0x055f5000
> 	needed_size: 0x055f5000
> 	
> 	Decompressing Linux...
> 	
> 	ZSTD-compressed data is corrupt
> 
> This could be reproduced by compiling the kernel with attached config,
> and use QEMU to boot it.
> 
> We suspect it could be related with the kernel size, as we only see
> it on big kernel, and some more info are:
> 
> * If we remove a lot of kernel config to build a much smaller kernel,
>  it will boot fine
> 
> * If we change the zstd algorithm from zstd22 to zstd19, the kernel will
>  boot fine with below patch
> 
> 	diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
> 	index 3962f59..8fe71ba 100644
> 	--- a/arch/x86/boot/compressed/Makefile
> 	+++ b/arch/x86/boot/compressed/Makefile
> 	@@ -147,7 +147,7 @@ $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE
> 	 $(obj)/vmlinux.bin.zst: $(vmlinux.bin.all-y) FORCE
> 	-	$(call if_changed,zstd22)
> 	+	$(call if_changed,zstd)
> 
> 
> Please let me know if you need more info, and sorry for the late report
> as we just tracked down to this point.

Thanks for the report, I will look into it today.

Best,
Nick

> Thanks,
> Feng
> 
> 
> 
> <zstd_x86_32.config>


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: PROBLEM: zstd bzImage decompression fails for some x86_32 config on 5.9-rc1
  2020-09-28 18:02 ` Nick Terrell
@ 2020-09-29  5:15   ` Nick Terrell
  2020-09-29  5:47     ` Feng Tang
  0 siblings, 1 reply; 5+ messages in thread
From: Nick Terrell @ 2020-09-29  5:15 UTC (permalink / raw)
  To: Feng Tang
  Cc: Linux Kernel Mailing List, X86 ML, Sedat Dilek, Ingo Molnar,
	Masahiro Yamada, Linux Kbuild mailing list, rong.a.chen,
	philip.li, Petr Malat



> On Sep 28, 2020, at 11:02 AM, Nick Terrell <terrelln@fb.com> wrote:
> 
> 
> 
>> On Sep 28, 2020, at 1:55 AM, Feng Tang <feng.tang@intel.com> wrote:
>> 
>> Hi Nick,
>> 
>> 0day has found some kernel decomprssion failure case since 5.9-rc1 (X86_32
>> build), and it could be related with ZSTD code, though initially we bisected
>> to some other commits.
>> 
>> The error messages are: 
>> 	
>> 	early console in setup code
>> 	Wrong EFI loader signature.
>> 	early console in extract_kernel
>> 	input_data: 0x046f50b4
>> 	input_len: 0x01ebbeb6
>> 	output: 0x01000000
>> 	output_len: 0x04fc535c
>> 	kernel_total_size: 0x055f5000
>> 	needed_size: 0x055f5000
>> 	
>> 	Decompressing Linux...
>> 	
>> 	ZSTD-compressed data is corrupt
>> 
>> This could be reproduced by compiling the kernel with attached config,
>> and use QEMU to boot it.
>> 
>> We suspect it could be related with the kernel size, as we only see
>> it on big kernel, and some more info are:
>> 
>> * If we remove a lot of kernel config to build a much smaller kernel,
>> it will boot fine
>> 
>> * If we change the zstd algorithm from zstd22 to zstd19, the kernel will
>> boot fine with below patch
>> 
>> 	diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
>> 	index 3962f59..8fe71ba 100644
>> 	--- a/arch/x86/boot/compressed/Makefile
>> 	+++ b/arch/x86/boot/compressed/Makefile
>> 	@@ -147,7 +147,7 @@ $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE
>> 	 $(obj)/vmlinux.bin.zst: $(vmlinux.bin.all-y) FORCE
>> 	-	$(call if_changed,zstd22)
>> 	+	$(call if_changed,zstd)
>> 
>> 
>> Please let me know if you need more info, and sorry for the late report
>> as we just tracked down to this point.
> 
> Thanks for the report, I will look into it today.

CC: Petr Malat

I’ve successfully reproduced, and found the issue. It turns out that this
patch [0] from Petr Malat fixes the issue. As I mentioned in that thread, his
fix corresponds to this upstream commit [1].

Can we get Petr's patch merged into v5.9?

This bug only happens when the window size is > 8 MB. A non-kernel workaround
would be to compress the kernel level 19 instead of level 22, which uses an
8 MB window size, instead of a 128 MB window size.

The reason it only shows up for large kernels, is that the code is only buggy
when an offset > 8 MB is used, so a kernel <= 8 MB can't trigger the bug.

Best,
Nick

[0] https://lkml.org/lkml/2020/9/14/94
[1] https://github.com/facebook/zstd/commit/8a5c0c98ae5a7884694589d7a69bc99011add94d

> Best,
> Nick
> 
>> Thanks,
>> Feng
>> 
>> 
>> 
>> <zstd_x86_32.config>


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: PROBLEM: zstd bzImage decompression fails for some x86_32 config on 5.9-rc1
  2020-09-29  5:15   ` Nick Terrell
@ 2020-09-29  5:47     ` Feng Tang
  2020-10-03 18:49       ` Sedat Dilek
  0 siblings, 1 reply; 5+ messages in thread
From: Feng Tang @ 2020-09-29  5:47 UTC (permalink / raw)
  To: Nick Terrell
  Cc: Linux Kernel Mailing List, X86 ML, Sedat Dilek, Ingo Molnar,
	Masahiro Yamada, Linux Kbuild mailing list, rong.a.chen,
	philip.li, Petr Malat

On Tue, Sep 29, 2020 at 05:15:38AM +0000, Nick Terrell wrote:
> 
> 
> > On Sep 28, 2020, at 11:02 AM, Nick Terrell <terrelln@fb.com> wrote:
> > 
> > 
> > 
> >> On Sep 28, 2020, at 1:55 AM, Feng Tang <feng.tang@intel.com> wrote:
> >> 
> >> Hi Nick,
> >> 
> >> 0day has found some kernel decomprssion failure case since 5.9-rc1 (X86_32
> >> build), and it could be related with ZSTD code, though initially we bisected
> >> to some other commits.
> >> 
> >> The error messages are: 
> >> 	Decompressing Linux...
> >> 	
> >> 	ZSTD-compressed data is corrupt
> >> 
> >> This could be reproduced by compiling the kernel with attached config,
> >> and use QEMU to boot it.
> >> 
> >> We suspect it could be related with the kernel size, as we only see
> >> it on big kernel, and some more info are:
> >> 
> >> * If we remove a lot of kernel config to build a much smaller kernel,
> >> it will boot fine
> >> 
> >> * If we change the zstd algorithm from zstd22 to zstd19, the kernel will
> >> boot fine with below patch
> >> 
> >> Please let me know if you need more info, and sorry for the late report
> >> as we just tracked down to this point.
> > 
> > Thanks for the report, I will look into it today.
> 
> CC: Petr Malat
> 
> I’ve successfully reproduced, and found the issue. It turns out that this
> patch [0] from Petr Malat fixes the issue. As I mentioned in that thread, his
> fix corresponds to this upstream commit [1].

Glad to know there is already a fix.

> Can we get Petr's patch merged into v5.9?
> 
> This bug only happens when the window size is > 8 MB. A non-kernel workaround
> would be to compress the kernel level 19 instead of level 22, which uses an
> 8 MB window size, instead of a 128 MB window size.
> 
> The reason it only shows up for large kernels, is that the code is only buggy
> when an offset > 8 MB is used, so a kernel <= 8 MB can't trigger the bug.
> 
> Best,
> Nick
> 
> [0] https://lkml.org/lkml/2020/9/14/94

With this patch, all the failed cases on my side could boot fine.

Tested-by: Feng Tang <feng.tang@intel.com>

Thanks,
Feng

> [1] https://github.com/facebook/zstd/commit/8a5c0c98ae5a7884694589d7a69bc99011add94d



^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: PROBLEM: zstd bzImage decompression fails for some x86_32 config on 5.9-rc1
  2020-09-29  5:47     ` Feng Tang
@ 2020-10-03 18:49       ` Sedat Dilek
  0 siblings, 0 replies; 5+ messages in thread
From: Sedat Dilek @ 2020-10-03 18:49 UTC (permalink / raw)
  To: Feng Tang
  Cc: Nick Terrell, Linux Kernel Mailing List, X86 ML, Ingo Molnar,
	Masahiro Yamada, Linux Kbuild mailing list, rong.a.chen,
	philip.li, Petr Malat

On Tue, Sep 29, 2020 at 7:47 AM Feng Tang <feng.tang@intel.com> wrote:
>
> On Tue, Sep 29, 2020 at 05:15:38AM +0000, Nick Terrell wrote:
> >
> >
> > > On Sep 28, 2020, at 11:02 AM, Nick Terrell <terrelln@fb.com> wrote:
> > >
> > >
> > >
> > >> On Sep 28, 2020, at 1:55 AM, Feng Tang <feng.tang@intel.com> wrote:
> > >>
> > >> Hi Nick,
> > >>
> > >> 0day has found some kernel decomprssion failure case since 5.9-rc1 (X86_32
> > >> build), and it could be related with ZSTD code, though initially we bisected
> > >> to some other commits.
> > >>
> > >> The error messages are:
> > >>    Decompressing Linux...
> > >>
> > >>    ZSTD-compressed data is corrupt
> > >>
> > >> This could be reproduced by compiling the kernel with attached config,
> > >> and use QEMU to boot it.
> > >>
> > >> We suspect it could be related with the kernel size, as we only see
> > >> it on big kernel, and some more info are:
> > >>
> > >> * If we remove a lot of kernel config to build a much smaller kernel,
> > >> it will boot fine
> > >>
> > >> * If we change the zstd algorithm from zstd22 to zstd19, the kernel will
> > >> boot fine with below patch
> > >>
> > >> Please let me know if you need more info, and sorry for the late report
> > >> as we just tracked down to this point.
> > >
> > > Thanks for the report, I will look into it today.
> >
> > CC: Petr Malat
> >
> > I’ve successfully reproduced, and found the issue. It turns out that this
> > patch [0] from Petr Malat fixes the issue. As I mentioned in that thread, his
> > fix corresponds to this upstream commit [1].
>
> Glad to know there is already a fix.
>
> > Can we get Petr's patch merged into v5.9?
> >
> > This bug only happens when the window size is > 8 MB. A non-kernel workaround
> > would be to compress the kernel level 19 instead of level 22, which uses an
> > 8 MB window size, instead of a 128 MB window size.
> >
> > The reason it only shows up for large kernels, is that the code is only buggy
> > when an offset > 8 MB is used, so a kernel <= 8 MB can't trigger the bug.
> >
> > Best,
> > Nick
> >
> > [0] https://lkml.org/lkml/2020/9/14/94
>
> With this patch, all the failed cases on my side could boot fine.
>
> Tested-by: Feng Tang <feng.tang@intel.com>
>

I applied this patch to see if it is OK with x86 64bit - Yes, it is.

Feel free to add my:

     Tested-by: Sedat Dilek <sedat.dilek@gmail.com>

- Sedat -

> Thanks,
> Feng
>
> > [1] https://github.com/facebook/zstd/commit/8a5c0c98ae5a7884694589d7a69bc99011add94d
>
>

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, back to index

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <20200928085505.GA22244@shbuild999.sh.intel.com>
2020-09-28  9:53 ` PROBLEM: zstd bzImage decompression fails for some x86_32 config on 5.9-rc1 Sedat Dilek
2020-09-28 18:02 ` Nick Terrell
2020-09-29  5:15   ` Nick Terrell
2020-09-29  5:47     ` Feng Tang
2020-10-03 18:49       ` Sedat Dilek

Linux-KBuild Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-kbuild/0 linux-kbuild/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-kbuild linux-kbuild/ https://lore.kernel.org/linux-kbuild \
		linux-kbuild@vger.kernel.org
	public-inbox-index linux-kbuild

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kbuild


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git