Kernel-4.14: With ubuntu-18.04 building rootfs images and booting gives SQUASHFS error: xz decompression failed, data probably corrupt

Kernel-4.14: With ubuntu-18.04 building rootfs images and booting gives SQUASHFS error: xz decompression failed, data probably corrupt

[Pintu Agarwal]

Hi,
Here are few details.
* Linux Kernel: 4.14
* Processor: Qualcomm Arm32 Cortex-A7
* Storage: NAND 512MB
* Platform: Simple busybox
* Filesystem: UBIFS, Squashfs
* Build system: Linux Ubuntu 18.04 with Yocto build system
* Consists of nand raw partitions, squashfs ubi volumes.

What we are trying to do:
We are trying to boot dm-verity enabled rootfs on our system.
The images for rootfs were generated on Ubuntu 18.04 machine using
Yocto build system.

Issue:
Earlier, when we were using Ubuntu 16.04 to generate our images, the
system was booting fine even with dm-verity enabled.
Recently we switched to Ubuntu 18.04 build machine, and now with the
same changes we are seeing the below squashfs error, just after the
mount.
Note: with 18.04 also our rootfs is mounting successfully and
dm-verity is also working fine.
We only get these squashfs errors flooded in the boot logs:
{{{
....
[    5.153479] device-mapper: init: dm-0 is ready
[    5.334282] VFS: Mounted root (squashfs filesystem) readonly on device 253:0.
....
[    8.954120] SQUASHFS error: xz decompression failed, data probably corrupt
[    8.954153] SQUASHFS error: squashfs_read_data failed to read block 0x1106
[    8.970316] SQUASHFS error: Unable to read data cache entry [1106]
[    8.970349] SQUASHFS error: Unable to read page, block 1106, size 776c
[    8.980298] SQUASHFS error: Unable to read data cache entry [1106]
[    8.981911] SQUASHFS error: Unable to read page, block 1106, size 776c
[    8.988280] SQUASHFS error: Unable to read data cache entry [1106]
....
}}}

Note: For dm-verity wwe are trying to append the verity-metadata as
part of our rootfs image like this:
...
218 +    with open(verity_md_img, "rb") as input_file:
219 +        with open(sparse_img, "ab") as out_file:
220 +            for line in input_file:
221 +                out_file.write(line)
....

These changes work fine when we build it with Ubuntu 16.04.
So, we are wondering what could be the issue with Ubuntu 18.04 build ?

If there is any history about it please let us know.


Regards,
Pintu

Re: Kernel-4.14: With ubuntu-18.04 building rootfs images and booting gives SQUASHFS error: xz decompression failed, data probably corrupt

[Pintu Agarwal]

Hi,

On Mon, 8 Nov 2021 at 20:00, Pintu Agarwal <pintu.ping@gmail.com> wrote:
>
> Hi,
> Here are few details.
> * Linux Kernel: 4.14
> * Processor: Qualcomm Arm32 Cortex-A7
> * Storage: NAND 512MB
> * Platform: Simple busybox
> * Filesystem: UBIFS, Squashfs
> * Build system: Linux Ubuntu 18.04 with Yocto build system
> * Consists of nand raw partitions, squashfs ubi volumes.
>
> What we are trying to do:
> We are trying to boot dm-verity enabled rootfs on our system.
> The images for rootfs were generated on Ubuntu 18.04 machine using
> Yocto build system.
>
> Issue:
> Earlier, when we were using Ubuntu 16.04 to generate our images, the
> system was booting fine even with dm-verity enabled.
> Recently we switched to Ubuntu 18.04 build machine, and now with the
> same changes we are seeing the below squashfs error, just after the
> mount.
> Note: with 18.04 also our rootfs is mounting successfully and
> dm-verity is also working fine.
> We only get these squashfs errors flooded in the boot logs:
> {{{
> ....
> [    5.153479] device-mapper: init: dm-0 is ready
> [    5.334282] VFS: Mounted root (squashfs filesystem) readonly on device 253:0.
> ....
> [    8.954120] SQUASHFS error: xz decompression failed, data probably corrupt
> [    8.954153] SQUASHFS error: squashfs_read_data failed to read block 0x1106
> [    8.970316] SQUASHFS error: Unable to read data cache entry [1106]
> [    8.970349] SQUASHFS error: Unable to read page, block 1106, size 776c
> [    8.980298] SQUASHFS error: Unable to read data cache entry [1106]
> [    8.981911] SQUASHFS error: Unable to read page, block 1106, size 776c
> [    8.988280] SQUASHFS error: Unable to read data cache entry [1106]
> ....
> }}}
>
Just one question:
Is there any history about these squashfs errors while cross-compiling
images on Ubuntu 18.04 or higher ?


Thanks,
Pintu

Re: Kernel-4.14: With ubuntu-18.04 building rootfs images and booting gives SQUASHFS error: xz decompression failed, data probably corrupt

[Pintu Agarwal]

On Tue, 9 Nov 2021 at 16:45, Pintu Agarwal <pintu.ping@gmail.com> wrote:
>
> Hi,
>
> On Mon, 8 Nov 2021 at 20:00, Pintu Agarwal <pintu.ping@gmail.com> wrote:
> >
> > Hi,
> > Here are few details.
> > * Linux Kernel: 4.14
> > * Processor: Qualcomm Arm32 Cortex-A7
> > * Storage: NAND 512MB
> > * Platform: Simple busybox
> > * Filesystem: UBIFS, Squashfs
> > * Build system: Linux Ubuntu 18.04 with Yocto build system
> > * Consists of nand raw partitions, squashfs ubi volumes.
> >
> > What we are trying to do:
> > We are trying to boot dm-verity enabled rootfs on our system.
> > The images for rootfs were generated on Ubuntu 18.04 machine using
> > Yocto build system.
> >
> > Issue:
> > Earlier, when we were using Ubuntu 16.04 to generate our images, the
> > system was booting fine even with dm-verity enabled.
> > Recently we switched to Ubuntu 18.04 build machine, and now with the
> > same changes we are seeing the below squashfs error, just after the
> > mount.
> > Note: with 18.04 also our rootfs is mounting successfully and
> > dm-verity is also working fine.
> > We only get these squashfs errors flooded in the boot logs:
> > {{{
> > ....
> > [    5.153479] device-mapper: init: dm-0 is ready
> > [    5.334282] VFS: Mounted root (squashfs filesystem) readonly on device 253:0.
> > ....
> > [    8.954120] SQUASHFS error: xz decompression failed, data probably corrupt
> > [    8.954153] SQUASHFS error: squashfs_read_data failed to read block 0x1106
> > [    8.970316] SQUASHFS error: Unable to read data cache entry [1106]
> > [    8.970349] SQUASHFS error: Unable to read page, block 1106, size 776c
> > [    8.980298] SQUASHFS error: Unable to read data cache entry [1106]
> > [    8.981911] SQUASHFS error: Unable to read page, block 1106, size 776c
> > [    8.988280] SQUASHFS error: Unable to read data cache entry [1106]
> > ....
> > }}}
> >
> Just one question:
> Is there any history about these squashfs errors while cross-compiling
> images on Ubuntu 18.04 or higher ?
>
One quick observation:
This issue is seen only when we enable dm-verity in our bootloader and
cross-building the bootloader/kernel (with Yocto 2.6 toolchain
arm-oe-linux-gnueabi-) on Ubuntu 18.04.
The issue is *NOT* seen (on the same device) when building the
dm-verity enabled kernel on Ubuntu 16.04.

Is it something to do with the cross-toolchain difference between
Ubuntu 16 and 18 ?


Thanks,
Pintu

Re: Kernel-4.14: With ubuntu-18.04 building rootfs images and booting gives SQUASHFS error: xz decompression failed, data probably corrupt

[Pintu Agarwal]

Hi,

On Tue, 9 Nov 2021 at 21:04, Pintu Agarwal <pintu.ping@gmail.com> wrote:

> > > We only get these squashfs errors flooded in the boot logs:
> > > {{{
> > > ....
> > > [    5.153479] device-mapper: init: dm-0 is ready
> > > [    5.334282] VFS: Mounted root (squashfs filesystem) readonly on device 253:0.
> > > ....
> > > [    8.954120] SQUASHFS error: xz decompression failed, data probably corrupt
> > > [    8.954153] SQUASHFS error: squashfs_read_data failed to read block 0x1106
> > > [    8.970316] SQUASHFS error: Unable to read data cache entry [1106]
> > > [    8.970349] SQUASHFS error: Unable to read page, block 1106, size 776c
> > > [    8.980298] SQUASHFS error: Unable to read data cache entry [1106]
> > > [    8.981911] SQUASHFS error: Unable to read page, block 1106, size 776c
> > > [    8.988280] SQUASHFS error: Unable to read data cache entry [1106]
> > > ....
> > > }}}
> > >

One more observation:
When I disable FEC flag in bootloader, I see the below error:
[    8.360791] device-mapper: verity: 253:0: data block 2 is corrupted
[    8.361134] device-mapper: verity: 253:0: data block 3 is corrupted
[    8.366016] SQUASHFS error: squashfs_read_data failed to read block 0x1106
[    8.379652] SQUASHFS error: Unable to read data cache entry [1106]
[    8.379680] SQUASHFS error: Unable to read page, block 1106, size 7770

Also, now I see that the decompress error is gone, but the read error
is still there.

This seems to me that dm-verity detects some corrupted blocks but with
FEC it auto corrects itself, how when dm-verity auto corrects itself,
the squashfs decompression algorithm somehow could not understand it.

So, it seems like there is some mis-match between the way FEC
correction and the squashfs decompression happens ?

Is this issue seen by anybody else here ?


Thanks,
Pintu

Re: Kernel-4.14: With ubuntu-18.04 building rootfs images and booting gives SQUASHFS error: xz decompression failed, data probably corrupt

[Pintu Agarwal]

+ Adding squashfs-devel to get opinion from squashfs side.

On Fri, 12 Nov 2021 at 12:48, Pintu Agarwal <pintu.ping@gmail.com> wrote:
>
> Hi,
>
> On Tue, 9 Nov 2021 at 21:04, Pintu Agarwal <pintu.ping@gmail.com> wrote:
>
> > > > We only get these squashfs errors flooded in the boot logs:
> > > > {{{
> > > > ....
> > > > [    5.153479] device-mapper: init: dm-0 is ready
> > > > [    5.334282] VFS: Mounted root (squashfs filesystem) readonly on device 253:0.
> > > > ....
> > > > [    8.954120] SQUASHFS error: xz decompression failed, data probably corrupt
> > > > [    8.954153] SQUASHFS error: squashfs_read_data failed to read block 0x1106
> > > > [    8.970316] SQUASHFS error: Unable to read data cache entry [1106]
> > > > [    8.970349] SQUASHFS error: Unable to read page, block 1106, size 776c
> > > > [    8.980298] SQUASHFS error: Unable to read data cache entry [1106]
> > > > [    8.981911] SQUASHFS error: Unable to read page, block 1106, size 776c
> > > > [    8.988280] SQUASHFS error: Unable to read data cache entry [1106]
> > > > ....
> > > > }}}
> > > >
>
> One more observation:
> When I disable FEC flag in bootloader, I see the below error:
> [    8.360791] device-mapper: verity: 253:0: data block 2 is corrupted
> [    8.361134] device-mapper: verity: 253:0: data block 3 is corrupted
> [    8.366016] SQUASHFS error: squashfs_read_data failed to read block 0x1106
> [    8.379652] SQUASHFS error: Unable to read data cache entry [1106]
> [    8.379680] SQUASHFS error: Unable to read page, block 1106, size 7770
>
> Also, now I see that the decompress error is gone, but the read error
> is still there.
>
> This seems to me that dm-verity detects some corrupted blocks but with
> FEC it auto corrects itself, how when dm-verity auto corrects itself,
> the squashfs decompression algorithm somehow could not understand it.
>
> So, it seems like there is some mis-match between the way FEC
> correction and the squashfs decompression happens ?
>
> Is this issue seen by anybody else here ?
>

The squashfs version used by Kernel:
[    0.355958] squashfs: version 4.0 (2009/01/31) Phillip Lougher

The squashfs version available on Ubuntu:
mksquashfs version 4.3-git (2014/06/09)

The squashfs version used by Yocto 2.6:
squashfs-tools/0001-squashfs-tools-Allow-setting-selinux-xattrs-through-.patch:61:
   printf("mksquashfs version 4.3-git (2014/09/12)\n");

We create dm-verity squashfs image using version 4.3 whereas, the
kernel uses 4.0 version to decompress it.
Is there something missing here?

When FEC (Forward Error Correction) comes into picture, then squashfs
decompress fails.
When we remove FEC flag from dm-verity then decompress works but read
error still occurs.
This seems as if something is missing either in FEC handling or either
in squashfs decompress logic.

Just wanted to know if there are any fixes already available in the
mainline for this ?


Thanks,
Pintu

Re: Kernel-4.14: With ubuntu-18.04 building rootfs images and booting gives SQUASHFS error: xz decompression failed, data probably corrupt

[Phillip Lougher]

On 14/11/2021 07:06, Pintu Agarwal wrote:
> + Adding squashfs-devel to get opinion from squashfs side.
> 
> On Fri, 12 Nov 2021 at 12:48, Pintu Agarwal <pintu.ping@gmail.com> wrote:
>>
>> Hi,
>>
>> On Tue, 9 Nov 2021 at 21:04, Pintu Agarwal <pintu.ping@gmail.com> wrote:
>>
>>>>> We only get these squashfs errors flooded in the boot logs:
>>>>> {{{
>>>>> ....
>>>>> [    5.153479] device-mapper: init: dm-0 is ready
>>>>> [    5.334282] VFS: Mounted root (squashfs filesystem) readonly on device 253:0.
>>>>> ....
>>>>> [    8.954120] SQUASHFS error: xz decompression failed, data probably corrupt
>>>>> [    8.954153] SQUASHFS error: squashfs_read_data failed to read block 0x1106
>>>>> [    8.970316] SQUASHFS error: Unable to read data cache entry [1106]
>>>>> [    8.970349] SQUASHFS error: Unable to read page, block 1106, size 776c
>>>>> [    8.980298] SQUASHFS error: Unable to read data cache entry [1106]
>>>>> [    8.981911] SQUASHFS error: Unable to read page, block 1106, size 776c
>>>>> [    8.988280] SQUASHFS error: Unable to read data cache entry [1106]
>>>>> ....
>>>>> }}}
>>>>>
>>
>> One more observation:
>> When I disable FEC flag in bootloader, I see the below error:
>> [    8.360791] device-mapper: verity: 253:0: data block 2 is corrupted
>> [    8.361134] device-mapper: verity: 253:0: data block 3 is corrupted
>> [    8.366016] SQUASHFS error: squashfs_read_data failed to read block 0x1106
>> [    8.379652] SQUASHFS error: Unable to read data cache entry [1106]
>> [    8.379680] SQUASHFS error: Unable to read page, block 1106, size 7770
>>
>> Also, now I see that the decompress error is gone, but the read error
>> is still there.
>>
>> This seems to me that dm-verity detects some corrupted blocks but with
>> FEC it auto corrects itself, how when dm-verity auto corrects itself,
>> the squashfs decompression algorithm somehow could not understand it.
>>
>> So, it seems like there is some mis-match between the way FEC
>> correction and the squashfs decompression happens ?
>>
>> Is this issue seen by anybody else here ?
>>
> 
> The squashfs version used by Kernel:
> [    0.355958] squashfs: version 4.0 (2009/01/31) Phillip Lougher
> 
> The squashfs version available on Ubuntu:
> mksquashfs version 4.3-git (2014/06/09)
> 
> The squashfs version used by Yocto 2.6:
> squashfs-tools/0001-squashfs-tools-Allow-setting-selinux-xattrs-through-.patch:61:
>     printf("mksquashfs version 4.3-git (2014/09/12)\n");
> 
> We create dm-verity squashfs image using version 4.3 whereas, the
> kernel uses 4.0 version to decompress it.
> Is there something missing here?
> 
> When FEC (Forward Error Correction) comes into picture, then squashfs
> decompress fails.
> When we remove FEC flag from dm-verity then decompress works but read
> error still occurs.
> This seems as if something is missing either in FEC handling or either
> in squashfs decompress logic.
> 
> Just wanted to know if there are any fixes already available in the
> mainline for this ?
> 
> 

As Squashfs maintainer I want you to stop randomly blaming anything and 
everything here.  You won't fix anything doing that.

In a previous email you stated


> 
> One quick observation:
> This issue is seen only when we enable dm-verity in our bootloader and
> cross-building the bootloader/kernel (with Yocto 2.6 toolchain
> arm-oe-linux-gnueabi-) on Ubuntu 18.04.
> The issue is *NOT* seen (on the same device) when building the
> dm-verity enabled kernel on Ubuntu 16.04.
>
> Is it something to do with the cross-toolchain difference between
> Ubuntu 16 and 18 ?
> 

If that is the case, then it is not an issue with Squashfs or any
kernel code, it is a build time issue and *that* is where you should
be concentrating your efforts.  Find out what differences are there.

You don't seem to understand that a Squashfs filesystem generated
by any Mksquashfs 4.X is mountable *without* errors on any kernel
since 2.6.29 (January 2009).  Looking for mismatches between
Mksquashfs and/or kernel version and blaming that for the above 
different behaviour is a complete waste of time.

Phillip

Re: Kernel-4.14: With ubuntu-18.04 building rootfs images and booting gives SQUASHFS error: xz decompression failed, data probably corrupt

[Pintu Agarwal]

On Mon, 15 Nov 2021 at 00:40, Phillip Lougher <phillip@squashfs.org.uk> wrote:
>
> On 14/11/2021 07:06, Pintu Agarwal wrote:
> > + Adding squashfs-devel to get opinion from squashfs side.
> >
> > On Fri, 12 Nov 2021 at 12:48, Pintu Agarwal <pintu.ping@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> On Tue, 9 Nov 2021 at 21:04, Pintu Agarwal <pintu.ping@gmail.com> wrote:
> >>
> >>>>> We only get these squashfs errors flooded in the boot logs:
> >>>>> {{{
> >>>>> ....
> >>>>> [    5.153479] device-mapper: init: dm-0 is ready
> >>>>> [    5.334282] VFS: Mounted root (squashfs filesystem) readonly on device 253:0.
> >>>>> ....
> >>>>> [    8.954120] SQUASHFS error: xz decompression failed, data probably corrupt
> >>>>> [    8.954153] SQUASHFS error: squashfs_read_data failed to read block 0x1106
> >>>>> [    8.970316] SQUASHFS error: Unable to read data cache entry [1106]
> >>>>> [    8.970349] SQUASHFS error: Unable to read page, block 1106, size 776c
> >>>>> [    8.980298] SQUASHFS error: Unable to read data cache entry [1106]
> >>>>> [    8.981911] SQUASHFS error: Unable to read page, block 1106, size 776c
> >>>>> [    8.988280] SQUASHFS error: Unable to read data cache entry [1106]
> >>>>> ....
> >>>>> }}}
> >>>>>
> >>
> >> One more observation:
> >> When I disable FEC flag in bootloader, I see the below error:
> >> [    8.360791] device-mapper: verity: 253:0: data block 2 is corrupted
> >> [    8.361134] device-mapper: verity: 253:0: data block 3 is corrupted
> >> [    8.366016] SQUASHFS error: squashfs_read_data failed to read block 0x1106
> >> [    8.379652] SQUASHFS error: Unable to read data cache entry [1106]
> >> [    8.379680] SQUASHFS error: Unable to read page, block 1106, size 7770
> >>
> >> Also, now I see that the decompress error is gone, but the read error
> >> is still there.
> >>
> >> This seems to me that dm-verity detects some corrupted blocks but with
> >> FEC it auto corrects itself, how when dm-verity auto corrects itself,
> >> the squashfs decompression algorithm somehow could not understand it.
> >>
> >> So, it seems like there is some mis-match between the way FEC
> >> correction and the squashfs decompression happens ?
> >>
> >> Is this issue seen by anybody else here ?
> >>
> >
> > The squashfs version used by Kernel:
> > [    0.355958] squashfs: version 4.0 (2009/01/31) Phillip Lougher
> >
> > The squashfs version available on Ubuntu:
> > mksquashfs version 4.3-git (2014/06/09)
> >
> > The squashfs version used by Yocto 2.6:
> > squashfs-tools/0001-squashfs-tools-Allow-setting-selinux-xattrs-through-.patch:61:
> >     printf("mksquashfs version 4.3-git (2014/09/12)\n");
> >
> > We create dm-verity squashfs image using version 4.3 whereas, the
> > kernel uses 4.0 version to decompress it.
> > Is there something missing here?
> >
> > When FEC (Forward Error Correction) comes into picture, then squashfs
> > decompress fails.
> > When we remove FEC flag from dm-verity then decompress works but read
> > error still occurs.
> > This seems as if something is missing either in FEC handling or either
> > in squashfs decompress logic.
> >
> > Just wanted to know if there are any fixes already available in the
> > mainline for this ?
> >
> >
>
> As Squashfs maintainer I want you to stop randomly blaming anything and
> everything here.  You won't fix anything doing that.
>
> In a previous email you stated
>
>
> >
> > One quick observation:
> > This issue is seen only when we enable dm-verity in our bootloader and
> > cross-building the bootloader/kernel (with Yocto 2.6 toolchain
> > arm-oe-linux-gnueabi-) on Ubuntu 18.04.
> > The issue is *NOT* seen (on the same device) when building the
> > dm-verity enabled kernel on Ubuntu 16.04.
> >
> > Is it something to do with the cross-toolchain difference between
> > Ubuntu 16 and 18 ?
> >
>
> If that is the case, then it is not an issue with Squashfs or any
> kernel code, it is a build time issue and *that* is where you should
> be concentrating your efforts.  Find out what differences are there.
>
> You don't seem to understand that a Squashfs filesystem generated
> by any Mksquashfs 4.X is mountable *without* errors on any kernel
> since 2.6.29 (January 2009).  Looking for mismatches between
> Mksquashfs and/or kernel version and blaming that for the above
> different behaviour is a complete waste of time.
>

I am sorry, but I am not blaming anybody here.
I am just trying to put my observation here and trying to understand
if someone else have seen a similar issue.
Toolchain side also, it seems the same as it comes from Yocto itself.

It seems there is some relation between dm-verity FEC correction and
squashfs decompression.
So I was looking for some clues from both sides.

Anyways, thank you so much for your suggestion.
Yes, we are analysing the Yocto side build difference as well between
Ubuntu 16 and 18.

Thank you!
Pintu