Kernel 4.14: SQUASHFS error: xz decompression failed, data probably corrupt

Kernel 4.14: SQUASHFS error: xz decompression failed, data probably corrupt

[Pintu Agarwal]

Hi All,

With Kernel 4.14 we are getting squashfs error during bootup resulting
in kernel panic.
The details are below:
Device: ARM-32 board with Cortex-A7 (Single Core)
Storage: NAND Flash 512MiB
Kernel Version: 4.14.170 (maybe with some Linaro updates)
File system: Simple busybox with systemd (without Android)
File system type: UBIFS + SQUASHFS
UBI Volumes supported: rootfs (ro), others (rw)
-------------------

When we try to flash the UBI images and then try to boot the device,
we observe the below errors:
{{{
[    5.608810] SQUASHFS error: xz decompression failed, data probably corrupt
[    5.608846] SQUASHFS error: squashfs_read_data failed to read block 0x4d7ffe
[    5.614745] SQUASHFS error: Unable to read data cache entry [4d7ffe]
[    5.621939] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
[    5.628274] SQUASHFS error: Unable to read data cache entry [4d7ffe]
[    5.634934] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
[    5.641309] SQUASHFS error: Unable to read data cache entry [4d7ffe]
[    5.647954] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
[    5.654304] SQUASHFS error: Unable to read data cache entry [4d7ffe]
[    5.660977] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
[    5.667309] SQUASHFS error: Unable to read data cache entry [4d7ffe]
[    5.673997] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
[    5.680497] Kernel panic - not syncing: Attempted to kill init!
exitcode=0x00007f00
[....]
}}}
We also observed that some of our Yocto build images will work and
boots fine, while sometimes the build images cause this issue.

So we wanted to know:
a) What could be the root cause of this issue ?
b) Is it related to squashfs ?
c) If yes, are there any fixes available already in the latest mainline ?
    Please share some references.

Please let us know if anybody encountered this similar issue with
squashfs and how did you handle it ?

Note:
Our current commit in fs/squashfs is pointing at:
Squashfs: Compute expected length from inode size rather than block length


Thanks,
Pintu

Kernel 4.14: SQUASHFS error: xz decompression failed, data probably corrupt

[Pintu Agarwal]

Hi All,

With Kernel 4.14 we are getting squashfs error during bootup resulting
in kernel panic.
The details are below:
Device: ARM-32 board with Cortex-A7 (Single Core)
Storage: NAND Flash 512MiB
Kernel Version: 4.14.170 (maybe with some Linaro updates)
File system: Simple busybox with systemd (without Android)
File system type: UBIFS + SQUASHFS
UBI Volumes supported: rootfs (ro), others (rw)
-------------------

When we try to flash the UBI images and then try to boot the device,
we observe the below errors:
{{{
[    5.608810] SQUASHFS error: xz decompression failed, data probably corrupt
[    5.608846] SQUASHFS error: squashfs_read_data failed to read block 0x4d7ffe
[    5.614745] SQUASHFS error: Unable to read data cache entry [4d7ffe]
[    5.621939] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
[    5.628274] SQUASHFS error: Unable to read data cache entry [4d7ffe]
[    5.634934] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
[    5.641309] SQUASHFS error: Unable to read data cache entry [4d7ffe]
[    5.647954] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
[    5.654304] SQUASHFS error: Unable to read data cache entry [4d7ffe]
[    5.660977] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
[    5.667309] SQUASHFS error: Unable to read data cache entry [4d7ffe]
[    5.673997] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
[    5.680497] Kernel panic - not syncing: Attempted to kill init!
exitcode=0x00007f00
[....]
}}}
We also observed that some of our Yocto build images will work and
boots fine, while sometimes the build images cause this issue.

So we wanted to know:
a) What could be the root cause of this issue ?
b) Is it related to squashfs ?
c) If yes, are there any fixes available already in the latest mainline ?
    Please share some references.

Please let us know if anybody encountered this similar issue with
squashfs and how did you handle it ?

Note:
Our current commit in fs/squashfs is pointing at:
Squashfs: Compute expected length from inode size rather than block length


Thanks,
Pintu

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Kernel 4.14: SQUASHFS error: xz decompression failed, data probably corrupt

[Florian Fainelli]

On 6/14/2021 3:39 AM, Pintu Agarwal wrote:
> Hi All,
> 
> With Kernel 4.14 we are getting squashfs error during bootup resulting
> in kernel panic.
> The details are below:
> Device: ARM-32 board with Cortex-A7 (Single Core)
> Storage: NAND Flash 512MiB
> Kernel Version: 4.14.170 (maybe with some Linaro updates)
> File system: Simple busybox with systemd (without Android)
> File system type: UBIFS + SQUASHFS
> UBI Volumes supported: rootfs (ro), others (rw)
> -------------------
> 
> When we try to flash the UBI images and then try to boot the device,
> we observe the below errors:

Someone in The OpenWrt community seems to have run into this problem,
possibly on the exact same QCOM SoC than you and came up with the following:

https://forum.openwrt.org/t/patch-squashfs-data-probably-corrupt/70480

> {{{
> [    5.608810] SQUASHFS error: xz decompression failed, data probably corrupt
> [    5.608846] SQUASHFS error: squashfs_read_data failed to read block 0x4d7ffe
> [    5.614745] SQUASHFS error: Unable to read data cache entry [4d7ffe]
> [    5.621939] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
> [    5.628274] SQUASHFS error: Unable to read data cache entry [4d7ffe]
> [    5.634934] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
> [    5.641309] SQUASHFS error: Unable to read data cache entry [4d7ffe]
> [    5.647954] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
> [    5.654304] SQUASHFS error: Unable to read data cache entry [4d7ffe]
> [    5.660977] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
> [    5.667309] SQUASHFS error: Unable to read data cache entry [4d7ffe]
> [    5.673997] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
> [    5.680497] Kernel panic - not syncing: Attempted to kill init!
> exitcode=0x00007f00
> [....]
> }}}
> We also observed that some of our Yocto build images will work and
> boots fine, while sometimes the build images cause this issue.
> 
> So we wanted to know:
> a) What could be the root cause of this issue ?
> b) Is it related to squashfs ?
> c) If yes, are there any fixes available already in the latest mainline ?
>     Please share some references.
> 
> Please let us know if anybody encountered this similar issue with
> squashfs and how did you handle it ?
> 
> Note:
> Our current commit in fs/squashfs is pointing at:
> Squashfs: Compute expected length from inode size rather than block length
> 
> 
> Thanks,
> Pintu
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
> 

-- 
Florian

Re: Kernel 4.14: SQUASHFS error: xz decompression failed, data probably corrupt

[Florian Fainelli]

On 6/14/2021 3:39 AM, Pintu Agarwal wrote:
> Hi All,
> 
> With Kernel 4.14 we are getting squashfs error during bootup resulting
> in kernel panic.
> The details are below:
> Device: ARM-32 board with Cortex-A7 (Single Core)
> Storage: NAND Flash 512MiB
> Kernel Version: 4.14.170 (maybe with some Linaro updates)
> File system: Simple busybox with systemd (without Android)
> File system type: UBIFS + SQUASHFS
> UBI Volumes supported: rootfs (ro), others (rw)
> -------------------
> 
> When we try to flash the UBI images and then try to boot the device,
> we observe the below errors:

Someone in The OpenWrt community seems to have run into this problem,
possibly on the exact same QCOM SoC than you and came up with the following:

https://forum.openwrt.org/t/patch-squashfs-data-probably-corrupt/70480

> {{{
> [    5.608810] SQUASHFS error: xz decompression failed, data probably corrupt
> [    5.608846] SQUASHFS error: squashfs_read_data failed to read block 0x4d7ffe
> [    5.614745] SQUASHFS error: Unable to read data cache entry [4d7ffe]
> [    5.621939] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
> [    5.628274] SQUASHFS error: Unable to read data cache entry [4d7ffe]
> [    5.634934] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
> [    5.641309] SQUASHFS error: Unable to read data cache entry [4d7ffe]
> [    5.647954] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
> [    5.654304] SQUASHFS error: Unable to read data cache entry [4d7ffe]
> [    5.660977] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
> [    5.667309] SQUASHFS error: Unable to read data cache entry [4d7ffe]
> [    5.673997] SQUASHFS error: Unable to read page, block 4d7ffe, size 7a3c
> [    5.680497] Kernel panic - not syncing: Attempted to kill init!
> exitcode=0x00007f00
> [....]
> }}}
> We also observed that some of our Yocto build images will work and
> boots fine, while sometimes the build images cause this issue.
> 
> So we wanted to know:
> a) What could be the root cause of this issue ?
> b) Is it related to squashfs ?
> c) If yes, are there any fixes available already in the latest mainline ?
>     Please share some references.
> 
> Please let us know if anybody encountered this similar issue with
> squashfs and how did you handle it ?
> 
> Note:
> Our current commit in fs/squashfs is pointing at:
> Squashfs: Compute expected length from inode size rather than block length
> 
> 
> Thanks,
> Pintu
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
> 

-- 
Florian

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Kernel 4.14: SQUASHFS error: xz decompression failed, data probably corrupt

[Pintu Agarwal]

On Tue, 15 Jun 2021 at 03:53, Florian Fainelli <f.fainelli@gmail.com> wrote:
>
>
>
> On 6/14/2021 3:39 AM, Pintu Agarwal wrote:
> > Hi All,
> >
> > With Kernel 4.14 we are getting squashfs error during bootup resulting
> > in kernel panic.
> > The details are below:
> > Device: ARM-32 board with Cortex-A7 (Single Core)
> > Storage: NAND Flash 512MiB
> > Kernel Version: 4.14.170 (maybe with some Linaro updates)
> > File system: Simple busybox with systemd (without Android)
> > File system type: UBIFS + SQUASHFS
> > UBI Volumes supported: rootfs (ro), others (rw)
> > -------------------
> >
> > When we try to flash the UBI images and then try to boot the device,
> > we observe the below errors:
>
> Someone in The OpenWrt community seems to have run into this problem,
> possibly on the exact same QCOM SoC than you and came up with the following:
>
> https://forum.openwrt.org/t/patch-squashfs-data-probably-corrupt/70480
>
Thanks!
Yes I have already seen this and even one more.
https://www.programmersought.com/article/31513579159/

But I think these changes are not yet in the mainline right ?

So, I wanted to know which are the exact patches which are already
accepted in mainline ?
Or, is it already mainlined ?

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/log/fs/squashfs?h=next-20210611
From here, I see that we are only till this:
==> 2018-08-02: Squashfs: Compute expected length from inode size
rather than block length



Thanks,
Pintu

Re: Kernel 4.14: SQUASHFS error: xz decompression failed, data probably corrupt

[Pintu Agarwal]

On Tue, 15 Jun 2021 at 03:53, Florian Fainelli <f.fainelli@gmail.com> wrote:
>
>
>
> On 6/14/2021 3:39 AM, Pintu Agarwal wrote:
> > Hi All,
> >
> > With Kernel 4.14 we are getting squashfs error during bootup resulting
> > in kernel panic.
> > The details are below:
> > Device: ARM-32 board with Cortex-A7 (Single Core)
> > Storage: NAND Flash 512MiB
> > Kernel Version: 4.14.170 (maybe with some Linaro updates)
> > File system: Simple busybox with systemd (without Android)
> > File system type: UBIFS + SQUASHFS
> > UBI Volumes supported: rootfs (ro), others (rw)
> > -------------------
> >
> > When we try to flash the UBI images and then try to boot the device,
> > we observe the below errors:
>
> Someone in The OpenWrt community seems to have run into this problem,
> possibly on the exact same QCOM SoC than you and came up with the following:
>
> https://forum.openwrt.org/t/patch-squashfs-data-probably-corrupt/70480
>
Thanks!
Yes I have already seen this and even one more.
https://www.programmersought.com/article/31513579159/

But I think these changes are not yet in the mainline right ?

So, I wanted to know which are the exact patches which are already
accepted in mainline ?
Or, is it already mainlined ?

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/log/fs/squashfs?h=next-20210611
From here, I see that we are only till this:
==> 2018-08-02: Squashfs: Compute expected length from inode size
rather than block length



Thanks,
Pintu

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Kernel 4.14: SQUASHFS error: xz decompression failed, data probably corrupt

[Pintu Agarwal]

On Tue, 15 Jun 2021 at 10:42, Pintu Agarwal <pintu.ping@gmail.com> wrote:
>
> On Tue, 15 Jun 2021 at 03:53, Florian Fainelli <f.fainelli@gmail.com> wrote:
> >
> >
> >
> > On 6/14/2021 3:39 AM, Pintu Agarwal wrote:
> > > Hi All,
> > >
> > > With Kernel 4.14 we are getting squashfs error during bootup resulting
> > > in kernel panic.
> > > The details are below:
> > > Device: ARM-32 board with Cortex-A7 (Single Core)
> > > Storage: NAND Flash 512MiB
> > > Kernel Version: 4.14.170 (maybe with some Linaro updates)
> > > File system: Simple busybox with systemd (without Android)
> > > File system type: UBIFS + SQUASHFS
> > > UBI Volumes supported: rootfs (ro), others (rw)
> > > -------------------
> > >
> > > When we try to flash the UBI images and then try to boot the device,
> > > we observe the below errors:
> >
> > Someone in The OpenWrt community seems to have run into this problem,
> > possibly on the exact same QCOM SoC than you and came up with the following:
> >
> > https://forum.openwrt.org/t/patch-squashfs-data-probably-corrupt/70480
> >
> Thanks!
> Yes I have already seen this and even one more.
> https://www.programmersought.com/article/31513579159/
>
> But I think these changes are not yet in the mainline right ?
>
> So, I wanted to know which are the exact patches which are already
> accepted in mainline ?
> Or, is it already mainlined ?
>
> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/log/fs/squashfs?h=next-20210611
> From here, I see that we are only till this:
> ==> 2018-08-02: Squashfs: Compute expected length from inode size
> rather than block length
>
@Phillip Lougher, do you have any suggestions/comments on these errors ?
Why do you think these errors occur ?
Also, I noticed that even if these errors occur, the device may boot normally.
However, for some people it does not boot at all.

Thanks,
Pintu

Re: Kernel 4.14: SQUASHFS error: xz decompression failed, data probably corrupt

[Pintu Agarwal]

On Tue, 15 Jun 2021 at 10:42, Pintu Agarwal <pintu.ping@gmail.com> wrote:
>
> On Tue, 15 Jun 2021 at 03:53, Florian Fainelli <f.fainelli@gmail.com> wrote:
> >
> >
> >
> > On 6/14/2021 3:39 AM, Pintu Agarwal wrote:
> > > Hi All,
> > >
> > > With Kernel 4.14 we are getting squashfs error during bootup resulting
> > > in kernel panic.
> > > The details are below:
> > > Device: ARM-32 board with Cortex-A7 (Single Core)
> > > Storage: NAND Flash 512MiB
> > > Kernel Version: 4.14.170 (maybe with some Linaro updates)
> > > File system: Simple busybox with systemd (without Android)
> > > File system type: UBIFS + SQUASHFS
> > > UBI Volumes supported: rootfs (ro), others (rw)
> > > -------------------
> > >
> > > When we try to flash the UBI images and then try to boot the device,
> > > we observe the below errors:
> >
> > Someone in The OpenWrt community seems to have run into this problem,
> > possibly on the exact same QCOM SoC than you and came up with the following:
> >
> > https://forum.openwrt.org/t/patch-squashfs-data-probably-corrupt/70480
> >
> Thanks!
> Yes I have already seen this and even one more.
> https://www.programmersought.com/article/31513579159/
>
> But I think these changes are not yet in the mainline right ?
>
> So, I wanted to know which are the exact patches which are already
> accepted in mainline ?
> Or, is it already mainlined ?
>
> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/log/fs/squashfs?h=next-20210611
> From here, I see that we are only till this:
> ==> 2018-08-02: Squashfs: Compute expected length from inode size
> rather than block length
>
@Phillip Lougher, do you have any suggestions/comments on these errors ?
Why do you think these errors occur ?
Also, I noticed that even if these errors occur, the device may boot normally.
However, for some people it does not boot at all.

Thanks,
Pintu

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Kernel 4.14: SQUASHFS error: xz decompression failed, data probably corrupt

[Pintu Agarwal]

Hi,

On Tue, 15 Jun 2021 at 15:31, Pintu Agarwal <pintu.ping@gmail.com> wrote:
>
> On Tue, 15 Jun 2021 at 10:42, Pintu Agarwal <pintu.ping@gmail.com> wrote:
> >
> > On Tue, 15 Jun 2021 at 03:53, Florian Fainelli <f.fainelli@gmail.com> wrote:
> > >
> > >
> > >
> > > On 6/14/2021 3:39 AM, Pintu Agarwal wrote:
> > > > Hi All,
> > > >
> > > > With Kernel 4.14 we are getting squashfs error during bootup resulting
> > > > in kernel panic.
> > > > The details are below:
> > > > Device: ARM-32 board with Cortex-A7 (Single Core)
> > > > Storage: NAND Flash 512MiB
> > > > Kernel Version: 4.14.170 (maybe with some Linaro updates)
> > > > File system: Simple busybox with systemd (without Android)
> > > > File system type: UBIFS + SQUASHFS
> > > > UBI Volumes supported: rootfs (ro), others (rw)
> > > > -------------------
> > > >
> > > > When we try to flash the UBI images and then try to boot the device,
> > > > we observe the below errors:
> > >
> > > Someone in The OpenWrt community seems to have run into this problem,
> > > possibly on the exact same QCOM SoC than you and came up with the following:
> > >
> > > https://forum.openwrt.org/t/patch-squashfs-data-probably-corrupt/70480
> > >
> > Thanks!
> > Yes I have already seen this and even one more.
> > https://www.programmersought.com/article/31513579159/
> >
> > But I think these changes are not yet in the mainline right ?
> >
> > So, I wanted to know which are the exact patches which are already
> > accepted in mainline ?
> > Or, is it already mainlined ?
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/log/fs/squashfs?h=next-20210611
> > From here, I see that we are only till this:
> > ==> 2018-08-02: Squashfs: Compute expected length from inode size
> > rather than block length
> >
> @Phillip Lougher, do you have any suggestions/comments on these errors ?
> Why do you think these errors occur ?
> Also, I noticed that even if these errors occur, the device may boot normally.
> However, for some people it does not boot at all.
>

It seems we have fixed this issue now from bootloader.
I think it is related to -F (free space fixup) flag for ubifs partition.
http://www.linux-mtd.infradead.org/faq/ubifs.html#L_free_space_fixup

During flashing, we are trying to remove -F flag from ubifs image, but
I think this should be avoided for squashfs image.
I guess this issue might occur if we are trying to mixup
squashfs/ubifs volumes together ?
Or, we are trying to flash squashfs image on UBIFS volume ?

Re: Kernel 4.14: SQUASHFS error: xz decompression failed, data probably corrupt

[Pintu Agarwal]

Hi,

On Tue, 15 Jun 2021 at 15:31, Pintu Agarwal <pintu.ping@gmail.com> wrote:
>
> On Tue, 15 Jun 2021 at 10:42, Pintu Agarwal <pintu.ping@gmail.com> wrote:
> >
> > On Tue, 15 Jun 2021 at 03:53, Florian Fainelli <f.fainelli@gmail.com> wrote:
> > >
> > >
> > >
> > > On 6/14/2021 3:39 AM, Pintu Agarwal wrote:
> > > > Hi All,
> > > >
> > > > With Kernel 4.14 we are getting squashfs error during bootup resulting
> > > > in kernel panic.
> > > > The details are below:
> > > > Device: ARM-32 board with Cortex-A7 (Single Core)
> > > > Storage: NAND Flash 512MiB
> > > > Kernel Version: 4.14.170 (maybe with some Linaro updates)
> > > > File system: Simple busybox with systemd (without Android)
> > > > File system type: UBIFS + SQUASHFS
> > > > UBI Volumes supported: rootfs (ro), others (rw)
> > > > -------------------
> > > >
> > > > When we try to flash the UBI images and then try to boot the device,
> > > > we observe the below errors:
> > >
> > > Someone in The OpenWrt community seems to have run into this problem,
> > > possibly on the exact same QCOM SoC than you and came up with the following:
> > >
> > > https://forum.openwrt.org/t/patch-squashfs-data-probably-corrupt/70480
> > >
> > Thanks!
> > Yes I have already seen this and even one more.
> > https://www.programmersought.com/article/31513579159/
> >
> > But I think these changes are not yet in the mainline right ?
> >
> > So, I wanted to know which are the exact patches which are already
> > accepted in mainline ?
> > Or, is it already mainlined ?
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/log/fs/squashfs?h=next-20210611
> > From here, I see that we are only till this:
> > ==> 2018-08-02: Squashfs: Compute expected length from inode size
> > rather than block length
> >
> @Phillip Lougher, do you have any suggestions/comments on these errors ?
> Why do you think these errors occur ?
> Also, I noticed that even if these errors occur, the device may boot normally.
> However, for some people it does not boot at all.
>

It seems we have fixed this issue now from bootloader.
I think it is related to -F (free space fixup) flag for ubifs partition.
http://www.linux-mtd.infradead.org/faq/ubifs.html#L_free_space_fixup

During flashing, we are trying to remove -F flag from ubifs image, but
I think this should be avoided for squashfs image.
I guess this issue might occur if we are trying to mixup
squashfs/ubifs volumes together ?
Or, we are trying to flash squashfs image on UBIFS volume ?

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/