Seeing ubifs unmap LEB error with linux 2.6.35 kernel

Seeing ubifs unmap LEB error with linux 2.6.35 kernel

[Kiran Bangalore Sathyanarayana (kibangal)]

Hi,
We are seeing the below ubifs errors with linux 2.6.35 Kernel:

3:2019 Jul 21 13:13:03 UTC:4.0(2h):kernel:-:<3>UBIFS error (pid 24831): ubifs_leb_unmap: unmap LEB -1 failed, error -22
4:2019 Jul 21 13:13:03 UTC:4.0(2h):kernel:-:<4>UBIFS warning (pid 24831): ubifs_ro_mode: switched to read-only mode, error -22
3:2019 Jul 21 13:13:03 UTC:4.0(2h):kernel:-:<3>UBIFS error (pid 24831): make_reservation: cannot reserve 4144 bytes in jhead 2, error -22
3:2019 Jul 21 13:13:03 UTC:4.0(2h):kernel:-:<3>UBIFS error (pid 24831): do_writepage: cannot write page 98 of inode 67, error -22
3:2019 Jul 21 13:13:03 UTC:4.0(2h):kernel:-:<3>UBIFS error (pid 24581): make_reservation: cannot reserve 4144 bytes in jhead 2, error -30
3:2019 Jul 21 13:13:03 UTC:4.0(2h):kernel:-:<3>UBIFS error (pid 24581): do_writepage: cannot write page 99 of inode 67, error -30
3:2019 Jul 21 13:13:03 UTC:4.0(2h):kernel:-:<3>Buffer I/O error on device loop2, logical block 1

The system has a NAND flash with ubifs filesystem installed. The above errors happened randomly at runtime and ubifs switched to read-only mode.
The NAND flash itself is fine and there are no badblocks detected.
"ubinfo -a" output is as below:
UBI version:                    1
Count of UBI devices:           1
UBI control device major/minor: 10:63
Present UBI devices:            ubi0

ubi0:
Volumes count:                           1
Logical eraseblock size:                 126976
Total amount of logical eraseblocks:     1024 (130023424 bytes, 124.0 MiB)
Amount of available logical eraseblocks: 0 (0 bytes)
Maximum count of volumes                 128
Count of bad physical eraseblocks:       0
Count of reserved physical eraseblocks:  10
Current maximum erase counter value:     536
Minimum input/output unit size:          2048 bytes
Character device major/minor:            235:0
Cisco UBI Flash Worn:                    No
Present volumes:                         0

Volume ID:   0 (on ubi0)
Type:        dynamic
Alignment:   1
Size:        1010 LEBs (128245760 bytes, 122.3 MiB)
State:       OK
Name:        data
Character device major/minor: 235:1

This has occurred on multiple systems now.
Can you please provide me more info on this error? Is this any known issue with linux 2.6.35 Kernel?
Please let me know how to proceed with fixing this(any specific patch that needs to be specifically applied on 2.6.35 kernel).

Regards,
Kiran.



______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Seeing ubifs unmap LEB error with linux 2.6.35 kernel

[Richard Weinberger]

On Thu, Oct 17, 2019 at 7:38 PM Kiran Bangalore Sathyanarayana
(kibangal) <kibangal@cisco.com> wrote:
>
> Hi,
> We are seeing the below ubifs errors with linux 2.6.35 Kernel:
>
> 3:2019 Jul 21 13:13:03 UTC:4.0(2h):kernel:-:<3>UBIFS error (pid 24831): ubifs_leb_unmap: unmap LEB -1 failed, error -22
> 4:2019 Jul 21 13:13:03 UTC:4.0(2h):kernel:-:<4>UBIFS warning (pid 24831): ubifs_ro_mode: switched to read-only mode, error -22
> 3:2019 Jul 21 13:13:03 UTC:4.0(2h):kernel:-:<3>UBIFS error (pid 24831): make_reservation: cannot reserve 4144 bytes in jhead 2, error -22
> 3:2019 Jul 21 13:13:03 UTC:4.0(2h):kernel:-:<3>UBIFS error (pid 24831): do_writepage: cannot write page 98 of inode 67, error -22
> 3:2019 Jul 21 13:13:03 UTC:4.0(2h):kernel:-:<3>UBIFS error (pid 24581): make_reservation: cannot reserve 4144 bytes in jhead 2, error -30
> 3:2019 Jul 21 13:13:03 UTC:4.0(2h):kernel:-:<3>UBIFS error (pid 24581): do_writepage: cannot write page 99 of inode 67, error -30
> 3:2019 Jul 21 13:13:03 UTC:4.0(2h):kernel:-:<3>Buffer I/O error on device loop2, logical block 1
>
> The system has a NAND flash with ubifs filesystem installed. The above errors happened randomly at runtime and ubifs switched to read-only mode.
> The NAND flash itself is fine and there are no badblocks detected.
> "ubinfo -a" output is as below:
> UBI version:                    1
> Count of UBI devices:           1
> UBI control device major/minor: 10:63
> Present UBI devices:            ubi0
>
> ubi0:
> Volumes count:                           1
> Logical eraseblock size:                 126976
> Total amount of logical eraseblocks:     1024 (130023424 bytes, 124.0 MiB)
> Amount of available logical eraseblocks: 0 (0 bytes)
> Maximum count of volumes                 128
> Count of bad physical eraseblocks:       0
> Count of reserved physical eraseblocks:  10
> Current maximum erase counter value:     536
> Minimum input/output unit size:          2048 bytes
> Character device major/minor:            235:0
> Cisco UBI Flash Worn:                    No
> Present volumes:                         0
>
> Volume ID:   0 (on ubi0)
> Type:        dynamic
> Alignment:   1
> Size:        1010 LEBs (128245760 bytes, 122.3 MiB)
> State:       OK
> Name:        data
> Character device major/minor: 235:1
>
> This has occurred on multiple systems now.
> Can you please provide me more info on this error? Is this any known issue with linux 2.6.35 Kernel?

Aren't there more logs of UBI or UBIFS?

> Please let me know how to proceed with fixing this(any specific patch that needs to be specifically applied on 2.6.35 kernel).

Can you please reproduce using a recent kernel?

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Seeing ubifs unmap LEB error with linux 2.6.35 kernel

[Richard Weinberger]

Kiran,

----- Ursprüngliche Mail -----
> Von: "Kiran B S" <bs.kiran81@gmail.com>
> An: "Richard Weinberger" <richard.weinberger@gmail.com>, "linux-mtd" <linux-mtd@lists.infradead.org>
> Gesendet: Freitag, 18. Oktober 2019 11:46:44
> Betreff: Re: Seeing ubifs unmap LEB error with linux 2.6.35 kernel

> Hi Richard,
> Also a reboot of the system recovered the ubifs filesystem.

Okay. This indicates that the problem is not fatal.
Initially we set the GC-Head to -1, unmapping the GC-Head is
a common operation in UBIFS, maybe there is some logic bug in the
replay code.

Can you please give the following change a try?
diff --git a/drivers/mtd/ubi/kapi.c b/drivers/mtd/ubi/kapi.c
index 69fa4ef03c53..03ce752d4d1e 100644
--- a/drivers/mtd/ubi/kapi.c
+++ b/drivers/mtd/ubi/kapi.c
@@ -610,8 +610,10 @@ int ubi_leb_unmap(struct ubi_volume_desc *desc, int lnum)
        if (desc->mode == UBI_READONLY || vol->vol_type == UBI_STATIC_VOLUME)
                return -EROFS;
 
-       if (lnum < 0 || lnum >= vol->reserved_pebs)
+       if (lnum < 0 || lnum >= vol->reserved_pebs) {
+               dump_stack();
                return -EINVAL;
+       }
 
        if (vol->upd_marker)
                return -EBADF;

Using dump_stack() we can see who tries to unmap a bad LEB entry.

After 2.6.35 UBIFS was changed a lot maybe the issue got fixed and
is not present in a recent kernel.

Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Seeing ubifs unmap LEB error with linux 2.6.35 kernel

[Richard Weinberger]

On Thu, Jan 9, 2020 at 9:37 AM Kiran B S <bs.kiran81@gmail.com> wrote:
>> On Tue, Jan 7, 2020 at 6:01 PM Kiran B S <bs.kiran81@gmail.com> wrote:
>>>
>>> Hi Richard,
>>> We could hit this error now on multiple systems. An attempt to write some data was done during this time.
>>> Please find below the stack dump trace as per the changes that you had suggested:
>>>
>>> Stack dump trace:
>>> [dump_backtrace_entry]:59:[<c04ca258>] (unwind_backtrace+0x0/0xe8) from [<c0663280>] (ubi_leb_unmap+0x68/0x70)
>>> [dump_backtrace_entry]:59:[<c0663280>] (ubi_leb_unmap+0x68/0x70) from [<c05d0088>] (ubifs_garbage_collect+0x2f4/0x308)
>>> [dump_backtrace_entry]:59:[<c05d0088>] (ubifs_garbage_collect+0x2f4/0x308) from [<c05bcaf4>] (make_reservation+0xec/0x3a0)
>>> [dump_backtrace_entry]:59:[<c05bcaf4>] (make_reservation+0xec/0x3a0) from [<c05bd718>] (ubifs_jnl_write_data+0xec/0x1b8)
>>> [dump_backtrace_entry]:59:[<c05bd718>] (ubifs_jnl_write_data+0xec/0x1b8) from [<c05be8d8>] (do_writepage+0x100/0x16c)

So we know now that garbage collection tries to unmap an invalid LEB.
Like I said before, your kernel is super old and unsupported.
You can try try to backport fixes which affect garbage collection in UBIFS.

But I'd strongly suggest to upgrade the kernel.

-- 
Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/