UBIFS ECC errors

UBIFS ECC errors

[Sergeant Peppercorn]

I've seen this type of error posted before but I seem to be having a
different issue. I am being plagued with the following problem while
trying to mount our rootfs which is a UBIFS NAND partition. The
differences I see between our occurrences and others is that this is
extremely intermittent, and in some cases, it recovers and then boots
correctly, but in other cases, it continues, making the unit unusable.
I have been postulating anything from bad NAND hardware to power
supply issues to UBI bugs, etc. I am running a very old kernel
(3.12.10 with TI patches for ARM Sitara AM355x CPUs) which works just
find except for this issue, but this issue seems to be happening also
with a 4.9.88 kernel with NXP iMX patches on different hardware, so I
have a difficult time accepting that it is due to the age of the
kernel. That being said, I am open to anything, as this seems to be an
impossible problem. Any tips as to how to debug this would be greatly
appreciated. I would rather not have to re-flash many thousands of
units in the field if there is any sort of other solution. A patched
kernel or uboot with different kernel boot parameters would be the
ideal solution, if it exists. Thanks in advance for any help.

UBIFS: background thread "ubifs_bgt0_0" started, PID 779
UBIFS: recovery needed
UBI warning: ubi_io_read: error -74 (ECC error) while reading 126976
bytes from PEB 679:4096, read only 126976 bytes, retry
UBI warning: ubi_io_read: error -74 (ECC error) while reading 126976
bytes from PEB 679:4096, read only 126976 bytes, retry
UBI warning: ubi_io_read: error -74 (ECC error) while reading 126976
bytes from PEB 679:4096, read only 126976 bytes, retry
UBI error: ubi_io_read: error -74 (ECC error) while reading 126976
bytes from PEB 679:4096, read 126976 bytes
UBI warning: ubi_io_read: error -74 (ECC error) while reading 126976
bytes from PEB 679:4096, read only 126976 bytes, retry
UBI warning: ubi_io_read: error -74 (ECC error) while reading 126976
bytes from PEB 679:4096, read only 126976 bytes, retry
UBI warning: ubi_io_read: error -74 (ECC error) while reading 126976
bytes from PEB 679:4096, read only 126976 bytes, retry
UBI error: ubi_io_read: error -74 (ECC error) while reading 126976
bytes from PEB 679:4096, read 126976 bytes
UBIFS error (pid 1): ubifs_recover_leb: corrupt empty space LEB
7:2048, corruption starts at 2241
UBIFS error (pid 1): ubifs_scanned_corruption: corruption at LEB 7:4289
UBIFS error (pid 1): ubifs_scanned_corruption: first 8192 bytes from LEB 7:4289
UBIFS error (pid 1): ubifs_recover_leb: LEB 7 scanning failed
UBIFS: background thread "ubifs_bgt0_0" stops
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: UBIFS ECC errors

[Richard Weinberger]

On Wed, Jul 31, 2019 at 6:24 PM Sergeant Peppercorn
<speppercorn109@gmail.com> wrote:
> UBIFS error (pid 1): ubifs_recover_leb: corrupt empty space LEB
> 7:2048, corruption starts at 2241

This is a good hint. Can it be that your empty space shows bitflips?
In the last years many drivers needed fixes to deal with btiflips in empty space
since ECC cannot fix these.

-- 
Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: UBIFS ECC errors

[Sergeant Peppercorn]

Hi, many thanks for the reply.

I have seen posts on bitflips in empty space issues before, but don't
remember seeing any fixes. Does this mean the kernel UBI drivers must
be patched? Can you elaborate on what fixes or where to find patches,
etc, or what to do about this, then? All I care about is getting this
to work, and hopefully without needing to reformat the root file
system on 50,000 units in the field.

Kind regards

On Thu, Aug 1, 2019 at 1:04 AM Richard Weinberger
<richard.weinberger@gmail.com> wrote:
>
> On Wed, Jul 31, 2019 at 6:24 PM Sergeant Peppercorn
> <speppercorn109@gmail.com> wrote:
> > UBIFS error (pid 1): ubifs_recover_leb: corrupt empty space LEB
> > 7:2048, corruption starts at 2241
>
> This is a good hint. Can it be that your empty space shows bitflips?
> In the last years many drivers needed fixes to deal with btiflips in empty space
> since ECC cannot fix these.
>
> --
> Thanks,
> //richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: UBIFS ECC errors

[Richard Weinberger]

----- Ursprüngliche Mail -----
> Von: "Sergeant Peppercorn" <speppercorn109@gmail.com>
> An: "Richard Weinberger" <richard.weinberger@gmail.com>
> CC: "linux-mtd" <linux-mtd@lists.infradead.org>
> Gesendet: Freitag, 2. August 2019 18:14:55
> Betreff: Re: UBIFS ECC errors

> Hi, many thanks for the reply.
> 
> I have seen posts on bitflips in empty space issues before, but don't
> remember seeing any fixes. Does this mean the kernel UBI drivers must
> be patched? Can you elaborate on what fixes or where to find patches,
> etc, or what to do about this, then? All I care about is getting this
> to work, and hopefully without needing to reformat the root file
> system on 50,000 units in the field.

Your kernel is pretty old, so I guess you don't have this commit:
730a43fbc135 ("mtd: nand: add nand_check_erased helper functions")
 
Bitflips in empty pages should be handled in the NAND layer.

Please don't blindly apply this patch, it may have dependencies or will
uncover other bugs. Such as fix:
fdf2e8210529 ("mtd: nand: gpmi: Fix failure when a erased page has a bitflip at BBM")

Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: UBIFS ECC errors

[Sergeant Peppercorn]

Thanks so much for your help. What do you think of this patch?
http://lists.infradead.org/pipermail/linux-mtd/2014-January/051357.html

I don't see it even in the 4.9.88 kernel. I see other things there for this.

On Sat, Aug 3, 2019 at 1:48 AM Richard Weinberger <richard@nod.at> wrote:
>
> ----- Ursprüngliche Mail -----
> > Von: "Sergeant Peppercorn" <speppercorn109@gmail.com>
> > An: "Richard Weinberger" <richard.weinberger@gmail.com>
> > CC: "linux-mtd" <linux-mtd@lists.infradead.org>
> > Gesendet: Freitag, 2. August 2019 18:14:55
> > Betreff: Re: UBIFS ECC errors
>
> > Hi, many thanks for the reply.
> >
> > I have seen posts on bitflips in empty space issues before, but don't
> > remember seeing any fixes. Does this mean the kernel UBI drivers must
> > be patched? Can you elaborate on what fixes or where to find patches,
> > etc, or what to do about this, then? All I care about is getting this
> > to work, and hopefully without needing to reformat the root file
> > system on 50,000 units in the field.
>
> Your kernel is pretty old, so I guess you don't have this commit:
> 730a43fbc135 ("mtd: nand: add nand_check_erased helper functions")
>
> Bitflips in empty pages should be handled in the NAND layer.
>
> Please don't blindly apply this patch, it may have dependencies or will
> uncover other bugs. Such as fix:
> fdf2e8210529 ("mtd: nand: gpmi: Fix failure when a erased page has a bitflip at BBM")
>
> Thanks,
> //richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: UBIFS ECC errors

[Sergeant Peppercorn]

Hi, so are you saying that I should try to merge in these two commits
for a proper fix for this?
730a43fbc135 ("mtd: nand: add nand_check_erased helper functions")
fdf2e8210529 ("mtd: nand: gpmi: Fix failure when a erased page has a
bitflip at BBM")

I am assuming these are from the mainline kernel?

If not, how do I fix this? this other patch, instead or with those two commits?
http://lists.infradead.org/pipermail/linux-mtd/2014-January/051357.html

The patch doesn't seem to be able to apply cleanly to my kernel so I
must do some deep merging. I have not looked for the two commits yet.

Are there any other alternatives?

Thenks in advance for any help.

On Sat, Aug 3, 2019 at 1:48 AM Richard Weinberger <richard@nod.at> wrote:
>
> ----- Ursprüngliche Mail -----
> > Von: "Sergeant Peppercorn" <speppercorn109@gmail.com>
> > An: "Richard Weinberger" <richard.weinberger@gmail.com>
> > CC: "linux-mtd" <linux-mtd@lists.infradead.org>
> > Gesendet: Freitag, 2. August 2019 18:14:55
> > Betreff: Re: UBIFS ECC errors
>
> > Hi, many thanks for the reply.
> >
> > I have seen posts on bitflips in empty space issues before, but don't
> > remember seeing any fixes. Does this mean the kernel UBI drivers must
> > be patched? Can you elaborate on what fixes or where to find patches,
> > etc, or what to do about this, then? All I care about is getting this
> > to work, and hopefully without needing to reformat the root file
> > system on 50,000 units in the field.
>
> Your kernel is pretty old, so I guess you don't have this commit:
> 730a43fbc135 ("mtd: nand: add nand_check_erased helper functions")
>
> Bitflips in empty pages should be handled in the NAND layer.
>
> Please don't blindly apply this patch, it may have dependencies or will
> uncover other bugs. Such as fix:
> fdf2e8210529 ("mtd: nand: gpmi: Fix failure when a erased page has a bitflip at BBM")
>
> Thanks,
> //richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: UBIFS ECC errors

[Sergeant Peppercorn]

Also, would this apply fo my 4.9.88 IMX kernel as well as the TI 3.12.10 kernel?

On Wed, Aug 7, 2019 at 9:51 AM Sergeant Peppercorn
<speppercorn109@gmail.com> wrote:
>
> Hi, so are you saying that I should try to merge in these two commits
> for a proper fix for this?
> 730a43fbc135 ("mtd: nand: add nand_check_erased helper functions")
> fdf2e8210529 ("mtd: nand: gpmi: Fix failure when a erased page has a
> bitflip at BBM")
>
> I am assuming these are from the mainline kernel?
>
> If not, how do I fix this? this other patch, instead or with those two commits?
> http://lists.infradead.org/pipermail/linux-mtd/2014-January/051357.html
>
> The patch doesn't seem to be able to apply cleanly to my kernel so I
> must do some deep merging. I have not looked for the two commits yet.
>
> Are there any other alternatives?
>
> Thenks in advance for any help.
>
> On Sat, Aug 3, 2019 at 1:48 AM Richard Weinberger <richard@nod.at> wrote:
> >
> > ----- Ursprüngliche Mail -----
> > > Von: "Sergeant Peppercorn" <speppercorn109@gmail.com>
> > > An: "Richard Weinberger" <richard.weinberger@gmail.com>
> > > CC: "linux-mtd" <linux-mtd@lists.infradead.org>
> > > Gesendet: Freitag, 2. August 2019 18:14:55
> > > Betreff: Re: UBIFS ECC errors
> >
> > > Hi, many thanks for the reply.
> > >
> > > I have seen posts on bitflips in empty space issues before, but don't
> > > remember seeing any fixes. Does this mean the kernel UBI drivers must
> > > be patched? Can you elaborate on what fixes or where to find patches,
> > > etc, or what to do about this, then? All I care about is getting this
> > > to work, and hopefully without needing to reformat the root file
> > > system on 50,000 units in the field.
> >
> > Your kernel is pretty old, so I guess you don't have this commit:
> > 730a43fbc135 ("mtd: nand: add nand_check_erased helper functions")
> >
> > Bitflips in empty pages should be handled in the NAND layer.
> >
> > Please don't blindly apply this patch, it may have dependencies or will
> > uncover other bugs. Such as fix:
> > fdf2e8210529 ("mtd: nand: gpmi: Fix failure when a erased page has a bitflip at BBM")
> >
> > Thanks,
> > //richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: UBIFS ECC errors

[Richard Weinberger]

On Wed, Aug 7, 2019 at 6:52 PM Sergeant Peppercorn
<speppercorn109@gmail.com> wrote:
>
> Also, would this apply fo my 4.9.88 IMX kernel as well as the TI 3.12.10 kernel?

We care only about mainline, sorry.

> On Wed, Aug 7, 2019 at 9:51 AM Sergeant Peppercorn
> <speppercorn109@gmail.com> wrote:
> >
> > Hi, so are you saying that I should try to merge in these two commits
> > for a proper fix for this?
> > 730a43fbc135 ("mtd: nand: add nand_check_erased helper functions")
> > fdf2e8210529 ("mtd: nand: gpmi: Fix failure when a erased page has a
> > bitflip at BBM")

No. I said you need most likely 730a43fbc135 and commits it depends on.
Commit fdf2e8210529 was just an example of a related bug.

> > I am assuming these are from the mainline kernel?

Sure.

> > If not, how do I fix this? this other patch, instead or with those two commits?
> > http://lists.infradead.org/pipermail/linux-mtd/2014-January/051357.html
> >
> > The patch doesn't seem to be able to apply cleanly to my kernel so I
> > must do some deep merging. I have not looked for the two commits yet.

This patch was AFAIK not applied because we decided to work around the issue
in NAND core.

Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/