UBIFS on write-protected NAND

UBIFS on write-protected NAND

[Leon Pollak]

I am sorry to disturb the list with the problem most probably already
solved ion later versions...

I am running Linux 2.6.32 from TI and my root FS is on NAND UBIFS.
Linux boots with command line:
root=ubi0:ubi_rootfs ro noinitrd ubi.mtd=4,2048 rootfstype=ubifs
Please, note the "ro" in the command line.
Also the HW write-protect line is always set to "protected" state.
UBIFS stays most of the time in write protected HW state (system
requirement) and RO mounted, except the very rare cases when some
update is required.

For this update purpose:
- HW write-protect is removed in SW;
- root FS is remounted to RW (mount -o remount,rw /);
- the change is performed;
- sync, sleep 3;
- mount -o,remount,ro / ;
- sleep 2, return HW write-protection;
- reboot.

For some unknown reason (may be you know?), sometimes something still
remains in journal and on the next boot we receive a bundle of error
messages with error codes -5 and -30. This happens despite the RO
state of the FS and effectively blocks all the system:
- after these errors detection, UBIFS switches to read-only state,
blocking any possible corrections/repairs.
- we can't remove HW protection to allow it to finish desired work as
it happens in the Linux boot, when initd is just starting.

Now, I suppose that this issue (that everything is RO and shouldn't be
tried to recover) is treated already in the new versions. My problem
is that I can't move to newer Linux because of TI HW.

So, my questions are:
1. Where in the code of UBI (UBIFS?) can I insert the HW write-protect
removal in order to allow the UBI/UBIFS to do its desired work?
2. When can I put write-protection back?

We already tested that when in this locked state, booting with NAND HW
unprotected repairs the problem. The issue is only where to do NAND
unlock/lock.

Many Thanks for any help.

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: UBIFS on write-protected NAND

[Richard Weinberger]

On Sun, Jun 2, 2019 at 4:08 PM Leon Pollak <leon.pollak@gmail.com> wrote:
>
> I am sorry to disturb the list with the problem most probably already
> solved ion later versions...
>
> I am running Linux 2.6.32 from TI and my root FS is on NAND UBIFS.
> Linux boots with command line:
> root=ubi0:ubi_rootfs ro noinitrd ubi.mtd=4,2048 rootfstype=ubifs
> Please, note the "ro" in the command line.
> Also the HW write-protect line is always set to "protected" state.
> UBIFS stays most of the time in write protected HW state (system
> requirement) and RO mounted, except the very rare cases when some
> update is required.
>
> For this update purpose:
> - HW write-protect is removed in SW;
> - root FS is remounted to RW (mount -o remount,rw /);
> - the change is performed;
> - sync, sleep 3;
> - mount -o,remount,ro / ;
> - sleep 2, return HW write-protection;
> - reboot.
>
> For some unknown reason (may be you know?), sometimes something still
> remains in journal and on the next boot we receive a bundle of error
> messages with error codes -5 and -30. This happens despite the RO

More details please. Can you share a full back trace?

But in general a re-mount to ro does not guarantee a clean journal.
All it does is making sure that no new files can be opened in write-mode,
it is a VFS thing. UBIFS tries to be nice as possible and disables further
writes. Maybe your kernel has a bug, it is very old. Dunno...

> state of the FS and effectively blocks all the system:
> - after these errors detection, UBIFS switches to read-only state,
> blocking any possible corrections/repairs.
> - we can't remove HW protection to allow it to finish desired work as
> it happens in the Linux boot, when initd is just starting.
>
> Now, I suppose that this issue (that everything is RO and shouldn't be
> tried to recover) is treated already in the new versions. My problem
> is that I can't move to newer Linux because of TI HW.
>
> So, my questions are:
> 1. Where in the code of UBI (UBIFS?) can I insert the HW write-protect
> removal in order to allow the UBI/UBIFS to do its desired work?
> 2. When can I put write-protection back?

Using a write protected NAND is not recommend.
You basically remove the wear-leveling feature from UBI.
Blocks can gain bit-flips also in a read-only environment, consider
read disturb or other influences such as temperature changes.

-- 
Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: UBIFS on write-protected NAND

[Leon Pollak]

Thank you, Richard.

On Sun, 2 Jun 2019 at 23:02, Richard Weinberger
<richard.weinberger@gmail.com> wrote:
>
> On Sun, Jun 2, 2019 at 4:08 PM Leon Pollak <leon.pollak@gmail.com> wrote:
> >
> > I am sorry to disturb the list with the problem most probably already
> > solved ion later versions...
> >
> > I am running Linux 2.6.32 from TI and my root FS is on NAND UBIFS.
> > Linux boots with command line:
> > root=ubi0:ubi_rootfs ro noinitrd ubi.mtd=4,2048 rootfstype=ubifs
> > Please, note the "ro" in the command line.
> > Also the HW write-protect line is always set to "protected" state.
> > UBIFS stays most of the time in write protected HW state (system
> > requirement) and RO mounted, except the very rare cases when some
> > update is required.
> >
> > For this update purpose:
> > - HW write-protect is removed in SW;
> > - root FS is remounted to RW (mount -o remount,rw /);
> > - the change is performed;
> > - sync, sleep 3;
> > - mount -o,remount,ro / ;
> > - sleep 2, return HW write-protection;
> > - reboot.
> >
> > For some unknown reason (may be you know?), sometimes something still
> > remains in journal and on the next boot we receive a bundle of error
> > messages with error codes -5 and -30. This happens despite the RO
> More details please. Can you share a full back trace?
This is an example:
UBI error: ubi_io_write: error -5 while writing 512 bytes to PEB
1600:2048, written 0 bytes
UBI: run torture test for PEB 1600
UBI error: do_sync_erase: cannot erase PEB 1600, error -5
UBI error: erase_worker: failed to erase PEB 1600, error -5
UBI: mark PEB 1600 as bad
UBI error: ubi_io_mark_bad: cannot mark PEB 1600 bad, error -30
UBI warning: ubi_ro_mode: switch to read-only mode
UBI error: do_work: work failed with error code -30
UBI error: ubi_thread: ubi_bgt0d: work failed with error code -30


> But in general a re-mount to ro does not guarantee a clean journal.
> All it does is making sure that no new files can be opened in write-mode,
> it is a VFS thing. UBIFS tries to be nice as possible and disables further
> writes. Maybe your kernel has a bug, it is very old. Dunno...
OK, I understand. Is there any way to flush ALL data?
If "sync" and "ro" doesn't do the job, is there something more? Some IOCTL?...
Solving this issue will be the best possible for us.

> > state of the FS and effectively blocks all the system:
> > - after these errors detection, UBIFS switches to read-only state,
> > blocking any possible corrections/repairs.
> > - we can't remove HW protection to allow it to finish desired work as
> > it happens in the Linux boot, when initd is just starting.
> >
> > Now, I suppose that this issue (that everything is RO and shouldn't be
> > tried to recover) is treated already in the new versions. My problem
> > is that I can't move to newer Linux because of TI HW.
> >
> > So, my questions are:
> > 1. Where in the code of UBI (UBIFS?) can I insert the HW write-protect
> > removal in order to allow the UBI/UBIFS to do its desired work?
> > 2. When can I put write-protection back?
>
> Using a write protected NAND is not recommend.
> You basically remove the wear-leveling feature from UBI.
> Blocks can gain bit-flips also in a read-only environment, consider
> read disturb or other influences such as temperature changes.
Well, as I said, these NAND is updated not more than 30-40 times in
all its life.
So, wear-leveling is not so relevant.
May be this is relics of the NOR past, but our HW engineers always
keep flashes write-protected to be on the safe side and avoid possible
false writes/disturbances/problems at the power spikes.
The main goal here is to keep the system bootable despite everything
in the world, except nuclear explosion...:-)

Thank you for your help!

Leon

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: UBIFS on write-protected NAND

[Leon Pollak]

Sorry, I think the full UBI log may be also useful:
UBI: attaching mtd4 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    126976 bytes
UBI: smallest flash I/O unit:    2048
UBI: sub-page size:              512
UBI: VID header offset:          2048 (aligned 2048)
UBI: data offset:                4096
UBI: max. sequence number:       85
UBI: attached mtd4 to ubi0
UBI: MTD device name:            "File System"
UBI: MTD device size:            200 MiB
UBI: number of good PEBs:        1601
UBI: number of bad PEBs:         0
UBI: number of corrupted PEBs:   0
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     1
UBI: available PEBs:             0
UBI: total number of reserved PEBs: 1601
UBI: number of PEBs reserved for bad PEB handling: 16
UBI: max/mean erase counter: 2/0
UBI: image sequence number:  126133844
UBI: background thread "ubi_bgt0d" started, PID 42
UBIFS: mounted UBI device 0, volume 0, name "ubi_rootfs"
UBIFS: mounted read-only
UBIFS: file system size:   199225344 bytes (194556 KiB, 189 MiB, 1569 LEBs)
UBIFS: journal size:       9023488 bytes (8812 KiB, 8 MiB, 72 LEBs)
UBIFS: media format:       w4/r0 (latest is w4/r0)
UBIFS: default compressor: lzo
UBIFS: reserved for root:  0 bytes (0 KiB)
VFS: Mounted root (ubifs filesystem) readonly on device 0:13.
UBI error: ubi_io_write: error -5 while writing 512 bytes to PEB
1600:2048, written 0 bytes
UBI: run torture test for PEB 1600
UBI error: do_sync_erase: cannot erase PEB 1600, error -5
UBI error: erase_worker: failed to erase PEB 1600, error -5
UBI: mark PEB 1600 as bad
UBI error: ubi_io_mark_bad: cannot mark PEB 1600 bad, error -30
UBI warning: ubi_ro_mode: switch to read-only mode
UBI error: do_work: work failed with error code -30
UBI error: ubi_thread: ubi_bgt0d: work failed with error code -30

And from here my rootfs is in ro mode only.

Thanks alot
Leon

On Mon, 3 Jun 2019 at 10:31, Leon Pollak <leon.pollak@gmail.com> wrote:
>
> Thank you, Richard.
>
> On Sun, 2 Jun 2019 at 23:02, Richard Weinberger
> <richard.weinberger@gmail.com> wrote:
> >
> > On Sun, Jun 2, 2019 at 4:08 PM Leon Pollak <leon.pollak@gmail.com> wrote:
> > >
> > > I am sorry to disturb the list with the problem most probably already
> > > solved ion later versions...
> > >
> > > I am running Linux 2.6.32 from TI and my root FS is on NAND UBIFS.
> > > Linux boots with command line:
> > > root=ubi0:ubi_rootfs ro noinitrd ubi.mtd=4,2048 rootfstype=ubifs
> > > Please, note the "ro" in the command line.
> > > Also the HW write-protect line is always set to "protected" state.
> > > UBIFS stays most of the time in write protected HW state (system
> > > requirement) and RO mounted, except the very rare cases when some
> > > update is required.
> > >
> > > For this update purpose:
> > > - HW write-protect is removed in SW;
> > > - root FS is remounted to RW (mount -o remount,rw /);
> > > - the change is performed;
> > > - sync, sleep 3;
> > > - mount -o,remount,ro / ;
> > > - sleep 2, return HW write-protection;
> > > - reboot.
> > >
> > > For some unknown reason (may be you know?), sometimes something still
> > > remains in journal and on the next boot we receive a bundle of error
> > > messages with error codes -5 and -30. This happens despite the RO
> > More details please. Can you share a full back trace?
> This is an example:
> UBI error: ubi_io_write: error -5 while writing 512 bytes to PEB
> 1600:2048, written 0 bytes
> UBI: run torture test for PEB 1600
> UBI error: do_sync_erase: cannot erase PEB 1600, error -5
> UBI error: erase_worker: failed to erase PEB 1600, error -5
> UBI: mark PEB 1600 as bad
> UBI error: ubi_io_mark_bad: cannot mark PEB 1600 bad, error -30
> UBI warning: ubi_ro_mode: switch to read-only mode
> UBI error: do_work: work failed with error code -30
> UBI error: ubi_thread: ubi_bgt0d: work failed with error code -30
>
>
> > But in general a re-mount to ro does not guarantee a clean journal.
> > All it does is making sure that no new files can be opened in write-mode,
> > it is a VFS thing. UBIFS tries to be nice as possible and disables further
> > writes. Maybe your kernel has a bug, it is very old. Dunno...
> OK, I understand. Is there any way to flush ALL data?
> If "sync" and "ro" doesn't do the job, is there something more? Some IOCTL?...
> Solving this issue will be the best possible for us.
>
> > > state of the FS and effectively blocks all the system:
> > > - after these errors detection, UBIFS switches to read-only state,
> > > blocking any possible corrections/repairs.
> > > - we can't remove HW protection to allow it to finish desired work as
> > > it happens in the Linux boot, when initd is just starting.
> > >
> > > Now, I suppose that this issue (that everything is RO and shouldn't be
> > > tried to recover) is treated already in the new versions. My problem
> > > is that I can't move to newer Linux because of TI HW.
> > >
> > > So, my questions are:
> > > 1. Where in the code of UBI (UBIFS?) can I insert the HW write-protect
> > > removal in order to allow the UBI/UBIFS to do its desired work?
> > > 2. When can I put write-protection back?
> >
> > Using a write protected NAND is not recommend.
> > You basically remove the wear-leveling feature from UBI.
> > Blocks can gain bit-flips also in a read-only environment, consider
> > read disturb or other influences such as temperature changes.
> Well, as I said, these NAND is updated not more than 30-40 times in
> all its life.
> So, wear-leveling is not so relevant.
> May be this is relics of the NOR past, but our HW engineers always
> keep flashes write-protected to be on the safe side and avoid possible
> false writes/disturbances/problems at the power spikes.
> The main goal here is to keep the system bootable despite everything
> in the world, except nuclear explosion...:-)
>
> Thank you for your help!
>
> Leon

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: UBIFS on write-protected NAND

[Richard Weinberger]

----- Ursprüngliche Mail -----
> Von: "Leon Pollak" <leon.pollak@gmail.com>
> An: "Richard Weinberger" <richard.weinberger@gmail.com>
> CC: "linux-mtd" <linux-mtd@lists.infradead.org>
> Gesendet: Montag, 3. Juni 2019 13:11:52
> Betreff: Re: UBIFS on write-protected NAND

> Sorry, I think the full UBI log may be also useful:
> UBI: attaching mtd4 to ubi0
> UBI: physical eraseblock size:   131072 bytes (128 KiB)
> UBI: logical eraseblock size:    126976 bytes
> UBI: smallest flash I/O unit:    2048
> UBI: sub-page size:              512
> UBI: VID header offset:          2048 (aligned 2048)
> UBI: data offset:                4096
> UBI: max. sequence number:       85
> UBI: attached mtd4 to ubi0
> UBI: MTD device name:            "File System"
> UBI: MTD device size:            200 MiB
> UBI: number of good PEBs:        1601
> UBI: number of bad PEBs:         0
> UBI: number of corrupted PEBs:   0
> UBI: max. allowed volumes:       128
> UBI: wear-leveling threshold:    4096
> UBI: number of internal volumes: 1
> UBI: number of user volumes:     1
> UBI: available PEBs:             0
> UBI: total number of reserved PEBs: 1601
> UBI: number of PEBs reserved for bad PEB handling: 16
> UBI: max/mean erase counter: 2/0
> UBI: image sequence number:  126133844
> UBI: background thread "ubi_bgt0d" started, PID 42
> UBIFS: mounted UBI device 0, volume 0, name "ubi_rootfs"
> UBIFS: mounted read-only
> UBIFS: file system size:   199225344 bytes (194556 KiB, 189 MiB, 1569 LEBs)
> UBIFS: journal size:       9023488 bytes (8812 KiB, 8 MiB, 72 LEBs)
> UBIFS: media format:       w4/r0 (latest is w4/r0)
> UBIFS: default compressor: lzo
> UBIFS: reserved for root:  0 bytes (0 KiB)
> VFS: Mounted root (ubifs filesystem) readonly on device 0:13.
> UBI error: ubi_io_write: error -5 while writing 512 bytes to PEB

Can you please place a dump_stack() in ubi_io_write() error path?
I wonder where the write comes from.

Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: UBIFS on write-protected NAND

[Steve deRosier]

Hi Leon,

On Mon, Jun 3, 2019 at 12:31 AM Leon Pollak <leon.pollak@gmail.com> wrote:

> > Using a write protected NAND is not recommend.
> > You basically remove the wear-leveling feature from UBI.
> > Blocks can gain bit-flips also in a read-only environment, consider
> > read disturb or other influences such as temperature changes.
> Well, as I said, these NAND is updated not more than 30-40 times in
> all its life.
> So, wear-leveling is not so relevant.
> May be this is relics of the NOR past, but our HW engineers always
> keep flashes write-protected to be on the safe side and avoid possible
> false writes/disturbances/problems at the power spikes.
> The main goal here is to keep the system bootable despite everything
> in the world, except nuclear explosion...:-)
>

Your HW engineers are wrong and did not read and _understand_ the NAND
datasheets. Nor do they understand the software and what it does. The
days of the HW guy designing something and throwing it over the wall
and asking the SW guy to make it work are long over.

If you want NAND to stay bootable "despite everything in the world",
you can't run it write protected. NAND will bit-rot over time. It is
the nature of NAND. UBIFS detects this and will move data around as
necessary to keep it readable. There are certain areas that really
only get read at boot time, so if it's write protected at that point,
you're sunk - UBIFS can't do the work of preserving the NAND that it
is designed to do.

If it were me, in u-boot (or whatever bootloader you're using), I'd
flip the GPIO holding the /WP line to make the NAND writable before I
booted the kernel and then I'd leave it there.

The other way requires more effort - you could go into your NAND
driver, find the low-level write sequences and flip the GPIO to write
and close it to protect after you're done. But, pay very close
attention to your datasheets to be sure you have your setup and hold
times correct if you're going to do that.

The final way to do it is to not use UBIFS at all.  Run a R/O image
like squashfs and run the NAND with way higher ECC than required and
hope that over the lifetime of the device you don't accumulate more
than that bit-flips in any sector that you care about.

- Steve

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: UBIFS on write-protected NAND

[Richard Weinberger]

----- Ursprüngliche Mail -----
> Your HW engineers are wrong and did not read and _understand_ the NAND
> datasheets. Nor do they understand the software and what it does. The
> days of the HW guy designing something and throwing it over the wall
> and asking the SW guy to make it work are long over.
> 
> If you want NAND to stay bootable "despite everything in the world",
> you can't run it write protected. NAND will bit-rot over time. It is
> the nature of NAND. UBIFS detects this and will move data around as
> necessary to keep it readable. There are certain areas that really
> only get read at boot time, so if it's write protected at that point,
> you're sunk - UBIFS can't do the work of preserving the NAND that it
> is designed to do.
> 
> If it were me, in u-boot (or whatever bootloader you're using), I'd
> flip the GPIO holding the /WP line to make the NAND writable before I
> booted the kernel and then I'd leave it there.
> 
> The other way requires more effort - you could go into your NAND
> driver, find the low-level write sequences and flip the GPIO to write
> and close it to protect after you're done. But, pay very close
> attention to your datasheets to be sure you have your setup and hold
> times correct if you're going to do that.
> 
> The final way to do it is to not use UBIFS at all.  Run a R/O image
> like squashfs and run the NAND with way higher ECC than required and
> hope that over the lifetime of the device you don't accumulate more
> than that bit-flips in any sector that you care about.

Amen. :-)

Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/