* UBIFS on write-protected NAND @ 2019-06-02 14:08 Leon Pollak 2019-06-02 20:02 ` Richard Weinberger 0 siblings, 1 reply; 7+ messages in thread From: Leon Pollak @ 2019-06-02 14:08 UTC (permalink / raw) To: linux-mtd I am sorry to disturb the list with the problem most probably already solved ion later versions... I am running Linux 2.6.32 from TI and my root FS is on NAND UBIFS. Linux boots with command line: root=ubi0:ubi_rootfs ro noinitrd ubi.mtd=4,2048 rootfstype=ubifs Please, note the "ro" in the command line. Also the HW write-protect line is always set to "protected" state. UBIFS stays most of the time in write protected HW state (system requirement) and RO mounted, except the very rare cases when some update is required. For this update purpose: - HW write-protect is removed in SW; - root FS is remounted to RW (mount -o remount,rw /); - the change is performed; - sync, sleep 3; - mount -o,remount,ro / ; - sleep 2, return HW write-protection; - reboot. For some unknown reason (may be you know?), sometimes something still remains in journal and on the next boot we receive a bundle of error messages with error codes -5 and -30. This happens despite the RO state of the FS and effectively blocks all the system: - after these errors detection, UBIFS switches to read-only state, blocking any possible corrections/repairs. - we can't remove HW protection to allow it to finish desired work as it happens in the Linux boot, when initd is just starting. Now, I suppose that this issue (that everything is RO and shouldn't be tried to recover) is treated already in the new versions. My problem is that I can't move to newer Linux because of TI HW. So, my questions are: 1. Where in the code of UBI (UBIFS?) can I insert the HW write-protect removal in order to allow the UBI/UBIFS to do its desired work? 2. When can I put write-protection back? We already tested that when in this locked state, booting with NAND HW unprotected repairs the problem. The issue is only where to do NAND unlock/lock. Many Thanks for any help. ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: UBIFS on write-protected NAND 2019-06-02 14:08 UBIFS on write-protected NAND Leon Pollak @ 2019-06-02 20:02 ` Richard Weinberger 2019-06-03 7:31 ` Leon Pollak 0 siblings, 1 reply; 7+ messages in thread From: Richard Weinberger @ 2019-06-02 20:02 UTC (permalink / raw) To: Leon Pollak; +Cc: linux-mtd On Sun, Jun 2, 2019 at 4:08 PM Leon Pollak <leon.pollak@gmail.com> wrote: > > I am sorry to disturb the list with the problem most probably already > solved ion later versions... > > I am running Linux 2.6.32 from TI and my root FS is on NAND UBIFS. > Linux boots with command line: > root=ubi0:ubi_rootfs ro noinitrd ubi.mtd=4,2048 rootfstype=ubifs > Please, note the "ro" in the command line. > Also the HW write-protect line is always set to "protected" state. > UBIFS stays most of the time in write protected HW state (system > requirement) and RO mounted, except the very rare cases when some > update is required. > > For this update purpose: > - HW write-protect is removed in SW; > - root FS is remounted to RW (mount -o remount,rw /); > - the change is performed; > - sync, sleep 3; > - mount -o,remount,ro / ; > - sleep 2, return HW write-protection; > - reboot. > > For some unknown reason (may be you know?), sometimes something still > remains in journal and on the next boot we receive a bundle of error > messages with error codes -5 and -30. This happens despite the RO More details please. Can you share a full back trace? But in general a re-mount to ro does not guarantee a clean journal. All it does is making sure that no new files can be opened in write-mode, it is a VFS thing. UBIFS tries to be nice as possible and disables further writes. Maybe your kernel has a bug, it is very old. Dunno... > state of the FS and effectively blocks all the system: > - after these errors detection, UBIFS switches to read-only state, > blocking any possible corrections/repairs. > - we can't remove HW protection to allow it to finish desired work as > it happens in the Linux boot, when initd is just starting. > > Now, I suppose that this issue (that everything is RO and shouldn't be > tried to recover) is treated already in the new versions. My problem > is that I can't move to newer Linux because of TI HW. > > So, my questions are: > 1. Where in the code of UBI (UBIFS?) can I insert the HW write-protect > removal in order to allow the UBI/UBIFS to do its desired work? > 2. When can I put write-protection back? Using a write protected NAND is not recommend. You basically remove the wear-leveling feature from UBI. Blocks can gain bit-flips also in a read-only environment, consider read disturb or other influences such as temperature changes. -- Thanks, //richard ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: UBIFS on write-protected NAND 2019-06-02 20:02 ` Richard Weinberger @ 2019-06-03 7:31 ` Leon Pollak 2019-06-03 11:11 ` Leon Pollak 2019-06-03 12:32 ` Steve deRosier 0 siblings, 2 replies; 7+ messages in thread From: Leon Pollak @ 2019-06-03 7:31 UTC (permalink / raw) To: Richard Weinberger; +Cc: linux-mtd Thank you, Richard. On Sun, 2 Jun 2019 at 23:02, Richard Weinberger <richard.weinberger@gmail.com> wrote: > > On Sun, Jun 2, 2019 at 4:08 PM Leon Pollak <leon.pollak@gmail.com> wrote: > > > > I am sorry to disturb the list with the problem most probably already > > solved ion later versions... > > > > I am running Linux 2.6.32 from TI and my root FS is on NAND UBIFS. > > Linux boots with command line: > > root=ubi0:ubi_rootfs ro noinitrd ubi.mtd=4,2048 rootfstype=ubifs > > Please, note the "ro" in the command line. > > Also the HW write-protect line is always set to "protected" state. > > UBIFS stays most of the time in write protected HW state (system > > requirement) and RO mounted, except the very rare cases when some > > update is required. > > > > For this update purpose: > > - HW write-protect is removed in SW; > > - root FS is remounted to RW (mount -o remount,rw /); > > - the change is performed; > > - sync, sleep 3; > > - mount -o,remount,ro / ; > > - sleep 2, return HW write-protection; > > - reboot. > > > > For some unknown reason (may be you know?), sometimes something still > > remains in journal and on the next boot we receive a bundle of error > > messages with error codes -5 and -30. This happens despite the RO > More details please. Can you share a full back trace? This is an example: UBI error: ubi_io_write: error -5 while writing 512 bytes to PEB 1600:2048, written 0 bytes UBI: run torture test for PEB 1600 UBI error: do_sync_erase: cannot erase PEB 1600, error -5 UBI error: erase_worker: failed to erase PEB 1600, error -5 UBI: mark PEB 1600 as bad UBI error: ubi_io_mark_bad: cannot mark PEB 1600 bad, error -30 UBI warning: ubi_ro_mode: switch to read-only mode UBI error: do_work: work failed with error code -30 UBI error: ubi_thread: ubi_bgt0d: work failed with error code -30 > But in general a re-mount to ro does not guarantee a clean journal. > All it does is making sure that no new files can be opened in write-mode, > it is a VFS thing. UBIFS tries to be nice as possible and disables further > writes. Maybe your kernel has a bug, it is very old. Dunno... OK, I understand. Is there any way to flush ALL data? If "sync" and "ro" doesn't do the job, is there something more? Some IOCTL?... Solving this issue will be the best possible for us. > > state of the FS and effectively blocks all the system: > > - after these errors detection, UBIFS switches to read-only state, > > blocking any possible corrections/repairs. > > - we can't remove HW protection to allow it to finish desired work as > > it happens in the Linux boot, when initd is just starting. > > > > Now, I suppose that this issue (that everything is RO and shouldn't be > > tried to recover) is treated already in the new versions. My problem > > is that I can't move to newer Linux because of TI HW. > > > > So, my questions are: > > 1. Where in the code of UBI (UBIFS?) can I insert the HW write-protect > > removal in order to allow the UBI/UBIFS to do its desired work? > > 2. When can I put write-protection back? > > Using a write protected NAND is not recommend. > You basically remove the wear-leveling feature from UBI. > Blocks can gain bit-flips also in a read-only environment, consider > read disturb or other influences such as temperature changes. Well, as I said, these NAND is updated not more than 30-40 times in all its life. So, wear-leveling is not so relevant. May be this is relics of the NOR past, but our HW engineers always keep flashes write-protected to be on the safe side and avoid possible false writes/disturbances/problems at the power spikes. The main goal here is to keep the system bootable despite everything in the world, except nuclear explosion...:-) Thank you for your help! Leon ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: UBIFS on write-protected NAND 2019-06-03 7:31 ` Leon Pollak @ 2019-06-03 11:11 ` Leon Pollak 2019-06-03 11:31 ` Richard Weinberger 2019-06-03 12:32 ` Steve deRosier 1 sibling, 1 reply; 7+ messages in thread From: Leon Pollak @ 2019-06-03 11:11 UTC (permalink / raw) To: Richard Weinberger; +Cc: linux-mtd Sorry, I think the full UBI log may be also useful: UBI: attaching mtd4 to ubi0 UBI: physical eraseblock size: 131072 bytes (128 KiB) UBI: logical eraseblock size: 126976 bytes UBI: smallest flash I/O unit: 2048 UBI: sub-page size: 512 UBI: VID header offset: 2048 (aligned 2048) UBI: data offset: 4096 UBI: max. sequence number: 85 UBI: attached mtd4 to ubi0 UBI: MTD device name: "File System" UBI: MTD device size: 200 MiB UBI: number of good PEBs: 1601 UBI: number of bad PEBs: 0 UBI: number of corrupted PEBs: 0 UBI: max. allowed volumes: 128 UBI: wear-leveling threshold: 4096 UBI: number of internal volumes: 1 UBI: number of user volumes: 1 UBI: available PEBs: 0 UBI: total number of reserved PEBs: 1601 UBI: number of PEBs reserved for bad PEB handling: 16 UBI: max/mean erase counter: 2/0 UBI: image sequence number: 126133844 UBI: background thread "ubi_bgt0d" started, PID 42 UBIFS: mounted UBI device 0, volume 0, name "ubi_rootfs" UBIFS: mounted read-only UBIFS: file system size: 199225344 bytes (194556 KiB, 189 MiB, 1569 LEBs) UBIFS: journal size: 9023488 bytes (8812 KiB, 8 MiB, 72 LEBs) UBIFS: media format: w4/r0 (latest is w4/r0) UBIFS: default compressor: lzo UBIFS: reserved for root: 0 bytes (0 KiB) VFS: Mounted root (ubifs filesystem) readonly on device 0:13. UBI error: ubi_io_write: error -5 while writing 512 bytes to PEB 1600:2048, written 0 bytes UBI: run torture test for PEB 1600 UBI error: do_sync_erase: cannot erase PEB 1600, error -5 UBI error: erase_worker: failed to erase PEB 1600, error -5 UBI: mark PEB 1600 as bad UBI error: ubi_io_mark_bad: cannot mark PEB 1600 bad, error -30 UBI warning: ubi_ro_mode: switch to read-only mode UBI error: do_work: work failed with error code -30 UBI error: ubi_thread: ubi_bgt0d: work failed with error code -30 And from here my rootfs is in ro mode only. Thanks alot Leon On Mon, 3 Jun 2019 at 10:31, Leon Pollak <leon.pollak@gmail.com> wrote: > > Thank you, Richard. > > On Sun, 2 Jun 2019 at 23:02, Richard Weinberger > <richard.weinberger@gmail.com> wrote: > > > > On Sun, Jun 2, 2019 at 4:08 PM Leon Pollak <leon.pollak@gmail.com> wrote: > > > > > > I am sorry to disturb the list with the problem most probably already > > > solved ion later versions... > > > > > > I am running Linux 2.6.32 from TI and my root FS is on NAND UBIFS. > > > Linux boots with command line: > > > root=ubi0:ubi_rootfs ro noinitrd ubi.mtd=4,2048 rootfstype=ubifs > > > Please, note the "ro" in the command line. > > > Also the HW write-protect line is always set to "protected" state. > > > UBIFS stays most of the time in write protected HW state (system > > > requirement) and RO mounted, except the very rare cases when some > > > update is required. > > > > > > For this update purpose: > > > - HW write-protect is removed in SW; > > > - root FS is remounted to RW (mount -o remount,rw /); > > > - the change is performed; > > > - sync, sleep 3; > > > - mount -o,remount,ro / ; > > > - sleep 2, return HW write-protection; > > > - reboot. > > > > > > For some unknown reason (may be you know?), sometimes something still > > > remains in journal and on the next boot we receive a bundle of error > > > messages with error codes -5 and -30. This happens despite the RO > > More details please. Can you share a full back trace? > This is an example: > UBI error: ubi_io_write: error -5 while writing 512 bytes to PEB > 1600:2048, written 0 bytes > UBI: run torture test for PEB 1600 > UBI error: do_sync_erase: cannot erase PEB 1600, error -5 > UBI error: erase_worker: failed to erase PEB 1600, error -5 > UBI: mark PEB 1600 as bad > UBI error: ubi_io_mark_bad: cannot mark PEB 1600 bad, error -30 > UBI warning: ubi_ro_mode: switch to read-only mode > UBI error: do_work: work failed with error code -30 > UBI error: ubi_thread: ubi_bgt0d: work failed with error code -30 > > > > But in general a re-mount to ro does not guarantee a clean journal. > > All it does is making sure that no new files can be opened in write-mode, > > it is a VFS thing. UBIFS tries to be nice as possible and disables further > > writes. Maybe your kernel has a bug, it is very old. Dunno... > OK, I understand. Is there any way to flush ALL data? > If "sync" and "ro" doesn't do the job, is there something more? Some IOCTL?... > Solving this issue will be the best possible for us. > > > > state of the FS and effectively blocks all the system: > > > - after these errors detection, UBIFS switches to read-only state, > > > blocking any possible corrections/repairs. > > > - we can't remove HW protection to allow it to finish desired work as > > > it happens in the Linux boot, when initd is just starting. > > > > > > Now, I suppose that this issue (that everything is RO and shouldn't be > > > tried to recover) is treated already in the new versions. My problem > > > is that I can't move to newer Linux because of TI HW. > > > > > > So, my questions are: > > > 1. Where in the code of UBI (UBIFS?) can I insert the HW write-protect > > > removal in order to allow the UBI/UBIFS to do its desired work? > > > 2. When can I put write-protection back? > > > > Using a write protected NAND is not recommend. > > You basically remove the wear-leveling feature from UBI. > > Blocks can gain bit-flips also in a read-only environment, consider > > read disturb or other influences such as temperature changes. > Well, as I said, these NAND is updated not more than 30-40 times in > all its life. > So, wear-leveling is not so relevant. > May be this is relics of the NOR past, but our HW engineers always > keep flashes write-protected to be on the safe side and avoid possible > false writes/disturbances/problems at the power spikes. > The main goal here is to keep the system bootable despite everything > in the world, except nuclear explosion...:-) > > Thank you for your help! > > Leon ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: UBIFS on write-protected NAND 2019-06-03 11:11 ` Leon Pollak @ 2019-06-03 11:31 ` Richard Weinberger 0 siblings, 0 replies; 7+ messages in thread From: Richard Weinberger @ 2019-06-03 11:31 UTC (permalink / raw) To: Leon Pollak; +Cc: linux-mtd ----- Ursprüngliche Mail ----- > Von: "Leon Pollak" <leon.pollak@gmail.com> > An: "Richard Weinberger" <richard.weinberger@gmail.com> > CC: "linux-mtd" <linux-mtd@lists.infradead.org> > Gesendet: Montag, 3. Juni 2019 13:11:52 > Betreff: Re: UBIFS on write-protected NAND > Sorry, I think the full UBI log may be also useful: > UBI: attaching mtd4 to ubi0 > UBI: physical eraseblock size: 131072 bytes (128 KiB) > UBI: logical eraseblock size: 126976 bytes > UBI: smallest flash I/O unit: 2048 > UBI: sub-page size: 512 > UBI: VID header offset: 2048 (aligned 2048) > UBI: data offset: 4096 > UBI: max. sequence number: 85 > UBI: attached mtd4 to ubi0 > UBI: MTD device name: "File System" > UBI: MTD device size: 200 MiB > UBI: number of good PEBs: 1601 > UBI: number of bad PEBs: 0 > UBI: number of corrupted PEBs: 0 > UBI: max. allowed volumes: 128 > UBI: wear-leveling threshold: 4096 > UBI: number of internal volumes: 1 > UBI: number of user volumes: 1 > UBI: available PEBs: 0 > UBI: total number of reserved PEBs: 1601 > UBI: number of PEBs reserved for bad PEB handling: 16 > UBI: max/mean erase counter: 2/0 > UBI: image sequence number: 126133844 > UBI: background thread "ubi_bgt0d" started, PID 42 > UBIFS: mounted UBI device 0, volume 0, name "ubi_rootfs" > UBIFS: mounted read-only > UBIFS: file system size: 199225344 bytes (194556 KiB, 189 MiB, 1569 LEBs) > UBIFS: journal size: 9023488 bytes (8812 KiB, 8 MiB, 72 LEBs) > UBIFS: media format: w4/r0 (latest is w4/r0) > UBIFS: default compressor: lzo > UBIFS: reserved for root: 0 bytes (0 KiB) > VFS: Mounted root (ubifs filesystem) readonly on device 0:13. > UBI error: ubi_io_write: error -5 while writing 512 bytes to PEB Can you please place a dump_stack() in ubi_io_write() error path? I wonder where the write comes from. Thanks, //richard ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: UBIFS on write-protected NAND 2019-06-03 7:31 ` Leon Pollak 2019-06-03 11:11 ` Leon Pollak @ 2019-06-03 12:32 ` Steve deRosier 2019-06-03 12:58 ` Richard Weinberger 1 sibling, 1 reply; 7+ messages in thread From: Steve deRosier @ 2019-06-03 12:32 UTC (permalink / raw) To: Leon Pollak; +Cc: Richard Weinberger, linux-mtd Hi Leon, On Mon, Jun 3, 2019 at 12:31 AM Leon Pollak <leon.pollak@gmail.com> wrote: > > Using a write protected NAND is not recommend. > > You basically remove the wear-leveling feature from UBI. > > Blocks can gain bit-flips also in a read-only environment, consider > > read disturb or other influences such as temperature changes. > Well, as I said, these NAND is updated not more than 30-40 times in > all its life. > So, wear-leveling is not so relevant. > May be this is relics of the NOR past, but our HW engineers always > keep flashes write-protected to be on the safe side and avoid possible > false writes/disturbances/problems at the power spikes. > The main goal here is to keep the system bootable despite everything > in the world, except nuclear explosion...:-) > Your HW engineers are wrong and did not read and _understand_ the NAND datasheets. Nor do they understand the software and what it does. The days of the HW guy designing something and throwing it over the wall and asking the SW guy to make it work are long over. If you want NAND to stay bootable "despite everything in the world", you can't run it write protected. NAND will bit-rot over time. It is the nature of NAND. UBIFS detects this and will move data around as necessary to keep it readable. There are certain areas that really only get read at boot time, so if it's write protected at that point, you're sunk - UBIFS can't do the work of preserving the NAND that it is designed to do. If it were me, in u-boot (or whatever bootloader you're using), I'd flip the GPIO holding the /WP line to make the NAND writable before I booted the kernel and then I'd leave it there. The other way requires more effort - you could go into your NAND driver, find the low-level write sequences and flip the GPIO to write and close it to protect after you're done. But, pay very close attention to your datasheets to be sure you have your setup and hold times correct if you're going to do that. The final way to do it is to not use UBIFS at all. Run a R/O image like squashfs and run the NAND with way higher ECC than required and hope that over the lifetime of the device you don't accumulate more than that bit-flips in any sector that you care about. - Steve ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: UBIFS on write-protected NAND 2019-06-03 12:32 ` Steve deRosier @ 2019-06-03 12:58 ` Richard Weinberger 0 siblings, 0 replies; 7+ messages in thread From: Richard Weinberger @ 2019-06-03 12:58 UTC (permalink / raw) To: Steve deRosier; +Cc: Richard Weinberger, linux-mtd, Leon Pollak ----- Ursprüngliche Mail ----- > Your HW engineers are wrong and did not read and _understand_ the NAND > datasheets. Nor do they understand the software and what it does. The > days of the HW guy designing something and throwing it over the wall > and asking the SW guy to make it work are long over. > > If you want NAND to stay bootable "despite everything in the world", > you can't run it write protected. NAND will bit-rot over time. It is > the nature of NAND. UBIFS detects this and will move data around as > necessary to keep it readable. There are certain areas that really > only get read at boot time, so if it's write protected at that point, > you're sunk - UBIFS can't do the work of preserving the NAND that it > is designed to do. > > If it were me, in u-boot (or whatever bootloader you're using), I'd > flip the GPIO holding the /WP line to make the NAND writable before I > booted the kernel and then I'd leave it there. > > The other way requires more effort - you could go into your NAND > driver, find the low-level write sequences and flip the GPIO to write > and close it to protect after you're done. But, pay very close > attention to your datasheets to be sure you have your setup and hold > times correct if you're going to do that. > > The final way to do it is to not use UBIFS at all. Run a R/O image > like squashfs and run the NAND with way higher ECC than required and > hope that over the lifetime of the device you don't accumulate more > than that bit-flips in any sector that you care about. Amen. :-) Thanks, //richard ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2019-06-03 12:58 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-06-02 14:08 UBIFS on write-protected NAND Leon Pollak 2019-06-02 20:02 ` Richard Weinberger 2019-06-03 7:31 ` Leon Pollak 2019-06-03 11:11 ` Leon Pollak 2019-06-03 11:31 ` Richard Weinberger 2019-06-03 12:32 ` Steve deRosier 2019-06-03 12:58 ` Richard Weinberger
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.